Easy MFA and Profile Switching in AWS CLI

UPDATE 2019-01-15: I have released a much-improved v2.0 of the awscli-mfa.sh script. Its location has also changed; it can now be found at https://github.com/vwal/awscli-mfa

An updated blog entry is forthcoming; the article below discusses version 1.0 which was completely rewritten in this new iteration.

As I alluded in the previous post, I have released another AWS script that works in conjunction with the AWS CLI Key Rotation script. This one is for easily activating an MFA session while on the command line. It also makes it easy to switch between the different configured AWS profiles.

While MFA is perhaps more frequently used with the web console sessions, in some ways the API keys are even more of a security risk. Whereas people usually memorize the console password, or keep it in a password manager, the API keys sit in plaintext in `~/.aws/credentials` file. All it takes is a malware that dumps the content of that known file location to a hacker, or perhaps a stolen laptop, or a stolen unencrypted flash drive (that you used to transfer the API keys to your new laptop, say), and a third party will have access to as much of your AWS assets as the IAM policy associated with the key allows. Admin keys? Full access! For this reason, I advocate the use of MFA especially for the keys with wide access, and that are intended for interactive use. Of course roles should be used as much as possible, and as much as possible should be automated, but still, in most environments people use the API keys for various command line tasks and to run utility scripts from the admins’ laptops.

Enter awscli-mfa.sh! This script makes it easy to select a profile that has MFA configured, to initialize an MFA session, and to make the MFA session default for the current console session (via an environment variable). The script is obviously interactive since it’ll prompt for an MFA one-time passcode. When you run it, it’ll query the AWS profiles configured in `~/.aws/credentials`, and list them out, along with the actual IAM username associated with each profile (since the profile names are arbitrary, and as such can be set to anything). The script will also indicate whether a virtual MFA has been configured for the profile. You can select to activate a profile that doesn’t have MFA configured, but obviously then an MFA code is not prompted for, and MFA is not used. When the chosen profile does have MFA configured, the script will prompt for the one-time passcode (e.g. from Authy or Google Authenticator), initialize the session, and output the environment export variable for easy cut-and-paste to the console (when ran on a Mac, or in a Linux GUI that has `xclip` installed, the export string is automatically copied on the clipboard). Paste on the command line, hit Enter, and the MFA profile is active!

Once an MFA session has been activated and you run the script again, you’ll see the newly initialized MFA profile under its base profile. You can switch away from it to another profile, and then back again. The MFA profile is indicated either in “OK” or “LIMITED” status. The latter may suggest that the IAM policy session duration has been set to a shorter period than what was configured in this script (you can adjust the token lifetime duration in the script on line 16. It should be set to match the session duration set in the MFA enforcement policy (if the policy duration times out first, the key appears still valid, but it’s no longer authorized for any transaction on the AWS). Note that if the IAM policy for the profile is limited so that it isn’t allowed to query `aws iam get-user` for itself, you’ll see the “LIMITED” status even when the MFA session is valid. Once the session token expires, the MFA profile disappears from the session list (even though you can see it still if you `cat ~/.aws/credentials` file.

A good write-up and an example MFA enforcement policy can be found here. I used their policy with a slight modification: I set the maximum MFA session duration to six hours by altering the Condition for the last two Sids in the policy, like so:

"Condition": {
  "NumericGreaterThanIfExists": {
    "aws:MultiFactorAuthAge": "21600"
  }
}

As I mentioned in the previous post (“AWS CLI Key Rotation Script for IAM Users revisited“), you can use an MFA session initialized with this script to rotate the keys of the MFA session’s base profile that doesn’t have permissions for anything without an active associated MFA session. As long as you have an MFA session initialized for the profile whose keys you want to rotate, the key rotation script will detect the presence of the MFA session, and ask if you would like to use it to execute the key rotation for the base profile. The MFA sessions are not visible in the key rotation session list because their keys are transient, only valid for the length of an MFA session.

The script was written for macOS, but portability for Ubuntu (and so, probably most/all Linux distros) was added.

The awscli-mfa.sh script is available from its GitHub repository.

I recommend Authy for the virtual MFA (it’s available both for iOS and Android).

As always, comments or improvement suggestions are welcome (either as comments to this post, or as PRs to the repository)!

UPDATE 2018-02-19: I’ve updated the script, fixed some minor bugs for some edge cases, cleaned up its formatting, clarified the output, and added a companion script (source-to-clear-AWS-envvars.sh) that can be used to clear any AWS secrets/variables that may be set in the environment through the use of the awscli-mfa.sh script.

A Simple AWS CLI KMS encrypt/decrypt example

This would have saved me an hour or two, so I’m posting it here for posterity. 🙂 Maybe it will save some time for someone else.

A quick example of how to use the AWS CLI to encrypt a file using a KMS with a key identified by the `key-id`. The output is saved into 76-column wrapped ASCII-armored file, and then decrypt the same back into cleartext. I couldn’t find a way to column-wrap the output from `aws kms encrypt`, so the base64 encoding is first undone, and then re-applied with the [default] column width of 76.

Replace the below fake `key-id` with your own (obviously 🙂 ) that your AWS credentials have access to. For this to work, you have to have awscli installed and configured (run `aws configure` after you’ve installed `awscli`).

** NOTE: On macOS, you have to use capital `-D` with `base64` to decrypt.

First, to encrypt the contents of a file, and then output the decrypted content back into a file, use the following format:

aws kms encrypt --key-id 3c436c82-eabe-4b58-996f-6ca3f808f237 --plaintext fileb://my-secret-key.pem --query CiphertextBlob --output text | base64 -d | base64 -w 76 > encrypted.asc

aws kms decrypt --ciphertext-blob fileb://<(cat encrypted.asc | base64 -d) --output text --query Plaintext | base64 -d > decrypted.txt

Then, to encrypt/decrypt a string without first saving it into a file (who came up with the decryption format?!). This works in `bash`, `zsh`, and alike (`ksh`..?):

aws kms encrypt --key-id 3c436c82-eabe-4b58-996f-6ca3f808f237 --plaintext 'Secret message' --query CiphertextBlob --output text

aws kms decrypt --ciphertext-blob fileb://<(echo 'AQECAHiuImqexTQGWMAtOjKMcH5UIxXuSZ5WSGx3WKO+vsUI3AAAAKIwgZ8GCSqGSIb3DQEHBqCBkTCBjgIBADCBiAYJKoZIhvcNAQcBMB4GCWCGSAFlAwQBLjARBAwT3cwVGtUYHz02irsCARCAW8a4TP7pL+inl7Je7x1xEr84Q4lN11t3dNFvycpMZALe185DYow4i1GLaJnJnB7g6V1ZaiB+b+Diap/5auM/K3bjLmcTq0molBnn2TG3r0uj70lP0FSQP+XwQ+8=' | base64 -d) --output text --query Plaintext | base64 -d

Finally, if you want to enter the encrypted content into a JSON structure, simply leave the `base64` undo/redo out, and change the output type to JSON, like so:

aws kms encrypt --key-id 3c436c82-eabe-4b58-996f-6ca3f808f237 --plaintext fileb://my-secret-key.pem --query CiphertextBlob --output json

Without redirection both the encrypt and decrypt commands output to standard output (as in the first example).

Protecting Internet-facing SSH access with MFA on Ubuntu 16.04 (while running standard SSH for allowed addresses)

When it comes to secure access to a remote server, such as an AWS EC2 instance, you have couple of options. The preferred option is to have the instance (or the server in a data center or other similar environment) within a private network (such as a VPC in AWS), only accessible by SSH over a VPN (either your own OpenVPN setup, or IPsec available from AWS). However, not having a fall-back SSH connectivity is not always practical or feasible, even if it be just to access a gateway instance that likely serves as a bastion host. This is obviously not applicable to environments where nothing is strictly ever done over SSH, and everything is only ever done over configuration management, but such environments are far and few between.

The following outlines my preferred method of setting up SSH access on the gateway instance. Because the configuration parameters differ between the MFA-protected, but IP-unrestricted SSH server, and the one that servers connections from the allowed addresses/CIDR ranges, it is best to run two separate SSH daemons.

Before starting this process, make sure that the normal OpenSSH access to your server/instance has been configured, as this article mainly outlines the deltas from the standard SSH setup. So let’s get started!

  1. Install `libpam-google-authenticator`:
    sudo apt-get install libpam-google-authenticator
  2. Make a copy of the existing `sshd_config`:
    sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config_highport
  3. Modify the newly created sshd_config_highpoprt:
    • Select a different port, such as 22222:
      Port 22222
    • Allow users who should be allowed to use the MFA-protected, but IP-unrestricted SSH access:
      AllowUsers myusername
    • Set Google Authenticator -compatible authentication method:
      # require Google Authenticator after pubkey
      AuthenticationMethods publickey,keyboard-interactive
    • Set `ChallengeResponseAuthentication`:
      ChallengeResponseAuthentication yes
    • If you have configured any `Match address` or `Match user` entries in your primary SSH server’s sshd_config file (whose copy we’re editing), remove them from the copy. For example, you might have something like this configured for the primary SSH instance:
      Match address 10.10.10.0/24
      PasswordAuthentication yes
      
      Match User root Address 10.10.10.0/24
      PermitRootLogin yes
      
      Match User root Address 100.100.100.50/32
      PermitRootLogin prohibit-password
      

      If you do, remove them from `sshd_config_highport`.

  4. Make a copy of `/etc/pam.d/sshd`, and modify the copy, like so:
    sudo cp /etc/pam.d/sshd /etc/pam.d/sshd2

    Then add on top of the `sshd2` file:

    auth required pam_google_authenticator.so

    .. and comment out the following line in the file:

    @include common-auth

    like so:

    # @include common-auth
  5. Now run `google-authenticator` as the user who you want to be able to log in as over the MFA-protected, but IP-unrestricted SSH-service. Do not use root; use an unprivileged user account instead! Once you run it, you will be prompted to answer: `Do you want authentication tokens to be time-based (y/n)`. Answer `yes`, and the system will display a QR code. If you don’t have a Google Authenticator -compatible app on your smart phone/tablet yet, install one. I recommend Authy (Android / iOS). Once you have installed it and created an account in it, scan the QR code off the screen, and also write down the presented five “emergency scratch codes” in a safe place. Then answer the remaining questions, all affirmative:

    – Do you want me to update your “~/.google_authenticator” file (y/n) y
    – Do you want to disallow multiple uses of the same authentication token? .. y
    – By default, tokens are good for 30 seconds and in order to compensate .. y
    – .. Do you want to enable rate-limiting (y/n) .. y

  6. Create a `ssh2` symlink to the existing `ssh` executable (a requirement by service autostart):
    sudo ln -s /usr/sbin/sshd /usr/sbin/sshd2
  7. Add the following in `/etc/default/ssh`:
    # same for the highport SSH daemon
    SSHD2_OPTS=
  8. Create a new `systemd` service file for the `ssh2` daemon:
    sudo cp /lib/systemd/system/ssh.service /etc/systemd/system/sshd2.service

    .. then modify the `sshd2.service` file; modify the existing ExecStart and Alias lines, like so:

    ExecStart=/usr/sbin/sshd2 -D $SSHD2_OPTS -f /etc/ssh/sshd_config_highport
    Alias=sshd-highport.service

    Note that the Alias name (as set above) must be different from the file name for the service. In other words, as in the example above, the file name is `sshd2.service`, the “Alias” must be set to something else that doesn’t previously exist in the `/etc/systemd/system` directory (i.e. in the above example `sshd-highport.service`).

    Then start the service, test it, and finally enable it (to persist reboots):

    sudo systemctl daemon-reload
    sudo systemctl start sshd2
    sudo systemctl status sshd2
    
    sudo systemctl enable sshd2
    sudo systemctl status sshd2

When you enable the service with `sudo systemctl enable sshd2`, a symlink is created by the name of the alias you defined. Following the above example, it would be like so:

`/etc/systemd/system/sshd-highport.service` [symlink] -> `/etc/systemd/system/sshd2.service` [physical file]

You’re all set! Other considerations: If you’re running this in AWS, remember to configure the Security Groups to allow access to the high port SSH from any source (assuming you want it to be accessed from any source), as well as adjust the ACLs and iptables/ufw if you use them. Furthermore, since the MFA-protected SSH port will be accessible publicly, it’s a good idea to install sshguard (some install info here) to prevent port knocking (besides stressing the system some, brute force attacks aren’t much of a threat since the port is now protected by the MFA which also implements access rate limiting).

Finally, since the MFA is time-based, it’s a good idea to make sure `ntp` is running on the your server/instance. Additionally I run `ntpdate` from system crontab once a day to make sure a possible greater drift than what the maximum ntp slew rate can correct in a reasonable amount of time is corrected at least once a day:

sudo apt-get update
sudo apt-get install ntp
sudo systemctl enable ntp

.. and in `/etc/crontab:`

# Force time-sync once a day (in case of a time difference too big for stepped adjustment via ntp)
0       5       *       *       *       root   service ntp stop && ntpdate -u us.pool.ntp.org && service ntp start

(`ntpdate` won’t run if `ntp` service is active)

And there you have it! Now when you connect to the server/instance over SSH from a random external IP (such as from a hotel) using a public key to your chosen user account, you will be prompted for an MFA code (which you have to enter from the Authy app) before the connection is authorized. And because the primary SSH daemon still serves the default SSH port 22, and because it is restricted to the private networks (plus possibly some strictly limited public IPs), those “known”, or “internal” IPs are not presented with an MFA challenge in order to get connected.