Interactive AWS CLI Query Filtering with JSONPath

Since I seem to be on a kick posting AWS articles, here’s one more!

Rather than using shell shenanigans (`grep`, `awk`, `sed`, and friends) to filter AWS CLI output from `text` output type, the integrated JMESPath query command makes it easy to filter the output before `aws` command even spits it out. However, a JMESPath query can become pretty complex when using it to dig some information deep in the output structure. Of course, while a query can be complex, digging the same information out with a string of piped `greps`, `seds`, and `awks` is even more messy, not to mention fragile.

A very handy trick exists that makes building JMESPath queries a lot easier! This technique can be used for ad hoc queries when you simply need to find some information from a large output, but it can just as well be used as a testing ground for a query to be embedded in a script. Here’s how it works (assuming you’re on some kind of *nix command line… macOS, Ubuntu, something; though it may even work in Windows command prompt since it’s Python based). First, install `jmespath-terminal`:

sudo pip install jmespath-terminal

Now (with `aws` CLI obviously configured), retrieve the unfiltered output from which you wish to dig the important nuggets of information. Here I’m listing all the security groups. Note that the output format needs to be set to `json` (if you have configured output format to something else (i.e. `table` or `text`) in `~/.aws/config`, here `–output json` will override it (on the other hand, `json` is the default output format if you haven’t set it otherwise). Save the output into a file. Then launch `jpterm` with the file name as the sole argument.

aws ec2 describe-security-groups --output json > ~/securitygroups.json
jpterm ~/securitygroups.json

And…

jmespath

Now you’ll have an interactive text-based console, where you can experiment with JMESPath queries! In the left pane is your original data, while on the right you’ll see the result of the JMESpath query immediately as you type!

Check out the official JMESPath Specification for everything you can do with the embedded JMESPath queries. And while you’re reading, also check out James Lawson’s 2015 article on JMESPath queries in the AWS CLI for more insights.

One more thing. I mentioned above that you must use `–output json` to get the output in a format that `jsonpath-terminal` can understand. That is accurate, but if you’re building the JMESPath query for a script, once you’re satisfied with the query, you can switch the output format to `–output text`, and AWS will use the query just fine, while the final output will be in `text` format (i.e. no quotes around a string, no JSON artifacts). This can be useful especially if the final output result set is very small, such as a single value, or a simple list of IDs, as such output may then be easier to process further with your favorite shell commands.

Easy MFA and Profile Switching in AWS CLI

UPDATE 2019-01-15: I have released a much-improved v2.0 of the awscli-mfa.sh script. Its location has also changed; it can now be found at https://github.com/vwal/awscli-mfa

An updated blog entry is forthcoming; the article below discusses version 1.0 which was completely rewritten in this new iteration.

As I alluded in the previous post, I have released another AWS script that works in conjunction with the AWS CLI Key Rotation script. This one is for easily activating an MFA session while on the command line. It also makes it easy to switch between the different configured AWS profiles.

While MFA is perhaps more frequently used with the web console sessions, in some ways the API keys are even more of a security risk. Whereas people usually memorize the console password, or keep it in a password manager, the API keys sit in plaintext in `~/.aws/credentials` file. All it takes is a malware that dumps the content of that known file location to a hacker, or perhaps a stolen laptop, or a stolen unencrypted flash drive (that you used to transfer the API keys to your new laptop, say), and a third party will have access to as much of your AWS assets as the IAM policy associated with the key allows. Admin keys? Full access! For this reason, I advocate the use of MFA especially for the keys with wide access, and that are intended for interactive use. Of course roles should be used as much as possible, and as much as possible should be automated, but still, in most environments people use the API keys for various command line tasks and to run utility scripts from the admins’ laptops.

Enter awscli-mfa.sh! This script makes it easy to select a profile that has MFA configured, to initialize an MFA session, and to make the MFA session default for the current console session (via an environment variable). The script is obviously interactive since it’ll prompt for an MFA one-time passcode. When you run it, it’ll query the AWS profiles configured in `~/.aws/credentials`, and list them out, along with the actual IAM username associated with each profile (since the profile names are arbitrary, and as such can be set to anything). The script will also indicate whether a virtual MFA has been configured for the profile. You can select to activate a profile that doesn’t have MFA configured, but obviously then an MFA code is not prompted for, and MFA is not used. When the chosen profile does have MFA configured, the script will prompt for the one-time passcode (e.g. from Authy or Google Authenticator), initialize the session, and output the environment export variable for easy cut-and-paste to the console (when ran on a Mac, or in a Linux GUI that has `xclip` installed, the export string is automatically copied on the clipboard). Paste on the command line, hit Enter, and the MFA profile is active!

Once an MFA session has been activated and you run the script again, you’ll see the newly initialized MFA profile under its base profile. You can switch away from it to another profile, and then back again. The MFA profile is indicated either in “OK” or “LIMITED” status. The latter may suggest that the IAM policy session duration has been set to a shorter period than what was configured in this script (you can adjust the token lifetime duration in the script on line 16. It should be set to match the session duration set in the MFA enforcement policy (if the policy duration times out first, the key appears still valid, but it’s no longer authorized for any transaction on the AWS). Note that if the IAM policy for the profile is limited so that it isn’t allowed to query `aws iam get-user` for itself, you’ll see the “LIMITED” status even when the MFA session is valid. Once the session token expires, the MFA profile disappears from the session list (even though you can see it still if you `cat ~/.aws/credentials` file.

A good write-up and an example MFA enforcement policy can be found here. I used their policy with a slight modification: I set the maximum MFA session duration to six hours by altering the Condition for the last two Sids in the policy, like so:

"Condition": {
  "NumericGreaterThanIfExists": {
    "aws:MultiFactorAuthAge": "21600"
  }
}

As I mentioned in the previous post (“AWS CLI Key Rotation Script for IAM Users revisited“), you can use an MFA session initialized with this script to rotate the keys of the MFA session’s base profile that doesn’t have permissions for anything without an active associated MFA session. As long as you have an MFA session initialized for the profile whose keys you want to rotate, the key rotation script will detect the presence of the MFA session, and ask if you would like to use it to execute the key rotation for the base profile. The MFA sessions are not visible in the key rotation session list because their keys are transient, only valid for the length of an MFA session.

The script was written for macOS, but portability for Ubuntu (and so, probably most/all Linux distros) was added.

The awscli-mfa.sh script is available from its GitHub repository.

I recommend Authy for the virtual MFA (it’s available both for iOS and Android).

As always, comments or improvement suggestions are welcome (either as comments to this post, or as PRs to the repository)!

UPDATE 2018-02-19: I’ve updated the script, fixed some minor bugs for some edge cases, cleaned up its formatting, clarified the output, and added a companion script (source-to-clear-AWS-envvars.sh) that can be used to clear any AWS secrets/variables that may be set in the environment through the use of the awscli-mfa.sh script.

AWS CLI Key Rotation Script for IAM Users revisited

In April of this year I published a Bash script for rotating the default AWS API keys configured in `~/.aws/credentials` file. Now I have improved on the original script, adding the following functionality:

  • The script is now fully interactive, and supports multiple profiles. It was an interactive script before, potentially requiring user input during its run, and as such was intended for manual execution. Now the script presents a list of configured profiles from which you can choose a profile whose keys you want to rotate.
  • The script works with Multi-Factor Authentication. At work we have been moving increasingly to MFA/2FA, and have also started enforcing MFA use from the users (this policy works great for that purpose). Since MFA enforcement for AWS console and for the CLI API cannot be separated for a given IAM user for the most part (since the console is just a GUI for the API), there had to be a solution for relatively convenient use of MFA on the command line. My second script (more detail in the next post) offers that capability, and now the key rotation script is aware of it. If you’re rotating the keys for an IAM account whose MFA use is enforced, the script now detects an existing MFA profile (created by my CLI MFA script), and can use it to authorize key rotation for the base profile, which might otherwise not be authorized to execute the key rotation operation.
  • The listing of the configured profiles includes the current keys (two concurrent keys is the recommended maximum by policy), the ages of the keys, the actual IAM username (since the profile name is arbitrary, and as such can be set to anything), and the access status of the profile (‘OK’, or ‘LIMITED’; the latter is displayed when the profile doesn’t appear to have normal access during the query process — for example, it may result from MFA enforcement)
  • Many AWS errors are also masked and translated to more user-friendly outputs. If a profile doesn’t have a valid key, the script handles it gracefully and displays: “CHECK CREDENTIALS!” next to that profile (and obviously it cannot give more detail about such profile, or offer the option to rotate its keys).
  • The script was originally written for macOS, but it has now been tested on Ubuntu, and portability has been added (hence it likely works on other Linux distros as well).

The latest version of the aws-iam-rotate-keys.sh script is available from the same repository as before.

Workflow magic (or so it seems :-)

A few days ago I came across the solution to a workflow issue I had been for long wishing could be done. Today I solved another one, and since they’re connected, I outline them below. Maybe they’ll make someone’s day! 🙂

I often work with remote servers that are almost always some flavor of Linux, most frequently Ubuntu/Debian, but also CentOS/Redhat. Despite of being a lifelong geek, I really dislike `vi`, finding it massively unintuitive. Maybe I haven’t figured out its intricacies, but I doubt it – it just doesn’t jive with me. In the 90’s I remembered most of Emacs’ chords by heart (or more likely by muscle memory), and really enjoyed using it, but for some reason I eventually stopped using it and only now am picking it up again. For now, however, my go-to editor in Linux is `joe`, which is handy enough (more so than `pico` or `nano`), but none of them still are as comfortable and flexible to use than a good GUI editor. In Windows & macOS my editor of choice is Sublime Text 3, and a few days ago I came across this answer in StackOverflow. Enter rmate/rsub! To make the magic happen, first install a small `rmate` script on your server (but call it `rsub` since you’re using Sublime Text rather than TextMate):

sudo curl -L --output /usr/local/bin/rsub https://raw.github.com/aurora/rmate/master/rmate
sudo chmod +x /usr/local/bin/rsub

Or perhaps like this (requires `git`, but makes it easier to keep rmate/rsub up-to-date by executing `git pull` in the cloned directory):

cd /opt
git clone https://github.com/aurora/rmate.git
ln -s /opt/rmate/rmate /usr/local/bin/rsub

Then add a few lines in `~/.ssh/config` on your Mac (Windows users, keep reading, you haven’t been forgotten):

Host *
  RemoteForward 52698 127.0.0.1:52698
  UseKeyChain no

To exclude some domains where you’ll never use `rsub`, like GitHub:

Host github.com
  Hostname github.com
  User myUserName
  ForwardAgent no
  ClearAllForwardings yes
  IdentityFile ~/.ssh/myGitHubKey

(the significant line above is `ClearAllForwardings yes`)

Then install `rsub` package in Sublime (easiest done with Package Control), and now `rsub someFileName` command on the remote opens `someFileName` in Sublime on Mac! Like magic!! No need to mess with FTPS. Just one word of caution.. if you put the Mac to sleep, the “rsubbed” file may become disconnected from the remote, so when you make the first save after resuming work, verify that the changes are indeed saved, or close the file and reopen it from the remote.

`rsub` also works with Sublime Text in Windows. The only part of the above setup that differs on Windows is the SSH config since Microsoft’s effort to create a native OpenSSH port for Windows is slow going (“non-production ready pre-release v0.0.17.0” with 161 open issues as of writing of this post). I usually use the excellent commercial SSH client VanDyke’s SecureCRT as the SSH terminal in Windows. I initially thought reverse port forwarding would not be possible in it, but alas, I stand corrected! Van Dyke’s always helpful and knowledgeable technical support pointed out how to achieve the desired port forwarding in it (the config section in question was titled: “Port Forwarding – Remote/X11”, and that indeed meant “remote OR X11”, not “remote X11”, as I had read it. So regardless of whether you use PuTTY (or its derivatives like KiTTY) or SecureCRT, you setting up the port forwarding is a snap. I’ll outline below PuTTY’s setup first, then SecureCRT’s equivalent configuration.

Install and fire up PuTTY. Just to make sure you start from a clean slate, click on “Default Settings” and then “Load”:

putty-1-load-defaults

Set the name (or IP) of the server you’ll be running `rsub` from:

putty-2-set-server-name

If you’re not using PuTTY’s key agent Pageant to authenticate, enter the path for your private RSA key. Note that it must be in `PPK` format (if you have PEM format key, you need to convert it with PuTTYgen, another PuTTY utility program):

putty-3-set-key

Add the remote listener port:

putty-4-add-remote-listener-port

[OPTIONAL] If you’re NOT using this PuTTY session for the shell (i.e. if you’re using for that purpose some another program that lacks the reverse forwarding capability), disable the shell for this connection:

putty-5-do-not-start-shell

[OPTIONAL] Similarly, if you’re not using this PuTTY session for the shell, set the window size to a small value:

putty-6-set-window-size

Finally, back in the Session tab, save the session: give it a name, then click on “Save”. If this is a shell-less RSUB session only, give a descriptive name:

putty-7-save-session

Now you have configured a RSUB session in PuTTY. For convenience, you may want to create a desktop link to activate the [RSUB] session. If so, right click on the desktop and select “New > Shortcut”, then enter the path to `putty.exe`, followed by `-load`, and the name you gave the session in PuTTY:

putty-8-create-shortcut

Finally, give a descriptive name for the desktop link:

putty-9-name-shortcut

And now you have a link that opens the RSUB channel to your remote server:

putty-10-link-image

Click on it, and it opens a small session window. As long as you keep it open, you can use the `rsub` command on the remote to open and edit remote files on your local Sublime Text!

And now, the same for SecureCRT:

Once you have configured a profile otherwise (set the hostname, username, and authentication information – [preferably] either a PEM format key, or a password [if allowed by your server]), head to Connection > Port Forwarding > Remote/X11, like so:

rsub-scrt-1-add-remote-port-forwarding

Click on “Add”, and enter a name for this tunnel, such as “RSUB” here, and the port 52698. When you enter it in the “Remote > Port” field, the “Local > Port” is automatically filled out for you:

rsub-scrt-2-add-remote-port-forwarding

And now you have the `rsub` port forwarding in place in SecureCRT!

rsub-scrt-3-remote-port-forwarding-added

Now when you open the session, you automatically have a `rsub` port open as well, and once you open Sublime Text on Windows, you can proceed to type `rsub someFileName` on the server, and it opens in Sublime.

So far so good (remote editing is working great!). Today I needed to diff an old configuration file against the new one, and the CLI `diff` output was too tedious to decipher. Then it occurred to me that maybe it would be possible to somehow use my favorite GUI diff utility, Beyond Compare with Sublime — and it is! The only thing to note is that when you open Beyond Compare from the files open in Sublime, you’ll need to save the completed diff first in Beyond Compare, and then in Sublime. Note that the file change indicator doesn’t light up in Sublime even when it has received the changes resulting from the diff reconciliation in Beyond Compare. The changes are received in Sublime when you save them in Beyond Compare, but not saved from Sublime [to the server over `rsub`] until you explicitly hit save in Sublime.

Now.. the files opened in Sublime from a remote using `rsub` can be diffed by this method with Beyond Compare! Working with remote files just became a lot more fun! 🙂

Update 2017-11-07: The article was updated with the latest `rsub` installation details, and PuTTY configuration instructions were added.

Update 2017-11-09: Van Dyke’s technical support pointed out to me that the reverse port forwarding can be accomplished in SecureCRT as well, so I corrected the article, and added the configuration details for SecureCRT.

Update 2017-11-24: I just noticed you can open two files from two different servers (using `rsub`), or one file from a server, and one file locally, and then compare them with Beyond Compare as described above. So cool!

Update 2018-01-17: Note that if you have a connection open from multiple machines, the first one that reverse-forwards the port `52698` receives the file when you run `rsub someFileName` on the remote. This can be confusing if you work on same remotes from multiple systems, so if you so choose, you can map a different port from different laptops/desktops, like so:

laptop – `RemoteForward 52698 127.0.0.1:52698`
desktop – `RemoteForward 52699 127.0.0.1:52698`

(NOTE: The first port value is the forwarded remote port; the second value is the local port which always remains at `52698` as it corresponds to the Sublime’s rsub extension port set in Sublime > Preferences > Package Settings > rsub > port)

Then on the remote system set up couple of aliases, respectively (here we assume the `rmate` repo was cloned in `/opt/rmate` rather than downloaded with `curl`):

alias rsubl='/opt/rmate/rmate --port 52698'
alias rsubd='/opt/rmate/rmate --port 52699'

If you set up multiple `rsub` alternatives like this, you may not want to copy/symlink the `rmate` executable to `/usr/local/bin/rsub` as that way you have to use the the port-specific aliased commands instead.