macOS: ‘dig’ vs. ‘dscacheutil’ while using split DNS with Viscosity VPN client

If you’re using Viscosity VPN client on Mac, and have enabled the split DNS for the VPN domains, `dig` doesn’t work quite as someone familiar with Linux/*NIX would expect. This is because Apple has replaced the traditional resolver on macOS with something “more advanced”. Granted, it handles the split DNS gracefully, but as a result, using `dig` for the VPN domains without defining a DNS server for a query doesn’t produce any result even though resolution works otherwise in macOS.

This works:

~ dig +short random.ac a
52.4.179.30

But this does not:

~ dig +short internal.ville.sh a

However, this does:

~ dscacheutil -q host -a name internal.ville.sh
name: internal.ville.sh
ip_address: 10.50.50.10

To make things easier, I have created the following `zsh` alias:

alias dnsquery='dscacheutil -q host -a name'

However, I’ve forgotten this a few times when some time has passed since the last encounter with dig on internal domains. And then time goes down the drain trying to figure out if something is wrong with the DNS.. but it’s working all along! So I added a reminder for myself in form of another `zsh` alias:

dig() { echo && echo -e '\033[0;97m\033[41m Remember, this is a Mac! Use "dnsquery" instead! \033[0m' && /usr/bin/dig $@ } #macOS

And so:

~ dig +short internal.ville.sh a

 Remember, this is a Mac! Use "dnsquery" instead!

#oh right! :-)

~ dnsquery internal.ville.sh
name: internal.ville.sh
ip_address: 10.50.50.10

Remove DRM Easily (?) from Your Audible Purchases

NOTE: The following instructions are for Windows; I may later figure out the equivalent on macOS.

NOTE: The only way to remove DRM from an Audible audiobook is to play and record it digitally with your computer. If it’s a 17 hour book, the conversion takes 17 hours!

I listen to audiobooks a lot, and mostly I buy them from Audible. However, Audible’s own player sucks. Unless it has changed recently, its bookmarking system is non-existent, and it also couldn’t keep track of multiple books in progress at the same time. So I prefer to use the excellent Listen Audiobook Player on my Android phone (on iOS devices you might consider Bookmobile Audiobook Player). But it won’t play the Audible’s DRM-protected files, so somehow I have to get rid of the protection first. I’ve been using Aimersoft’s DRM Media Converter for the task for the last couple of years, but today I found out that it no longer works after a recent Windows update. It invokes iTunes (which is authorized to play the Audible DRM content) as it should, but for some reason it no longer ‘hears’ what iTunes plays back to it, so the mp3 is never written.

I spent a couple of hours figuring out a new solution. Even though this new solution requires a few of more components, it’s more stable and the required components are free/donationware, so you can get accomplish the task for free, or at least at a cost acceptable to you (I recommend pitching in to the Virtual Audio Cable folks for their fine piece of software!) ?

So let’s get started.

  1. Log in to your Audible account, then click on “Hi, {Yourname}! ˅” (on top of the Audible page), and navigate to Account Details > Update Settings, then scroll to the bottom of the page, and clear “Check for Audible Download Manager” checkbox if it is checked.
  2. Download and install Audible Manager. You’ll only use the Audible Download Manager from this installation; you’ll find it in the “AudibleManager” folder in the Start menu.
  3. Download and install iTunes. Once installed confirm that you’re using “Windows Audio Session” (which means system sound) in iTunes Preferences > Playback > “Play Audio Using”.
  4. Download and install Virtual Audio Cable. If you have a 64-bit system (which is likely), select the 64-bit installer when starting the installation (both are included in the download). Note that it needs to be installed as the Administrator, so you may need to right click on the installer, and select “Run as administrator”. Virtual Audio Cable is donationware, so once you have this setup working, consider returning to the VB-AUDIO Software website, and sending them a donation of your choice.
  5. Download and install Aktiv Mp3 Recorder. It is free.
  6. Open Aktiv Mp3 Recorder and configure it:

    aktiv_mp3

    1. Check detect silence
    2. Set silence duration at least to 20 seconds. If the audiobook you’re converting has long stretches of silence, set this to a longer amount so that the recording won’t stop prematurely. However, there will be this amount of silence at the end of the recording before the recording is automatically stopped when the book ends.
    3. Select CABLE Output (VB-Audio Virtual…) for the input device. If it doesn’t show up on the list (after you installed Virtual Audio Cable earlier during this process), reboot your computer and continue from this step.
    4. Select the output folder. The file name you’ll want to of course set for each book you’re converting.
    5. Select 128kBPS, 44Hz Stereo for the output format.
    6. Click on “Set Default” so that you don’t have to reconfigure this every time (the only thing that you need to change is the Target File name).
    7. When you’re ready to start recording, click on “Record” (but not yet!)
    8. Close the program (note that you have to exit the program in the systray where it remains after you close the main window).

The above steps need to be done only once to set up. The following steps are repeated whenever you download new content from Audible, and process the conversion:

  1. Once you have purchased an audiobook in Audible, go to your “Library” in Audible, and click on the “DOWNLOAD” button for that book. A file with name “admhelper.adh” will be downloaded. While each “DOWNLOAD” action (for any audiobook you have purchased) downloads an “admhelper.adh” file, each file is specific to the book which you downloaded it for. When you double-click on an “admhelper.adh” file, Audible Download Manager will open and immediately start downloading the corresponding DRM-protected audio file. The file has an “.aax” extension.
  2. Open iTunes and click on the drop-down in the top left corner. By default it is set to “Music”, but choose “Audiobooks” instead. Then drag and drop the downloaded audiobook file (with the “.aax” file extension) into iTune’s library pane on the right. If you haven’t played Audible content with iTunes previously, you will be prompted to authorize iTunes with Audible. Proceed to do so, and a browser window will open. Log in to Audible with the same credentials you purchased the book with, and click on the link “Click here to complete your activation!”. You are now ready to play the audiobook in iTunes, and you can test-play it if you like.
  3. To prepare for conversion, set the system audio input/output: go to the Control Panel, open “Sound”, and in “Playback” tab set the default to “CABLE Input” (make note of what was the previous default as you’ll want to return to it after the conversion is complete). Then go to the “Recording” tab, and set the default to “CABLE Output”. NOTE: If you use DFX or other sound enhancer that iTunes plays through, you should set the output (to “CABLE Input”) in that program’s settings.
  4. To prepare for conversion, open the Volume Mixer from the system tray (or hit Win+R, then type “sndvol”, and click “OK” to open the mixer). Select “CABLE Input” device from the Device drop-down if it’s not already selected, and mute System Sounds, and preferably the browser and other sound sources besides iTunes (any sounds made by the system or the browser will otherwise be included in the converted audiobook).
  5. Ready to go! Open iTunes, make sure the audiobook you want to convert is on the Audiobooks library page. Open Aktiv Mp3 Recorder, make sure the settings are as you set them earlier, then enter the audiobook’s name in the File name field. Click on “Record”, then switch (within 20 seconds or whatever you configured as Silence Duration) to iTunes, and start playing of the audiobook. Switch back to the Aktiv Mp3 Recorder, and make sure that the audio is coming in (the level indicators should be moving). Then wait. And wait. ? Note that you can go to the iTunes window and see on the top, next to the playing book’s title, how much time is remaining. After the entire book has played through (you won’t hear it while it’s playing), the Aktiv Mp3 Recorder will notice once there the set Silence Duration (20 seconds if you followed my steps) has elapsed, and stop the recording. You can then take the Mp3 file and tag it as you see fit (say, with Foobar2000, or any of the million other Mp3 players/taggers), copy it to your mobile device, and play it with the player of your choice!

Well, this wasn’t totally simple. ? But once you have set it up and ran through it once, it’s pretty quick to do. Obviously, after you’ve done with the conversion, you’ll want to unmute the system and browser sounds, and switch the playback/recording default devices as they were before you altered them.

I hope this is useful to someone out there! ?

Exploring GitHub Flavored Markdown local preview

I’ve been writing various lengthy documents for GitHub repositories recently, and as I push them to GitHub, and then view them in the browser, there’s almost always something wrong with the layout resulting in additional commits or amends/forced updates which I would rather avoid. I was initially using MacDown to edit/preview Markdown, but its rendering of GitHub Flavored Markdown (GFM) was way off. There had a to be a better way, and there is! I found two solutions that work reasonably well: the first one is Node-based vmd for general use (it watches the given file, and thus works with any editor), and the second is a set of plugins for Sublime Text 3. The former is now my choice for offline GFM previewing (such as on a plane), while the latter is my preferred choice because 1) Sublime Text 3 is already my go-to editor, 2) the Sublime Text plugins provide incredibly handy macros for, and context-highlighting of the Markdown being edited and, most importantly 3) the GFM is rendered through the GitHub API, ensuring that the rendered output matches the way GitHub renders it 100%. Both `vmd` and the Sublime Text solution work on macOS, Windows, and Linux Desktop.

Some other options that I looked at included Grip (a cool idea, but it kept crashing on me), or Atom‘s GFM preview (which, even with the right extensions installed, wasn’t accurately GFM, which is strange considering that Atom is made by GitHub; if you know how to make Atom render GFM through the GitHub API, please comment this article).

First, `vmd`. It’s easy to set up, simply first install Node.js on your platform, then add `vmd` with `npm` as a global:

npm install -g vmd

Now simply point `vmd` to the file you’re editing, and it gets re-rendered every time the file is saved as long as `vmd` is running.

vmd SomeFileName

That’s all. You can use any editor to edit the file.

However, if you’re using Sublime Text 3, give the following a try; it makes Markdown editing outright fun. The following instructions are for ST3.

  1. Install Sublime package manager if it’s not yet installed. Then restart Sublime.
  2. Install the following packages with the Package Manager:
    `Ctrl-Shift-p [Win] / Cmd-Shift-p [macOS] > Package Control: Install Package`

    Restart Sublime.

  3. Install monokaiC theme (read the installation instructions on its GitHub page).The target location depends on the platform, see this StackOverflow answer for details, but use the User subdir under the directory listed for your platform.
  4. SETTINGS

    1. In Markdown Preview User Settings:
      `Preferences > Package Settings > Markdown Preview > Settings — User`

      Add:

      {
         "git_oauth_token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
         "github_mode": "gfm",
         "enable_autoreload": true
      }

      ** Get the `git_oauth_token` from GitHub. Log in to your GitHub account, then:
      `Settings > Developer Settings > Personal Access Tokens`, or use generatetoken.sh script. No scope permissions are required for a token for it to be valid for Markdown rendering. Note that without a token set you’re limited to about 60 renders per hour, while with the token set you can make 5000 requests per hour (which you’ll never hit, since the re-render only occurs on file save).

      ** Make sure you have a comma at the end of each line, except the last parameter line in the settings

    2. In Markdown Editing User Settings for GFM:
      `Preferences > Package Settings > Markdown Editing > Markdown GFM Settings — User`

      Add:

      {
        "color_scheme": "Packages/User/ME-MonokaiC.tmTheme",
        "extensions":
        [
          "md"
        ]
      }
    3. In Key bindings
      `Preferences > Key Bindings — User`

      Add:

      { "keys": ["alt+l"], "command": "markdown_preview", "args": {"target": "browser", "parser":"markdown"} },
      { "keys": ["alt+m"], "command": "markdown_preview", "args": {"target": "browser", "parser":"github"} }
      

    Restart Sublime.

  5. Enable Simple Reload:
    `Ctrl-Shift-p [Win] / Cmd-Shift-p [macOS] > Live Reload: Enable/Disable Plugins > Enable Simple Reload` You need to reactivate Simple Reload like this whenever you restart Sublime, and want to use the auto-reload.

All set! Open an existing markdown (.md) file, or start a new file and save it with `.md` extension. Then hit `alt-m` for GitHub-rendered preview, or `alt-l` for locally rendered (but less accurate for GFM) Markdown preview. When you make changes and save the file, the preview gets automatically refreshed in your browser using the originally selected method (i.e. GitHub API or local).

Be sure to read the MarkdownEditing documentation for awesomely useful keyboard macros and other support for Markdown! This will also give you syntax highlighting in fenced code blocks when you postfix the first triple-backtick with supported language identifier, like so:

```sql

Happy GitHub Markdown editing! Octocat

Update 2018-03-02: The `git_oauth_token` must be set in Markdown Editing package settings, not in global Sublime settings. This correction has been reflected in the above instructions.

Interactive AWS CLI Query Filtering with JSONPath

Since I seem to be on a kick posting AWS articles, here’s one more!

Rather than using shell shenanigans (`grep`, `awk`, `sed`, and friends) to filter AWS CLI output from `text` output type, the integrated JMESPath query command makes it easy to filter the output before `aws` command even spits it out. However, a JMESPath query can become pretty complex when using it to dig some information deep in the output structure. Of course, while a query can be complex, digging the same information out with a string of piped `greps`, `seds`, and `awks` is even more messy, not to mention fragile.

A very handy trick exists that makes building JMESPath queries a lot easier! This technique can be used for ad hoc queries when you simply need to find some information from a large output, but it can just as well be used as a testing ground for a query to be embedded in a script. Here’s how it works (assuming you’re on some kind of *nix command line… macOS, Ubuntu, something; though it may even work in Windows command prompt since it’s Python based). First, install `jmespath-terminal`:

sudo pip install jmespath-terminal

Now (with `aws` CLI obviously configured), retrieve the unfiltered output from which you wish to dig the important nuggets of information. Here I’m listing all the security groups. Note that the output format needs to be set to `json` (if you have configured output format to something else (i.e. `table` or `text`) in `~/.aws/config`, here `–output json` will override it (on the other hand, `json` is the default output format if you haven’t set it otherwise). Save the output into a file. Then launch `jpterm` with the file name as the sole argument.

aws ec2 describe-security-groups --output json > ~/securitygroups.json
jpterm ~/securitygroups.json

And…

jmespath

Now you’ll have an interactive text-based console, where you can experiment with JMESPath queries! In the left pane is your original data, while on the right you’ll see the result of the JMESpath query immediately as you type!

Check out the official JMESPath Specification for everything you can do with the embedded JMESPath queries. And while you’re reading, also check out James Lawson’s 2015 article on JMESPath queries in the AWS CLI for more insights.

One more thing. I mentioned above that you must use `–output json` to get the output in a format that `jsonpath-terminal` can understand. That is accurate, but if you’re building the JMESPath query for a script, once you’re satisfied with the query, you can switch the output format to `–output text`, and AWS will use the query just fine, while the final output will be in `text` format (i.e. no quotes around a string, no JSON artifacts). This can be useful especially if the final output result set is very small, such as a single value, or a simple list of IDs, as such output may then be easier to process further with your favorite shell commands.