Second post in this series reviewed the basics of getting ssh working with GoldKey tokens. Here we focus on edge-cases and Mac specific problems.
Fine-print: compatibility with other applications
Invoking ssh from the command line is not the only use-case for SSH authentication. Many applications leverage SSH tunnels to protect otherwise unauthenticated/weakly-authenticated connections. One example already covered is the source-control system git. git invokes ssh in a straightforward manner, with some customization possible via an environment variable. Because all of this happens in a terminal, PIN prompt also appears inline on the same terminal session and everything works as expected after user enters correct PIN.
Such automatic compatibility does not hold true for more complex use cases, and depends to some extent on the way SSH authentication is invoked. Interestingly GUI applications can also work with cryptographic hardware if they are designed properly. For example Sequel Pro is a popular SQL client for OS X. It supports accessing databases over SSH tunnels through bastion hosts. When using a GoldKey for the SSH credential, PIN prompt will appear in a modal dialog:
This is compliments of the custom ssh-agent built into OS X, which includes changes from Apple to interface with the system keychain for collecting PIN/passphrase to unlock keys. Unfortunately this “value-added” version from Apple has a downside: it does not work correctly with hardware key.
Macintrash strikes back: key-agent troubles
One problem quickly discovered by Airbnb engineers during this pilot concerned SSH key-agent: adding credentials from the token to SSH agent does not work. In fact, it is worse than simply failing: it fork-bombs the machine with copies of ssh-pkcs11-helper process until no additional processes can be created. (pkill to the rescue for terminating all of them by name.) No one can accuse Apple for having tested enterprise scenario.
This creates two problems:
- Usability: GoldKey PIN must be entered each time. Arguably this is beneficial for security; guidelines for PIV clients even discourage PIN caching, since it can lead to problems when a computer is left unattended with the token attached. But there is also an inconvenience element to frequent PIN entry. This is more of a concern when the token is used for scenarios such as accessing a git repo, when frequent server interactions are common. (SSH to multiple targets in quick succession is rare although there are applications such as massively parallel SSH designed to do exactly that.)
- Functional limitation: Putting aside the security vs usability debate for PIN caching, there is a more fundamental problem with key-agent not supporting hardware tokens: it breaks scenarios relying on SSH agent-forwarding. For example, when the user connects to server A and from there tries connecting to server B, they need an agent running on their local machine and that agent to be forwarded to server A in order to authenticate to B using credentials on the GoldKey token. A less obvious example of “nested” connection involves logging into a remote machine and then executing a git pull there to sync a repository, which involves SSH authentication under the covers.
Luckily the limitation turns out to be a function of openssh binaries included out-of-the-box with Mavericks. Upgrading to 6.6p1 and making sure that all of the binaries executed– including ssh-agent and ssh-pkcs11-helper– point to this latest version solves the problem.
$ ssh-add -s /usr/lib/opensc-pkcs11.so Enter passphrase for PKCS#11: Card added: /usr/lib/opensc-pkcs11.so $ ssh-add -e /usr/lib/opensc-pkcs11.so Card removed: /usr/lib/opensc-pkcs11.so
The collateral damage however is losing PIN collection for GUI applications, such as the Sequel Pro example above. That functionality is dependent on custom OS X keychain integration which the vanilla open-source ssh-agent lacks. When the application is invoked manually from the terminal, we can see the PIN prompt appear on stdout. Not good.
One work-around is to manually add the key to the agent before invoking GUI applications. A better solution is provided by the Homebrew version of openssl which combines the keychain patch with latest openssh sources to satisfy both use cases.
The final post in this series will conclude with a discussion of setup, key-generation/credential provisioning and maintenance operations.