SSH, Vagrant and Private Key -> Host mapping

I've been using a lot of different Linux systems (Linuxes?) lately so it's been a case of Vagrant to the rescue… however, there's a problem. Great as Vagrant is, and it is, the problem is that I'm working with multiple Vagrant instances that I need to ssh into… directly (because Ansible) and this causes issues with ssh and my known_hosts.

Why Ansible? Well to install and configure different Swift builds on different platforms

The biggest bug bears are different private keys for each Vagrant instance and multiple entires with different host fingerprints for the same hostname

Defining host with unique keys

The first problem is simple to fix, for each Vagrant instance we create a Host entry in ~/.ssh/config. The config file saves you a lot of time and brain space by letting you setup a Host definition that you can use anywhere ssh is used (like rsync or Ansible).

Back to our problem, for each Vagrant instance we can define a seperate Host entry that includes a useful alias, the username to log in with, the port to ssh to and the location of the private key to use, something like this:

#Development
IdentitiesOnly=yes

#Vagrant instance in Development/Websites
Host localweb
  HostName localhost
  Port 2222
  User vagrant
  IdentityFile ~/Development/Websites/.vagrant/machines/default/virtualbox/private_key

#Vagrant instance in Development/Ansible
Host localansible
  HostName localhost
  Port 2223
  User vagrant
  IdentityFile ~/Development/Ansible/.vagrant/machines/default/virtualbox/private_key

#Vagrant instance in Development/SwiftServer
Host localswift
  HostName localhost
  Port 9001
  User vagrant
  IdentityFile ~/Development/SwiftServer/.vagrant/machines/default/virtualbox/private_key

N.B. Different ports are only needed for each instance if you intend on having more than one running at the same time.

Problematic

Now that the first issue is resolved, it's clear I've not explained the second problem very well… maybe an example will help. In the following snippet I'm ssh'ing into a CentOS Vagrant instance… using specific port and identity parameters.

me:CentOS7: ssh vagrant@127.0.0.1 -p 2222 -i .vagrant/machines/default/virtualbox/private_key
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:EDR3Dex/fIdTsB4Siewx+wiXVho+fxAFl/18yLE89ss.
Please contact your system administrator.
Add correct host key in /Users/me/.ssh/known_hosts to get rid of this message.
Offending RSA key in /Users/me/.ssh/known_hosts:114
RSA host key for [127.0.0.1]:2222 has changed and you have requested strict checking.
Host key verification failed.
me:CentOS7:

The reason this error is being thrown is because I have previously used almost the identical settings to ssh into a Ubuntu instance, which of course has a different key associated with it.

So the problem is that a line in the ~/.ssh/known_hosts matches the current request (erroneously but anyway) based on the host identity. If you cat your know_hosts file you'll see that each line starts with a host identity, a space and then the host key signature… similar to these:

[127.0.0.1]:2222 randomletters+thatarethekeyfingerprintiidentifyingthehost
[127.0.0.1]:8080 randomlettersthat+arethekeyfingerprintiidentifyingthehost
[::1]:8080       randomlettersthat+arethekeyfingerprintiidentifyingthehost
localhost:2222   randomlettersthatare+thekeyfingerprintiidentifyingthehost

As you can see these are all references to the loopback interface (i.e. the local computer), this is a problem because the entries are added sequentially and the lookup isn't that smart for resolving if the host you're connecting to is actually known… As far as I can tell it stops on the first match so when you're running multiple virtual machines with similar configurations on your local computer, you will eventually hit this problem.

In the error above, on line 114, it finds a match for a host at 127.0.0.1 it's just not the right match for that host.

A Solution

Before you read any further — remember this solution is only for your local development machine and these configuration changes should not be used on anything else, definitely not on a production, staging or QA system.

My first thought was to find a way to set things up so the correct match was made (in this case on line 116) — but that's not possible.

It turns out the solution for this type of development issue is pretty straight forward, it's just that a little reverse thinking is required, which is why the solution isn't obvious.

Since we can't make it find the right entry in known_hosts the trick is to make sure there isn't a match in the first place in a way that doesn't stop Ansible or whatever other tool we're using from working.

To this, we go back to that friend of SSH users everywhere, the config file. So, as you should know the config file lets you setup hosts with all their unique attributes including user, port, identity file, hostname and a host alias (amongst many other things) Here's part of my config, as an example:

#Development
IdentitiesOnly=yes

#Vagrant instance in Development/Websites
Host localweb
  HostName localhost
  Port 2222
  User vagrant
  IdentityFile ~/Development/Websites/.vagrant/machines/default/virtualbox/private_key
  UserKnownHostsFile /dev/null

#Vagrant instance in Development/Ansible
Host localansible
  HostName localhost
  Port 2223
  User vagrant
  IdentityFile ~/Development/Ansible/.vagrant/machines/default/virtualbox/private_key
  UserKnownHostsFile /dev/null

#Vagrant instance in Development/SwiftServer
Host localswift
  HostName localhost
  Port 9001
  User vagrant
  IdentityFile ~/Development/SwiftServer/.vagrant/machines/default/virtualbox/private_key
  UserKnownHostsFile /dev/null

As you can see I've defined three Vagrant instances that all point to my local machine (i.e. HostName localhost) but I've given each a different alias (e.g. Host localswift) and notably, different ports and the IdentityFile is the one in each of the instances .vagrant directory. So far all pretty basic stuff - so how do these configurations fix my problem?

Have a look at that last line for each Host, this option is defined as follows:

UserKnownHostsFile

Specifies a file to use for the user host key database instead of ~/.ssh/known_hosts.

The trick is to specify /dev/null as the file to use — this works because the null device is a valid file descriptor or in Wikipedia's words "null device is device file"

So now the entries for the localhost/127.0.0.1/::1 won't be added anywhere that can be checked and thus solving our original issue —  but now we have another problem.

As a result of the UserKnownHostsFile change, the ssh agent, rightly, stops everything and asks if it's OK to use this "new" host as because, hey, I don't have this fingerprint on file anywhere. So, when I ssh localweb I (or Ansible or some other tool) has to interact with the ssh agent:

ssh localweb
The authenticity of host '[localhost]:2222 ([127.0.0.1]:2222)' can't be established.
ECDSA key fingerprint is SHA256:eUgqRdjBZBO61tKa3fLJwCAu5E2lOZEy+c1Xer/DOY8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[localhost]:2222' (ECDSA) to the list of known hosts.

Ok, the solution is to actually tell ssh to not be so strict about hostkey fingerprints for this particular host. That's where the StrictHostKeyChecking no option comes in. With that option added, you still get a warning but that's just sent to stdout and it's easily ignored by you and tools like Ansible.

With this final option added our Host configurations now look like this:

#Vagrant instance in Development/Websites
Host localweb
  HostName localhost
  Port 2222
  User vagrant
  IdentityFile ~/Development/Websites/.vagrant/machines/default/virtualbox/private_key
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null

and ssh only outputs a warning about the key being added to the list of known_hosts, which is actually /dev/null

$ ssh localweb
Warning: Permanently added '[localhost]:2222' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 3.13.0-96-generic x86_64)

Summary

1. Use ~/.ssh/config to define each of your virtual hosts
2. In your Host definition point known_hosts to /dev/null
3. Turn off strict host key checking for your newly defined Host
4. SSH to the host alias defined in your ~/.ssh/config file e.g. ssh localweb