NixOps: How to deploy to an existing NixOS VM? - linux

Question

I have almost the same problem as in this question, but it was never answered:
nixops: how to use local ssh key when deploying on machine with existing nixos (targetEnv is none)?
I'm not using Terraform though. Just NixOS + NixOps. So far, I:
Created a new VM on Vultr
Did a standard NixOS install from the current iso (20.09 pre something), setting a root password
Enabled ssh with root password authentication and did a nixos-rebuild switch
Manually generated an ssh keypair on my laptop
sshed into the VM with the password and added the public key to /root/.ssh/authorized_keys
Now I can ssh into the VM manually with the new key, as expected:
ssh -i .secrets/vultrtest1_rsa root#XXX.XXX.XXX.XXX
Cool. Next, I copied the existing NixOS config files to my laptop and tried to wire them up to NixOps. I tried a minimal test1.nix, as well as adding the deployment."none" and/or users.users.root.openssh sections below.
vultrtest1
├── configuration.nix
└── hardware-configuration.nix
test1.nix
# test1.nix
{
network.description = "vultr test 1";
network.enableRollback = true;
vultrtest1 = { config, pkgs, ... } : {
deployment.targetHost = "XXX.XXX.XXX.XXX";
imports = [ ./vultrtest1/configuration.nix ];
# deployment.targetEnv = "none"; # existing nixos vm
# same result with or without this section:
deployment."none" = {
sshPrivateKey = builtins.readFile ./secrets/vultrtest1_rsa;
sshPublicKey = builtins.readFile ./secrets/vultrtest1_rsa.pub;
sshPublicKeyDeployed = true;
};
# same result with or without this:
users.users.root.openssh.authorizedKeys.keyFiles = [ ./secrets/vultrtest1_rsa.pub ];
};
}
In all cases, when I try to create and deploy the network NixOps tries to generate another SSH key, then fails to log in with it:
$ nixops create test1.nix -d test1
created deployment ‘b4ac25fa-c842-11ea-9a84-00163e5e6c00’
b4ac25fa-c842-11ea-9a84-00163e5e6c00
$ nixops list
+--------------------------------------+-------+------------------------+------------+------+
| UUID | Name | Description | # Machines | Type |
+--------------------------------------+-------+------------------------+------------+------+
| b4ac25fa-c842-11ea-9a84-00163e5e6c00 | test1 | Unnamed NixOps network | 0 | |
+--------------------------------------+-------+------------------------+------------+------+
$ nixops deploy -d test1
vultrtest1> generating new SSH keypair... done
root#XXX.XXX.XXX.XXX: Permission denied (publickey,keyboard-interactive).
vultrtest1> could not connect to ‘root#XXX.XXX.XXX.XXX’, retrying in 1 seconds...
root#XXX.XXX.XXX.XXX: Permission denied (publickey,keyboard-interactive).
vultrtest1> could not connect to ‘root#XXX.XXX.XXX.XXX’, retrying in 2 seconds...
root#XXX.XXX.XXX.XXX: Permission denied (publickey,keyboard-interactive).
vultrtest1> could not connect to ‘root#XXX.XXX.XXX.XXX’, retrying in 4 seconds...
root#XXX.XXX.XXX.XXX: Permission denied (publickey,keyboard-interactive).
vultrtest1> could not connect to ‘root#XXX.XXX.XXX.XXX’, retrying in 8 seconds...
root#XXX.XXX.XXX.XXX: Permission denied (publickey,keyboard-interactive).
Traceback (most recent call last):
File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/bin/..nixops-wrapped-wrapped", line 991, in <module>
args.op()
File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/bin/..nixops-wrapped-wrapped", line 412, in op_deploy
max_concurrent_activate=args.max_concurrent_activate)
File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 1063, in deploy
self.run_with_notify('deploy', lambda: self._deploy(**kwargs))
File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 1052, in run_with_notify
f()
File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 1063, in <lambda>
self.run_with_notify('deploy', lambda: self._deploy(**kwargs))
File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 996, in _deploy
nixops.parallel.run_tasks(nr_workers=-1, tasks=self.active_resources.itervalues(), worker_fun=worker)
File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/parallel.py", line 44, in thread_fun
result_queue.put((worker_fun(t), None, t.name))
File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 979, in worker
os_release = r.run_command("cat /etc/os-release", capture_stdout=True)
File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/backends/__init__.py", line 337, in run_command
return self.ssh.run_command(command, self.get_ssh_flags(), **kwargs)
File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/ssh_util.py", line 280, in run_command
master = self.get_master(flags, timeout, user)
File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/ssh_util.py", line 200, in get_master
compress=self._compress)
File "/nix/store/kybdy5m979h4kvswq2gx3la3rpw5cq5k-nixops-1.7/lib/python2.7/site-packages/nixops/ssh_util.py", line 57, in __init__
"‘{0}’".format(target)
nixops.ssh_util.SSHConnectionFailed: unable to start SSH master connection to ‘root#XXX.XXX.XXX.XXX’
What am I missing? Perhaps I can manually add the key NixOps just generated?
Update: I used SQLiteBrowser to look inside the NixOps state database and pasted the generated public key into authorized_keys. Now I can ssh in with the newly generated key manually, but NixOps still fails to deploy.

Answer 1

Solved it temporarily, in a not-very-satisfying way:
browsed the database for the public + private key NixOps generated
manually added those to authorized_keys on the VM
also added the old key to the local ~/.ssh with an entry in ~/.ssh/config
No idea why NixOps uses the local ssh config, or how to prevent that. The entry that works looks like:
Host XXX.XXX.XXX.XXX
HostName XXX.XXX.XXX.XXX
Port 22
User root
IdentityFile ~/.ssh/vultrtest1_rsa
Will wait a couple days, then mark this as the solution unless anyone can explain how to tell NixOps to use the local key from .secrets instead of ~/.ssh.

Answer 2

Looking at the source at
https://github.com/NixOS/nixops/blob/master/nix/options.nix
there is deployment.provisionSSHKey option
which says.
deployment.provisionSSHKey = mkOption {
type = types.bool;
default = true;
description = ''
This option specifies whether to let NixOps provision SSH deployment keys.
NixOps will by default generate an SSH key, store the private key in its state file,
and add the public key to the remote host.
Setting this option to <literal>false</literal> will disable this behaviour
and rely on you to manage your own SSH keys by yourself and to ensure
that <command>ssh</command> has access to any keys it requires.
'';
};
Maybe this can help? Once i'll get back to my Nixops machine, I'll give it a try.