How to setup SSH keys for buildbot user - linux

Question

I'm trying to setup SSH keys for Buildbot's buildbot user, so it can clone my private repo without needing a username and password.
This is how I'm testing:
sudo -u buildbot bash
export HOME=/var/lib/buildbot
cd $HOME
mkdir git
cd git
git clone git#bitbucket.org:myproject/myrepo.git
the last step fails with:
Cloning into 'myrepo'...
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Why am I getting this error? I've confirmed my SSH key is correct with:
/var/lib/buildbot/.ssh folder exists with the correct permissions
/var/lib/buildbot/.ssh/id_rsa file exists with the correct permissions
/var/lib/buildbot/.ssh/id_rsa.pub file exists with the correct permissions
the content of /var/lib/buildbot/.ssh/id_rsa.pub file has been registered in my Bitbucket account as my SSH key
If I run:
ssh-agent bash -c 'ssh-add /var/lib/buildbot/.ssh/id_rsa; git clone git#bitbucket.org:myproject/myrepo.git'
then it clones my repo perfectly. Why isn't the normal command able to access my SSH key?
Edit: The output of ssh -vvv git#bitbucket.org is:
OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g 1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "bitbucket.org" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to bitbucket.org [104.192.143.2] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/buildbot/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/buildbot/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/buildbot/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/buildbot/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/buildbot/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/buildbot/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/buildbot/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/buildbot/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
debug1: Remote protocol version 2.0, remote software version conker_1.0.288-e9ef8ea app-123
debug1: no match: conker_1.0.288-e9ef8ea app-123
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to bitbucket.org:22 as 'git'
debug3: hostkeys_foreach: reading file "/home/buildbot/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/buildbot/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from bitbucket.org
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01#openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ssh-rsa-cert-v01#openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01#openssh.com,ecdsa-sha2-nistp384-cert-v01#openssh.com,ecdsa-sha2-nistp521-cert-v01#openssh.com,ssh-ed25519-cert-v01#openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib#openssh.com,zlib
debug2: compression stoc: none,zlib#openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-dss,ssh-rsa
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,arcfour256,arcfour128
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,arcfour256,arcfour128
debug2: MACs ctos: hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: MACs stoc: hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256#libssh.org
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-rsa SHA256:zzXQOXSRBEiUtuE8AikJYKwbHaxvSc0ojez9YXaGp1A
debug3: hostkeys_foreach: reading file "/home/buildbot/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/buildbot/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from bitbucket.org
debug3: hostkeys_foreach: reading file "/home/buildbot/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/buildbot/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from 104.192.143.2
debug1: Host 'bitbucket.org' is known and matches the RSA host key.
debug1: Found key in /home/buildbot/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug2: set_newkeys: mode 0
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS received
debug2: key: /home/buildbot/.ssh/id_rsa ((nil))
debug2: key: /home/buildbot/.ssh/id_dsa ((nil))
debug2: key: /home/buildbot/.ssh/id_ecdsa ((nil))
debug2: key: /home/buildbot/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/buildbot/.ssh/id_rsa
debug3: no such identity: /home/buildbot/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /home/buildbot/.ssh/id_dsa
debug3: no such identity: /home/buildbot/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/buildbot/.ssh/id_ecdsa
debug3: no such identity: /home/buildbot/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/buildbot/.ssh/id_ed25519
debug3: no such identity: /home/buildbot/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).
Edit: Updating buildbot's home directory in /etc/passwd to point to /var/lib/buildbot causes more problems. Now I can't even access buildbot's home directory as buildbot:
$ sudo -u buildbot bash
$ echo $HOME
/home/ubuntu
$ export HOME=/var/lib/buildbot
$ cd $HOME
bash: cd: /var/lib/buildbot: Permission denied

Answer 1

Buildbot now supports specifying private SSH keys (these changes will be released as part of buildbot 1.3.0). See https://github.com/buildbot/buildbot/pull/4178 and https://github.com/buildbot/buildbot/pull/4160
Disclaimer: I'm the author of these PRs :-)

Answer 2

OpenSSH is not using environment variable $HOME, as you can see from the log, but uses directly the getpwnam() functions to get the real home directory of running user (/home/buildbot/).
You either have to fix it in /etc/passwd or use the ssh-agent or IdentityFile option/config value to the ssh to make ssh use this key.