EC2 keypair works in one instance but fails on other - Permission denied (publickey) - linux

Question

I have read many posts on this subject but none helped me solve my issue.
I have a machine amazon ec2 which I connect using this SSH command:
ssh -i /Library/AWS/glrpopulis.pem ec2-user#54.225.154.23
I've never had problems with this command until now. It just stopped working, the following message is displayed: Permission denied (publickey). out of nowhere!
I really can't understand why suddenly the same command I use almost everyday is failing to work. Probably I've changed something I wasn't supposed to, but I'm having a really hard time figuring out what.
I was creating a service for a web application (atlassian bamboo) when that happened the first time, but I'm not sure if this relates to the error.
I have reboot the machine a couple of times and tried over and over again, with no success.
The complete output with the -v option is displayed bellow:
mac-pipo:~ felipereis$ ssh -v -i /Library/AWS/glrpopulis.pem ec2-user#54.225.154.23
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: Connecting to 54.225.154.23 [54.225.154.23] port 22.
debug1: Connection established.
debug1: identity file /Users/felipereis/.ssh/id_rsa type 1
debug1: identity file /Users/felipereis/.ssh/id_rsa-cert type -1
debug1: identity file /Users/felipereis/.ssh/id_dsa type -1
debug1: identity file /Users/felipereis/.ssh/id_dsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.2
debug1: match: OpenSSH_6.2 pat OpenSSH*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm#openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm#openssh.com none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 19:ef:f1:2b:56:dd:86:ec:42:65:ff:1d:6b:64:0f:f3
debug1: Host '54.225.154.23' is known and matches the RSA host key.
debug1: Found key in /Users/felipereis/.ssh/known_hosts:12
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/felipereis/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Offering RSA public key: /Library/AWS/glrpopulis.pem
debug1: Authentications that can continue: publickey
debug1: Trying private key: /Users/felipereis/.ssh/id_dsa
debug1: No more authentication methods to try.
Permission denied (publickey).
UPDATE:
* I have just tested and I'm able to use the same key (glrpopulis.pem) to connect to a different ec2 instance, so maybe is something going on the first machine

Answer 1

Sounds like the keys under ~/.ssh/authorized_keys got messed up or the file got deleted.
Try the following:
Stop your EC2 instance
Detach your root Volume (/dev/sda1) -- Assuming this is Volume A
Spin up a new EC2 instance of the same type and same credentials.
Attach Volume A to that new instance as /dev/sdf
ssh connect to his new instance.
mkdir -p /mnt/xvdf
mount /dev/xvdf /mnt/xvdf
cp ~/.ssh to /mnt/xvdf/home/ec2-user/.
chmod 700 /mnt/xvdf/home/ec2-user
chmod 600 /mnt/xvdf/home/ec2-user/authorized_keys
Shutdown new instance
Detach Volume A on new instance
Reattach Volume A on /dev/sda1 on original instance.
Start original instance.
You should be able to login now.

Answer 2

Depending on your AMI, the public key might be being added to the authorized_keys file of a different user to ec2-user.
To find out, you can view the boot log for the instance in the EC2 console, and it should output the username that cloud-init is using as the "default user". Mine has a line like this:
ci-info: +++++++++++++++++++++Authorized keys from /home/ec2-user/.ssh/authorized_keys for user ec2-user++++++++++++++++++++++
You can also try logging in as root as that will sometimes give an error like 'Please login as the user "ec2-user" rather than the user "root".'
This happened to me, and it was because I had updated my version of cloud-init, which is what adds the public key to authorized_keys. The default config file (/etc/cloud/cloud.cfg) was replaced, causing the default user to change from "ec2-user" to "cloud-user".
I fixed this issue by changing the system_info section of the new /etc/cloud/cloud.cfg to this:
...
system_info:
...
default_user:
name: ec2-user
sudo: ALL=(ALL) NOPASSWD:ALL
...
You can then create a new AMI from that instance, and it should setup ec2-user correctly again.