Using Chef & Amazon with a Non-standard ssh port - linux

Question

Question: How can I allow inbound SSH traffic on a non-standard port when using Amazon Security Groups and also provisioning with Chef?
Amazon EC2: Allow inbound ssh traffic on port 999 instead of 22 by adding this rule to a Security Group.
Custom TCP Rule Port Range: 999
Chef: Create a new server with access to ssh on a non-standard port via:
knife ec2 server create .... -p 999 ....
Ubuntu: Allow ssh access on a non-standard port by editing /etc/ssh/sshd_config
Port 999
Easy enough? Why doesn't it work?
When using knife ec2 server create ... -p 999 ... the instance creates. However it hangs on "Waiting for sshd...". Eventually that errors out. The instance is not available using ssh -p 999 username#ip-address nor ssh -p 22 username#ip-address.

Answer 1

As some of the commenters have said, you are being locked out because the instance is listening on port 22, but your security groups are only allowing TCP connections on port 999.
There is no way to connect to the instance in this state.
There are three solutions to this problem that I can see.
Solution 1: Make a new AMI
Create a new AMI which has sshd configured to listen on 999.
Once you have done so, create your EC2 servers using your custom AMI and everything should work.
There are fancy-pants ways of doing this using cloudinit to let you customize the port later, but it hardly seems worth the effort.
Just hardcode "999" instead of "22" into /etc/ssh/sshd_config
The obvious downside is that for any new AMI that you want to use, you'll have to bake a new base AMI that uses the desired port instead of 22.
Furthermore, this diverges away from the Cheffy philosophy of layering configuration on a base image.
Solution 2: Icky Icky Security Groups
You can hack your way around this by modifying your security groups every time you bring up a new server.
Simply add an exception that allows your machine to SSH into the box for the duration of your bootstrapping process, then remove this from the security group(s) that you are using once you are done.
The downside here is that security groups in EC2 are not dynamic, so you either have to create a new security group for each machine (ick!), or let this open port 22 for your workstation on all of your servers during the bootstrapping window (also ick!).
Solution 3: Tunneling
The last option that I can think of is to allow connections on port 22 amongst your live servers.
You can then tunnel your connection through another server by connecting to it, opening an SSH tunnel (i.e. ssh -L ...), and doing the knife ec2 actions through the tunnel.
Alternatively, you can pass through one of the other servers manually on your way to the target, and not use knife to bootstrap the new node.
The downside here is that you need to trust your servers' security, since you'll have to do some agent forwarding or other shenanigans to successfully connect to the new node.
This is probably the solution that I would choose because it does not require new AMIs, doesn't open port 22 to the outside world, and requires a minimal amount of effort for a new team member to learn how to do.