I'm using node.js and AWS with autoscaling. A javascript SDK solution is preferable but at this point I'll take anything.
I'm hoping this is super easy to do and that I'm just an idiot, but how does one go about getting the public IP addresses of instances that are undergoing the scaling event?
I'm trying to keep a list of active public IP's within a specific application tier so I can circumvent ELB for websocket connections, but I can't figure out how to programmatically get the public IP addresses of the instances that have just been added/removed.
For me, and Sensu Client config, I add a basic sensu client config in the base AMI for my instances with "this_hostname" "this_ip" and "this_role." Then I just add some simple sed's in my cloudformation user_data script that curl the aws endpoint for the public ip's as the instance boots. Each cloudformation script sets/exports the APP_TYPE(downcased) in the same user_data script prior to my sed lines, so I reuse that as the role for sensu:
"sed -i \"s/this_hostname/$(curl http://169.254.169.254/latest/meta-data/public-ipv4)/\" /etc/sensu/conf.d/client.json\n",
"sed -i \"s/this_ip/$(hostname -i)/\" /etc/sensu/conf.d/client.json\n",
"sed -i \"s/this_role/${APP_TYPE,,}/\" /etc/sensu/conf.d/client.json\n",
You can also use the internal IP for both, or external for both hostname/IP, to which you can see examples of both above...
For Shutdown, I use a simple /etc/rc0.d/S01Instance_Termination script that is symbolically linked from /etc/init.d/instance_termination that runs a similar curl to remove itself from the host on instance shutdown:
http://pastebin.com/6He1mQTH
Related
Background Info
I have a Node.js app running on a Managed Instance Group of VM's on GCP's Compute Engine.
New VM's are generated from a template with a startup script. The script does the usual stuff, install Node.js, curl, git clone the app code, etc.
This application is set to auto-scale, which is why I need configurations to happen pro grammatically - namely setting host and port in the .env file for the Node.js project.
How I have tried to solve the problem
The only way I can think about doing this programmatically in my startup.sh script is by running the command: gcloud compute instaces list
This returns something like this
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
VM-1 us-central1-a n1-standard-1 XX.XXX.X.X XX.XXX.XXX.XXX RUNNING
VM-2 us-central1-a n1-standard-1 XX.XXX.X.X XX.XXX.XXX.XXX RUNNING
VM-3 us-central1-a n1-standard-1 XX.XXX.X.X XX.XXX.XXX.XXX RUNNING
Then from that, I would want to pull out the EXTERNAL_IP of the current machine the script is running on, and somehow write it to an environment variable / write it to the .env file for my project.
The .env file only has this information in it:
{
"config" : {
"host" : "VM_EXTERNAL_IP"
"port" : 3019
}
}
This method, I think, would require some sort of regex to grab the correct IP out of the commands output, then stored in Environment variable, then written to .env.
This seems like unnecessary work as I surely not the first person to want to do something like this. Is there a standard way of doing this? I'm not a Node.js expert by any means, and even less of a GCP expert. Perhaps there is a mechanism on GCP to deal with this? - some metadata API that can easily grab the IP to use in code? Maybe on the Node.js side there is a better way of configuring the host? Any recommendations is appreciated.
There are many methods to determine the external IP address. Note that your instance does not have an external public IP address. Google implements a 1-to-1 NAT which handles IP address translation which maps the public IP address to the private IP address.
The CLI supports the command line option --format json. You can use tools that parse json such as jq.
gcloud compute instances list --format json | jq -r ".[].networkInterfaces[0].accessConfigs[0].natIP"
Get your public IP address from several sources, which may or may not be the same as your instance:
https://checkip.amazonaws.com/.
curl https://checkip.amazonaws.com/
Use the CLI with options to get just the external address
gcloud compute instances describe [INSTANCE_NAME] --format='get(networkInterfaces[0].accessConfigs[0].natIP)'
Read the metadata server from inside the instance:
curl http://metadata/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip -H "Metadata-Flavor: Google"
Once you have determined the method to get your external IP address, you can use tools such as awk to replace the value in your file.
https://www.grymoire.com/Unix/Awk.html
I have setup a cloud test bed using OpenStack. I used the 3 node architecture.
The IP assigned to each node is as given below
Compute Node : 192.168.9.19/24
Network Node : 192.168.9.10/24
Controller Node : 192.168.9.2/24
The link of instance created is like this :
http://controller:6080/vnc_auto.html?token=2af0b9d8-0f83-42b9-ba64-e784227c119b&title=hadoop14%28f53c0d89-9f08-4900-8f95-abfbcfae8165%29
At first this instance was accessible only when I substitutes controller:8090 with 192.168.9.2:8090. I solved this by setting a local DNS server and resolving 192.168.9.2 to controller.local. Now instead of substituting the IP it works when I substitute controller.local.
Is there any other way to do it?? Also how can I access this instance from another subnet other than 192.168.9.0/24, without specifying the IP.
If I understood your question correctly, yes there is another way, you don't need to set up a DNS server!
On the machine that you would like to access the link, perform the operations below:
Open /etc/hosts file with a text editor.
Add this entry: 192.168.9.2 controller
Save the file, and that's it.
I suggest you to do these on all your nodes so that you can use these hostnames on your OpenStack configuration files instead of their IPs. This would also save you from tons of modifications if you have to make a change on the subnet IPs.
So for example your /etc/hosts files on your nodes should look like these:
#controller
192.168.9.2 controller
#network
192.168.9.10 network
#compute
192.168.9.19 compute
First, I wish to use purely gcloud commands to acheive my objective - NOT the GCE interface - so please don't provide answers using the GUI!
I created an image from a disk attached to a VM instance. In order to do so, I had to delete the instance, per the Google documentation for creating images. After that, I recreated my instance using the image.
Almost everything seems to have worked perfectly from that process except http and https traffic is now disabled in the instance! I can no longer browse to the website hosted on the VM. I also cannot get a response by pinging the domain anymore.
When I look in the GCE gui (just looking - not modifying anything!) I can see that the checkboxes for the "Allow http traffic" and "Allow https traffic" are not checked for the instance. It seems that must be related to my problem.
I checked the firewall rules on the server (ipTables), and on the Google network assiocated with the VM. There is nothing wrong with either of those (and the VM is definitely assiocated with that network). If I listen on port 80 using tcpdump on the server and I browse to my domain, I can see the requests are reaching server, so they aren't blocked by an incoming firewall. I also explictly restarted Apache, just be make sure that wasn't the problem.
So, is there something I need to do to unblock port 80 and 443 on an outgoing basis instead? Is this possibley an SELinux thing? Since the image should represent exactly what was on the disk it shouldn't be. It seems this must be on the GCE side...
What do those checkboxes actually do for the instance if they don't edit iptables on the server or the firewall rules on the Google network? What is the gcloud command to set those switches, or ideally specify that with an instance create command?
Solved. I don't entirely understand what is going on behind the scenes, but the solution to this requires the use of "tags" which associate firewall rules on the network with the VM instance. As far as I can see at this point, this is only pertinent for http and https. Other ports that are open on the network and the VM seem to work without this additional piece.
If you view your firewall rules, you'll probably see the port 80 and 443 rules have the tags "http-server" and "https-server" respectively. If they don't, you'll need to add those (or other tags of your choosing). It turns out the instance needs those tags added to it as well.
To add the tags to an existing VM instance, use this gcloud command:
gcloud compute instances add-tags [YOUR_INSTANCE_NAME] --tags http-server,https-server
To add the tags at the time of the instance creation, include that flag in your statement:
gcloud compute instances create [YOUR_INSTANCE_NAME] --tags http-server,https-server
If you look in the GCE gui, you'll see those "Allow http traffic" and "Allow https traffic" checkboxes are checked after doing that. Requests and responses then flow across ports 80 and 443 as expected.
One of the super helpful things the Google Cloud Console offers is a link at the bottom of the create for most resources for the REST API and command line to create the same resource. I am challenging myself to be able to do everything I can do in the console from SDK command line, so I use this often when I have a question like yours.
Having the same question as above, in the console I created a VM and selected "Allow HTTP traffic". Looking at the command line for this, you will see two commands. The first is the create command with the tag as noted above (http-server):
gcloud beta compute --project=XXXX instances create cgapperi-vm1 \
--zone=XXXXX --machine-type=f1-micro --subnet=default \
--tags=http-server --image=debian-10-buster-v20200413 \
--image-project=debian-cloud --boot-disk-size=10GB \
--boot-disk-type=pd-standard --boot-disk-device-name=cgapperi-vm1 \
--no-shielded-secure-boot --shielded-vtpm --shielded-integrity-monitoring \
--reservation-affinity=any
The second actually creates the firewall rule (default-allow-http) for you, and sets the target for requests to the http-server tag (--target-tags=http-server) on tcp port 80 (--rules=tcp:80) from incoming requests (--direction=INGRESS) from all sources (--source-ranges=0.0.0.0/0):
gcloud compute --project=XXXX firewall-rules create default-allow-http \
--direction=INGRESS --priority=1000 --network=default --action=ALLOW \
--rules=tcp:80 --source-ranges=0.0.0.0/0 --target-tags=http-server
I hope this is helpful for anyone else.
NOTE: I did reduce the output of the gcloud compute instance create to relevant bits in order to reduce the clutter.
As per the details in this link:
https://cloud.google.com/vpc/docs/special-configurations
"By selecting these checkboxes, the VPC network automatically creates a default-http or default-https rule that applies to all instances with either the http-server or https-server tags. Your new instance is also tagged with the appropriate tag depending your checkbox selection."
So ticking these boxes tags your server and creates the necessary firewall rule for you that will apply to all servers with that tag. From a gcloud perspective I guess you would need to ensure the tag is created and applied and the rule is also created for it to do what the console option does for you.
Last several days I'm struggling with a problem.
I have two instances(ubuntu server) on gcloud and I want to assign them their external IP.
And I can ping and ssh to my instances but when I try to do telnet it is not performed.
On gcloud all instances have one internal ip and one external IP.
And they does not know their ip. I get it from gcloud console.
How could I assign it to them?
Also I've tried sudo ifconfig eth0:0 130.211.95.1 up
You can do something like this to add the external IP to a local interface:
ip addr add 130.211.95.1/32 dev eth0 label eth0:shortlabel
Replace 'add' with 'del' to remove it once you are done with it.
shortlabel can be any string up to a certain (short) length.
Update: also see this GCE support issue for related information.
A feature request for this is already filed on GCE public issue tracker, however it is not yet implemented. You can star it to get notification if any update posted on the thread.
May you also mention what's your use case? so I can probably provide you with a workaround.
I want to set up a 3 node Rabbit cluster on EC2 (amazon linux). We'd like to have recovery implemented so if we lose a server it can be replaced by another new server automagically. We can set the cluster up manually easily using the default hostname (ip-xx-xx-xx-xx) so that the broker id is rabbit#ip-xx-xx-xx-xx. This is because the hostname is resolvable over the network.
The problem is: This hostname will change if we lose/reboot a server, invalidating the cluster. We haven't had luck in setting a custom static hostname because they are not resolvable by other machines in the cluster; thats the only part of that article that doens't make sense.
Has anyone accomplished a RabbitMQ Cluster on EC2 with a recovery implementation? Any advice is appreciated.
You could create three A records in an external DNS service for the three boxes and use them in the config. E.g., rabbit1.alph486.com, rabbit2.alph486.com and rabbit3.alph486.com. These could even be the ec2 private IP addresses. If all of the boxes are in the same region it'll be faster and cheaper. If you lose a box, just update the DNS record.
Additionally, you could assign an elastic IPs to the three boxes. Then, when you lose a box, all you'd need to do is assign the elastic IP to it's replacement.
Of course, if you have a small number of clients, you could just add entries into the /etc/hosts file on each box and update as needed.
From:
http://www.rabbitmq.com/ec2.html
Issues with hostname
RabbitMQ names the database directory using the current hostname of the system. If the hostname changes, a new empty database is created. To avoid data loss it's crucial to set up a fixed and resolvable hostname. For example:
sudo -s # become root
echo "rabbit" > /etc/hostname
echo "127.0.0.1 rabbit" >> /etc/hosts
hostname -F /etc/hostname
#Chrskly gave good answers that are the general consensus of the Rabbit community:
Init scripts that handle DNS or identification of other servers are mainly what I hear.
Elastic IPs we could not get to work without the aid of DNS or hostname aliases because the Internal IP/DNS on amazon still rotate and the public IP/DNS names that stay static cannot be used as the hostname for rabbit unless aliased properly.
Hosts file manipulations via an script are also an option. This needs to be accompanied by a script that can identify the DNS's of the other servers upon launch so doesn't save much work in terms of making things more "solid state" config wise.
What I'm doing:
Due to some limitations on the DNS front, I am opting to use bootstrap scripts to initialize the machine and cluster with any other available machines using the default internal dns assigned at launch. If we lose a machine, a new one will come up, prepare rabbit and lookup the DNS names of machines to cluster with. It will then remove the dead node from the cluster for housekeeping.
I'm using some homebrew init scripts in Python. However, this could easily be done with something like Chef/Puppet.
Update: Detail from Docs
From:
http://www.rabbitmq.com/ec2.html
Issues with hostname
RabbitMQ names the database directory using the current hostname of
the system. If the hostname changes, a new empty database is created.
To avoid data loss it's crucial to set up a fixed and resolvable
hostname. For example:
sudo -s # become root
echo "rabbit" > /etc/hostname
echo "127.0.0.1 rabbit" >> /etc/hosts
hostname -F /etc/hostname