If a docker enabled VM is restarted, e.g. due to Azure patching the VM or for whatever reason, the node can get a new IP address (VirtualBox can cause this, and Azure too)
Which in turn results in the cert no longer being valid and Docker fails to start on that machine.
If I use Docker Swarm, the result is that the restarted node will be stuck in status Pending indefinitely.
If I then do a docker-machine regenerate-certs mymachine then it starts working again.
How should I reason around this?
I guess there is no way around having nodes being restarted, so how do you deal with this?
Regarding Azure you can ensure your VM keeps its public IP address after restart by using "Reserved IP" addresses. Please note using reserved IPs on Azure (as with other cloud providers) may incur additional charges. https://azure.microsoft.com/en-us/documentation/articles/virtual-networks-reserved-public-ip/
Another way to handle this is using discovery. Swarm offers a discovery mechanism which support etcd, consul and zookeeper. Find more details here:
https://docs.docker.com/swarm/discovery/
Related
i have several Linux Vms in MS Azure within the same security group and i can access all of them over SSH expect one. Here i need to restart the VM 5 to 10 times before i also can access it via SSH.
anyone has an idea whats wrong with this VM?
If the problem seems specific to this VM alone, you might want to check the VM's Resource Health first. Ensure that the VM reports as being healthy. If you have boot diagnostics enabled, verify the VM is not reporting boot errors in the logs.
If that looks clean, you might consider redeploying the VM. This redeploys a VM to another node within Azure, which may correct any underlying networking issues.
Do note that post this operation completes, ephemeral disk data is lost and dynamic IP addresses that are associated with the virtual machine are updated.
Additional troubleshooting guidance can be found here.
Azure Container Services has the option now to run in containers.
To register the billing you have to give your API key + Billing URL.
Even though I configured everything correctly and the service works locally, my calls are not registered as quoata's.
PS: Dont try to run the container without an internet connection, it will block the calls then ;)
Willem,
Here's what I think is going on: The problem is the linux container host picks an IP address range for the container that includes the IP addresses of your local DNS servers. This makes it impossible for the container to resolve names as requests for that range just end up on the local container network and won't go to the DNS servers.
The problem is described in this entry along with several solutions. The best solution seems to be at the very bottom which is also described in the docker documentation. The short version of this is to update the routing table on the host with the reserved IP range so that docker won’t pick it for the container.
Hope this helps,
Henrik
I have an MSDN account and created a VM (Windows 2012 R2) to which I have added several software packages for a proof of concept.
The problem is that with my limited $50/month credit, I'm forced to shutdown the VM at night, especially over weekends. When you shutdown a VM from the admin console (the only way they stop charging you) you will also lose the IP address.
The problem occurs when you restart the VM the next day, it will get a different IP. If you try to lookup the machine through a ping, their advertised external IP retrieved through DNS does NOT match the external IP address of the VM.
This makes it impossible for the software to resolve the host.
I figured it might take a couple hours for DNS to push the change, but it's been all day now and the DNS address is still incorrect.
Anyone else run into this?
(I cant create a support ticket because I only have an MSDN subscription.)
Where are you looking up the IP address? When you shutdown the machine and restart it you are correct in that it assigns a new external ip address. This only happens if this is the only VM running in that cloud service. This sounds like your setup.
The IP address you should be looking for is the one attached to the cloud service that your VM is a part of. If you go to the dashboard of the cloud service it will show you the public virtual IP address along with the FQDN. This IP will also match your VM's public IP.
See below
One thing I would suggest is to use Reserved IP option. It allows you to hook up semi permanent IP address to your cloud service.
https://azure.microsoft.com/en-us/documentation/articles/virtual-networks-reserved-public-ip/
Worth mentioning that Public IP may be returned back to the pool under certain circumstances, i.e. if your subscription run out of money. This is public cloud service after all, and however big the pool is, it is still limited.
Worth noting that I'm using the same approach shutting down my VMs when I'm not using them, but I don't experience IP change too often. Possibly because I have A0 domain controller VM always running.
We are currently evaluating azure, to see if we can use it for our stress and production environments.
Our environment is pretty complex, including web servers, mysql servers, hadoop and cassandra servers, as well as monitoring and deployment servers.
To set the stress environment, we need to install the environment, and then load large amounts of data into it, before we can run a stress test. This takes time and effort, and so, since we pay by the hour, we would like to be able to completely shutdown the environment, and start it up again ready to go when we want to run additional stress tests.
Here's our issue - we could not find a way to set a fixed internal ip address for a vm in azure. In AWS it is possible with VPC, but in azure, even if you define a virtual network, there seems to be no way to set a fixed internal ip (at least none that we can find).
This creates several issues for us -
1. Hadoop relies on all nodes in the cluster being able to translate all the modes hostnames to ip addresses.
2. A cassandra cluster that has all the ip addresses in the cluster change at once freaks out. We actually lost data in a test cassandra cluster because of this.
Our questions are:
1. Is there a way to set a fixed internal ip for a vm in azure?
2. If not, did anyone have an experience with running hadoop and cassandra on azure? How did you handle the changing ip addresses when the cluster is shut down?
Any advice on these issues will be much appreciated,
Thanks
Amir
Please note that the portal doesn't always expose all the capabilities of Azure. Some of the features in Azure are only possible through the REST API and PowerShell.
If you take a look at the new release of the PowerShell Cmdlets, you'll notice there is a new option for Static IPs in VNets.
https://github.com/WindowsAzure/azure-sdk-tools
Background. I have a service that is crawling a partners website. Their website blocks "bot like" behaviour by IP Address. They don't control their servers themselves, so they want me to change IP address of my crawler often (e.g. every minute). The crawler is currently running as an Azure Cloud Service.
So my question is. How would you programmatically change the IP address of an Azure server?
I could use a Proxy... but the site is running on SSL, so I'm not sure this is an option.
I could use a VPN connection... but I'm not sure if it's possible to programmatically dial a VPN from within a Cloud Service/VM? I'm using C#/.NET.
Finally. I tried to do a VIP Swap, which actually gives me a new IP in seconds. But unfortunately this only works when swapping form Staging to Production. I get an error when trying to swap from Production to Staging, if there is no staging running. Why is this blocked?
You cannot programmatically change a deployment's IP address, aside from shutting down the deployment (which releases the IP address) and then re-starting it (and that you can do programmatically, or through PowerShell). Also note that if you have several Virtual Machines in the same deployment, each VM will share the same public IP address.
Note that with Virtual Machines (vs. web/worker role cloud services), if you have a single Virtual Machine, you can stop the VM, then restart it, and you'll get a new IP address (ok, actually, there's no guarantee you'll get a different IP address, so more accurately you'll be assigned an IP address) while retaining the state of your VM. Shutdown and startup are not an instantaneous operatios however, and it's not going to provide you a per-minute IP address (and your VM will end up being offline shutting down or starting up pretty much all the time), so it won't satisfy your requirement.