Tldr:
The issue is that the IP address the agent uses doesn't match the public IP address attached to the container Instance. It's like it is using some hidden internal IP address that isn't visible to an end user.
Background:
I setup a self-hosted DevOps agent in an Azure Container Instance. I added a public IP so I could whitelist it in my subscription, and the DevOps agent would have access to alter resource per Terraform templates.
Problem:
The agent is unable to alter resources that have firewall rules, despite adding the container Instance IP to the whitelist. The error message states failure due to network access from an unapproved IP address. The issue is that the IP address the agent uses doesn't match the public IP address attached to the container Instance. It's like it is using some hidden internal IP address that isn't visible to an end user.
Anyone encountered this before? The whole reason I decided on a self-hosted agent was to avoid the IP hopping problem caused by using DevOps hosted agents.
I just tried with a simple curl container, outgoing IP is different with the IP the container is exposed on. This is something I'd expect to see as well (I believe it would be using Kubernetes underneath). Nothing you can do about that.
You can create a container in the VNet, hence you won't need to whitelist random IP addresses. or you can just create a VM assign a Public IP to it and whitelist that IP (that will always work).
Related
I was following this article - Azure virtual machine scale set agents. Everything is good and I am able to run my custom images to perform builds and deployments.
One thing that is missing there - what if agent needs to access some IP restricted resource? Previously (with a simple VM running as agent) it was easy to whitelist public IP of a given agent. With those scale sets (and --load-balancer "") I am not sure how to get them to use a static IP (preferably only 1 outgoing public IP for all VMs in a set)
Any suggestions?
When you use a Load Balancer in front of the VMSS with only one static public IP address, then all the VMSS instances only can access outside through that one static public IP address. It's easy to achieve it, you just need to configure the public IP address as static. Then you put that public IP address in the whitelist for the restricted resource.
In this way, all the instances access the restricted resource via the static public IP address and it's in the whitelist.
I followed the documentation guide here to configure an Azure DNS zone for a virtual network.
I then created a virtual machine on that virtual network and provisioned a virtual network gateway to allow my Azure web apps to communicate with the virtual machine using VNet integration. My web apps are then able to resolve against the virtual machines private IP as expected.
I then created an 'A' type record set within the DNS zone resolving a service name against the private IP of the virtual machine just like here.
However, when I then attempt to access the DNS configured service name that should resolve against the private IP address of the virtual machine, I get the following error:
curl: (6) Could not resolve host: xxxx.local
Am I missing something obvious here?
I can not reproduce this error following your steps:
Create an Azure VNet with a DNS private zone like private.test.com.
Create a VM and a VPN gateway on that VNet.
Enable VNet integration with my Azure web app service.
My web apps are able to resolve against the virtual machines private IP as expected. This could verify the Azure private DNS zone should work. Then I also create an A record for a custom name against the VM private IP address. Both scenarios work.
You could check if an A record is something like below picture in the private DNS zone.
Then you could verify if curl with http:// or without that, or without the specific port 9200, the error is still the same.
I suggest using SET WEBSITE_DNS_ command. This command will output the current DNS server that is being used by the web app. If the error Environment variable WEBSITE_DNS_ not defined is received, no custom DNS servers are configured for the web app. See more details about networking Related Commands for Azure App Services.
I've recently setup a point to site VPN on Azure. In doing so I have created:
Azure Virtual Network
Azure Virtual Network Gateway
Azure Virtual Network Gateway IP (static)
Azure VM
I was under the impression that any VM set to use the above said Virtual Network would all share the same public IP? However it appears that the Azure VM i setup is instead using its own dynamic public IP.
Is there a way to route any VM on the Custom Virtual Network I created through one public IP? IF so - how can I do this?
According to your description, you have create a point-to-side VPN. in this scenario, we can remove the public IP addresses from VMs, then you can access your VMs with private IP addresses via VPN.
You are correct - I have the VPN working correctly. My only concern is this, it'd be great that if any of those VM's in that private network shared the same static IP - that way I could grant access to that IP in my firewall to other systems i have in other hosting platforms (as they are all typically locked down by ip).
It sounds like your reference to availablity sets may be the phrase i was looking for to further google.. i will check it out and let me know
please let me know if anything I added above changes your answer/recommendations :)
I was under the impression that any VM set to use the above said
Virtual Network would all share the same public IP?
You are talking about Azure ASM module (classic). In ASM module, VMs can use the same public IP address(NAT).
However it appears that the Azure VM i setup is instead using its own
dynamic public IP.
In Azure resource module, we can create Azure VMs with different Public IP addresses.
Is there a way to route any VM on the Custom Virtual Network I created
through one public IP?
According to your description, you have create a point-to-side VPN. in this scenario, we can remove the public IP addresses from VMs, then you can access your VMs with private IP addresses via VPN.
If you want to access those VMs from Internet and use one public IP address, we can create VMs in the same availability set, and deploy a Internet facing load balancer, then add NAT rules to them, in this way, we can access those VMs use the same public IP address and different ports.
I have an Azure Cloud Service, mywebapp.cloudapp.net, that consists of two Azure VMs - mywebappvm1 and mywebappvm2. Both VMs are in the same Availability Set and have the same DNS name.
I also have a Regional Reserved IP address assigned to the Cloud Service so that I can give our clients a guaranteed IP address that our app uses.
Part of the app uses a private background process, currently only running on one of the VMs. I want to be able to make a connection to that process over TCP running on mywebappvm1 from mywebappvm2. I could use the public IP and an endpoint on mywebappvm1 but I don't want the background service to be publicly accessible.
I'm currently using the private IP address, but is that safe? Will the private IP of each VM change if it's rebooted? I can't see an easy way of fixing the private IP of each VM - that seems like something you can do with a VNET but I can't find any information on how to do it with a cloud service and an availability group as well.
Is there perhaps another way to run a web app on multiple load-balanced VMs within an availability set that would make this easier?
What you do is absolutely safe and actually a recommended best practice. You should not go out to public IP address in order to communicate between the Virtual Machines.
It is also a recommended best practice to organize your Virtual Machines into Virtual Network and sub-nets.
This excellent blog post describes how can you even use static IP addresses for the VMs, so you are always 100% sure that mywebappvm1 always get XXX.XXX.XXX.XXX IP Address andyour mywebappvm2 always get YYY.YYY.YYY.YYY IP Address.
Please note that if you do not use Static IP Address assigned to the VM, it is guaranteed that the IP Address of the VM may change.
The IP for a webRole VM instance will not change for the lifetime of the deployment regardless of reboot, update or swapping. The IP will be released only if you delete the deployment, detailed here
I had a need to add additional public IP addresses to an Azure VM and found a working solution here:
Azure VM: More than one Public IP
Essentially this creates a reserved IP in Azure and then adds the reserved IP to a cloud service. Once it's bound to a cloud service it can be mapped to a VM endpoint.
This all works great but there is one bit I don't understand - The IP address of the reserved IP and the resultant VM endpoint don't match. I have to set up DNS to point to the IP address of the endpoint to make this work. Is there something I am not doing right, or is this just the way reserved VMs work?
It looks like this unanswered question is the same issue:
azure reserved IP for VM is diffrent than the given
Thanks!
The "Azure Cloud Service" is a container that provides internet connectivity to "Azure VMs". Thus, you assign the Internet facing Public IP to the Cloud Service. This article is relatively good at explaining the relationship: Azure Cloud Services
From above link:
Here’s a definition of an Azure IaaS cloud service that will make it easy for you to understand what it is in the context of Azure Infrastructure Services:
A cloud service is a network container where you can place virtual machines.
All virtual machines in that container can communicate with each other directly through Azure (and therefore don’t have to go out to the Internet to communicate with each other).
This container is also assigned a DNS name that is reachable from the Internet.
A rudimentary DNS server is created and can provide name resolution for all virtual machines within the same cloud service container (note that name resolution provided by the DNS server is only available to the virtual machines that are located within the cloud service).
One or more Virtual IP Addresses (VIPs) are assigned to the container and these IP addresses can be used to allow inbound connections from the Internet to the virtual machines.
Certain services (like FTP) may require your vm have a public IP: Azure VM Public IP
(IaaS v1) An Azure cloud service comes with a permanent DNS name - something.cloudapp.net - and has a single VIP allocated whenever there are VMs deployed in it OR whenever a reserved IP address is associated with it. Traffic is either load balanced or NATted (port forwarded) to the VM from the Azure Load Balancer sitting on the VIP. You can also associate a public instance-level IP address (PIP) with a VM, which gives it an additional IP address. The VIP always has a DNS name (something.cloudapp.net) while the PIP has one only if you specifically add it, I did a post which goes into these differences.
(IaaS v2) VMs are not deployed into cloud services and only have a public IP address if one is specifically added - either by configuring a PIP on the NIC of the VM (and optionally giving it a cloudapp.azure.com DNS name) or by configuring a load balancer and either load balancing or NATting traffic to it. This load balancer is configured with a public IP address and can optionally have a cloudapp.azure.com DNS name associated with it. (Ignoring internal load balancers in this discussion.)