I am using Azure Container Services (AKS - not ACS) to stand up some API's - some of which are for public consumption, some of which are not.
For the public access route everything is as you might expect, a load-balancer service bound to a public IP is created, DNS zone contains our A record forwarding to the public IP, traffic is routed through to an NGINX controller and then onwards to the correct internal service endpoints.
Currently the preview version assigns a new VNET to place the AKS resource group within, moving forwards I will place the AKS instance inside an already existing VNET which houses other components (App Services, on an App Service Environment).
My question is how to grant access to the private APIs to other components inside the same VNET, as well as components in other VNETS?
I believe AKS supports an ILB-type load balancer, which I think might be what is required for routing traffic from other VNETS? But what about where the components reside already inside the same VNET?
Thank you in advance!
If you need to access these services from other services outside the AKS cluster, you still need an ILB to load balance across your service on the different nodes in your cluster. You can either use the ILB created by using the annotation in your service. The alternative is using NodePort and then stringing up your own way to spread the traffic across all the nodes that host the endpoints.
I would use ILB instead of trying to make your own using NodePort service types. The only thing would be perhaps using some type of API Gateway VM inside your vnet where you can define the backend Pool, that may be a solution if you are hosting API's or something through a 3rd party API Gateway hosted on an Azure VM in the same VNet.
Eddie Villalba
MCSD: Azure Solutions Architect | CKA: Certified Kubernetes Administrator
Related
I want to restrict access to my Azure Container App with an Api Management in Azure.
I successfully linked the Api Management with the Container App and I have activated a Subscription with an Api Key that will prevent public access over the Api Management Service Url. The problem, however, is that the Container App can still be accessed over the public Url of the Container App.
There is still the option to set the Ingress Traffic in the Container App to Limited to Container Apps Environment but then the Api Management will not have access to the Container App as well.
What is the correct way to properly secure the Container App behind an Api Management Service?
For Azure Container Instances, you don't have the option to configure IP restrictions similar to Azure App Services. Instead you will have to first create a virtual network and configure a Network Security Group to Deny all traffic from the internet and allow only from APIM, and then deploy your Azure Container Instance to this virtual network.
See here for deploying an azure container instance to a virtual network : https://learn.microsoft.com/en-us/azure/container-instances/container-instances-vnet
For configuring network security groups in your virtual network see : https://learn.microsoft.com/en-us/azure/virtual-network/manage-network-security-group#work-with-security-rules
You app service is still accessible over the public internet because you haven't configured Access Restrictions in your App Service's Network.
What you need to do is go to your App service. Then select Networking from the left menu and Turn on Access Restrictions for inbound traffic.
Create an access restriction rule to deny from the internet.
Next create a second acccess rule to allow access from the APIM. Ensure the priority on this one is higher.
Read the Microsoft Docs on how to set app service IP restrictions here : https://learn.microsoft.com/en-us/azure/app-service/app-service-ip-restrictions
Assuming your API management service has a static IP (not a consumption plan), you would need to use you own VNET:
Networking architecture in Azure Container Apps
Then using NSG, you could add an inbound rule to only allow traffic from the APIM service IP on HTTPS (TCP 443).
Azure container apps do now seem to have the ability to restrict inbound ip addresses
https://azure.microsoft.com/en-gb/updates/public-preview-inbound-ip-restrictions-support-in-azure-container-apps/
We have are looking at a similar architecture with a similar dilemma. Everything we have is secured with Azure b2c but if I want to make an internal container/microservice accessible to Azure Api Management I think I'd have to drop b2c (api management has no UI to log into b2c) and make it publicly accessible via the Ingress. If the inbound ip addresses are restricted to api management maybe that is ok. It does worry me that ip addresses can be spoofed although you'd hope Microsoft have thought of that.
Another alternative which I've not investigated but which does work for Azure functions is managed identities. This might not work at all with container apps though
https://www.svenmalvik.com/azure-apim-function-msi/
First, I think that it is good to explain networking architecture in Azure Container Apps.
Azure Container Apps run in the context of an environment, which is supported by a virtual network (VNET). When you create an environment, you can provide a custom VNET, otherwise a VNET is automatically generated for you.
There are two ways to deploy Container Apps environments:
External - Container Apps environments deployed as external resources are available for public requests. External environments are deployed with a virtual IP on an external, public facing IP address.
Internal - When set to internal, the environment has no public endpoint. Internal environments are deployed with a virtual IP (VIP) mapped to an internal IP address. The internal endpoint is an Azure internal load balancer (ILB) and IP addresses are issued from the custom VNET's list of private IP addresses.
I attach the image from Azure portal to show above two options:
Now going further, if you want your container app to restrict all outside access, create an internal Container Apps environment.
Now when it comes to deployment of the Container Apps to the Container Apps Environment, accessibility level you selected for the environment will impact the available ingress options for your container app deployments.
If you are deploying to an external environment, you have two options for configuring ingress traffic to your container app:
Limited to Container Apps Environment - to allow only traffic from other container apps deployed within the shared Container Apps environment.
Accepting traffic from anywhere - to allow the application to be accessible from the public internet.
If you are deploying to an internal environment, you also have two options for configuring ingress traffic to your container app:
Limited to Container Apps Environment - to allow only traffic from other container apps deployed within the shared Container Apps environment.
Limited to vNET (Virtual Network) - to allow traffic from the VNET to make container app to be accessible from other Azure resources or applications within the virtual network or connected to the virtual network through Peering or some type of VPN connectivity
Now in you case, what you are looking for is the architecture where you enable access to Azure Container Apps only through the Azure API Management. In this case you have to deploy Azure Container Apps Environment with Internal mode and set ingress traffic to Limited to VNet (Virtual Network).
I assume that Azure API Management can be accessible from the Internet. In this case you have to deploy Azure API Management inside an Azure Virtual Network. There are two possible modes: internal, and external. In you scenario, you can use external mode. More details can be found here. When API Management instance in the external mode, the developer portal, API gateway, and other API Management endpoints are accessible from the public internet, and backend services are located in the Azure Virtual Network.
Here I also attach the solution architecture to show how all these components are connected together. I also have Azure Front Door here but API Management is deployed with external mode. Please remember that you will also need private DNS Zone for your Azure Container Apps Environment domain, to make it possible to refer to specific APIs from the Azure API Management using URLs, example:
https://ca-tmf-mip-vc-api--v-01.blacklacier-cf61414b.westeurope.azurecontainerapps.io
Helpful links:
Repo with Bicep files to deploy Azure Container App with internal mode
Azure Container Apps Virtual Network Integration
I've created an Azure container instance with a Private IP. This is connected to a VNET so my Web Apps can communicate with it in a secure way. (This API has Bearer tokens also but I don't want to make it public).
However, when restarting the Container I get a new IP. Therefore I have to update the Env and restart my apps.
Is there a way to implement service discovery within Azure, so my Web Apps (and other services) know where this Container Instance is, especially when the container gets a new IP.
I am used to dealing with Pivotal and Consul but under Azure I don't have these tools.
In Pivotal I was able to fire up multiple instances and the platform would auto discover and load balance. At the moment, Azure feels very manual :(
Does azure have the ability to register a service under a host name that can then auto resolve?
Does Azure support load balancing when multiple instances are started with the same name?
For this kind of work maybe the best solution is to create an Azure service web app. The web app as public static IP link to your plan which does not change.
Inbound and outbound IP addresses in Azure App Service
With this solution your IP never change.
According to the documentation linked, you can find your inbound IP like this:
nslookup <app-name>.azurewebsites.net
and the outbound IP in powershell:
(Get-AzWebApp -ResourceGroup <group_name> -name <app_name>).OutboundIpAddresses
Can we have a single Application Gateway for all VMSS created in different regions?
If yes please share the possible options.
As the comment mentioned, we could not have a single Application gateway for all VMSS created in a different region since Application Gateway is always deployed in a virtual network subnet and it directly supports to deploy the VMSS as the backends in the same region and virtual network as the Application gateway.
As a workaround, you could use a public IP address as the backend for communicating with instances outside of the virtual network as long as there is IP connectivity. Read more details about backend pools. So you may use a public-facing load balancer associated with the VMSS.
Furthermore, you also could use Traffic Manager to distribute traffic across multiple Application Gateways in different datacenters. Or use Azure Front Door Service provides a scalable and secure entry point for fast delivery of your global web applications.
I'm struggling with how to use Azure API management together with App Service Environment v1 (aka ASEv1), i.e. how to set the things via Azure portal.
I have ASE with one app service (target is several of them) and I have APIM gateway pointing to the public host name of the service.
What I have to do to make the app services in ASE VNet to be accessible only through the APIM gateway, to keep the back-end services hidden?
I need to be able to manage APIM from Azure portal and to be able to access the services directly via FTPS for deployment, collecting logs, etc.
--
I've created a new subnet for the APIM and put it in the ASE VNet , where already ASE subnet was. Now I probably have to set some NSG rules but I'm not sure how and if it is all I'm supposed to do.
The second thing I'm not sure is how to change the APIM API settings. It now points to a public URL of the service - I do not know if it will be available after NSG changes.
There are a couple of options, depending on what you really want. If you're fine having your backend services visible by outside but not callable, you can employ any means of authentication between APIM and backend services:
Shared secret - header/query param
Client certificate authentication
IP filtering on the side of backend services
If you want to really hide backend services from outside, you'll have to put APIM and ASE into same VNET.
Following seems to work, although it may need some fine tuning.
In short, each layer put into separate external VNet with own Network Security Group (NSG), using caller's public IP address to manage access to VNet:
back-end service web app in ASEv1 VNet "vnet-ase", with associated NSG "nsg-ase"
APIM gateway put to a separate VNET "vnet-apim", with associated NSG "nsg-apim"
in nsg-ase allow access from public IP of APIM GW, to vnet-ase
nsg-apim will be used to manage connections to the APIM gateway
I would like to create a simple architecture on Azure. My high level design is very similar to the picture below (source: https://www.import.io/post/using-amazon-lambda-and-api-gateway/)
I do want to access the internal services via the Azure API Management. What I can see on Microfos documentation page is that this simple and secure architecture is not mentioned as a reference:
https://learn.microsoft.com/en-us/azure/container-service/container-service-kubernetes-walkthrough
I have the following issues:
API Management cannot be assigned to a Virtual Network if there is at least one NIC is using the same network (why?)
Even with peered Virtual Networks I cannot access 10.244.X.0/24 network (pods' network) because only 10.240.0.0/16 is owned by the k8s Virtual Network. How can I access cluster ips (10.0.0.0/16) and pod ips (10.244.0.0/16)?
Well, you don't need an Extra VNET, but just an extra Subnet. That Subnet could lie within your existing VNET. The Size of Subnet can be the smallest /29 which Azure supports.
The Extra Subnet requirement for API Management comes from the fact, that it is built on PAAS V1 (Classic) technology. While we can deploy into a Resource Manager VNET (V2 layer), there are consequences to that. The Classic deployment model in Azure are not tightly coupled with Resource Manager model and so if you create a resource in V2 stuff, the V1 doesn't know about it and problems can happen such as API Management trying to use an IP that is already allocated to a NIC (built on V2).
To learn more about difference of Classic and Resource Manager models in Azure refer to blog difference between Classic and ResourceManager models
The answer is basically YES although the setup is not trivial.
You need:
One extra VNet for the API Management (EDIT: an extra subnet is enough)
One service (kubernetes terminology)
Steps:
Peer the Kubernetes VNet and the extra VNet you have created (test it)
API Management -> Virtual network: change to External
Choose as Virtual Network the one extra VNet (lets call it 'apimgmntvnet') and a Subnet
Save it! Drink a beer because it took me 1h!
Meanwhile expose your deployment internally:
kubectl expose deployment app --port=<serviceport> --name=app --target-port=<containerport> --type=NodePort (NodePort is important!LoadBalancer type triggers kubernetes to dynamically configure the Azure External LB for Kubernetes install)
Check node IP:PORT on kubernetes (kubectl proxy) BUI
API Management -> Publisher portal: modify your API to the IP address (AgentIP:30361)
Theoretically it should work. It is advised to start with a VM in the apimgmntvnet and try peering first from the VM and than delete it (API Management cannot be part of a VNet where at least one NIC is present (?!) ).