We have an Azure Function that is integrated with VNET. Function uses Key Vault to read configuration values from. MSI has been enabled on function and access policy is added in the KV to allow function to access it. Traffic to Key Vault is restricted to be allowed from the said VNET. We have received this error in the diagnostics yesterday, which has affected our execution flow. The App was offline for a few minutes resulting in failures. This is what the diagnostic report says:
Detected host offline in your function app.
Description
A host error has occurred during startup operation 'd732fccb-40ae-40a1-8d14-ea3cdba0e725'.
Last offline at
5/4/2021 10:32:13 AM
Details
Microsoft.Azure.KeyVault.Models.KeyVaultErrorException : Client address is not authorized and caller is not a trusted service.
Client address: x.x.x.x
The function started working properly after being down for about 4 minutes. This could be an issue with VNET integration.
Has anyone encountered this? Is there any way to prevent this?
In this case, you could add the application settings WEBSITE_VNET_ROUTE_ALL = 1 of your azure function, this will force all outbound traffic from your azure function into that VNet.
When you route all of your outbound traffic into your VNet, it's
subject to the NSGs and UDRs that are applied to your integration
subnet. When WEBSITE_VNET_ROUTE_ALL is set to 1, outbound traffic is
still sent from the addresses that are listed in your app properties,
unless you provide routes that direct the traffic elsewhere.
Reference from https://learn.microsoft.com/en-us/azure/azure-functions/functions-networking-options#regional-virtual-network-integration
Related
Currently I am trying to use Api Management to expose an Azure Function http trigger that is within a vnet.
After setting everything up, I tried a simple GET request and I'm getting a 500 error telling me
"messages": [
"Error occured while calling backend service.",
"The remote name could not be resolved: '<function-name>.azurewebsites.net'"
]
My Azure function was:
Created with a premium plan.
enabled inbound traffic with a private endpoint.
deployed in the same Vnet as my api management service.
Added a httpTrigger template from the portal
Was set up with a Azure managed DNS name
My api management service was created in external mode and I linked my function to Api Management in the portal.
Things I've tried:
I've double checked that the correct security group rules have been opened for api management to work
I've ensured that both my api management (external) is in the same vnet as my Serverless function
I deployed a vm to the same virtual network and was unable to resolve the dns name of my function there. Used ping, telnet, nslookup.
Tried adding application settings to my function to specify the azure dns server, among other settings.
Ive check that my private dns zone is linked with my vnet.
Seems like a DNS name resolving issue, but I can't seem to fix it. Any ideas on what could be causing this error?
update
So it seems to be an issue with my private endpoint. Every function I create without one works fine, but I would still like to have the private endpoint so it isn't accessible from the internet.
I know I could probably lockdown my function to only be called from the api management gateway ip, but I would rather not have to hard code IPs if I don't have too.
Thank you MayankBargali-MSFT | Microsoft Docs Posting your suggestion as answer to help other community members.
As per the error, the APIM is not able to resolve your azure function
app. Can you please verify if the custom DNS is correctly setup and
you can refer to this
document
for more details. Outbound access on port 53 is required for
communication with DNS servers. If a custom DNS server exists on the
other end of a VPN gateway, the DNS server must be reachable from the
subnet hosting API Management. I will also suggest you to review this
document for the setup part.
Reference: External api management service can't resolve Azure function service name - Microsoft Q&A
In my current situation, we have several Azure Function apps that talk to an Azure SQL database (using Entity Framework, should it matter), using functions that trigger on an Azure ServiceBus trigger.
In the last weeks we have been improving security by using a VNET and subnets to only allow access to the Azure Database server by only the Function apps that need to use it. However, I have run into a strange issue. It seems that now the database server is set to disallow traffic apart from the defined subnets in which my fuction apps run, the function apps start giving intermittent SQLExceptions when connecting to the database with the message that some specific IP is not allowed by the database Firewall rules. The weird thing is that this error is not consistent. I would expect either for the function app to be declined at the firewall for it's IP, or be allowed all the time, but not randomly as is currently the case.
Question
Is there something that I am missing with my setup? Or, how do I force my function apps in subnets to use their internal IP that is allowed by the database server firewall rules, and not some other outbound IP address that is not in the database firewall rules?
Alternatively: What can possibly explain that access to the database sometimes succeed (indicating a proper internal IP used by the funcation app), and sometimes fail on the firewall (with an unknown IP address), seemingly at random.
Hopefully somebody can help!
Detailed Description of situation
The Function App has a function that is triggered by a Service Bus trigger. The Function App is running with a P1v2 Premium service plan with Vnet integration on.
The app is running inside a Subnet in our environment with a defined IP adress range with a /26 subnet mask. If I check the environment variables of the funciton app in Kudu I can see the PRIVATE_IP_ADDRESS setting is in the subnet range. The database firewall is set up to disallow all traffic, apart from the subnet of my function app as follows:
Triggering the function app which will write some stuff into the database works sometimes (Indicating that the access to the database is working at least when the IP address of the function is the correct one) however, there are also a lot of SQLExceptions with the following error:
Cannot open server 'database-server-name' requested by the login. Client with IP address 'XXX.XXX.XXX.XXX' is not allowed to access the server. To enable access, use the Windows Azure Management Portal or run sp_set_firewall_rule on the master database to create a firewall rule for this IP address or address range. It may take up to five minutes for this change to take effect.
The IP address mentioned in the error is NOT the internal IP defined on the Vnet or subnet IP range. It is also not even one of the IP's that show up in the Possible Outbound Ip addresses of the function app,
These happen more or less randomly. Sometimes they don't appear for x triggers, sometimes it fails for hundres of calls in a row.
Enabling the Datbaase server setting "Allow Azure services and resources to access this database server" stops the error from occuring, but of course that is counter to configuring the firewall to allow certain subnets
What I have tried
Setting the configuration settings WEBSITE_VNET_ROUTE_ALL and WEBSITE_DNS_SERVER on the function app to force traffic to use the VNET route as metioned here: Unable to connect Azure Function with Azure SQL using private endpoint
Adding Storage as service endpoint to the subnet as mentioned here: Unable to connect to Azure Function App after integrating into VNET
Restarting and stopping/starting the Function app
Changing the Function app Scaling to force a change in outbound IP addresses as mentioned here: https://learn.microsoft.com/en-us/azure/azure-functions/ip-addresses#outbound-ip-address-changes
Your Azure Function will only use a private outbound IP address picked from your VNet when you have VNet integration enabled. The VNet integration option is only available in the Function Premium Plan.
Additionally, the environment variables WEBSITE_VNET_ROUTE_ALL and WEBSITE_DNS_SERVER you already mentioned should be set as well as mentioned here (assuming your SQL server is in the same VNet).
Of course a new day and fresh perspective brings the probable answer.
While I havent changed anything about the setup I mentioned in my question, I was reading this post again: https://learn.microsoft.com/en-us/azure/azure-functions/ip-addresses#outbound-ip-address-changes
It mentioned switching the service plan temporary force an IP change. I initially misinterpreted this as switching between the DEV and premium plans, which cannot happen because the DEV plans don't support VNET integration. So I switched plans between the P1v2 and P1v3 plans. This however did not work.
What was meant here was to switch beween S1 and P1v2 plans. The Standard plans are hidden behind this link:
It also shows this small message net to the Apply button.
After switching between the S1 and P1v2 plans for a moment and resubmitting deadlettered servicebus messages to the function app, everything started working again.
I assume the IP switch was necessary, but switching between P-plans is not what triggers it. It has to be between standard and premium.
I have been using azure timer based function in my project.
Since I started using the function, I was getting error 403 related to access forbidden from Azure storage account.
I tried adding AzureWebJobsStorage Key in function's configuration
I tried adding outbound IPs of Allowed ip ranges of azure storage accounts
I tried checking the option of allowing azure managed services to the storage account.
However, I'm still getting the error in that particular timer function of my function app. The other functions run properly.
Although, if I allow all the networks in storage account firewall and VN settings, my function runs proper.
Both the function and storage account are in same region.
I want to enable it somehow that I do not need to choose the allow all networks. What can be done?
In case anyone else is searching for this... I had a similar issue. I had a function app that I had created a private endpoint and regional VNet integration back with the VNet interacting with a Storage Account that also had a private endpoint with the same VNet. The Storage Account's network/firewall settings only allowed connections from the VNet (no external traffic allowed). Both the storage account and function app reside in the same region.
Attempt at fix #1 (not ideal):
I added code to determine what IP the function app was running from. That led me to add all of the IP's in the portal under function app --> Properties --> Additional Outbound IP Addresses. This is exposed by Terraform if using that.
Attempt at fix #2 (better):
The resolution is to ensure you have the proper function app settings set.
See: Microsoft documentation
Setting
Suggested value
Description
WEBSITE_CONTENTOVERVNET
1
Create this app setting. A value of 1 enables your function app to scale when your storage account is restricted to a virtual network.
WEBSITE_DNS_SERVER
168.63.129.16
Create this app setting. When your app integrates with a virtual network, it will use the same DNS server as the virtual network. Your function app needs this setting so it can work with Azure DNS private zones. It's required when you use private endpoints. This setting and WEBSITE_VNET_ROUTE_ALL will send all outbound calls from your app into your virtual network.
WEBSITE_VNET_ROUTE_ALL
1
Create this app setting. When your app integrates with a virtual network, it uses the same DNS server as the virtual network. Your function app needs this setting so it can work with Azure DNS private zones. It's required when you use private endpoints. This setting and WEBSITE_DNS_SERVER will send all outbound calls from your app into your virtual network.
Note: The 168.63.129.16 is a static value for Azure DNS.
After setting all of these, my function app was able to connect to the storage account through the VNet as expected.
There're already some answers about this issue, you can see here and here.
In short, if the function and storage account are in same region, they communicate in an internal way without going through outboundIpAddresses.
The workaround is that create them in different regions.
I can't get vNet integration between an Azure App Service / WebJob and it's connected Storage Account to work correctly. I have added both the App Service and the storage account to the same vNet and enabled service endpoint binding for Microsoft.Storage. From what I can see in the docs, this is what is needed to get the communication to work between the services. Unfortunately, I get this error in the WebJob log when I try to run it:
"Unhandled Exception: Microsoft.WindowsAzure.Storage.StorageException:
The remote server returned an error: (403) Forbidden. --->
System.Net.WebException: The remote server returned an error: (403)
Forbidden."
If I disable the firewall on the storage account everything works just fine.
All the services are placed in the same resource group and region (West Europe).
I have tried both the "normal" (with gateway) and the new (Preview) vNet integration version on the App Service and both are failing the same way.
I am not using ASE (App Service Environment).
I have also added the following exceptions on the storage firewall:
Storage firewall exceptions
First, when you allow the set of trusted Microsoft services to bypass the network rules. These services will then use strong authentication to access the storage account.
In this case, you may think Microsoft.Networking service will be allowed in firewall rule. But basically, VNet Integration does not mean App Service is inside a VNet. VNet Integration gives your web app access to resources in your virtual network but doesn't grant private access to your web app from the virtual network. We usually used to securely access the resource in a VNet.
If you want to put your App Service in a VNet, you need to deploy it in App Service Environment, then it's already in a VNet.
Alternatively, you could whitelist the outbound web app service IP address in the firewall rule of the storage account.
Hope this helps, feel free to let me know if you have any concerns.
I've struggled myself on this topic.
First off, if you're restricting access to storage using the firewall you need to add your VNET to the storage firewall. Use your delegate subnet when adding your VNET.
When you add the private link between your storage and your VNET, Azure creates a private DNS zone but your app doesn't automatically use it, even when integrated to the VNET.
You can find more here https://learn.microsoft.com/en-us/azure/app-service/web-sites-integrate-with-vnet
But specifically this passage applies:
After your app integrates with your VNet, it uses the same DNS server that your VNet is configured with. By default, your app won't work with Azure DNS private zones. To work with Azure DNS private zones, you need to add the following app settings:
WEBSITE_DNS_SERVER with value 168.63.129.16
WEBSITE_VNET_ROUTE_ALL with value 1
These settings send all of your outbound calls from your app into your VNet and enable your app to access an Azure DNS private zone. With these settings, your app can use Azure DNS by querying the DNS private zone at the worker level.
You can test the DNS resolution your app is using by opening the debug console and using the NameResolver command
NameResolver mystorageaccount.blob.core.windows.net or
NameResolver mystorageaccount.queue.core.windows.net
should return your private address. If not you have a DNS issue.
We have time triggered Azure Function deployed on portal to perform some iterative task at specific time. Our azure function uses the database deployed on Azure VM via connection string provided at AppSettings. The function throws following error on running:
MySql.Data: Authentication to host 'xxx' for user 'xxx using method 'mysql_native_password' failed with message:
Client with IP address 'x.x.x.x' is not allowed to connect to this MySQL server. MySql.Data: Client with IP address 'x.x.x.x' is not allowed to connect to this MySQL server*
When we white-list the IP mentioned in error message, the function runs successfully. But since the azure function has no determined work-station or PC of same IP that handles the execution, whenever the function runs from new IP, it throws the error back. Therefore, we require a mechanism by which we can white-list all IPs of PCs that will be running our function app OR some better mechanism to authenticate and allow azure function to access our database hosted on Azure VM.
What we tried?
We white-listed the Virtual IP address of the function app. But it doesn't work every time.
We tried to white-list the IP ranges obtained from Microsoft Datacenter of the region in which our function app is deployed. But this method also didn't work.
Azure application can't access database on Azure VM?
Thus, is there any way by which azure function can access our database deployed on Virtual machine securely?
I have opened up the issue on github but no reply yet from there.
Finally, after thorough researching, found the solution.
One need to white-list all Outbound IPs of Function App in Virtual Machine where DB is deployed. The outbound IP address can be found from resources.azure.com. On searching your resource (in my case Function app's name), there will be a long json output from where you have to pick possibleOutboundIpAddresses parameter as shown in the image. Whitelist all Ips and then your azure function app can access your database deployed on virtual machine.
Also, I searched whether these IPs tend to change on regular basis or not. While I didn't find any official word from it, but from various internet sources I came to know that even if the IPs gets planned to be changed ever, everyone will get enough notifications before that to prevent any problems.
You need to set up a Virtual Network (VNet) where both your App Service Plan hosting the Azure Function and the VM participate.
Then from Azure Functions, view All Properties > Networking and you should see the virtual network to connect.
This method doesn't require you to whitelist IP addresses for your VM and secures your VM by allowing only internal network traffice.
Note that your Azure Function must be set up on an App Service Plan, rather than on a Consumption plan.