I have been using azure timer based function in my project.
Since I started using the function, I was getting error 403 related to access forbidden from Azure storage account.
I tried adding AzureWebJobsStorage Key in function's configuration
I tried adding outbound IPs of Allowed ip ranges of azure storage accounts
I tried checking the option of allowing azure managed services to the storage account.
However, I'm still getting the error in that particular timer function of my function app. The other functions run properly.
Although, if I allow all the networks in storage account firewall and VN settings, my function runs proper.
Both the function and storage account are in same region.
I want to enable it somehow that I do not need to choose the allow all networks. What can be done?
In case anyone else is searching for this... I had a similar issue. I had a function app that I had created a private endpoint and regional VNet integration back with the VNet interacting with a Storage Account that also had a private endpoint with the same VNet. The Storage Account's network/firewall settings only allowed connections from the VNet (no external traffic allowed). Both the storage account and function app reside in the same region.
Attempt at fix #1 (not ideal):
I added code to determine what IP the function app was running from. That led me to add all of the IP's in the portal under function app --> Properties --> Additional Outbound IP Addresses. This is exposed by Terraform if using that.
Attempt at fix #2 (better):
The resolution is to ensure you have the proper function app settings set.
See: Microsoft documentation
Setting
Suggested value
Description
WEBSITE_CONTENTOVERVNET
1
Create this app setting. A value of 1 enables your function app to scale when your storage account is restricted to a virtual network.
WEBSITE_DNS_SERVER
168.63.129.16
Create this app setting. When your app integrates with a virtual network, it will use the same DNS server as the virtual network. Your function app needs this setting so it can work with Azure DNS private zones. It's required when you use private endpoints. This setting and WEBSITE_VNET_ROUTE_ALL will send all outbound calls from your app into your virtual network.
WEBSITE_VNET_ROUTE_ALL
1
Create this app setting. When your app integrates with a virtual network, it uses the same DNS server as the virtual network. Your function app needs this setting so it can work with Azure DNS private zones. It's required when you use private endpoints. This setting and WEBSITE_DNS_SERVER will send all outbound calls from your app into your virtual network.
Note: The 168.63.129.16 is a static value for Azure DNS.
After setting all of these, my function app was able to connect to the storage account through the VNet as expected.
There're already some answers about this issue, you can see here and here.
In short, if the function and storage account are in same region, they communicate in an internal way without going through outboundIpAddresses.
The workaround is that create them in different regions.
Related
In my current situation, we have several Azure Function apps that talk to an Azure SQL database (using Entity Framework, should it matter), using functions that trigger on an Azure ServiceBus trigger.
In the last weeks we have been improving security by using a VNET and subnets to only allow access to the Azure Database server by only the Function apps that need to use it. However, I have run into a strange issue. It seems that now the database server is set to disallow traffic apart from the defined subnets in which my fuction apps run, the function apps start giving intermittent SQLExceptions when connecting to the database with the message that some specific IP is not allowed by the database Firewall rules. The weird thing is that this error is not consistent. I would expect either for the function app to be declined at the firewall for it's IP, or be allowed all the time, but not randomly as is currently the case.
Question
Is there something that I am missing with my setup? Or, how do I force my function apps in subnets to use their internal IP that is allowed by the database server firewall rules, and not some other outbound IP address that is not in the database firewall rules?
Alternatively: What can possibly explain that access to the database sometimes succeed (indicating a proper internal IP used by the funcation app), and sometimes fail on the firewall (with an unknown IP address), seemingly at random.
Hopefully somebody can help!
Detailed Description of situation
The Function App has a function that is triggered by a Service Bus trigger. The Function App is running with a P1v2 Premium service plan with Vnet integration on.
The app is running inside a Subnet in our environment with a defined IP adress range with a /26 subnet mask. If I check the environment variables of the funciton app in Kudu I can see the PRIVATE_IP_ADDRESS setting is in the subnet range. The database firewall is set up to disallow all traffic, apart from the subnet of my function app as follows:
Triggering the function app which will write some stuff into the database works sometimes (Indicating that the access to the database is working at least when the IP address of the function is the correct one) however, there are also a lot of SQLExceptions with the following error:
Cannot open server 'database-server-name' requested by the login. Client with IP address 'XXX.XXX.XXX.XXX' is not allowed to access the server. To enable access, use the Windows Azure Management Portal or run sp_set_firewall_rule on the master database to create a firewall rule for this IP address or address range. It may take up to five minutes for this change to take effect.
The IP address mentioned in the error is NOT the internal IP defined on the Vnet or subnet IP range. It is also not even one of the IP's that show up in the Possible Outbound Ip addresses of the function app,
These happen more or less randomly. Sometimes they don't appear for x triggers, sometimes it fails for hundres of calls in a row.
Enabling the Datbaase server setting "Allow Azure services and resources to access this database server" stops the error from occuring, but of course that is counter to configuring the firewall to allow certain subnets
What I have tried
Setting the configuration settings WEBSITE_VNET_ROUTE_ALL and WEBSITE_DNS_SERVER on the function app to force traffic to use the VNET route as metioned here: Unable to connect Azure Function with Azure SQL using private endpoint
Adding Storage as service endpoint to the subnet as mentioned here: Unable to connect to Azure Function App after integrating into VNET
Restarting and stopping/starting the Function app
Changing the Function app Scaling to force a change in outbound IP addresses as mentioned here: https://learn.microsoft.com/en-us/azure/azure-functions/ip-addresses#outbound-ip-address-changes
Your Azure Function will only use a private outbound IP address picked from your VNet when you have VNet integration enabled. The VNet integration option is only available in the Function Premium Plan.
Additionally, the environment variables WEBSITE_VNET_ROUTE_ALL and WEBSITE_DNS_SERVER you already mentioned should be set as well as mentioned here (assuming your SQL server is in the same VNet).
Of course a new day and fresh perspective brings the probable answer.
While I havent changed anything about the setup I mentioned in my question, I was reading this post again: https://learn.microsoft.com/en-us/azure/azure-functions/ip-addresses#outbound-ip-address-changes
It mentioned switching the service plan temporary force an IP change. I initially misinterpreted this as switching between the DEV and premium plans, which cannot happen because the DEV plans don't support VNET integration. So I switched plans between the P1v2 and P1v3 plans. This however did not work.
What was meant here was to switch beween S1 and P1v2 plans. The Standard plans are hidden behind this link:
It also shows this small message net to the Apply button.
After switching between the S1 and P1v2 plans for a moment and resubmitting deadlettered servicebus messages to the function app, everything started working again.
I assume the IP switch was necessary, but switching between P-plans is not what triggers it. It has to be between standard and premium.
I have an Azure App Service which uses Blob Storage to host it's static files. For this particular app (staging environment) we only want to allow access from the app service and a specific set of ip addresses.
What is the best way to go about this? I tried providing an identity for the app service with the role of Owner but it keeps getting a 403 when trying to access blob storage.
I can reproduce your problem:
Why you still get 403 is because you are missing the necessary steps.
You have two way.
1, add service outbound ip to the firewall of storage.
2, put the app under a virtual network and then allow the VNET to access.
Common solution:
First of all, create a virtual network on azure.
Then add the network to the firewall setting of storage:
After above configuration, my Azure App Service can access the storage with no problem. Please have a try on your side.(Pleace notice that VNET configuration will not work immediately, you need wait for a little time.)
I am currently working with a client that requires access to all Azure resource locking down as much as possible and I am having problems with the Storage Account that is utilised by our Azure Functions.
With the Firewalls and Virtual Networks blade in portal set to "All Networks" I am able to deploy to the Function App and it runs without issue.
However once I enable the access restriction by checking "Selected Networks"no matter what virtual network subnets I enter or IP Addresses I can not get the communication to work
I have entered the Outbound IP Addresses of our Consumption based Function App and also check that the additional IP Addresses from the Powershell and all have been added to the whitelist. I have also added all the CIDR IP ranges of the local Azure datacenter but again it does not work.
The problem we have is that once the access restrictions have been put into place we are unable to deploy to the Function App and the app no longer runs. Is this scenario supported and what is the mechanism for tying down access to the Storage Account so that only the Function App can utilise it.
As far as I know, you have two options to restrict access to your storage account from your function app or web app.
Whitelist the outboundIpAddresses and possibleOutboundIpAddresses of the function app in the firewall of the storage account. However, it does not work if the Azure function app and Azure storage located in the same region refer to Sam's answer.
when you hit the storage account from your function, because they are
in the same region as each other, all the traffic goes over the
internal Azure network on internal IP's, not the public IPs listed in
the web app, and so is not allowed over the firewall.
If your resources were in different regions, you could use the network section of function app to allow function app to access resources in a VNet, then enable service endpoint for Microsoft.Storage in this app integration subnet. But you need Azure Functions Premium plan referring to this tutorial: integrate Functions with an Azure virtual network.
Sometimes, the deployment order for networking is important. In this case, you will deploy the followings:
Firstly, you could deploy new VNet integration with an unused subnet. After the VNet Integration is completed and the function app is restarted, you could enable service endpoint for this subnet. In the end, you could add the subnet in the firewall of the storage account.
Note that the new version is in Preview, currently. You could also check these characteristics and get more references from this thread.
I can't get vNet integration between an Azure App Service / WebJob and it's connected Storage Account to work correctly. I have added both the App Service and the storage account to the same vNet and enabled service endpoint binding for Microsoft.Storage. From what I can see in the docs, this is what is needed to get the communication to work between the services. Unfortunately, I get this error in the WebJob log when I try to run it:
"Unhandled Exception: Microsoft.WindowsAzure.Storage.StorageException:
The remote server returned an error: (403) Forbidden. --->
System.Net.WebException: The remote server returned an error: (403)
Forbidden."
If I disable the firewall on the storage account everything works just fine.
All the services are placed in the same resource group and region (West Europe).
I have tried both the "normal" (with gateway) and the new (Preview) vNet integration version on the App Service and both are failing the same way.
I am not using ASE (App Service Environment).
I have also added the following exceptions on the storage firewall:
Storage firewall exceptions
First, when you allow the set of trusted Microsoft services to bypass the network rules. These services will then use strong authentication to access the storage account.
In this case, you may think Microsoft.Networking service will be allowed in firewall rule. But basically, VNet Integration does not mean App Service is inside a VNet. VNet Integration gives your web app access to resources in your virtual network but doesn't grant private access to your web app from the virtual network. We usually used to securely access the resource in a VNet.
If you want to put your App Service in a VNet, you need to deploy it in App Service Environment, then it's already in a VNet.
Alternatively, you could whitelist the outbound web app service IP address in the firewall rule of the storage account.
Hope this helps, feel free to let me know if you have any concerns.
I've struggled myself on this topic.
First off, if you're restricting access to storage using the firewall you need to add your VNET to the storage firewall. Use your delegate subnet when adding your VNET.
When you add the private link between your storage and your VNET, Azure creates a private DNS zone but your app doesn't automatically use it, even when integrated to the VNET.
You can find more here https://learn.microsoft.com/en-us/azure/app-service/web-sites-integrate-with-vnet
But specifically this passage applies:
After your app integrates with your VNet, it uses the same DNS server that your VNet is configured with. By default, your app won't work with Azure DNS private zones. To work with Azure DNS private zones, you need to add the following app settings:
WEBSITE_DNS_SERVER with value 168.63.129.16
WEBSITE_VNET_ROUTE_ALL with value 1
These settings send all of your outbound calls from your app into your VNet and enable your app to access an Azure DNS private zone. With these settings, your app can use Azure DNS by querying the DNS private zone at the worker level.
You can test the DNS resolution your app is using by opening the debug console and using the NameResolver command
NameResolver mystorageaccount.blob.core.windows.net or
NameResolver mystorageaccount.queue.core.windows.net
should return your private address. If not you have a DNS issue.
We have time triggered Azure Function deployed on portal to perform some iterative task at specific time. Our azure function uses the database deployed on Azure VM via connection string provided at AppSettings. The function throws following error on running:
MySql.Data: Authentication to host 'xxx' for user 'xxx using method 'mysql_native_password' failed with message:
Client with IP address 'x.x.x.x' is not allowed to connect to this MySQL server. MySql.Data: Client with IP address 'x.x.x.x' is not allowed to connect to this MySQL server*
When we white-list the IP mentioned in error message, the function runs successfully. But since the azure function has no determined work-station or PC of same IP that handles the execution, whenever the function runs from new IP, it throws the error back. Therefore, we require a mechanism by which we can white-list all IPs of PCs that will be running our function app OR some better mechanism to authenticate and allow azure function to access our database hosted on Azure VM.
What we tried?
We white-listed the Virtual IP address of the function app. But it doesn't work every time.
We tried to white-list the IP ranges obtained from Microsoft Datacenter of the region in which our function app is deployed. But this method also didn't work.
Azure application can't access database on Azure VM?
Thus, is there any way by which azure function can access our database deployed on Virtual machine securely?
I have opened up the issue on github but no reply yet from there.
Finally, after thorough researching, found the solution.
One need to white-list all Outbound IPs of Function App in Virtual Machine where DB is deployed. The outbound IP address can be found from resources.azure.com. On searching your resource (in my case Function app's name), there will be a long json output from where you have to pick possibleOutboundIpAddresses parameter as shown in the image. Whitelist all Ips and then your azure function app can access your database deployed on virtual machine.
Also, I searched whether these IPs tend to change on regular basis or not. While I didn't find any official word from it, but from various internet sources I came to know that even if the IPs gets planned to be changed ever, everyone will get enough notifications before that to prevent any problems.
You need to set up a Virtual Network (VNet) where both your App Service Plan hosting the Azure Function and the VM participate.
Then from Azure Functions, view All Properties > Networking and you should see the virtual network to connect.
This method doesn't require you to whitelist IP addresses for your VM and secures your VM by allowing only internal network traffice.
Note that your Azure Function must be set up on an App Service Plan, rather than on a Consumption plan.