Access denied to storage account from Azure Data Factory - azure

My goal is to run an exe file stored in a private Azure Blob container.
The exe is simple : it creates a text file, write the current datetime in it, and then push it to the private Azure Blob container.
This has to be sent from Azure Data Factory. To do this, here is my environment :
Azure Data Factory running with the simple pipeline :
https://i.stack.imgur.com/txQ9r.png
Private storage account with the following configuration :
https://i.stack.imgur.com/SJrGX.png
A linked service connected to the storage account :
https://i.stack.imgur.com/8xW5l.png
A private managed virtual network approved :
https://i.stack.imgur.com/G2DH3.png
A linked service connected to an Azure Batch :
https://i.stack.imgur.com/Yaq6C.png
A batch account linked to the right storage account
A pool running on this batch account
Two things that I need to add in context :
When I set the storage account to public, it works and I find the text file in my blob storage. So the process works well, but there is a security issue somewhere I can't find.
All the resources (ADF, Blob storage, Batch account) used have a role has contributor/owner of the blob with a managed identity.
Here is the error I get when I set the storage account to private :
{
"errorCategory":0,
"code":"BlobAccessDenied",
"message":"Access for one of the specified Azure Blob(s) is denied",
"details":[
{
"Name":"BlobSource",
"Value":"https://XXXXXXXXXXXXXXXXX/testv2.exe?sv=2018-03-28&sr=b&sig=XXXXXXXXXXXXXXXXXX&sp=r"
},
{
"Name":"FilePath",
"Value":"D:\\batch\\tasks\\workitems\\XXXXXXXXXXX\\job-1\\XXXXXXXXXXXXXXXXXXXXXXXX\\testv2.exe"
}
]
}
Thank you for your help!

Solution found Azure community support :
Check Subnet information under Network Configuration from the Azure portal > Batch Account > Pool > Properties. Take note and write the information down.
Navigate to the storage account, and select Networking. In the Firewalls and virtual networks setting, select Enable from selected virtual networks and IP addresses for Public network access. Add the Batch pool's subnet in the firewall allowlist.
If the subnet doesn't enable the service endpoint, when you select it, a notification will be displayed as follows:
The following networks don't have service endpoints enabled for 'Microsoft.Storage'. Enabling access will take up to 15 minutes to complete. After starting this operation, it is safe to leave and return later if you don't wish to wait.
Therefore, before you add the subnet, check it in the Batch virtual network to see if the service endpoint for the storage account is enabled.
After you complete the configurations above, the Batch nodes in the pool can access the storage account successfully.

Related

For VM diagnostics agent, we create SA, if the SA is not publicly allowed, do we need to create Private end point for both Blob and Table for this SA?

As per this Microsoft documentation the VM uses both blob and table for storing the diagnostic logs.
As described here that the StorageType can be Table, Blob, or TableAndBlob. And let say we are using TableAndBlob.
The storage account that we are creating for this purpose will only be accessible via private endpoints. So do, I need to create private endpoint for both blob and table with private DNS zones privatelink.blob.core.windows.net and privatelink.table.core.windows.net?
Also, if I choose StorageType as Table, I just need the table(privatelink.table.core.windows.net) private endpoint?
I tried to reproduce the same in my environment public access does not allow in the traffic. we can able to create a private endpoint for both blob and table as shown below:
In your storage account -> networking under Security + networking ->private endpoint connection -> create
Make sure in Firewalls and virtual networks try to enable Allow Azure services on the trusted services list to access this storage account. as below
while creating Private endpoint connection in a target sub resource you can able to create a blob and table as below.
And DNS also automatically created in Private endpoint connection as privatelink.blob.core.windows.net and i have created for both blob and table.
You can check on your DNS configuration under setting once you click on your table or blob.
Additionally, you can connect in azure account through storage explore to get access storage account -> right click connect -> next
You can make use of Azure Private Link for storage Accounts through virtual network

Connecting Blob Storage to a Synapse Workspace with Public Network Workspace Access Disabled

I'm trying to connect blob storage from my RG storage account to the Data tab in my synapse workspace, but I get the following error: "The public network interface on this Workspace is not accessible. To connect to this Workspace, use the Private Endpoint from inside your virtual network or enable public network access for this workspace."
Public network access to my workspace must be disabled for company reasons. I made private endpoint connections on my synapse resource to Dev, Sql, and Sql-On-Demand, but I'm not sure where to go from there.
Thanks!
Go to Azure Synapse -> Manage -> Managed private endpoints -> +New and add private endpoints.
Accessing blob storage : If you already created linked service the follow below image. If not, please created and follow this Ms Doc for creating linked service.
Fastest way to access Azure blob storage
For for information follow this reference by Dennes Torres.

Trying to set a link services through registerd app to azure data lake storage and keep getting 24200 error

I am new to azure. We have azure data lake storage set. I am trying to set the link services from the data factory to the azure data lake storage gen2. It keeps failing when I test the link service to the data lake storage. As far as I can see, I have granted the "Storage blob contributor" role to the user in the azure data lake storage. I still keep getting permission denied error when I test the link services
ADLS Gen2 operation failed for: Storage operation '' on container 'testconnection' get failed with 'Operation returned an invalid status code 'Forbidden''. Possible root causes: (1). It's possible because the service principal or managed identity don't have enough permission to access the data. (2). It's possible because some IP address ranges of Azure Data Factory are not allowed by your Azure Storage firewall settings. Azure Data Factory IP ranges please refer https://learn.microsoft.com/en-us/azure/data-factory/azure-integration-runtime-ip-addresses.. Account: 'dlsisrdatapoc001'. ErrorCode: 'AuthorizationFailure'. Message: 'This request is not authorized to perform this operation.'.
What I could observe is that when I open the network to all (public) in the data lake storage, it works, when I set the firewall with CIDR it fails. Couldn't narrow the cause of the problem. I do have the "Allow azure services on the trusted services list to access this account" checked.
Completely lost
As mentioned in the error description, the error usually occurs if you don't have sufficient permissions to perform the action or if you don't add the required IPs in the firewall settings of your storage account.
To resolve the error, please check if you added the Storage Blob Data Contributor role to your managed identity along with the user like below:
Go to Azure Portal -> Storage Accounts -> Your Storage Account -> Access Control (IAM) ->Add role assignment
Make sure to select the managed identity, based on the authentication method you selected while creating linked service.
As mentioned in this MsDoc, make sure to add all the required IPs based on your resource location and service tag.
Download the JSON file to know the IP range for service tag in your resource location and add them in the firewall settings like below:
Make sure to select the Resource type as
Microsoft.DataFactory/factories while choosing CIDR.
For more in detail, please refer below links:
Error when I am trying to connect between Azure Data factory and Azure Data lake Gen2 by Anushree Garg
Storage Accoung V2 access with firewall, VNET to data factory V2 by Cindy Pau

Azure function connection to Azure Blob storage behind Vnet issue

We are currently migrating to a new Azure Subscription and are having issues executing Azure Functions that worked as expected in our old Azure Subscription. The man difference between our old Subscription and our new Subscription is that we have set up a Virtual Network with Subnets and have deployed our Resources behind the Subnets.
We have also had to migrate from an Azure App Service in the old Subscription to a Azure App Environment in the new Subscription.
Our Azure environment consist of:
App Service Environment
App Service Plan I1
The Azure App Environment and Storage Containers are on the same Virtual Network but different Sub Nets. The Function is using a Managed Identity which has Owner Role on Storage Account.
The code listed below worked just fine in our old environment which did not contain the Virtual Network, but fails in our new environment.
Any guidance would be greatly appreciated.
The Azure function which connects to Azure Storage works when run locally from Visual Studio 2019, but fails when run from Azure portal.
Code Snippet below:
This section works just fine:
string storageConnectionString = XXXXConn.ConnectionETLFileContainer();//Get Storage connection string
var myDirectory = "XXXX/Uploads"; ///XXXX-etl-file-ingest/ABSS/Uploads/ CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageConnectionString);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();// Create a CloudBlobClient object for credentialed access to Azure Blob. CloudBlobContainer blobContainer = blobClient.GetContainerReference("XXXX-etl-blobfile-ingest");// Get a reference to the Blob Container we created previously. CloudBlobDirectory blobDirectory = blobContainer.GetDirectoryReference(myDirectory);// Get a reference to the Blob Directory.
var blobs = blobDirectory.ListBlobs(useFlatBlobListing: true); //set useFlatBlobListing as true
This statement fails: Failure occurs when trying to iterate through the Blob files and get specific file info.
foreach (var myblob in blobs)
In the azure portal open storage account blade under that go to configuration blade , you will be able to see the list of networks for which your storage account has allowed access to.Once you have the allowed network list kindly check if the function app is on one of those networks if not then you need to get the network on which your function app is hosted added to the list.
Update 2:
The simplest explanation/cause that I found is when an App Service or Function App has the setting WEBSITE_VNET_ROUTE_ALL set to 1, all traffic to public endpoints is blocked. So if your Storage Account has no private endpoint configured, requests to it will fail.
Docs: "To block traffic to public addresses, you must have the application setting WEBSITE_VNET_ROUTE_ALL set to 1."
https://learn.microsoft.com/en-us/azure/app-service/web-sites-integrate-with-vnet#network-security-groups
Update 1:
My answer below was only a workaround for my problem. Turns out I did not link the Private DNS Zone (this is created for you when you create a new Private Endpoint) to my VNET.
To do this, go to your Private DNS Zone in the Azure Portal and click on Virtual network links in the left menu bar. There add a new link to the VNET your Function is integrated in.
This may not have been relevant for the OP, but hopefully it will help others.
Original answer:
In my case this was solved by enabling the Microsoft.Storage Service Endpoint on the App Service's subnet (dedicated subnet).

How to read a blob in Azure databricks with SAS

I'm new to Databricks. I write sample code to read Storage Blob in Azure Databricks.
blob_account_name = "sars"
blob_container_name = "mpi"
blob_sas_token =r"**"
ini_path = "58154388-b043-4080-a0ef-aa5fdefe22c8"
inputini = 'wasbs://%s#%s.blob.core.windows.net/%s' % (blob_container_name, blob_account_name, ini_path)
spark.conf.set("fs.azure.sas.%s.%s.blob.core.windows.net"% (blob_container_name, blob_account_name), blob_sas_token)
print(inputini)
ini=sc.textFile(inputini).collect()
It throw error:
Container mpi in account sars.blob.core.windows.net not found
I guess it doesn't attach the SAS token in WASBS link, so that it doesn't permission to read the data.
How to attach the SAS in wasbs link.
This is excepted behaviour, you cannot access the read private storage from Databricks. In order to access private data from storage where firewall is enabled or when created in a vnet, you will have to Deploy Azure Databricks in your Azure Virtual Network then whitelist the Vnet address range in the firewall of the storage account. You could refer to configure Azure Storage firewalls and virtual networks.
WITH PRIVATE ACCESS:
When you have provided access level to "Private (no anonymous access)".
Output: Error message
shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Container carona in account cheprasas.blob.core.windows.net not found, and we can't create it using anoynomous credentials, and no credentials found for them in the configuration.
WITH CONTAINER ACCESS:
When you have provided access level to "Container (Anonymous read access for containers and blobs)".
Output: You will able to see the output without any issue.
Reference: Quickstart: Run a Spark job on Azure Databricks using the Azure portal.

Resources