I have a linked service to connect to development environment of Azure data lake storage account.
The connection to the storage account is using Azure key vault. while doing the deployment from DEV to UAT, the ARMTemplate has the URL string for the data lake storage account to override with UAT data lake storage account URL, but does not have the property to override the secret name.
Related
I have created Azure Storage account, inside storage account created container and uploaded files to my container.
When I tried to access the Container from browser. Getting: This request is not authorized to perform this operation. ,Assigned Storage blob Contributer and also Storage Blob Owner but same issue.
I tried to reproduce the same in my environment to access the blob Storage. As I got below
I created storage account with container, like below.
Azure Portal > Storage accounts > Create a Storage Account
Note: If you want to access Azure Blob from public Internet, Select Enable public access from all networks under Networking section or if you select Enable public access from selected Virtual network and IP Address, it will access only particular Virtual Network.
**Container Creation: **
Azure Storage Account > Containers > Container >
Assigned Storage Blob Data Contributor to my storage account, Like below.
Azure Storage Account > Select your Storage Account > Access Control (IAM)> Add > Add role Assignment > Storage Blob Data Contributor.
My blob is accessible over the Public Internet
Those files/container are private, this means only authorised users can access to them. To access a particular file with your browser, you need to grant permissions to that file. One of many solutions is to use SAS token (Shared Access Signature). In resume, this will generate a url with a token and this url can be pasted in a browser to access the file.
SAS token can be configured at some levels:
User delegation
Service
Account
One of the benefits (it has a lot more) of SAS token is that you can define an expiration policy for that SAS token, so it not be externally accesible for eternity.
You should configure Share Access Policies if those files have sensible content or block some paths to protect them from external access.
Please, refer to the official documentation for more info about SAS Token usage:
What is SAS
Grant limited access to Azure Storage resources using SAS
I have an external web application which has the option to access a storage account using the service principal.
I want to access Azure storage account/blob by the external application loading the data directly into the datalake account.
So here is what I am trying to do:
Set up a service principal (using Azure AD app registration)
Create a Storage account and store the access key in Azure Key Vault.
Add the service principal secret to the same key vault.
Create a policy within Key vault for the service principal to have access to read Keys and Secrets within Key Vault.
Also create a policy within Key Vault for service principal to have contributor role to access storage account.
Also grant access to storage account container to service principal.
But I cannot connect, and I am unable to authorize the connection.
I am confused on which steps I am missing to resolve this.
As you want to access the storage account using service principal, you do not need to store the storage account access in the key vault.
The steps you can follow up to access storage account by service principal:
Create a service principal (Azure AD App Registration)
Create a storage account
Assign Storage Blob Data Contributor role to the service principal
Now you would be able to access the Azure Storage Blob data using your service principal
Note: You do not need to store the service principal client secret in the key vault. Because you need the client secret again to access the key vault first.
Thanks #RamaraoAdapa-MT
This works
Finally, I setup like you said,,
SAS -> service principle -> permission to storage account -> storage account.
In this case, no need for Key vault.
Thanks you Guys,
Anupam Chand, RamaraoAdapa-MT
Here is what I have:
1 VNet with Subnet1 and Subnet2.
1 Storage Account with Private Endpoint in Subnet1
1 Azure Data Factory with Private Endpoint in Subnet2
Public network access disabled for both of them.
I am trying to read and write a blob in the Storage Account using a Data Factory pipeline (Copy Data).
With the above setup, the Pipleline times-out, which I believe is because it is unable to resolve the Private IP for Storage Account.
What step(s) am I missing to correctly use the Private Endpoints in my setup above to be able to R/W blob via Data Factory?
Note: If I create Managed Private Endpoint in the Data Factory to connect to the Storage Account, the pipeline works and is able to read/write blobs.
Ref: https://learn.microsoft.com/en-us/azure/data-factory/managed-virtual-network-private-endpoint
Are Managed Private Endpoints the only way to connect to the Storage Account? If not, how do I configure the normal Private Endpoints?
Apart from managed private endpoints option there is another way to access Blob inside a VNET from ADF.
You can add Managed Identity ID of Datafactory in Blob Account > Access Control (IAM) and grant the ID "Storage Blob Data Contributor" role.
I have Storage account kagsa1 with container cont1 inside and need it to accessible (mounted) via Databricks
If I use storage account key in KeyVault it works correctly:
configs = {
"fs.azure.account.key.kagsa1.blob.core.windows.net":dbutils.secrets.get(scope = "kv-db1", key = "storage-account-access-key")
}
dbutils.fs.mount(
source = "wasbs://cont1#kagsa1.blob.core.windows.net",
mount_point = "/mnt/cont1",
extra_configs = configs)
dbutils.fs.ls("/mnt/cont1")
..but if I'm trying to connect using Azure Active Directory credentials:
configs = {
"fs.azure.account.auth.type": "CustomAccessToken",
"fs.azure.account.custom.token.provider.class": spark.conf.get("spark.databricks.passthrough.adls.gen2.tokenProviderClassName")
}
dbutils.fs.ls("abfss://cont1#kagsa1.dfs.core.windows.net/")
..it fails:
ExecutionError: An error occurred while calling z:com.databricks.backend.daemon.dbutils.FSUtils.ls.
: GET https://kagsa1.dfs.core.windows.net/cont1?resource=filesystem&maxResults=5000&timeout=90&recursive=false
StatusCode=403
StatusDescription=This request is not authorized to perform this operation using this permission.
ErrorCode=AuthorizationPermissionMismatch
ErrorMessage=This request is not authorized to perform this operation using this permission.
Databrics Workspace tier is Premium,
Cluster has Azure Data Lake Storage Credential Passthrough option enabled,
Storage account has hierarchical namespace option enabled,
Filesystem was initialized with
spark.conf.set("fs.azure.createRemoteFileSystemDuringInitialization", "true")
dbutils.fs.ls("abfss://cont1#kagsa1.dfs.core.windows.net/")
spark.conf.set("fs.azure.createRemoteFileSystemDuringInitialization", "false")
and I have full access to container in storage account:
What am I doing wrong?
Note: When performing the steps in the Assign the application to a role, make sure to assign the Storage Blob Data Contributor role to the service principal.
As part of repro, I have provided owner permission to the service principal and tried to run the “dbutils.fs.ls("mnt/azure/")”, returned same error message as above.
Now assigned the Storage Blob Data Contributor role to the service principal.
Finally, able to get the output without any error message after assigning Storage Blob Data Contributor role to the service principal.
For more details, refer “Tutorial: Azure Data Lake Storage Gen2, Azure Databricks & Spark”.
Reference: Azure Databricks - ADLS Gen2 throws 403 error message.
We have an issue configuring the MapReduce activity in ADF.
If we use Linked Service with Service Principal as an Authentication method, it causes the following issue: Activity Selector failed: The storage connection string is invalid.
The Linked Service's test connection is successful.
MapReduce activity allows browsing the storage.
The Service Principal has enough rights assigned in Storage Account's resource group: "Storage Blob Data Owner"
When Linked Service configured with Account Key - everything works correctly.