I want to do operations in Azure datalake gen2 using rest operations. I have a service principal with client secret and having owner access on storage account.
I am confused how to construct the request for operations. I can't find any proper example demonstrating it.
The way which I want is to:
Get access token
Make a put request with bearer authentication method
Below are documents which I am referring
Access token
Put blob operation
I want to do it through postman. It would be really helpful if someone can suggest it
I tried to reproduce the same in my environment and got below results:
I created one service principal named DataLake and added API permissions as below:
Now, I granted Storage Blob Data Contributor role to that service principal at storage account level like below:
Go to Azure Portal -> Storage Accounts -> Your storage account -> Access Control (IAM) -> Add role assignment -> Storage Blob Data Contributor
To generate the access token via Postman, I used below parameters:
POST https://login.microsoftonline.com/<tenantID>/oauth2/v2.0/token
client_id:<appID>
grant_type:client_credentials
client_secret:<secret>
scope: https://storage.azure.com/.default
Response:
When I ran the below query by including above Bearer token, I got Status 201 Created like below:
PUT https://<storageaccname>.blob.core.windows.net/<container_name>/test.txt
Authorization:Bearer <token>
x-ms-version:2017-11-09
x-ms-blob-type:BlockBlob
Response:
You need to attach the file in Postman before running the query like below:
When I checked the same in Azure Portal, file uploaded to storage account successfully like below:
Related
I created an Azure AD Service Principal and to generate token I am using Client Credential Flow:
POST https://login.microsoftonline.com/tenantID/oauth2/token
&client_id = redacted
&grant_type = client_credentials
&resource = https://storage.azure.com
&client_secret = redacted
Token generated successfully but when I tried to list the containers in my storage account, it threw me Authorization Permission Mismatched error.
To list containers, I used below query
GET https://storageaccname.blob.core.windows.net/?comp=list
Error Details:
<?xml version="1.0" encoding="utf-8"?>
<Error>
<Code>AuthorizationPermissionMismatch</Code>
<Message>This request is not authorized to perform this operation using this permission.
RequestId:
Time:2022-11-20T08:12:24.9827677Z</Message>
</Error>
I am not sure what permissions I am missing after some Google search I found Storage Blob Data Contributor Role is required. I assigned this role to the Service Principal I created.
But still the same error, any workaround to resolve my issue??
I tried to reproduce the same in my environment and got the same error as below:
GET https://StorageAccName.blob.core.windows.net/?comp=list
Check whether you have assigned Storage Blob Data Contributor Role to the Service Principal and like below:
Go to Azure Portal -> Storage Accounts -> Your Storage Account -> Access Control (IAM) -> Add role assignment
To resolve the error try generating the token using v2.0 token endpoint like below:
POST https://login.microsoftonline.com/Tenant_ID/oauth2/v2.0/token
&client_id = Client_ID
&grant_type = client_credentials
&resource = https://storage.azure.com
&client_secret = Client_Secret
Using the above generated access token, I am able to get the list of containers successfully like below:
I want create APIM subscriptions through rest api, And was able to do it successfully by following this Microsoft doc, https://learn.microsoft.com/en-us/rest/api/apimanagement/current-ga/subscription.
And for Authentication I am generating a bearer token using ROPC grant type(My UserName & Password). Everything works fine with this flow.
But i dont want to configure my username & password in a application to get a bearer token, instead i followed Client-Credentials grant type(get token by client id & secret), i am able to generate token, but when i use that token to create subscription in APIM, i am getting a exception
The client '0--e' with object id '0--e' does not have authorization to perform action 'Microsoft.ApiManagement/service/subscriptions/write'
Is it possible to add a AAD application inside APIM AccessControl(IAM) to grant permission.
Or is this any other way to do this? or ROPC is the only way?
Can someone please help.
Yes, you can grant permission to AAD application (service principal) in APIM Access Control (IAM) by assigning it API Management Service Contributor role.
I tried to reproduce the same in my environment and got the below results:
I have generated one access token using Client-Credentials grant type like below:
When I used the above token to create APIM subscription with below query, I got the same error:
PUT https://management.azure.com/subscriptions/subid/resourceGroups/rg1/providers/Microsoft.ApiManagement/service/apimService1/subscriptions/testsub?api-version=2021-12-01
{
"properties": {
"ownerId": "/subscriptions/subid/resourceGroups/rgname/providers/Microsoft.ApiManagement/service/servicename/users/xxxxxxxxxxx",
"scope": "/subscriptions/subid/resourceGroups/rgname/providers/Microsoft.ApiManagement/service/servicename/products/xxxxxxxxxxx",
"displayName": "testsub"
}
}
Response:
To resolve the error, you need to grant API Management Service Contributor role for that application like below:
Go to Azure Portal -> APIM Services -> Your APIM -> Access control (IAM) -> Add role assignment
After granting the above role, I generated the access token again and ran the same query as below:
PUT https://management.azure.com/subscriptions/subid/resourceGroups/rg1/providers/Microsoft.ApiManagement/service/apimService1/subscriptions/testsub?api-version=2021-12-01
Response:
When I checked the Portal, APIM subscription got created successfully like below:
Reference:
How to use Role-Based Access Control in Azure API Management | Microsoft Docs
I have a nodejs application using the express framework. The user must sign in via Oauth2 protocol against their MS Azure Active Directory credentials. This is done using the passport-azure-ad-oauth2 npm package
I have successfully got the application working so that I can upload files in blob form to an azure storage container. The container access level is set to private. I have assigned user roles for the container so that certain users within the AD have 'reader and data access'. So my understanding is that when these users are authenticated via Oauth2, they should be able to access the files when retrieving the file's URL. However, after authentication, I am not able to access the files. I get the following error.
ResourceNotFound
The specified resource does not exist. RequestId:6341ef80-f01e-0011-6442-08f7c2000000 Time:2021-02-21T11:14:26.3475641Z
I have also followed the steps to grant the application permissions to azure storage here
What do I need to do so that authenticated users can go to the specific url for each blob and get access? Do I need to pass a token in the request? If so, how do you do this?
Any help would be much appreciated.
The 404 error(The specified resource does not exist) is always related to your request URL, but not the access token.
For example to get blob, you need to use GET Method, and some of the Request Headers are required.
GET https://myaccount.blob.core.windows.net/mycontainer/myblob
headers:
Authorization: Bear <access-token>
x-ms-version: 2020-04-08
x-ms-date: Fri, 26 Jun 2015 23:39:12 GMT
Get access token by application permission without user:
POST https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token
client_id={client_id}
&client_secret={client_secret}
&scope=http://storage.azure.com/.default // change to http://storage.azure.com/ for resource
&grant_type=client_credentials
Get access token by delegated permission with a signed-in user:
GET https://login.microsoftonline.com/{tenant}/oauth2/v2.0/authorize?
client_id={client_id}
&response_type=token
&redirect_uri=https://localhost:44300/
&scope=https://storage.azure.com/user_impersonation
&response_mode=fragment
&state=12345
&nonce=678910
Note: Navigate to your storage account -> Access Control (IAM) -> Add role assignment -> select Storage Blob Data XXX role and your login account. I add Storage Blob Data Contributor in my side, it might take up to 5 minutes to propagate the RBAC rule.
I would need to start an Azure Data Factory pipeline from REST API as per https://learn.microsoft.com/en-us/rest/api/datafactory/pipelines/createrun#code-try-0
I have created an AAD app, and given it the Azure Service Management API Permission. However when the client through the OAuth2 Implicit flow receives an id_token and invokes that API to start a pipeline I get
{
"error": {
"code": "InvalidAuthenticationToken",
"message": "The access token is invalid."
}
}
Am i using proper API pemrission? thanks.
If you just want to use OAuth2 flow to get the token to call the REST API, the client credentials flow is more suitable than the Implicit flow in this case.
Please follow the steps below.
1.Get values for signing in and create a new application secret.
2.Navigate to the data factory -> Access control (IAM) -> Add -> add your AD App as an RBAC role e.g. Contributor, Owner, Data Factory Contributor, details follow this.
3.In the postman, follow the screenshot below, fix the request body got from step 1, then use the token to call REST API, it will work fine.
POST https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token
client_id=<client_id>
&scope=https://management.azure.com/.default
&client_secret=<client_secret>
&grant_type=client_credentials
I cant test it right now but I would assume that having Data Factory Contributor should be enough for this.
https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#data-factory-contributor
I have a storage account in azure with ADLS gen2 (hierarchy enabled). I have created a app and a service principal. I have also created a container in the storage and a folder inside that container. Using ACL I have given execute permission to container and read and write permission to the folder.
I am able to copy data using azcopy to the directory I have created inside the container.
I would like to list the data I have copied. Since azcopy list does not work, I tried using rest api
Using Post req
https://login.microsoftonline.com//oauth2/v2.0/token
client_id : app_id
client_secret : secret
grant_type : client_credentials
scope : https://storage.azure.com/.default
I get a token. When I use this token here
https:///?directory=&resource=filesystem&recursive=true
Authorization = Bearer
x-ms-version : 2019-02-02
I get this error
{
"error": {
"code": "AuthorizationPermissionMismatch",
"message": "This request is not authorized to perform this operation using this permission.\nRequestId:0cd2e016-601f-003b-5fbe-5870a7000000\nTime:2019-08-22T07:50:06.5848071Z"
}
}
What am I missing here?
It requires **Read and Execute** permission to list the content of the directory.
I would suggest you to give Read & Execute both permission to application.
Reference:
https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/storage/blobs/data-lake-storage-access-control.md
Please try and see if it works.