I’m able to create files and folders in ADLS using PowerShell and ADLS Gen 2 REST API. However I’m having trouble renaming the file. I am using “x-ms-rename-source” in the header but its throwing exception.
code:
$n = '`n'
$stringToSign +=
#SECTION: CanonicalizedHeaders + “\n” #
“x-ms-date:$date” + $n +
“x-ms-version:2018-11-09” + $n +
“x-ms-rename-source:/adlsg2filesystemname/folderpath/filename” + $n
$stringToSign +=
# SECTION: CanonicalizedResource + “\n” #
“/$StorageAccountName/$FilesystemName” + $PathToCreate + $n
$sharedKey = [System.Convert]::FromBase64String($AccessKey)
$hasher = New-Object System.Security.Cryptography.HMACSHA256
$hasher.Key = $sharedKey
$signedSignature = [System.Convert]::ToBase64String($hasher.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($stringToSign)))
$authHeader = “SharedKey ${StorageAccountName}:$signedSignature”
$headers = #{“x-ms-date”=$date}
$headers.Add(“x-ms-version”,”2018-11-09″)
$headers.Add(“x-ms-rename-source”,”/adlsg2filesystemname/folderpath/filename”)
$headers.Add(“Authorization”,$authHeader)
$headers.Add(“If-None-Match”,”*”) # To fail if the destination already exists, use a conditional request with If-None-Match: “*”
$URI = “https://$StorageAccountName.dfs.core.windows.net/” + $FilesystemName + $PathToCreate
I am getting below exception:
Invoke-RestMethod : {"error":{"code":"AuthenticationFailed","message":"Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly
including the signature.\nRequestId:ddfd851b-501f-0057-3f88-7e0a7d000000\nTime:2019-10-09T09:59:53.7708781Z"}}
Any help will be truly appreciated. Thanks.
According to my test, we can use Azure AD authentication to call Azure data lake storage Gen2 REST API. For more details, please refer to https://social.msdn.microsoft.com/Forums/en-US/45be0931-379d-4252-9d20-164261cc64c5/error-while-calling-adls-gen-2-rest-api-to-create-file?forum=AzureDataLake.
Create Azure AD service principal and assign a RABC role to it. For further information, please refer to https://learn.microsoft.com/en-us/azure/storage/common/storage-auth-aad.
Connect-AzAccount
$password=''
$credentials = New-Object Microsoft.Azure.Commands.ActiveDirectory.PSADPasswordCredential -Property #{ StartDate=Get-Date; EndDate=Get-Date -Year 2024; Password=$password}
$sp = New-AzAdServicePrincipal -DisplayName jimtest1 -PasswordCredential $credentials
New-AzRoleAssignment -ApplicationId $sp.ApplicationId -RoleDefinitionName "Storage Blob Data Owner" -Scope "your scope such as your storage account scope"
get access token
$TeantID='hanxia.onmicrosoft.com'
$TokenResult = Invoke-RestMethod -Method Post -ContentType 'application/x-www-form-urlencoded' -Uri "https://login.microsoftonline.com/$($TeantID)/oauth2/token" -Body #{
client_id = $sp.ApplicationId # the application id of service principal
resource = 'https://storage.azure.com'
grant_type = 'client_credentials'
client_secret = $password # you use it in step 1
}
Call the rest api
$StorageAccountName =''
$FilesystemName =''
$PathToCreate=''
$URI = “https://$StorageAccountName.dfs.core.windows.net/” + $FilesystemName +"/"+$PathToCreate
Invoke-RestMethod -Method Put -Uri $URI -Headers #{
'Authorization' = "Bearer "+ $TokenResult.access_token
'x-ms-rename-source' = ' '
}
Related
I am calling the storage REST API to get container names using
Invoke-WebRequest -Method GET -Uri $storage_url -Headers $headers
This command often returns 'remote name could not be resolved error', even when the storage account exists and is reachable. Just running the command again gives correct result.
Invoke-WebRequest : The remote name could not be resolved: '<storageAccountName>.blob.core.windows.net'
At line:1 char:1
+ Invoke-WebRequest -Method GET -Uri $storage_url -Headers $headers #In ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
From the information you provided, you use the client credential flow to get the access token, then use the token to call the Storage Rest API - List Containers.
You could use the script below, it works for me.
Make sure the service principal you used has a RBAC role e.g. Contributor/Owner in your storage account -> Access Control, if not, click the Add to add it.
$ClientID = "xxxxxxx"
$ClientSecret = "xxxxxxx"
$tennantid = "xxxxxxx"
$storageaccountname = "joystoragev2"
$TokenEndpoint = {https://login.microsoftonline.com/{0}/oauth2/token} -f $tennantid
$Resource = "https://storage.azure.com/"
$Body = #{
'resource'= $Resource
'client_id' = $ClientID
'grant_type' = 'client_credentials'
'client_secret' = $ClientSecret
}
$params = #{
ContentType = 'application/x-www-form-urlencoded'
Headers = #{'accept'='application/json'}
Body = $Body
Method = 'Post'
URI = $TokenEndpoint
}
$token = Invoke-RestMethod #params
$accesstoken = $token.access_token
$url = {https://{0}.blob.core.windows.net/?comp=list} -f $storageaccountname
$header = #{
'Authorization' = 'Bearer ' + $accesstoken
'x-ms-version' = '2019-02-02'
}
$response = Invoke-WebRequest –Uri $url –Headers $header –Method GET
$response.RawContent
I tried to use the below uri for REST call, but getting error(403 Forbidden)
https://$storageAccount.table.core.windows.net/$tableName()?$filter=PartitionKey%20eq%20'Key1'
Is there other way? Please help.
According to my test, we can use share key to call the Azure table rest api
$accesskey="<storage account key>"
$storageAccount = "<account name>"
$version = "2017-04-17"
$resource = "table name"
$key="Jim"
$table_url = "https://$storageAccount.table.core.windows.net/$($resource)?`$filter=PartitionKey%20eq%20'$($key)'"
# create share key
$GMTTime = (Get-Date).ToUniversalTime().AddYears(1).toString('R')
$stringToSign = "$GMTTime`n/$storageAccount/$resource"
$hmacsha = New-Object System.Security.Cryptography.HMACSHA256
$hmacsha.key = [Convert]::FromBase64String($accesskey)
$signature = $hmacsha.ComputeHash([Text.Encoding]::UTF8.GetBytes($stringToSign))
$signature = [Convert]::ToBase64String($signature)
$headers = #{
'x-ms-date' = $GMTTime
"Authorization" = "SharedKeyLite " + $storageAccount + ":" + $signature
"x-ms-version" = $version
"Accept" = "application/json"
}
$item = Invoke-RestMethod -Method GET -Uri $table_url -Headers $headers -ContentType application/json
$item.value
Update
Regarding how to create sas token via Azure Portal, please refer to the following steps
Create sas token
Test
GET https://myaccount.table.core.windows.net/mytable()
?$filter=<>
&sv=2019-02-02&ss=t&srt=o&sp=r&se=2020-03-27T13:01:24Z&st=2020-03-27T05:01:24Z&spr=https&sig=OFUNXShu6kTojIp3SU...TkG%2BXAVZXJ8sqc%3D
I'm trying to implement Azure Active Directory in my API Management instance using the Protect an API by using OAuth 2.0 with Azure Active Directory and API Management doc. The doc suggests that in order to get the access token I need to use the Developer Portal.
My problem is: An external application is going to communicate with API Management. Is there a way to omit the Developer Portal and get the access token programmatically?
It's a pain but thanks to Jos Lieben I am able to do it with this Powershell function
It's specifically for granting API access on behalf of the Org, but as you can see you can extract the commands to get and use the API token.
Original Author Link: https://www.lieben.nu/liebensraum/2018/04/how-to-grant-oauth2-permissions-to-an-azure-ad-application-using-powershell-unattended-silently/
Function Grant-OAuth2PermissionsToApp{
Param(
[Parameter(Mandatory=$true)]$Username, #global administrator username
[Parameter(Mandatory=$true)]$Password, #global administrator password
[Parameter(Mandatory=$true)]$azureAppId #application ID of the azure application you wish to admin-consent to
)
$secpasswd = ConvertTo-SecureString $Password -AsPlainText -Force
$mycreds = New-Object System.Management.Automation.PSCredential ($Username, $secpasswd)
$res = login-azurermaccount -Credential $mycreds
$context = Get-AzureRmContext
$tenantId = $context.Tenant.Id
$refreshToken = #($context.TokenCache.ReadItems() | where {$_.tenantId -eq $tenantId -and $_.ExpiresOn -gt (Get-Date)})[0].RefreshToken
$body = "grant_type=refresh_token&refresh_token=$($refreshToken)&resource=74658136-14ec-4630-ad9b-26e160ff0fc6"
$apiToken = Invoke-RestMethod "https://login.windows.net/$tenantId/oauth2/token" -Method POST -Body $body -ContentType 'application/x-www-form-urlencoded'
$header = #{
'Authorization' = 'Bearer ' + $apiToken.access_token
'X-Requested-With'= 'XMLHttpRequest'
'x-ms-client-request-id'= [guid]::NewGuid()
'x-ms-correlation-id' = [guid]::NewGuid()
}
$script:url = "https://main.iam.ad.ext.azure.com/api/RegisteredApplications/$azureAppId/Consent?onBehalfOfAll=true"
Invoke-RestMethod -Uri $url -Headers $header -Method POST -ErrorAction Stop
}
I Have a problem. Could you please help me view list Cloud Service Classic use PowerShell and Azure Rest API. When I used script for Web APP I show list Web APP, but when I used scrip for Cloud Service Classic I show error.
# Variables
$TenantId = "" # Enter Tenant Id.
$ClientId = "" # Enter Client Id.
$ClientSecret = "" # Enter Client Secret.
$Resource = "https://management.core.windows.net/"
$SubscriptionId = "" # Enter Subscription Id.
$RequestAccessTokenUri = "https://login.microsoftonline.com/$TenantId/oauth2/token"
$body = "grant_type=client_credentials&client_id=$ClientId&client_secret=$ClientSecret&resource=$Resource"
$Token = Invoke-RestMethod -Method Post -Uri $RequestAccessTokenUri -Body $body -ContentType 'application/x-www-form-urlencoded'
Write-Host "Print Token" -ForegroundColor Green
Write-Output $Token
# Get Azure Resource Groups
$ResourceGroupApiUri = "https://management.core.windows.net/$SubscriptionId/services/hostedservices"
$Headers = #{}
$Headers.Add("Authorization","$($Token.token_type) "+ " " + "$($Token.access_token)")
$ResourceGroups = Invoke-RestMethod -Method Get -Uri $ResourceGroupApiUri -Headers $Headers
Write-Host "Print Resource groups" -ForegroundColor Green
Write-Output $ResourceGroups
Invoke-RestMethod : ForbiddenErrorThe server failed to authenticate the request. Verify that the certificate is valid and
is associated with this subscription.
Actually, there is a built-in ASM PowerShell to list the cloud services associated with the current subscription.
Get-AzureService
Reference - https://learn.microsoft.com/en-us/powershell/module/servicemanagement/azure/get-azureservice?view=azuresmps-4.0.0
Besides, if you insist on calling the ASM rest api with powershell, you could refer to this article, the sample calls the Get Deployment api, just change it to List Cloud Services.
#Request Headers required to invoke the GET DEPLOYMENT REST API
$method
=
“GET”
$headerDate
= ‘2009-10-01’
$headers
= #{“x-ms-version”=“$headerDate“}
#Retrieving the subscription ID
$subID
= (Get-AzureSubscription
-Current).SubscriptionId
$URI
=
https://management.core.windows.net/$subID/services/hostedservices/kaushalz/deployments/4f006bb7d2874dd4895f77a97b7938d0
#Retrieving the certificate from Local Store
$cert
= (Get-ChildItem
Cert:\CurrentUser\My
|
?{$_.Thumbprint -eq
“B4D460D985F1D07A6B9F8BFD67E36BC53A4490FC”}).GetRawCertData()
#converting the raw cert data to BASE64
body
=
“<Binary>—–BEGIN CERTIFICATE—–`n$([convert]::ToBase64String($cert))`n—–END CERTIFICATE—–</Binary>”
#Retrieving the certificate ThumbPrint
$mgmtCertThumb
= (Get-AzureSubscription
-Current).Certificate.Thumbprint
#Passing all the above parameters to Invoke-RestMethod cmdlet
Invoke-RestMethod
-Uri
$URI
-Method
$method
-Headers
$headers
-CertificateThumbprint
” B4D460D985F1D07A6B9F8BFD67E36BC53A4490FC”
-ContentType
$ContentType
In our school we use the Azure AD. Currently we have two custom applications A and B.
We should assign application A to all the users with mail address *#student.example.com and the users with #example.com to application B.
How can we assign the users based on this criteria without doing in manually?
You can use Graph API to automate this process. Here is a PowerShell Script I wrote to use the Graph API.
Add-Type -Path 'C:\Program Files\Microsoft Azure Active Directory Connect\Microsoft.IdentityModel.Clients.ActiveDirectory.dll'
# Some common fields to log into your tenant.
$tenantID = "<your tenantID>"
$loginEndpoint = "https://login.windows.net/"
# The default redirect URI and client id.
# No need to change them.
$redirectURI = New-Object System.Uri ("urn:ietf:wg:oauth:2.0:oob")
$clientID = "1950a258-227b-4e31-a9cf-717495945fc2"
$username = "<a global user of your tenant>"
$email_prefix1 = "*#student.example.com"
$email_prefix2 = "*#example.com"
# The display name of your AD apps, It's better if one does not contain another,
# because I am using the filter "startwith".
$apps1 = "<the display name of you first AD application>"
$apps2 = "<the display name of you second AD application>"
$resource = "https://graph.windows.net/"
# logging into your tenant to get the authorization header.
$authString = $loginEndpoint + $tenantID
$authenticationContext = New-Object Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext ($authString, $false)
$promptBehaviour = [Microsoft.IdentityModel.Clients.ActiveDirectory.PromptBehavior]::Auto
$userIdentifierType = [Microsoft.IdentityModel.Clients.ActiveDirectory.UserIdentifierType]::RequiredDisplayableId
$userIdentifier = New-Object Microsoft.IdentityModel.Clients.ActiveDirectory.UserIdentifier ($username, $userIdentifierType)
$authenticationResult = $authenticationContext.AcquireToken($resource, $clientID, $redirectURI, $promptBehaviour, $userIdentifier);
# construct authorization header for the REST API.
$authHeader = $authenticationResult.AccessTokenType + " " + $authenticationResult.AccessToken
$headers = #{"Authorization"=$authHeader; "Content-Type"="application/json"}
# getting the service principal object id of the 2 AD apps.
$uri = "https://graph.windows.net/$tenantID/servicePrincipals?api-version=1.5&`$filter=startswith(displayName,'$apps1')"
$apps = Invoke-RestMethod -Method Get -Uri $uri -Headers $headers
$app1_objectId = $apps.value[0].objectId
$uri = "https://graph.windows.net/$tenantID/servicePrincipals?api-version=1.5&`$filter=startswith(displayName,'$apps2')"
$apps = Invoke-RestMethod -Method Get -Uri $uri -Headers $headers
$app2_objectId = $apps.value[0].objectId
# getting the users in the tenant.
$uri = "https://graph.windows.net/$tenantID/users?api-version=1.5"
$users = Invoke-RestMethod -Method Get -Uri $uri -Headers $headers
# loop through the whole user list to assign the AD apps.
foreach ($user in $users.value){
$userID = $user.objectId
if ($user.otherMails[0] -like $email_prefix1){
$resourceId = $app1_objectId
}
elseif ($user.otherMails[0] -like $email_prefix2){
$resourceId = $app2_objectId
}
else{
continue
}
# Leave the id to be 00000000-0000-0000-0000-000000000000.
# This is exactly how Azure Classic Portal handles user assigning.
# That means if you assign a user to an AD application in the portal,
# the appRoleAssignment will have the id 00000000-0000-0000-0000-000000000000.
$body = #"
{"id": "00000000-0000-0000-0000-000000000000",
"principalId": "$userID",
"resourceId": "$resourceId"
}
"#
$uri = "https://graph.windows.net/$tenantID/users/$userID/appRoleAssignments?api-version=1.5"
Invoke-RestMethod -Method Post -Uri $uri -Headers $headers -Body $body
}
Notice that I am using the email address in otherMails. If you are using Live id, that email address is just the user's live id. If you are using organization id, you can have it set in the classic portal as field Alternate email address.