I tried to use the below uri for REST call, but getting error(403 Forbidden)
https://$storageAccount.table.core.windows.net/$tableName()?$filter=PartitionKey%20eq%20'Key1'
Is there other way? Please help.
According to my test, we can use share key to call the Azure table rest api
$accesskey="<storage account key>"
$storageAccount = "<account name>"
$version = "2017-04-17"
$resource = "table name"
$key="Jim"
$table_url = "https://$storageAccount.table.core.windows.net/$($resource)?`$filter=PartitionKey%20eq%20'$($key)'"
# create share key
$GMTTime = (Get-Date).ToUniversalTime().AddYears(1).toString('R')
$stringToSign = "$GMTTime`n/$storageAccount/$resource"
$hmacsha = New-Object System.Security.Cryptography.HMACSHA256
$hmacsha.key = [Convert]::FromBase64String($accesskey)
$signature = $hmacsha.ComputeHash([Text.Encoding]::UTF8.GetBytes($stringToSign))
$signature = [Convert]::ToBase64String($signature)
$headers = #{
'x-ms-date' = $GMTTime
"Authorization" = "SharedKeyLite " + $storageAccount + ":" + $signature
"x-ms-version" = $version
"Accept" = "application/json"
}
$item = Invoke-RestMethod -Method GET -Uri $table_url -Headers $headers -ContentType application/json
$item.value
Update
Regarding how to create sas token via Azure Portal, please refer to the following steps
Create sas token
Test
GET https://myaccount.table.core.windows.net/mytable()
?$filter=<>
&sv=2019-02-02&ss=t&srt=o&sp=r&se=2020-03-27T13:01:24Z&st=2020-03-27T05:01:24Z&spr=https&sig=OFUNXShu6kTojIp3SU...TkG%2BXAVZXJ8sqc%3D
Related
Trying to use the Databricks API to work with resources programmatically. I am using this microsoft documentto authenticate with a service principal.
https://learn.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/aad/service-prin-aad-token
But I'm getting the following error
"Invoke-RestMethod : {"error":"invalid_resource","error_description":"AADSTS500011: The resource principal named
https://management.core.azure.com was not found in the tenant"
This is my full script. What am I missing?
$ApiCommand = "clusters/get"
$DataBrick = "https://adb-3522222096750220.0.azuredatabricks.net"
$DataBricksResourceID = ""
$VaultName = ""
$KeyName = ""
$apiEndpointUri = "https://management.core.azure.com"
$tenantId = ""
$applicationId = ""
$secret = Get-AzKeyVaultSecret -VaultName $VaultName -Name $KeyName -AsPlainText
$RequestAccessTokenUri = "https://login.microsoftonline.com/$tenantId/oauth2/token"
$body = "grant_type=client_credentials&client_id=$applicationId&client_secret=$secret&resource=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d"
$Managementbody = "grant_type=client_credentials&client_id=$applicationId&client_secret=$secret&resource=$apiEndpointUri"
$contentType = 'application/x-www-form-urlencoded'
$AccessToken = Invoke-RestMethod -Method Post -Uri $RequestAccessTokenUri -Body $body -ContentType $contentType
Write-Output $AccessToken
$ManagementToken = Invoke-RestMethod -Method Post -Uri $RequestAccessTokenUri -Body $Managementbody -ContentType $contentType
Write-Output $ManagementToken
$apiuri = $DataBrick +"/api/2.0/$ApiCommand"
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("Authorization", "Bearer " + $AccessToken.access_token)
$headers.Add("X-Databricks-Azure-SP-Management-Token", $ManagementToken.access_token)
$headers.Add("X-Databricks-Azure-Workspace-Resource-Id", $DataBricksResourceID)
Invoke-RestMethod -Uri $apiuri -Headers $headers
The trailing / character in the management endpoint URI is really important - you need to specify it as in the documentation: https://management.core.windows.net/
You can also add this SP into the workspace itself, then you will need to get only one AAD token (see the docs).
I trying to call the storage REST API using powershell but facing errors. PFB details-
Script
$version = "2017-04-17"
$storageAccount = "{storageAccountName}"
$accesskey= “{storageAccountAccessKey}"
$resource = "?comp=list"
$storage_url = "https://$storageAccount.blob.core.windows.net/$resource"
$GMTTime = (Get-Date).ToUniversalTime().toString('R')
$stringToSign = "GET`nx-ms-date:$GMTTime`nx-ms-version:2017-04-17`n/$storageAccount/$resource"
$hmacsha = New-Object System.Security.Cryptography.HMACSHA256
$hmacsha.key = [Convert]::FromBase64String($accesskey)
$signature = $hmacsha.ComputeHash([Text.Encoding]::UTF8.GetBytes($stringToSign))
$signature = [Convert]::ToBase64String($signature)
$headers = #{
'x-ms-date' = $GMTTime
'Authorization' = "SharedKeyLite " + $storageAccount + ":" + $signature
'x-ms-version' = $version
}
Invoke-RestMethod -Method GET -Uri $storage_url -Headers $headers
Error Message
Invoke-RestMethod : AuthenticationFailedServer failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the
signature.
RequestId:ee5459f7-501e-004b-0426-46de36000000
Time:2020-06-19T10:45:36.7846714ZThe MAC signature found in the HTTP request <signature>' is not the same as any computed signature.
Server used following string to sign: 'GET
x-ms-date:Fri, 19 Jun 2020 10:37:00 GMT
x-ms-version:2017-04-17
/<storageAccount>/?comp=list'.
Can someone please help me understand what is missing here.
Documentation being followed for authorization- https://learn.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key
Please use the following code:
$version = "2017-04-17"
$storageAccount = "{storageAccountName}"
$accesskey= “{storageAccountAccessKey}"
$resource = "?comp=list"
$storage_url = "https://$storageAccount.blob.core.windows.net/$resource"
$GMTTime = (Get-Date).ToUniversalTime().toString('R')
$ContentMd5 = ""
$ContentType = ""
$CanonicalizedHeaders = "x-ms-date:$GMTTime`nx-ms-version:$version`n"
$CanonicalizedResource = "/$storageAccount/$resource"
$stringToSign = "GET`n$ContentMd5`n$ContentType`n`n$CanonicalizedHeaders$CanonicalizedResource"
$hmacsha = New-Object System.Security.Cryptography.HMACSHA256
$hmacsha.key = [Convert]::FromBase64String($accesskey)
$signature = $hmacsha.ComputeHash([Text.Encoding]::UTF8.GetBytes($stringToSign))
$signature = [Convert]::ToBase64String($signature)
$headers = #{
'x-ms-date' = $GMTTime
'Authorization' = "SharedKeyLite " + $storageAccount + ":" + $signature
'x-ms-version' = $version
}
Invoke-RestMethod -Method GET -Uri $storage_url -Headers $headers
Trying to submit a message to a service bus queue I have set up, and keep getting a 401 Unauthorized return.
I've tried configuring the SAS token myself using this method
$ResourceGroupName = 'myResourceGroup'
$NameSpaceName = "serviceBusNameSpace"
$QueueName = "myQueueName"
$PolicyName = "RootManageSharedAccessKey"
$body = "test message"
$Namespace = (Get-AzServiceBusNamespace -ResourceGroupName $ResourceGroupName -Name $namespacename).Name
$key = (Get-AzServiceBusKey -ResourceGroupName $ResourceGroupName -Namespace $namespacename -Name $PolicyName).PrimaryKey
$origin = [DateTime]"1/1/1970 00:00"
$Expiry = (Get-Date).AddMinutes(5)
#compute the token expiration time.
$diff = New-TimeSpan -Start $origin -End $Expiry
$tokenExpirationTime = [Convert]::ToInt32($diff.TotalSeconds)
#Create a new instance of the HMACSHA256 class and set the key to UTF8 for the size of $Key
$hmacsha = New-Object -TypeName System.Security.Cryptography.HMACSHA256
$hmacsha.Key = [Text.Encoding]::UTF8.GetBytes($Key)
$scope = "https://$Namespace.servicebus.windows.net/"
#create the string that will be used when cumputing the hash
$stringToSign = [Web.HttpUtility]::UrlEncode($scope) + "`n" + $tokenExpirationTime
#Compute hash from the HMACSHA256 instance we created above using the size of the UTF8 string above.
$hash = $hmacsha.ComputeHash([Text.Encoding]::UTF8.GetBytes($stringToSign))
#Convert the hash to base 64 string
$signature = [Convert]::ToBase64String($hash)
$fullResourceURI = "https://$Namespace.servicebus.windows.net/$QueueName"
#create the token
$token = [string]::Format([Globalization.CultureInfo]::InvariantCulture, `
"SharedAccessSignature sr={0}sig={1}&se={2}&skn={3}", `
[Web.HttpUtility]::UrlEncode($fullResourceURI), `
[Web.HttpUtility]::UrlEncode($signature), `
$tokenExpirationTime, $PolicyName)
$headers = #{ "Authorization" = "$token"; "Content-Type" = "application/atom+xml;type=entry;charset=utf-8" }
$uri = "https://$Namespace.servicebus.windows.net/$QueueName/messages"
$headers.Add("BrokerProperties", "{}")
#Invoke-WebRequest call.
Invoke-WebRequest -Uri $uri -Headers $headers -Method Post -Body $body -UseBasicParsing
I've also tried generating it through a built in cmdlet in Az.ServiceBus
$ResourceGroupName = 'myResourceGroup'
$NameSpaceName = "serviceBusNameSpace"
$QueueName = "myQueueName"
$PolicyName = "RootManageSharedAccessKey"
$body = "test message"
$expiry = (Get-Date).AddHours(2)
$authRule = Get-AzServiceBusAuthorizationRule -ResourceGroupName $ResourceGroupName -Namespace $NamespaceName
$token = New-AzServiceBusAuthorizationRuleSASToken -AuthorizationRuleId $authRule.Id -KeyType Primary -ExpiryTime $Expiry
$headers = #{ "Authorization" = "SharedAccessSignature $($token.SharedAccessSignature)"; "Content-Type" = "application/atom+xml;type=entry;charset=utf-8" }
$uri = "https://$Namespace.servicebus.windows.net/$QueueName/messages"
$headers.Add("BrokerProperties", "{}")
#Invoke-WebRequest call.
Invoke-WebRequest -Uri $uri -Headers $headers -Method Post -Body $body -UseBasicParsing
Both give me a 401 unauthorized error
Invoke-WebRequest : The remote server returned an error: (401) Unauthorized.
At line:9 char:17
+ ... $response = Invoke-WebRequest -Uri $uri -Headers $headers -Method Pos ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
I'm not sure what else to do. Is there a setting I need to configure for my queue within the azure portal?
Have found solution. UTC time was originally expiring token before even sending, in addition to malformed SAS signature
Final code edit below
$key = (Get-AzServiceBusKey -ResourceGroupName $ResourceGroupName -Namespace $namespacename -Name $PolicyName).PrimaryKey
$origin = [DateTime]"1/1/1970 00:00"
$Expiry = (Get-Date).AddMinutes(20)
$Expiry = $Expiry.ToUniversalTime()
#compute the token expiration time.
$diff = New-TimeSpan -Start $origin -End $Expiry
$tokenExpirationTime = [Convert]::ToInt32($diff.TotalSeconds)
$uri = "https://$Namespace.servicebus.windows.net/$QueueName/messages"
$scope = "https://$Namespace.servicebus.windows.net/$QueueName"
#create the string that will be used when cumputing the hash
$stringToSign = [Web.HttpUtility]::UrlEncode($scope) + "`n" + $tokenExpirationTime
#Create a new instance of the HMACSHA256 class and set the key to UTF8 for the size of $Key
$hmacsha = New-Object -TypeName System.Security.Cryptography.HMACSHA256
$hmacsha.Key = [Text.Encoding]::UTF8.GetBytes($Key)
#Compute hash from the HMACSHA256 instance we created above using the size of the UTF8 string above.
$hash = $hmacsha.ComputeHash([Text.Encoding]::UTF8.GetBytes($stringToSign))
#Convert the hash to base 64 string
$signature = [Convert]::ToBase64String($hash)
#create the token
$token = [string]::Format([Globalization.CultureInfo]::InvariantCulture, `
"SharedAccessSignature sr={0}&sig={1}&se={2}&skn={3}", `
[Web.HttpUtility]::UrlEncode($scope), `
[Web.HttpUtility]::UrlEncode($signature), `
$tokenExpirationTime, $PolicyName)
$headers = #{ "Authorization" = "$token"}
$headers.Add("Content-Type", "application/atom+xml;type=entry;charset=utf-8")
#Invoke-WebRequest call.
Invoke-WebRequest -Uri $uri -Headers $headers -Method Post -Body $body -UseBasicParsing
I have made some changes in your script and it is working fine.
$ResourceGroupName = 'myResourceGroup'
$Namespace = "serviceBusNameSpace"
$QueueName = "myQueueName"
$PolicyName = "RootManageSharedAccessKey"
$body = "test message"
$key = (Get-AzServiceBusKey -ResourceGroupName $ResourceGroupName -Namespace $Namespace -Name $PolicyName).PrimaryKey
$origin = [DateTime]"1/1/1970 00:00"
$Expiry = (Get-Date).AddMinutes(5)
#compute the token expiration time.
$diff = New-TimeSpan -Start $origin -End $Expiry
$tokenExpirationTime = [Convert]::ToInt32($diff.TotalSeconds)
#Create a new instance of the HMACSHA256 class and set the key to UTF8 for the size of $Key
$hmacsha = New-Object -TypeName System.Security.Cryptography.HMACSHA256
$hmacsha.Key = [Text.Encoding]::UTF8.GetBytes($Key)
#create the string that will be used when cumputing the hash
$stringToSign = [Web.HttpUtility]::UrlEncode($Namespace) + "`n" + $tokenExpirationTime
#Compute hash from the HMACSHA256 instance we created above using the size of the UTF8 string above.
$hash = $hmacsha.ComputeHash([Text.Encoding]::UTF8.GetBytes($stringToSign))
#Convert the hash to base 64 string
$signature = [Convert]::ToBase64String($hash)
#create the token
$token = [string]::Format([Globalization.CultureInfo]::InvariantCulture, `
"SharedAccessSignature sr={0}&sig={1}&se={2}&skn={3}", `
[Web.HttpUtility]::UrlEncode($Namespace), `
[Web.HttpUtility]::UrlEncode($signature), `
$tokenExpirationTime, $PolicyName)
$headers = #{ "Authorization" = "$token"; "Content-Type" = "application/atom+xml;type=entry;charset=utf-8" }
$uri = "https://$Namespace.servicebus.windows.net/$QueueName/messages"
$headers.Add("BrokerProperties", "{}")
#Invoke-WebRequest call.
Invoke-WebRequest -Uri $uri -Headers $headers -Method Post -Body $body -UseBasicParsing
The changes which I have made are:
You don't need to create scope variable. You need to pass the $Namespace to stringToSign.
You don't need to use Get-AzServiceBusNamespace to get namespace name as you are already taking this as user input.
See post edit.
Token expiration time wasn't converted to UTC, making it always expired, in addition to not having the SaS token configuration string formed correctly.
I've tried to authenticate to Azure Batch using REST API, to do so I wrote following PowerShell code
$Key = 'key'
$region = "region"
$sharedKey = [System.Convert]::FromBase64String($Key)
$date = [System.DateTime]::UtcNow.ToString("R")
$stringToSign = "GET`n`n`n`n`n`n`n`n`n`n`n`nocp-date:$date`n /$batchAccount/jobs`napi-version:2019-08-01.10.0`ntimeout:20"
[byte[]]$dataBytes = ([System.Text.Encoding]::UTF8).GetBytes($stringToSign)
$hmacsha256 = New-Object System.Security.Cryptography.HMACSHA256
$hmacsha256.Key = [Convert]::FromBase64String($key)
$sig = [Convert]::ToBase64String($hmacsha256.ComputeHash($dataBytes))
$authhdr = "SharedKey $BatchAccount`:$sig"
$headers = #{
"ocp-date" = $date;
"Authorization" = "$authhdr";
}
Invoke-restmethod -Headers $headers -Uri 'https://$BatchAccount.$region.batch.azure.com/jobs?api-version=2019-08-01.10.0'
please note that I know that I can
use OAuth2 as alternative authentication mechanism
use Az.Batch powershell modules
I just wanted to do this using REST and SharedKey scheme as described here
https://learn.microsoft.com/en-us/rest/api/batchservice/authenticate-requests-to-the-azure-batch-service
for this API
https://learn.microsoft.com/en-us/rest/api/batchservice/job/list
But for some reason it doesn't work
I get this error but everything seems to be folowing the docs
"message":{
"lang":"en-US",
"value":"Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:eb5134f2-2821-4244-ac5b-066bf19bec10\nTime:2019-11-24T21:08:20.3223384Z"
},
"values":[
{
"key":"AuthenticationErrorDetail",
"value":"The MAC signature found in the HTTP request 'signature-goes-here' is not the same as any computed signature. Server used following string to sign: 'GET\n\n\n\n\napplication/json; odata=minimalmetadata; charset=utf-8\n\n\n\n\n\n\nocp-date:Sun, 24 Nov 2019 21:08:20 GMT\n/name-goes-here/jobs\napi-version:2019-08-01.10.0'."
}
]
There is something wrong with $stringToSign . Try this :
$Key = "your key"
$region = "your region"
$BatchAccount = "your account name"
$BatchAccountURL = "Https://$BatchAccount.$region.batch.azure.com"
$sharedKey = [System.Convert]::FromBase64String($Key)
$date = [System.DateTime]::UtcNow.ToString("R")
$stringToSign = "GET`n`n`n`n`n`n`n`n`n`n`n`nocp-date:$date`n/$BatchAccount/jobs`napi-version:2019-08-01.10.0"
[byte[]]$dataBytes = ([System.Text.Encoding]::UTF8).GetBytes($stringToSign)
$hmacsha256 = New-Object System.Security.Cryptography.HMACSHA256
$hmacsha256.Key = [Convert]::FromBase64String($key)
$sig = [Convert]::ToBase64String($hmacsha256.ComputeHash($dataBytes))
$authhdr = "SharedKey $BatchAccount`:$sig"
$headers = #{
"ocp-date" = $date;
"Authorization" = "$authhdr";
}
Invoke-restmethod -Headers $headers -Uri "$BatchAccountURL/jobs?api-version=2019-08-01.10.0"
Result :
I’m able to create files and folders in ADLS using PowerShell and ADLS Gen 2 REST API. However I’m having trouble renaming the file. I am using “x-ms-rename-source” in the header but its throwing exception.
code:
$n = '`n'
$stringToSign +=
#SECTION: CanonicalizedHeaders + “\n” #
“x-ms-date:$date” + $n +
“x-ms-version:2018-11-09” + $n +
“x-ms-rename-source:/adlsg2filesystemname/folderpath/filename” + $n
$stringToSign +=
# SECTION: CanonicalizedResource + “\n” #
“/$StorageAccountName/$FilesystemName” + $PathToCreate + $n
$sharedKey = [System.Convert]::FromBase64String($AccessKey)
$hasher = New-Object System.Security.Cryptography.HMACSHA256
$hasher.Key = $sharedKey
$signedSignature = [System.Convert]::ToBase64String($hasher.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($stringToSign)))
$authHeader = “SharedKey ${StorageAccountName}:$signedSignature”
$headers = #{“x-ms-date”=$date}
$headers.Add(“x-ms-version”,”2018-11-09″)
$headers.Add(“x-ms-rename-source”,”/adlsg2filesystemname/folderpath/filename”)
$headers.Add(“Authorization”,$authHeader)
$headers.Add(“If-None-Match”,”*”) # To fail if the destination already exists, use a conditional request with If-None-Match: “*”
$URI = “https://$StorageAccountName.dfs.core.windows.net/” + $FilesystemName + $PathToCreate
I am getting below exception:
Invoke-RestMethod : {"error":{"code":"AuthenticationFailed","message":"Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly
including the signature.\nRequestId:ddfd851b-501f-0057-3f88-7e0a7d000000\nTime:2019-10-09T09:59:53.7708781Z"}}
Any help will be truly appreciated. Thanks.
According to my test, we can use Azure AD authentication to call Azure data lake storage Gen2 REST API. For more details, please refer to https://social.msdn.microsoft.com/Forums/en-US/45be0931-379d-4252-9d20-164261cc64c5/error-while-calling-adls-gen-2-rest-api-to-create-file?forum=AzureDataLake.
Create Azure AD service principal and assign a RABC role to it. For further information, please refer to https://learn.microsoft.com/en-us/azure/storage/common/storage-auth-aad.
Connect-AzAccount
$password=''
$credentials = New-Object Microsoft.Azure.Commands.ActiveDirectory.PSADPasswordCredential -Property #{ StartDate=Get-Date; EndDate=Get-Date -Year 2024; Password=$password}
$sp = New-AzAdServicePrincipal -DisplayName jimtest1 -PasswordCredential $credentials
New-AzRoleAssignment -ApplicationId $sp.ApplicationId -RoleDefinitionName "Storage Blob Data Owner" -Scope "your scope such as your storage account scope"
get access token
$TeantID='hanxia.onmicrosoft.com'
$TokenResult = Invoke-RestMethod -Method Post -ContentType 'application/x-www-form-urlencoded' -Uri "https://login.microsoftonline.com/$($TeantID)/oauth2/token" -Body #{
client_id = $sp.ApplicationId # the application id of service principal
resource = 'https://storage.azure.com'
grant_type = 'client_credentials'
client_secret = $password # you use it in step 1
}
Call the rest api
$StorageAccountName =''
$FilesystemName =''
$PathToCreate=''
$URI = “https://$StorageAccountName.dfs.core.windows.net/” + $FilesystemName +"/"+$PathToCreate
Invoke-RestMethod -Method Put -Uri $URI -Headers #{
'Authorization' = "Bearer "+ $TokenResult.access_token
'x-ms-rename-source' = ' '
}