I'm trying to upload a file to Azure Storage Account as Blob and I have a client provided by this "github.com/Azure/azure-storage-blob-go/azblob" package. As I saw in documentation there should be possibility to communicate with Storage using SAS Token creating anonymous credential with
credential := azblob.NewAnonymousCredential()
po := azblob.PipelineOptions{
Log: pipeline.LogOptions{
Log: func(s pipeline.LogLevel, m string) {
log.Tracef("pipeline message: %s", m)
},
ShouldLog: func(level pipeline.LogLevel) bool {
return level <= pipeline.LogError
},
},
}
pipeline := azblob.NewPipeline(credential, po)
However, I don't see an option to pass SAS Token which I receive from other service after I ask for access.
I also tried do it 'manually' using Azure Storage Account REST API, so my URL was like https://servicename.blob.core.windows.net/containerID/BlobID?sasToken... but all I get was 400, 411 and 501 HTTP codes depending on request header.
For example with
req.Header.Add("Accept", "*/*")
req.Header.Add("Accept-Language", "en-US,en;q=0.5 --compressed")
req.Header.Add("Accept-Encoding", "gzip, deflate, br")
req.Header.Add("content-type", "application/octet-stream")
req.Header.Add("x-ms-version", "2019-02-02")
req.Header.Add("x-ms-blob-type", "BlockBlob")
req.Header.Add("x-ms-client-request-id", "someID")
req.Header.Add("Connection", "keep-alive")
req.Header.Add("Content-Length", "512000")
req.Header.Add("Transfer-Encoding", "gzip, chunked, deflate")
I receive 400 code with
<?xml version="1.0" encoding="utf-8"?>
<Error>
<Code>MissingRequiredHeader</Code>
<Message>
An HTTP header that's mandatory for this request is not specified.
RequestId:someId
Time:2020-02-14T13:47:58.8383371Z
</Message>
<HeaderName>x-ms-original-content-length</HeaderName>
</Error>
Adding x-ms-original-content-length header changes nothing.
The funny fact is that it only happens when I try it in Go code. When I tried any REST Client it was working with these headers.
Summarizing, my need is to put a file into Azure Storage Account as a blob and the second solution which should simply work, does not work, and first one is not completed because I don't see the way to pass SAS Token. What am I missing?
So in the 1st case the problem was that SAS token is passed nowhere in this package. It should be added to URL later on during url creation like:
URL, err := url.Parse(blobURL + "/" + containerName + "/" + blobName + "?token as query"
And in the second case everything was about Content-Length which is not changeable from Header side. It's automatically set during http.NewRequest(...) but it must be one of following types *bytes.Buffer, *bytes.Reader or *strings.Reader. Otherwise it's 0. However http.NewRequest(...) accepts io.Reader as body, so it will compile with everything implementing io.Reader interface like *os.File but it will not set Content-Length which is required in case of Azure Storage Account. When I switched to one of three given types I listed above it started working.
Related
I have requirement where it has to be done programmatically using POSTMAN REST API, where I have to upload a file/blob to Azure storage account and retrieve the unique URL of the specific file that I have uploaded, and I have to share that URL to third party so that they can view it on browser.
This is what I have done in the POSTMAN
Request:
PUT https://{storage-account-name}.blob.core.windows.net/{container-name}/{file-name}{SAS-token}
Headers:
x-ms-version: 2020-04-08
x-ms-blob-type: BlockBlob
x-mock-response-name: Upload Blob
Body: Attached a file from my local
Response:
I have received 200 code and file is successfully uploaded. However, in the response headers I don't see any URL or unique SAS token that I can share to my third-party client.
I have also tried adding se and sp to sas token, I got the below error
AuthenticationFailed
Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:65282b4e-401e-0050-2337-43ee90000000 Time:2023-02-18T01:20:28.3522177Z
**Signature did not match. String to sign used was r 2023-02-18T09:12:15Z /blob/storage-account-name/container-name/file-name.txt 2021-06-08 b **
Note: We don't want to generate SAS token manually from Azure portal for each file and construct the URL and share it to the client due to high traffic coming in. Once it is successful using POSTMAN. I have to implement the same in IBM App Connect enterprise, ESQL coding*
All the suggestions are much appreciated. Thank you in advance.
Retrieve the unique URL of the specific file that I have uploaded programmatically and share that URL with third party so that they can view it on browser.
In addition to the se and sp parameters, the following parameters are required to construct the correct SAS URL:
signed version (sv)
signed resource (sr)
signature
Your error message says that the signature does not match the rest of the URL. Signature a hash-based message authentication code (HMAC) that you compute over the string-to-sign and key by using the SHA256 algorithm, and then encode by using Base64 encoding
You can find how to construct the string-to-sign and signature depending on the version on this documentation page.
Postman has a built-in JavaScript library that can help you to calculate HMAC:
CryptoJS.HmacSHA1("string-to-sign", "key").toString()
I'm trying to use a service principal to access and list the contents of an Azure storage container using PowerShell. I've been able to get a token for the service principal and use that to access the storage account but when I run this command
$iwrParams = #{
'Uri' = 'https://myaccount.blob.core.windows.net/mycontainer?restype=container&comp=list'
'UseBasicParsing' = $true
'ContentType' = 'application/xml'
'Headers' = #{
'Authorization' = "Bearer $($token.access_token)"
'x-ms-version' = '2017-11-09'
}
}
Invoke-WebRequest #iwrParams
I get a return from this but the return xml content never shows the URL for each returned blob. If you look at the return at https://learn.microsoft.com/en-us/rest/api/storageservices/enumerating-blob-resources#list-blobs-and-snapshots it shows that for each blob it should return a Name, Url and various other properties. When I run the code above I get everything but the Url.
Now the really interesting thing is if I change the container access to anonymous public and run this code, I get the Url returned as expected.
$iwrParams = #{
'Uri' = 'https://myaccount.blob.core.windows.net/mycontainer?comp=list'
'UseBasicParsing' = $true
'ContentType' = 'application/xml'
}
Invoke-WebRequest #iwrParams
The issue honestly seems to be having to include restype=container when accessing with any sort of authentication.
My question is does anyone know a way to get the URL for each blob returned when not using anonymous access to list container contents?
I'd like to do this without resorting to the Az modules.
The reason you're not able to see the URL returned in the response is because of the storage REST API version used by your code (2017-11-09). Essentially the Blob URL property was removed from the response starting with REST API version 2013-08-15. From this link:
In version 2013-08-15 and newer, the EnumerationResults element
contains a ServiceEndpoint attribute specifying the blob endpoint, and
a ContainerName field specifying the name of the container. In
previous versions these two attributes were combined together in the
ContainerName field. Also in version 2013-08-15 and newer, the Url
element under Blob has been removed.
Regarding your comment about why you can see the URL property if you list blobs anonymously, this is happening because if no REST API version is specified in the request, Storage Service uses the oldest REST API version to process the request if default version has not been set. From this link:
If an anonymous request to a general-purpose storage account does not
specify the x-ms-version header, and the default version for the
service has not been set using Set Blob Service Properties, then the
service uses the earliest possible version to process the request.
Considering you would want to use Azure AD based authorization, the earliest version you will be able to use is 2017-11-09 thus it will not be possible to get the Blob URL returned in the response body.
One option would be to manually construct the Blob URL by using Blob Container URL and Blob name. Other option would be to use Shared Key Authorization instead of Azure AD authorization and specifying a version earlier than 2013-08-15 for x-ms-version request header in your requests. You will need to manually compute Authorization header value in this case using instructions provided here.
I have got details to send data to Azure SAS.
Storage Account Name: acountname123
Blob Container Name - containername123
SAS URI:
https://acountname123.blob.core.windows.net/containername123?sv=2019-12-12&si=xxinboundpolicy&sr=c&sig=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
I tried to call this from soapui however its giving error related to authorization header.
Can you pl help me with what I need to pass in authorization header.
I followed below links
https://learn.microsoft.com/en-us/rest/api/storageservices/put-blob
https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview?toc=%2fazure%2fstorage%2fblobs%2ftoc.json
Thanks
Yatan
I am trying to list the Queues/Topics in an Azure Service Bus using the REST API.
When I try to connect I just get back a blank feed saying "This is the list of publicly-listed services currently available".
I am using the RootManageSharedAccessKey in the portal (for dev only, I can create a more restricted key later) so it should have all the access rights that I need, I just can't seem to get it to return anything. This documentation seems to suggest that this will work, but there's no actual working examples, just theoretical responses.
I have tried doing a GET request with the signature in the URL like this:
https://myservicebusnamespace.servicebus.windows.net/$Resources/Queues;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=MYSHAREDACCESSKEY
I have also tried doing it like this:
https://myservicebusnamespace.servicebus.windows.net/$Resources
and then setting the Authorization header to
WRAP access_token="MYSHAREDACCESSKEY="
Both times I just get this back
<feed xmlns="http://www.w3.org/2005/Atom">
<title type="text">Publicly Listed Services</title>
<subtitle type="text">This is the list of publicly-listed services currently available.</subtitle>
<id>uuid:6a5d438d-1793-451b-be41-XXXXXXXXXXXX;id=XXXXXX</id>
<updated>2020-06-28T13:03:04Z</updated>
<generator>Service Bus 1.1</generator>
</feed>
If I change the url slightly to be:
https://myservicebusnamespace.servicebus.windows.net/$Resources/Queues/
I get a slightly different response back of:
<Error>
<Code>401</Code>
<Detail>claim is empty. TrackingId:c40a2bd2-490d-4b5b-adde-33bc89aa84ff_G36, SystemTracker:myservicebusnamespace.servicebus.windows.net:$Resources/Queues, Timestamp:2020-06-28T13:27:40</Detail>
</Error>
Which seems to suggest that I am not authorised, or I am missing something. If I add an acutual queue name to the end of that url, it goes back to the original response.
I believe there is another way to get this information by using subscription ids and pem keys... using the management urls (https://management.core.windows.net/{subscription ID}/services/ServiceBus/Namespaces/{Namespace}/Topics/)
but this should all be possible using the format above, I just can't figure out the exact format required.
EDIT/UPDATE: If I don't include my auth claim, the result is exactly the same, suggesting that it's not seeing my auth claim or it's invalid. However if I include it, and just make it the token, without the WRAP bit at the start, I get an exception saying
<Error>
<Code>401</Code>
<Detail>MalformedToken: Invalid authorization header: The request is missing WRAP authorization credentials. TrackingId:7be2d7f0-c165-4658-8bf1-ea104c43defc_G28, SystemTracker:NoSystemTracker, Timestamp:2020-06-28T13:33:09</Detail>
</Error>
So it's like it's reading it then ignoring it?
If you want to list queues or topics we can use Azure service bus service rest api or Azure Resource Manager Rest API. For more details, please refer to the following steps
Azure service bus service rest api
Generate SAS token. For more details, please refer to the document
For example, I use python to create sas token
import hmac
import time
import hashlib
import base64
import urllib
sb_name='bowmantest'
// your entity path such as $Resources/topics (list topics) $Resources/queues(list queues)
topic='$Resources/topics'
url=urllib.parse.quote_plus("https://{}.servicebus.windows.net/{}".format(sb_name,topic))
sas_value='' // your share access key
sas_name='RootManageSharedAccessKey' // your share access rule name
expiry = str(int(time.time() + 10000))
to_sign =(url + '\n' + expiry).encode('utf-8')
sas = sas_value.encode('utf-8')
signed_hmac_sha256 = hmac.HMAC(sas, to_sign, hashlib.sha256)
signature = urllib.parse.quote(base64.b64encode(signed_hmac_sha256.digest()))
auth_format = 'SharedAccessSignature sig={0}&se={1}&skn={2}&sr={3}'
auth=auth_format.format(signature,expiry,sas_name,url)
print(auth)
Call the rest API
1). list Queues
GET https://<namespace name>.servicebus.windows.net/$Resources/queues
Authorization <sas token>
2). List topics
GET https://<namespace name>.servicebus.windows.net/$Resources/topics
Authorization <sas token>
Azure Resource Manager Rest API
create a service principal and assign Azure RABC role to the sp(I use Azure CLI)
az login
#it will create a service principal and assign contributor role to the sp
az ad sp create-for-rbac -n "jonsp2"
Get Azure AD token
POST /{tenant}/oauth2/v2.0/token HTTP/1.1 //Line breaks for clarity
Host: login.microsoftonline.com
Content-Type: application/x-www-form-urlencoded
client_id=<app id>
&scope=https://management.azure.com/.default
&client_secret=<app password>
&grant_type=client_credentials
call the rest API
List Queues
GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.ServiceBus/namespaces/{namespaceName}/queues?api-version=2017-04-01
Authorization Bearer <AD token>
I am attempting to generate a url to a blob with a Shared Access Signature using BlobService.getBlobURL() in the Azure library on Node.js on my local machine. But when I try to retrieve the blob via the generated URL, I'm getting an Authentication Error saying that the "Signature did not match". Downloading the same blob from the Azure Management Portal works fine.
Below is the code I'm using to generate the URL:
process.env['AZURE_STORAGE_ACCOUNT'] = "[MY_ACCOUNT_NAME]";
process.env['AZURE_STORAGE_ACCESS_KEY'] = "[MY_ACCESS_KEY]";
var azure = require('azure');
var blobs = azure.createBlobService();
blobs.getBlobUrl('[CONTAINER_NAME]', "[BLOB_NAME]", { AccessPolicy: {
Start: Date.now(),
Expiry: azure.date.minutesFromNow(60),
Permissions: azure.Constants.BlobConstants.SharedAccessPermissions.READ
}});
The URL generated by this function is:
https://[MY_ACCOUNT_NAME].blob.core.windows.net:443/[CONTAINER_NAME]/
[ENCODED_BLOB_NAME]
?st=2013-10-28T18%3A34%3A23Z
&se=2013-10-28T19%3A34%3A23Z
&sp=r
&sr=b
&sv=2012-02-12
&sig=rLB%2FEOAWzijkkWcseju8TJLAxzeE5e3Pvq1i68i5Erc%3D
When I try to paste this URL into a browser, I get the following error message:
<Error>
<Code>AuthenticationFailed</Code>
<Message>
Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:9fe3d3ed-97f4-43d1-8c65-c95ce6b15a08 Time:2013-10-28T18:34:43.3015398Z
</Message>
<AuthenticationErrorDetail>
Signature did not match. String to sign used was r 2013-10-28T18:34:23Z 2013-10-28T19:34:23Z /[MY_ACCOUNT_NAME]/[CONTAINER_NAME]/[BLOB_NAME] 2012-02-12
</AuthenticationErrorDetail>
</Error>
Then I tried logging on to the Azure Management Portal, selecting the same blob, and downloading it. This worked. URL provided from the Management Portal was:
http://[MY_ACCOUNT_NAME].blob.core.windows.net/[CONTAINER_NAME]/
[ENCODED_BLOB_NAME]
?sv=2012-02-12
&st=2013-10-28T18%3A35%3A16Z
&se=2013-10-28T18%3A42%3A16Z
&sr=b
&sp=r
&sig=kcjV%2BkrNAaWOj%2F7NFwmHefXJEiEyu61U7mUTsw3pw7w%3D
It appears that as of the Azure Node.js Library version 0.7.16, there is a bug causing this behavior. When a Blob name includes spaces, BlobService.getBlobURL() fails to generate a correct signature. To resolve, upload a new blob without any spaces in its name, and call BlobService.getBlobURL() again with the name of the new blob. The URL produced this time will be valid. You can check in on this issue on Github.