How to upload files into Azure Blob Storage with Postman? - azure

I am trying to upload base64 string as image file to Azure Blob Storage. Using https://learn.microsoft.com/en-us/rest/api/storageservices/put-blob documentation tried to create blob.
Request Syntax:
PUT https://myaccount.blob.core.windows.net/mycontainer/myblockblob HTTP/1.1
Request Headers:
x-ms-version: 2015-02-21
x-ms-date: <date>
Content-Type: text/plain; charset=UTF-8
x-ms-blob-content-disposition: attachment; filename="fname.ext"
x-ms-blob-type: BlockBlob
x-ms-meta-m1: v1
x-ms-meta-m2: v2
Authorization: SharedKey myaccount:YhuFJjN4fAR8/AmBrqBz7MG2uFinQ4rkh4dscbj598g=
Content-Length: 11
Request Body:
hello world
I am getting response as below,
<?xml
version="1.0" encoding="utf-8"?>
<Error>
<Code>AuthenticationFailed</Code>
<Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:a5d32623-f01e-0040-4275-c1880d000000
Time:2020-11-23T08:45:49.6994297Z</Message>
<AuthenticationErrorDetail>The MAC signature found in the HTTP request 'YhuFJjN4fAR8/AmBrqBz7MG2uFinQ4rkh4dscbj598g=' is not the same as any computed signature. Server used following string to sign: 'PUT
11
text/plain; charset=UTF-8
x-ms-blob-content-disposition:attachment; filename="demo.txt"
x-ms-blob-type:BlockBlob
x-ms-date:Mon, 23 Nov 2020 13:08:11 GMT
x-ms-encryption-key:YhuFJjN4fAR8/AmBrqBz7MG2uFinQ4rkh4dscbj598g=
x-ms-meta-m1:v1
x-ms-meta-m2:v2
x-ms-version:2015-02-21
/<myaccount>/<mycontainer>/<myblob>'.</AuthenticationErrorDetail>
</Error>
How to resolve this issue?

A simple way to upload a blob is to use the sas token.
Nav to azure portal -> your storage account -> Shared access signature, then select the following options in the screenshot -> then click the Generate SAS and connection string button. The screenshot is as below:
Then copy the SAS token, and append it to the url. Then the new url looks like this: https://myaccount.blob.core.windows.net/mycontainer/myblockblob?sv=2019-12-12&ss=b&srt=coxxxxx
Next, in the postman, paste the new url. And in the Headers, you can remove Authorization field.
The test result is as below:

#sathishKumar
If you look closely in this article Authorize with Shared Key
The syntax is as below :
Authorization="[SharedKey|SharedKeyLite] <AccountName>:<Signature>"
It is the signature that is passed along and not the Account key.
Signature is a Hash-based Message Authentication Code (HMAC) constructed from the request and computed by using the SHA256 algorithm, and then encoded by using Base64 encoding.
There are detailed steps how to construct the same mentioned on the above document.
Also, came across the post which talks about a PowerShell script which creates an Signature string through the Powershell that could be useful for you.
Sample Powershell Script
C# Implementation :
internal static AuthenticationHeaderValue GetAuthorizationHeader(
string storageAccountName, string storageAccountKey, DateTime now,
HttpRequestMessage httpRequestMessage, string ifMatch = "", string md5 = "")
{
// This is the raw representation of the message signature.
HttpMethod method = httpRequestMessage.Method;
String MessageSignature = String.Format("{0}\n\n\n{1}\n{5}\n\n\n\n{2}\n\n\n\n{3}{4}",
method.ToString(),
(method == HttpMethod.Get || method == HttpMethod.Head) ? String.Empty
: httpRequestMessage.Content.Headers.ContentLength.ToString(),
ifMatch,
GetCanonicalizedHeaders(httpRequestMessage),
GetCanonicalizedResource(httpRequestMessage.RequestUri, storageAccountName),
md5);
// Now turn it into a byte array.
byte[] SignatureBytes = Encoding.UTF8.GetBytes(MessageSignature);
// Create the HMACSHA256 version of the storage key.
HMACSHA256 SHA256 = new HMACSHA256(Convert.FromBase64String(storageAccountKey));
// Compute the hash of the SignatureBytes and convert it to a base64 string.
string signature = Convert.ToBase64String(SHA256.ComputeHash(SignatureBytes));
// This is the actual header that will be added to the list of request headers.
AuthenticationHeaderValue authHV = new AuthenticationHeaderValue("SharedKey",
storageAccountName + ":" + signature);
return authHV;
}

Related

Authentication errors using Azure blob storage and Azure CDN

I'm having intermittent 403 errors trying to access a blob storage, via Azure CDN, with the symmetric access key. It seems that sometimes there's a header added for "Range", in the format of "bytes=xxx". The full error message is below:
{'Date': 'Mon, 12 Dec 2022 13:07:40 GMT', 'Content-Type': 'application/xml', 'Content-Length': '697', 'Connection': 'keep-alive', 'x-ms-request-id': '3f89c2c1-e01e-0050-132a-0eeb42000000', 'x-ms-error-code': 'AuthenticationFailed', 'x-azure-ref': '20221212T130740Z-6rfkrgx8qt0shbtz3x46rwnhrn0000000630000000002ayd', 'X-Cache': 'TCP_MISS'}
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:3f89c2c1-e01e-0050-132a-0eeb42000000
Time:2022-12-12T13:07:40.7638741Z</Message><AuthenticationErrorDetail>The MAC signature found in the HTTP request 'xxxxxx=' is not the same as any computed signature. Server used following string to sign: 'GET
bytes=0-8388607
x-ms-date:Mon, 12 Dec 2022 13:07:36 GMT
x-ms-version:2020-04-08
/deviceimage2zgjscikl7kny/images/data-prod-1.1.packer'.</AuthenticationErrorDetail></Error>
I was able to reproduce the error by generating the MAC signature in Python, but I saw it originally using the Go SDK and az CLI.
We added a rule at the CDN to Bypass caching, and it seems to have improved the situation (problem happens less frequently), but we are still seeing it on occasion.
Has anyone else experienced this? And is there a workaround?
Trying to access a blob storage with an access key, via Azure CDN
I tried in my environment and got below results:
Initially, I got a same when I tried to access blob storage with CDN using Postman.
Postman:
The above error states that signature and date is incorrect. So, we can't pass directly storage access key. You need to create a signature string that represents the given request, sign the string with the HMAC-SHA256 algorithm (using your storage key to sign), and encode the result in base 64.
For creating signature, I used below .NET code:
using System.Globalization;
using System.Net;
using System.Security.Cryptography;
class Program
{
static void Main(string[] args)
{
ListBlobs();
Console.WriteLine("done");
Console.ReadLine();
}
static void ListBlobs()
{
string Account = "venkat123";
string Key = "<Storage account key>";
string Container = "test";
string apiversion = "2021-06-08";
DateTime dt = DateTime.UtcNow;
string StringToSign = String.Format("GET\n"
+ "\n" // content encoding
+ "\n" // content language
+ "\n" // content length
+ "\n" // content md5
+ "\n" // content type
+ "\n" // date
+ "\n" // if modified since
+ "\n" // if match
+ "\n" // if none match
+ "\n" // if unmodified since
+ "\n" // range
+ "x-ms-date:" + dt.ToString("R") + "\nx-ms-version:" + apiversion + "\n" // headers
+ "/{0}/{1}\ncomp:list\nrestype:container", Account, Container);
string auth = SignThis(StringToSign, Key, Account);
Console.WriteLine($"the date is: {dt.ToString("R")}");
Console.WriteLine($"the auth token is: {auth}");
Console.WriteLine("*********");
string method = "GET";
string urlPath = string.Format("https://{0}.blob.core.windows.net/{1}?restype=container&comp=list", Account, Container);
Uri uri = new Uri(urlPath);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Method = method;
request.Headers.Add("x-ms-date", dt.ToString("R"));
request.Headers.Add("x-ms-version", apiversion);
request.Headers.Add("Authorization", auth);
Console.WriteLine("***list all the blobs in the specified container, in xml format***");
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
Console.WriteLine(reader.ReadToEnd());
}
}
}
private static String SignThis(String StringToSign, string Key, string Account)
{
String signature = string.Empty;
byte[] unicodeKey = Convert.FromBase64String(Key);
using (HMACSHA256 hmacSha256 = new HMACSHA256(unicodeKey))
{
Byte[] dataToHmac = System.Text.Encoding.UTF8.GetBytes(StringToSign);
signature = Convert.ToBase64String(hmacSha256.ComputeHash(dataToHmac));
}
String authorizationHeader = String.Format(
CultureInfo.InvariantCulture,
"{0} {1}:{2}",
"SharedKey",
Account,
signature);
return authorizationHeader;
}
}
Console:
Above executed code, date and signature which I copied and used in postman, and it worked successfully.
Postman:

set messagettl of a message in Azure Queue using python

I'm trying to post a message to azure queue service using python3 by making a POST request and specifying messagettl to -1 which indicates the message does not expire. In the doc https://learn.microsoft.com/en-us/rest/api/storageservices/put-message I have to specify the Authorization key and Date which indicates the time at which the response was initiated (both parameters are required), and the body must be an XML, here what I did:
url = "https://MyStorageAccountName.queue.core.windows.net/MyQueueName?messagettl=-1"
xml = """<?xml version='1.0' encoding='utf-8'?>
<QueueMessage>
<MessageText>First message</MessageText>
</QueueMessage> """
headers = {'Content-Type': 'application/xml',
'Authorization' : 'SharedKey MyStorageAccountName:MyKey1....==',
'Date' : str(datetime.utcnow())}
print(requests.post(url, data=xml, headers=headers).text)
And the response is an error:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>AuthenticationFailed</Code>
<Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:44d1fd4c-c003-001d-215...000
Time:2020-11-20T15:39:10.9730253Z</Message>
<AuthenticationErrorDetail>The Date header in the request is incorrect.</AuthenticationErrorDetail>
</Error>
which piece of the puzzle I am missing?
UPDATE:
In headers I fixed the issue by replacing str(datetime.utcnow()) with format_date_time(mktime(datetime.now().timetuple())) and fixed the related date error, but I have a new error and don't know how to sign my key:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>AuthenticationFailed</Code>
<Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:359305a5-a003-0034...
Time:2020-11-20T15:59:12.4611176Z</Message>
<AuthenticationErrorDetail>The MAC signature found in the HTTP request 'HACSNj/4PwH...MyKey...YJQ==' is not the same as any computed signature. Server used following string to sign: 'POST
application/xml
Fri, 20 Nov 2020 15:59:09 GMT
/MystorageAccount/MyQueueName'.</AuthenticationErrorDetail>
</Error>
I think using python SDK to do this is much easier, just try the code below:
from azure.storage.queue import QueueClient
connectionString = "<storage account connection string>"
queueName = "<queue name>"
queueClient = QueueClient.from_connection_string(connectionString, queueName)
queueClient.send_message(content = 'hello sdk', time_to_live=-1)
Result:
For info about python queue client sdk, just refer to this doc.

How to send a query to Azure Table REST API with Postman

I'm trying to make a GET request to Azure Table REST API with Postman.
I can make a working request with a C# program I found, but when I try to copy the same information into the Postman request it return the followign error:
Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
With the C# program I generate my UTC time and my Authorization code.
The program will give me the following output:
x-ms-date: Fri, 01 Nov 2019 10:13:26 GMT
Authorization: SharedKeyLite username:e4IREMOVEDSOMELETTERST4Ag=
Request URI: https://username.table.core.windows.net/MainTable(PartitionKey='akey',RowKey='130')
The generated output works in the C# program, because when I use:
result = await Client.GetAsync(requestUri);
The result will give me the information of (akey, 130).
When I pass them into postman it will still give me an error.
I do update the date in postman whenever I make a new authorized string.
My postman setup is as follows:
I eventually want to make this request with the ESP32, so it might be a bit unrelated, but the ESP is giving me the same error. Any tips on setting the headers correct either for Postman or the ESP are appreciated.
to make this work first create two variables in your environment :
{{utcDate}}
{{authToken}}
Then create a new Get request and setup your headers like this :
x-ms-version 2015-12-11
x-ms-date {{utcDate}}
Authorization SharedKey resourceName:{{authToken}}
DataServiceVersion 3.0;NetFx
MaxDataServiceVersion 3.0;NetFx
Accept application/json;odata=nometadata
Finally, define a Pre-request Script :
var now = new Date().toUTCString();
pm.environment.set("utcDate", now);
var hcar = "/resourceName/TableName";
var verb = request.method;
var cntMd5 = "";
var cntType = "";
var mKey="<Your service key goes here>";
var text = verb + "\n" + (cntMd5 || "") + "\n" + (cntType || "") + "\n" + now + "\n" + hcar;
var key = CryptoJS.enc.Base64.parse(mKey);
var signature = CryptoJS.HmacSHA256(text, key);
var base64Bits = CryptoJS.enc.Base64.stringify(signature);
pm.environment.set("authToken", base64Bits);
The reason for the variables is, authToken because you need a place holder to store the calculated token, utcDate because the same date in your header must be used to calculate your token.
I found that the problem was within Postman itself.
There has been an ongoing issue with the automatic URL encoding.
When I went directly to the MainTable the code of Mauricio worked.

microsoft azure table authentication stringtosign error

I am having a problem with stringtosign authentication for azure table pagination query.
This is the current stringtosign im using :
GET\n\n\nFri, 05 Sep 2014 03:57:11 GMT\n/mystorageaccount/mytablename\nNextPartitionKey:1!20!UmFjZSBNZW1iZXJfNA--\nNextRowKey:1!12!TmFtZV85ODE-
Is there anything wrong with this stringtosign authentication?
The rest of the Headers are exactly the same as Fiddle.
Example
GET /mytablename?NextPartitionKey=1%2120%21UmFjZSBNZW1iZXJfNA--&NextRowKey=1%2112%21TmFtZV85ODE- HTTP/1.1
Host: mystorageaccount.table.core.windows.net
x-ms-version: 2014-02-14
x-ms-date: Fri, 05 Sep 2014 05:49:19 GMT
Authorization: SharedKey mystorageaccount:GD2w4pqsllzIOixNF/AfFqLkZhYzLpjK67a8OI7j6Go=
Accept: application/atom+xml
Accept-Charset: UTF-8
DataServiceVersion: 3.0;NetFx
MaxDataServiceVersion: 3.0;NetFx
I have read through both
http://msdn.microsoft.com/library/azure/dd179428.aspx
http://msdn.microsoft.com/en-us/library/azure/dd135718.aspx
Hi Gaurav Mantri,
It still did not work. I'll paste the request, my stringtosign and the response below:
GET /mytablename?NextPartitionKey=1%2120%21UmFjZSBNZW1iZXJfNA--&NextRowKey=1%2112%21TmFtZV85ODE- HTTP/1.1
Host: mystorageaccount.table.core.windows.net
x-ms-version: 2014-02-14
x-ms-date: Fri, 05 Sep 2014 07:05:12 GMT
Authorization: SharedKey mystorageaccount:HSYfO1Baadqcd4bQO5Q6uN1hrr2aXtLcQbFPkWgIXuw=
Accept: application/atom+xml
Accept-Charset: UTF-8
DataServiceVersion: 3.0;NetFx
MaxDataServiceVersion: 3.0;NetFx
String to sign:
GET\n\n\nFri, 05 Sep 2014 07:05:12 GMT\n/mystorageaccount/mytablename\nnextpartitionkey:1!20!UmFjZSBNZW1iZXJfNA--\nnextrowkey:1!12!TmFtZV85ODE-
Response:
<?xml version=\"1.0\" encoding=\"utf-8\"?><m:error xmlns:m=\"http://schemas.microsoft.com/ado/2007/08/dataservices/metadata\"><m:code>AuthenticationFailed</m:code><m:message xml:lang=\"en-US\">Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:37272f11-0002-0014-5aa7-f7dd1c000000
Time:2014-09-05T07:05:09.5720897Z</m:message></m:error>
I had an opportunity to actually write the code and try it out. Basically when creating CanonicalizedResource string for table resources, you need not include the query string parameters other than comp querystring parameter. Essentially this is what you would need to follow from the documentation (http://msdn.microsoft.com/library/azure/dd179428.aspx):
2009-09-19 Shared Key Lite and Table Service Format
This format supports Shared Key and Shared Key Lite for all versions
of the Table service, and Shared Key Lite for the 2009-09-19 version
of the Blob and Queue services and 2014-02-14 of the File service.
This format is identical to that used with previous versions of the
storage services. Construct the CanonicalizedResource string in this
format as follows:
Beginning with an empty string (""), append a forward slash (/), followed by the name of the account that owns the resource being
accessed.
Append the resource's encoded URI path. If the request URI addresses a component of the resource, append the appropriate query
string. The query string should include the question mark and the comp
parameter (for example, ?comp=metadata). No other parameters should be
included on the query string.
Once you do that, your code should run just fine. Here's the sample code I wrote:
static void QueryTable()
{
var requestMethod = "GET";
var storageServiceVersion = "2014-02-14";
var date = DateTime.UtcNow.ToString("R");
var canonicalizedResource = string.Format("/{0}/{1}", StorageAccount, TableName);
var stringToSign = string.Format("{0}\n\n\n{1}\n{2}", requestMethod, date, canonicalizedResource);
var authorizationHeader = GetAuthorizationHeader(stringToSign);
using (var httpClient = new HttpClient())
{
httpClient.BaseAddress = new Uri(TableEndpoint);
httpClient.DefaultRequestHeaders.Clear();
httpClient.DefaultRequestHeaders.Add("x-ms-date", date);
httpClient.DefaultRequestHeaders.Add("x-ms-version", storageServiceVersion);
httpClient.DefaultRequestHeaders.Add("Authorization", authorizationHeader);
var result = httpClient.GetAsync(TableName + "?NextPartitionKey=1!48!VXwzMzg0MDAzOWYzMjQ0ZDgxOWZjZmM5M2EyMzNkM2IxOA--&NextRowKey=1!0!");
result.Wait();
}
}
static string GetAuthorizationHeader(string canonicalizedString)
{
var signature = string.Empty;
using (var hash = new HMACSHA256(Convert.FromBase64String(StorageAccountKey)))
{
var data = Encoding.UTF8.GetBytes(canonicalizedString);
signature = Convert.ToBase64String(hash.ComputeHash(data));
}
return string.Format(CultureInfo.InvariantCulture, "{0} {1}:{2}", "SharedKey", StorageAccount, signature);
}
Based on the documentation here: http://msdn.microsoft.com/library/azure/dd179428.aspx (2009-09-19 Shared Key Format Section, point #4), you would need to convert all query parameters to lowercase. So your canonicalized resource string should be:
GET\n\n\nFri, 05 Sep 2014 03:57:11 GMT\n/mystorageaccount/mytablename\nnextpartitionkey:1!20!UmFjZSBNZW1iZXJfNA--\nnextrowkey:1!12!TmFtZV85ODE-
Give it a try. That should take care of the problem.

Mandrill Webhooks - Security

For security purposes, I try to allow only Mandrill's IP(s) to access these urls.
Does anyone know them?
Mandrill's signature is located in the HTTP response header: Authenticating-webhook-requests
In the request header find: X-Mandrill-Signature. This is a base64 of the hashcode, signed using web-hook key. This key is secret to your webhook only.
We have a range of IPs used for webhooks, but they can (and likely will) change or have new ones added as we scale. An alternative would be to add a query string to the webhook URL you add in Mandrill, and then check for that query string when a POST comes in so you can verify it's coming from Mandrill.
Just replace the constants and use this function:
<?php
function generateSignature($post)
{
$signed_data = WEB_HOOK_URL;
ksort($post);
foreach ($post as $key => $value) {
$signed_data .= $key;
$signed_data .= $value;
}
return base64_encode(hash_hmac('sha1', $signed_data, WEB_HOOK_AUTH_KEY, true));
}
//---
if (generateSignature($_POST) != $_SERVER['HTTP_X_MANDRILL_SIGNATURE']) {
//Invalid
}
?>
As described in mandrill's docs, they provide a signature to check if the request really came from them. to build the request there's a few steps:
start with the exact url of your webhook (mind slashes and params)
sort the post variables by key (in case of mandrill, you'll only have one post parameter: mandrill_events)
add key and value to the url, without any delimiter
hmac the url with your secret key (you can get the key from the web-interface) and base64 it.
compare the result with the X-Mandrill-Signature header
here's a sample implementation in python:
import hmac, hashlib
def check_mailchimp_signature(params, url, key):
signature = hmac.new(key, url, hashlib.sha1)
for key in sorted(params):
signature.update(key)
signature.update(params[key])
return signature.digest().encode("base64").rstrip("\n")
205.201.136.0/16
I have just whitelisted them in my server's firewall.
We don't need to white list the Ip they are using. Instead of that they have provided their own way to authenticate the webhook request.
When you are creating the mandrill webhook it will generate the key. It will come under the response we are getting to our post URL which is provided in the webhook.
public async Task<IHttpActionResult> MandrillEmailWebhookResponse()
{
string mandrillEvents = HttpContext.Current.Request.Form["mandrill_events"].Replace("mandrill_events=", "");
// validate the request is coming from mandrill API
string url = ConfigurationManager.AppSettings["mandrillWebhookUrl"];
string MandrillKey = ConfigurationManager.AppSettings["mandrillWebHookKey"];
url += "mandrill_events";
url += mandrillEvents;
byte[] byteKey = System.Text.Encoding.ASCII.GetBytes(MandrillKey);
byte[] byteValue = System.Text.Encoding.ASCII.GetBytes(url);
HMACSHA1 myhmacsha1 = new HMACSHA1(byteKey);
byte[] hashValue = myhmacsha1.ComputeHash(byteValue);
string generatedSignature = Convert.ToBase64String(hashValue);
string mandrillSignature = HttpContext.Current.Request.Headers["X-Mandrill-Signature"].ToString();
if (generatedSignature == mandrillSignature)
{
// validation = "Validation successful";
// do the updating using the response data
}
}

Resources