Getting Response 403 When signing an Amazon Request - python-3.x

I am trying to send an email through amazon SES without the SDK so I can send emails asynchronously in Python. I am using amazon's v4 signing method on their site here: https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html, but I'm not having any luck sending emails without the SDK. The output is:
RESPONSE++++++++++++++++++++++++++++++++++++
Response code: 403
<ErrorResponse xmlns="http://ses.amazonaws.com/doc/2010-12-01/">
<Error>
<Type>Sender</Type>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.</Message>
</Error>
<RequestId>a19a5fa1-3228-11e9-b2bc-ddb6d8a1cb1c</RequestId>
</ErrorResponse>
Process finished with exit code 0
Here is the block of code generating that response:
import datetime
import hashlib
import hmac
import urllib.parse
import requests
method = 'GET'
service = 'ses'
host = 'email.us-east-1.amazonaws.com'
region = 'us-east-1'
endpoint = 'https://email.us-east-1.amazonaws.com/'
def sign(key, msg):
return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()
def getSignatureKey(key, dateStamp, regionName, serviceName):
kDate = sign(('AWS4' + key).encode('utf-8'), dateStamp)
kRegion = sign(kDate, regionName)
kService = sign(kRegion, serviceName)
kSigning = sign(kService, 'aws4_request')
return kSigning
access_key = '<my access_key here>'
secret_key = '<my secret_key here>'
my_email = 'my email here'
t = datetime.datetime.utcnow()
amz_date = t.strftime('%Y%m%dT%H%M%SZ')
datestamp = t.strftime('%Y%m%d')
canonical_uri = '/'
canonical_headers = 'host:' + host + '\n'
signed_headers = 'host'
algorithm = 'AWS4-HMAC-SHA256'
credential_scope = datestamp + '/' + region + '/' + service + '/' + 'aws4_request'
canonical_querystring = '''Action=SendEmail
&Source=%s%40gmail.com
&Destination.ToAddresses.member.1=%s%40gmail.com
&Message.Subject.Data=This%20is%20the%20subject%20line.
&Message.Body.Text.Data=Hello.%20I%20hope%20you%20are%20having%20a%20good%20day''' % (my_email, my_email)
canonical_querystring += '&X-Amz-Algorithm=AWS4-HMAC-SHA256'
canonical_querystring += '&X-Amz-Credential=' + urllib.parse.quote_plus(access_key + '/' + credential_scope)
canonical_querystring += '&X-Amz-Date=' + amz_date
canonical_querystring += '&X-Amz-Expires=30'
canonical_querystring += '&X-Amz-SignedHeaders=' + signed_headers
payload_hash = hashlib.sha256(('').encode('utf-8')).hexdigest()
canonical_request = method + '\n' + canonical_uri + '\n' + canonical_querystring + '\n' + canonical_headers + '\n' + signed_headers + '\n' + payload_hash
string_to_sign = algorithm + '\n' + amz_date + '\n' + credential_scope + '\n' + hashlib.sha256(canonical_request.encode('utf-8')).hexdigest()
signing_key = getSignatureKey(secret_key, datestamp, region, service)
signature = hmac.new(signing_key, (string_to_sign).encode("utf-8"), hashlib.sha256).hexdigest()
canonical_querystring += '&X-Amz-Signature=' + signature
request_url = endpoint + "?" + canonical_querystring
print('\nBEGIN REQUEST++++++++++++++++++++++++++++++++++++')
print('Request URL = ' + request_url)
r = requests.get(request_url)
print('\nRESPONSE++++++++++++++++++++++++++++++++++++')
print('Response code: %d\n' % r.status_code)
print(r.text)
This is basically Amazon's code for generating a signature copied and pasted from their docs. Does anyone know what I'm doing wrong in signing my requests to amazon?
Edit* I changed the canonical_querystring to be alphabetical like so:
canonical_querystring = '''Action=SendEmail
&Destination.ToAddresses.member.1={}%40gmail.com
&Message.Body.Text.Data=Hello.%20I%20hope%20you%20are%20having%20a%20good%20day
&Message.Subject.Data=This%20is%20the%20subject%20line.
&Source={}%40gmail.com'''.format(my_email, my_email)
It is still giving me the same error though. Everything else in the query string is alphabetized, or "canonically ordered".

Your canonical query string is not canonical.
You have Action... Source... Destination... Message but all of the parameters need to be lexically ordered.
Step 3: Create the canonical query string.
[...]
The parameters must be sorted by name.
This ordering is the reason why the value is called canonical. It's necessary because the relative positions of query string parameters aren't necessarily guaranteed. The parameters don't need to be sorted in the actual query string accompanying the request, but they do need to be sorted here, for signing.
Since a given request can have only one possible valid signature, the parameters are sorted before signing to remove the ambiguity that might otherwise arise if a user agent or proxy rearranged the query parameters or ordered them arbitrarily when building the URL (as might be expected if, for example, the parameters are passed to the UA as an unordered hash/dictionary structure).

Related

Hotbit REST API's PUT_LIMIT function gives error RET_SIGN_ERROR

So basically the hotbit's documentation you can find on
https://github.com/hotbitex/hotbit.io-api-docs/blob/master/readme_en.md
https://github.com/hotbitex/hotbit.io-api-docs/blob/master/rest_api_en.md#orderput_limit
a point need to be remember in hashing SIGN parameter is :
sort the strings that require to be signed according to the parameter names(first compare the first letter of all parameter names and sort them based on alphabetical order; in case that the first letter of more than one parameters is the same, sort these parameters based on the second letter of their names according to alphabetical order, and so on)
parameters that need to be hashed according to alphabets are api_key, secret_key, market, isfee, market, price, amount
import hashlib, requests
url = "https://api.hotbit.io/v2/p2/order.put_limit"
sign_string = "amount=" + str(amount) + "&api_key=" + str(api_key) + "&isfee=0&market=" + str(market) + "&price=" + str(price) + "&secret_key=" + str(secret_key) + "&side=" + str(side)
sign = hashlib.md5(sign_string.encode('utf-8')).hexdigest()
sign = sign.upper()
body = {
"api_key" : str(api_key),
"amount" : amount,
"isfee" : 0 ,
"market" : str(market),
"price" : price,
"side" : side,
"sign": str(sign) }
params = "?amount=" + str(amount) + "&api_key=" + str(api_key) + "&isfee=0&market=" + str(market) + "&price=" + str(price) + "&side=" + str(side) + "&sign=" + str(sign)
print (params)
response = requests.request('POST', url, data = body )
print (response.text)
and here is the response we get from Hotbit API
{"error":{"code":6,"message":"RET_SIGN_ERROR"},"result":null,"id":0}
Please help me solving this problem..!
Thanks in advance.!
Ohk so i got answer what was wrong in it you can compare your code , (sign_string was not configured properly in alphabetical order)
ORDER_PUT_LIMIT = "https://api.hotbit.io/v2/p2/order.put_limit"
def sell(self, amount=0, price=0, market=settings.MARKET, isfee=settings.ISFEE):
"""
Response:
{
"error": null,
"result":
{
"id":8688803, #order-ID
"market":"ETHBTC",
"source":"web", #The source identification of data request
"type":1, #Type of order pladement 1-limit order
"side":2, #Identification of buyers and sellers 1-Seller,2-buyer
"user":15731,
"ctime":1526971722.164765, #Time of order establishment(second)
"mtime":1526971722.164765, #Time of order update(second)
"price":"0.080003",
"amount":"0.4",
"taker_fee":"0.0025",
"maker_fee":"0",
"left":"0.4",
"deal_stock":"0",
"deal_money":"0",
"deal_fee":"0",
"status":0 , #Sign of order status when 0x8 is true, it means the current order is cancelled, when 0x80 is true, it means that the current order is deducted by deductable tokens "fee_stock":"HTB", #Name of deductable token
"alt_fee":"0.5", #The discount of deductable tokens
"deal_fee_alt":"0.123" #Amount deducted
},
"id": 1521169460
}
"""
side = 1 # 1 for sell and 2 for buy
sign_string = "amount=" + str(amount) + "&api_key=" + str(settings.API_KEY) + "&isfee=0&market=" + str(market) + "&price=" + str(price) + "&side=" + str(side) + "&secret_key=" + str(settings.SECRET_KEY)
sign = hashlib.md5(sign_string.encode('utf-8')).hexdigest()
sign = sign.upper()
body = {
"api_key" : str(settings.API_KEY),
"amount" : amount,
"isfee" : 0 ,
"market" : str(market),
"price" : price,
"side" : side,
"sign": str(sign) }
response = requests.post(ORDER_PUT_LIMIT, data=body, headers=HEADERS).json()
print(response)
return response
and my problem was solved..
The api_key and private key are saved in settings.py file.

How to handle ezproxy authorization to download data through API?

I have a token to have an access to download large files from the comtrade. The original webpage is http://comtrade.un.org/ however I have a premium access through my university library subscription. So,if I want to use the premium features the website automatically redirects me to this page and after pressing login button the URL is https://ezproxy.nu.edu.kz:5588/data/dev/portal/. I am trying to send request and download files with API(using requests). I am getting response from http://comtrade.un.org/ but in order to download I need to use https://ezproxy.nu.edu.kz:5588/data/dev/portal/. and when I tried to download urllib.error.HTTPError: HTTP Error 401: Unauthorized this error message appeared. How can I handle this problem?
px = 'px=HS&' #classification
freq = 'freq=A&' #annual
type = 'type=C&' #commodity
auth = 'https://comtrade.un.org/api/getUserInfo?token=ZF5TSW8giRQMFHuPmS5JwQLZ5FB%2BNO0NCcjxFQUJADrLzCRDCkG5F0ZPnZTYQWO3MPgj96gZNF7Z9iN8BwscUMYBbXuDVYVDvsTAVNzAJ6FNC2dnN7gtB1rt9qJShAO467zBegHTLwvmlRIBSpjjwg%3D%3D'
with open('reporterAreas.json') as json_file:
data = json.load(json_file)
ls = data['results']
list_year = [*range(2011, 2021,1)]
for years in list_year:
print(years)
ps = 'ps='+ str(years) + '&'
for item in ls:
r = item['id'] #report_country_id
report_country_txt = item['text']
if r == 'all':
req_url = 'r=' + r + '&' + px + ps + type + freq + token
request = url + req_url
response = requests.get(request)
if response.status_code == 200:
print("Response is OK!")
data = response.json()[0]
download_url = dwld_url + data['downloadUri']
print(download_url)
filename = str(years) + '_' + report_country_txt + '.zip'
urllib.request.urlretrieve(url, filename)
I'm not sure if Ezproxy provides an API or SDK way to authenticate a request but i don't think.
What you could do is to provide the Ezproxy session to your request and with that, you request will not be treated as unauthorized because you're passing a valid session and therefore your request will be treated as a valid one.
Notice that you can retrieve your Ezproxy session id from your cookies.
And finally, you have to make your request against the starting point url
Otherwise, you can use selenium to fill automatically the login form and retrieve the Ezproxy session id to pass it to the requests.
I hope this could help you !

cant get encoding to work in python 3

I have created a program and from what I understand from the error shown below and from other posts on Stack, I need to encode the object before it can be hashed.
I have tried several ways to do this but still keep getting the same error message. provided below is my code and also a list of changes I have tried.
I understand what needs to be done but I guess I'm putting the code in the wrong place or the syntax is wrong as what I am trying isn't working.
any help is much appreciated.
Error Message
ha1 = hashlib.md5(user + ':' + realm + ':' + password.strip()).hexdigest()
TypeError: Unicode-objects must be encoded before hashing
Code
import sys
import requests
import hashlib
realm = "Pentester Academy"
lines = [line.rstrip('\n') for line in open('wordl2.txt')]
print (lines)
for user in ['nick', 'admin']:
get_response = requests.get("http://pentesteracademylab.appspot.com/lab/webapp/digest2/1")
test_creds = get_response
print (test_creds)
for password in lines:
# not the correct way but works for this challenge
snounce = test_creds.headers.get('www-authenticate').split('"')
uri = "/lab/webapp/digest2/1"
# create the HTTP Digest
ha1 = hashlib.md5(user + ':' + realm + ':' + password.strip()).hexdigest()
ha2 = hashlib.md5("GET:" + uri).hexdigest()
response = hashlib.md5(ha1 + ':' + snounce + ':' + ha2).hexdigest()
header_string = 'Digest username="%s", realm="%s", nonce="%s", uri="%s", response="%s"' % (user, realm, snounce, uri, response)
headers = { 'Authorization' : header_string }
test_creds = requests.get("http://pentesteracademylab.appspot.com/lab/webapp/digest2/1", headers = headers)
if test_creds.status_code == 200:
print ("CRACKED: %s:%s" % (user,password))
break
elif test_creds.status_code == 401:
print ("FAILED: %s:%s" % (user,password))
else:
print ("unexpected Status code: %d " % test_creds.status_code)
Attempted Changes
password.encode(utf-8)
----------------------
hashlib.md5().update(password.encode(lines.encoding))
---------------
lines = [line.rstrip('\n') for line in open('wordl2.txt', "rb")]
I have managed to solve my own problem using the line
pass2 = str.encode(password)
just inside the password for loop

Azure Put Blob API returns with a non-matching size of file in canonicalized Header

I am trying to upload a blob (pdf) file from my laptop to a container in Azure storage account. I found it to be working but with one glitch.
I am calculating the file size using:
f_info = os.stat(file_path)
file_size = (f_info.st_size) # returns - 19337
Then I insert this value in below canonicalized header:
ch = "PUT\n\n\n"+str(file_size)+"\n\napplication/pdf\n\n\n\n\n\n\nx-ms-blob-type:BlockBlob" + "\nx-ms-date:" + date + "\nx-ms-version:" + version + "\n"
and send the PUT request to PUT Blob API, however, it returns an error saying, "Authentication failed because the server used below below string to calculate the signature"
\'PUT\n\n\n19497\n\napplication/pdf\n\n\n\n\n\n\nx-ms-blob-type:BlockBlob\nx-ms-date:[date]\nx-ms-version:[API version]
Looking at this string it obvious that authentication failed because file size which azure calculated returns a different value! I don't understand how its calculating this value of file size?!?!
FYI: If I replace 19337 with 19497 in canonicalized string and re run. It works!
Any suggestion on where I am making mistakes?
Below is the code:
storage_AccountName = '<storage account name>'
storage_ContainerName = "<container_name>"
storageKey='<key>'
fd = "C:\\<path>\\<to>\\<file_to_upload>.pdf"
URI = 'https://' + storageAccountName + '.blob.core.windows.net/<storage_ContainerName >/<blob_file_name.pdf>
version = '2017-07-29'
date = datetime.datetime.utcnow().strftime("%a, %d %b %Y %H:%M:%S GMT")
if os.path.isfile(fd):
file_info = os.stat(fd)
file_size = (file_info.st_size)
ch = "PUT\n\n\n"+str(file_size)+"\n\napplication/pdf\n\n\n\n\n\n\nx-ms-blob-type:BlockBlob" + "\nx-ms-date:" + date + "\nx-ms-version:" + version + "\n"
cr = "/<storage_AccountName>/<storage_Containername>/<blob_file_name.pdf>"
canonicalizedString = ch + cr
storage_account_key = base64.b64decode(storageKey)
byte_canonicalizedString=canonicalizedString.encode('utf-8')
signature = base64.b64encode(hmac.new(key=storage_account_key, msg=byte_canonicalizedString, digestmod=hashlib.sha256).digest())
header = {
'x-ms-blob-type': "BlockBlob",
'x-ms-date': date,
'x-ms-version': version,
'Authorization': 'SharedKey ' + storageAccountName + ':' + signature.decode('utf-8'),
#'Content-Length': str(19497), # works
'Content-Length': str(file_size), # doesn't work
'Content-Type': "application/pdf"}
files = {'file': open(fd, 'rb')}
result = requests.put(url = URI, headers = header, files = files)
print (result.content)
As mentioned in the comments, the reason you're getting the content length mismatched header is because instead of uploading the file, you're uploading an object which contains file contents and that is causing the content length to increase.
Please change the following line of codes:
files = {'file': open(fd, 'rb')}
result = requests.put(url = URI, headers = header, files = files)
to something like:
data = open(fd, 'rb') as stream
result = requests.put(url = URI, headers = header, data = data)
And now you're only uploading the file contents.

AWS4 Signing request signature does not match…even though it seems to

I appear to have correctly followed the procedure for generating a canonical string and string to sign for the AWS4 SDK. However I receive the error The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
The only clue I have is that when I'm not using the Moment.js utc() call, it accepts the signature as a match but (as expected) treats the signature as expired, so I suspect the UTC vs. local time is related to the issue.
Here is the code where I generate the timestamps.
var now = moment().utc();
var date_stamp = now.format('YYYYMMDD');
var amzn_date = now.format('YYYY-MM-DDTHH:mm:ssZ');
var string_to_sign_date = now.format('YYYYMMDDTHHmmssZ');
string_to_sign_date = string_to_sign_date.replace('+00:00', 'Z');
amzn_date = string_to_sign_date.replace('+00:00', 'Z');
Here is where I create the string_to_sign:
var string_to_sign = connectMeRequest.algorithm + '\n' + string_to_sign_date + '\n' + credential_scope + '\n' + cryptoJS.SHA256(canonical_request);
Here is my (console logged) vs. Amazon's signature. I didn't replace the newlines in their JSON res in case that is the issue.
My output for canonical string:
POST
/prod/makeEchoCallHandler
content-type:application/x-www-form-urlencoded
host:408wm9ltub.execute-api.us-west-2.amazonaws.com
x-amz-date:20160116T191451Z
x-amz-target:aws4_request
content-type;host;x-amz-date;x-amz-target
03a2c439264740e4883441d0049beaf9da4dc865ddd7169dbe9e747f28da6185
Their output:
POST\n/prod/makeEchoCallHandler\n\ncontent-type:application/x-www-form-urlencoded\nhost:408wm9ltub.execute-api.us-west-2.amazonaws.com\nx-amz-date:20160116T191451Z\nx-amz-target:aws4_request\n\ncontent-type;host;x-amz-date;x-amz-target\n03a2c439264740e4883441d0049beaf9da4dc865ddd7169dbe9e747f28da6185
My output for string to sign:
AWS4-HMAC-SHA256
20160116T191451Z
20160116/us-west-2/execute-api/aws4_request
ab63b72a190addcde39771097bbbc2e28c0d00c458fda9136d2d630e227e9074
Their output:
AWS4-HMAC-SHA256\n20160116T191451Z\n20160116/us-west-2/execute-api/aws4_request\nab63b72a190addcde39771097bbbc2e28c0d00c458fda9136d2d630e227e9074
The '\n' is officially part of the string to sign. You need to add it in explicitly. A good example is found here. The important part is:
StringToSign =
Algorithm + '\n' +
RequestDate + '\n' +
CredentialScope + '\n' +
HashedCanonicalRequest
Add those in and give it another try!
EDIT: As noted in the comments, this looks to be an error in the date formatting between the canonical string and the string to sign.

Resources