How to Validate a Xero webhook payload with HMACSHA256 python 3 - webhooks

Based on the instructions here (https://developer.xero.com/documentation/webhooks/configuring-your-server) for setting up and validating the intent to receive for the Xero webhook.
The computed signature should match the signature in the header for a correctly signed payload.
But, using python 3, the computed signature doesn't match the signature in the header in any way at all. Xero would send numerous requests to the subscribing webhook url for both correctly and incorrect. In my log, all those requests returned as 401. So, below is my test code which also proved to not match. I don't know what is missing or I what did wrong.
Don't worry about the key been show here, i have generated another key but this was the key assigned to me to use for hashing at this point.
based on their instruction, running this code should make the signature match one of the headers. But not even close at all.
XERO_KEY =
"lyXWmXrha5MqWWzMzuX8q7aREr/sCWyhN8qVgrW09OzaqJvzd1PYsDAmm7Au+oeR5AhlpHYalba81hrSTBeKAw=="
def create_sha256_signature(key, message):
message = bytes(message, 'utf-8')
return base64.b64encode(hmac.new(key.encode(), message,
digestmod=hashlib.sha256).digest()).decode()
# first request header (possibly the incorrect one)
header = "onoTrUNvGHG6dnaBv+JBJxFod/Vp0m0Dd/B6atdoKpM="
# second request header (possibly the correct one)
header = "onoTrUNvGHG6dnaBv+JBJxFodKVp0m0Dd/B6atdoKpM="
payload = {
'events':[],
'firstEventSequence':0,
'lastEventSequence':0,
'entropy':
'YSXCMKAQBJOEMGUZEPFZ'
}
payload = json.dumps(payload, separators=(",", ":")).strip()
signature = create_sha256_signature(XERO_KEY, str(payload))
if hmac.compare_digest(header, signature):
print(True)
return 200
else:
print(False)
return 401

The problem was because when I was receiving the request payload, I was using
# flask request
request.get_json()
this will automatically parse my request data into a json format, hence the reason why the calculated signature never matched
So, what I did was change the way I receive the request payload:
request.get_data()
This will get the raw data.

I still could not get this to work even with the OP's answer, which I found a little vague.
The method I found which worked was:
key = #{your key}
provided_signature = request.headers.get('X-Xero-Signature')
hashed = hmac.new(bytes(key, 'utf8'), request.data, hashlib.sha256)
generated_signature = base64.b64encode(hashed.digest()).decode('utf-8')
if provided_signature != generated_signature:
return '', 401
else:
return '', 200
found on https://github.com/davidvartanian/xero_api_test/blob/master/webhook_server.py#L34

Related

Using request_mock to dynamically set response based on request

I am trying to mock a simple POST request that creates a resource from the request body, and returns the resource that was created. For simplicity, let's assume the created resource is exactly as passed in, but given an ID when created. Here is my code:
def test_create_resource(requests_mock):
# Helper function to generate dynamic response
def get_response(request, context):
context.status_code = 201
# I assumed this would contain the request body
response = request.json()
response['id'] = 100
return response
# Mock the response
requests_mock.post('test-url/resource', json=get_response)
resource = function_that_creates_resource()
assert resource['id'] == 100
I end up with runtime error JSONDecodeError('Expecting value: line 1 column 1 (char 0)'). I assume this is because request.json() does not contain what I am looking for. How can I access the request body?
I had to hack up your example a little bit as there is some information missing - but the basic idea works fine for me. I think as mentioned something is wrong with the way you're creating the post request.
import requests
import requests_mock
with requests_mock.mock() as mock:
# Helper function to generate dynamic response
def get_response(request, context):
context.status_code = 201
# I assumed this would contain the request body
response = request.json()
response['id'] = 100
return response
# Mock the response
mock.post('http://example.com/test-url/resource', json=get_response)
# resource = function_that_creates_resource()
resp = requests.post('http://example.com/test-url/resource', json={'a': 1})
assert resp.json()['id'] == 100
This example is not complete and so we cannot truly see what is happening.
In particular, it would be useful to see a sample function_that_creates_resource.
That said, I think your get_response code is valid.
I believe that you are not sending valid JSON data in your post request in function_that_creates_resource.

How can I get the body of a gmail email with an attatchment gmail python API

This is my code to get the body of the email:
body = []
body.append(msg['payload']['parts'])
if 'data' in body[0][0]['body']:
print("goes path 1")
body = base64.urlsafe_b64decode(
body[0][0]['body']['data'])
else
print("goes path 2")
body = base64.urlsafe_b64decode(
body[0][1]['body']['data'])
else:
# What Do I do Here?
The reason i have the if elif statements is because sometimes the body is in different places so i have to try for both of them. When run through this an email that had an attachment resulted in a key error of data not existing meaning it's probably in a different place. The json object of body is in an image linked below because it is too big to paste here. How do I get the body of the email?
https://i.stack.imgur.com/Ufh5E.png
Edit:
The answers given by #fullfine aren't working, they output another json object the body of which can not be decoded for some reason:
binascii.Error: Invalid base64-encoded string: number of data characters (1185) cannot be 1 more than a multiple of 4
and:
binascii.Error: Incorrect padding
An example of a json object that i got from their answer is:
{'size': 370, 'data': 'PGRpdiBkaXI9Imx0ciI-WW91IGFyZSBpbnZpdGVkIHRvIGEgWm9vbSBtZWV0aW5nIG5vdy4gPGJyPjxicj5QbGVhc2UgcmVnaXN0ZXIgdGhlIG1lZXRpbmc6IDxicj48YSBocmVmPSJodHRwczovL3pvb20udXMvbWVldGluZy9yZWdpc3Rlci90Sll1Y3VpcnJEd3NHOVh3VUZJOGVEdkQ2NEJvXzhjYUp1bUkiPmh0dHBzOi8vem9vbS51cy9tZWV0aW5nL3JlZ2lzdGVyL3RKWXVjdWlyckR3c0c5WHdVRkk4ZUR2RDY0Qm9fOGNhSnVtSTwvYT48YnI-PGJyPkFmdGVyIHJlZ2lzdGVyaW5nLCB5b3Ugd2lsbCByZWNlaXZlIGEgY29uZmlybWF0aW9uIGVtYWlsIGNvbnRhaW5pbmcgaW5mb3JtYXRpb24gYWJvdXQgam9pbmluZyB0aGUgbWVldGluZy48L2Rpdj4NCg=='}
I figured out that i had to use base64.urlsafe_b64decode to decode the body which got me b'<div dir="ltr">You are invited to a Zoom meeting now. <br><br>Please register the meeting: <br>https://zoom.us/meeting/register/tJYucuirrDwsG9XwUFI8eDvD64Bo_8caJumI<br><br>After registering, you will receive a confirmation email containing information about joining the meeting.</div>\r\n'
How can I remove all the extra html tags while keeping the raw text?
Answer
The structure of the response body changes depending on the message itself. You can do some test to check how they look like in the documentation of the method: users.messages.get
How to manage it
Intial scenario:
Get the message with the id and define the parts.
msg = service.users().messages().get(userId='me', id=message_id['id']).execute()
payload = msg['payload']
parts = payload.get('parts')
Simple solution
You can find the raw version of the body message in the snippet, as the documentation says, it contains the short part of the message text. It's a simple solution that returns you the message without formatting or line breaks. Furthermore, you don't have to decode the result. If it does not fit your requirements, check the next solutions.
raw_message = msg['snippet']
Solution 1:
Add a conditional statement to check if any part of the message has a mimeType equal to multipart/alternative. If it is the case, the message has an attachment and the body is inside that part. You have to get the list of subparts inside that part. I attach you the code:
for part in parts:
body = part.get("body")
data = body.get("data")
mimeType = part.get("mimeType")
# with attachment
if mimeType == 'multipart/alternative':
subparts = part.get('parts')
for p in subparts:
body = p.get("body")
data = body.get("data")
mimeType = p.get("mimeType")
if mimeType == 'text/plain':
body_message = base64.urlsafe_b64decode(data)
elif mimeType == 'text/html':
body_html = base64.urlsafe_b64decode(data)
# without attachment
elif mimeType == 'text/plain':
body_message = base64.urlsafe_b64decode(data)
elif mimeType == 'text/html':
body_html = base64.urlsafe_b64decode(data)
final_result = str(body_message, 'utf-8')
Solution 2:
Use a recursive function to process the parts:
def processParts(parts):
for part in parts:
body = part.get("body")
data = body.get("data")
mimeType = part.get("mimeType")
if mimeType == 'multipart/alternative':
subparts = part.get('parts')
[body_message, body_html] = processParts(subparts)
elif mimeType == 'text/plain':
body_message = base64.urlsafe_b64decode(data)
elif mimeType == 'text/html':
body_html = base64.urlsafe_b64decode(data)
return [body_message, body_html]
[body_message, body_html] = processParts(parts)
final_result = str(body_message, 'utf-8')
Extra comments
If you need to get more data from your message I recommend you to use the documentation to see how the response body looks like.
You can also check the method in the API library of Python to see a detailed description of each element.
Do not use images in this way as DalmTo has said
edit
I tried the code with Python 2, it was my mistake. With Python 3, as you said, you have to use base64.urlsafe_b64decode(data) instead of base64.b64decode(data). I've already updated the code.
I added a simple solution that maybe fits your needs. It takes the message from the snippet key. It is a simplified version of the body message that does not need decoding.
I also don't know how you have obtained the text/html part with my code that does not handle that. If you want to get it, you have to add a second if statement, I updated the code so you can see it.
Finally, what you obtained using base64.urlsafe_b64decode is a bytes variable, to obtain the string you have to convert it using str(body_message, 'utf-8'). It is now in the code

Signing and Verifying of Signature using Pycryptodome always fails

Hi I'm using the Pycryptodome package to try and verify signatures of transactions in a Blockchain. My issue is that when trying to add a new transaction, I first create a signature to be passed into a verify transaction method but for some reason it always fails even though the logic seems to be right when I compare it to the documentation. If anyone could point me where I'm going wrong it would be much appreciated. I have 3 methods that handle all of this and i'm not sure where the issue is
The generate keys method
def generate_keys(self):
# generate private key pair
private_key = RSA.generate(1024, Crypto.Random.new().read)
# public key comes as part of private key generation
public_key = private_key.publickey()
# return keys as hexidecimal representation of binary data
return (binascii.hexlify(public_key.exportKey(format='DER')).decode('ascii'), binascii.hexlify(private_key.exportKey(format='DER')).decode('ascii'))
The sign transaction method
def sign_transaction(self, sender, recipient, amount, key):
# convert transaction data to SHA256 string
hash_signer = SHA256.new(
(str(sender) + str(recipient) + str(amount)).encode('utf-8'))
# sign transaction
signature = pkcs1_15.new(RSA.importKey(
binascii.unhexlify(key))).sign(hash_signer)
# return hexidecimal representation of signature
return binascii.hexlify(signature).decode('ascii')
and the verify transaction method
#staticmethod
def verify_transaction(transaction):
# convert public key back to binary representation
public_key = RSA.importKey(binascii.unhexlify(
transaction.sender))
try:
# create signature from transaction data
hash_signer = SHA256.new(
(str(transaction.sender) + str(transaction.recipient) + str(transaction.amount)).encode('utf-8'))
pkcs1_15.new(public_key).verify(
hash_signer, binascii.unhexlify(transaction.signature))
return True
except ValueError:
return False
Once i've generated my key pair and attempt to use them to sign and verify transactions it always fails. I know this because it always returns false from the verify method leading me to believe a value error is always raised. Thanks in advance hopefully someone can help me out.

Invalid hash, timestamp, and key combination in Marvel API Call

I'm trying to form a Marvel API Call.
Here's a link on authorization:
https://developer.marvel.com/documentation/authorization
I'm attempting to create a server-side application, so according to the link above, I need a timestamp, apikey, and hash url parameters. The hash needs be a md5 hash of the form: md5(timestamp + privateKey + publicKey) and the apikey url param is my public key.
Here's my code, I'm making the request in Python 3, using the request library to form the request, the time library to form the timestamp, and the hashlib library to form the hash.
#request.py: making a http request to marvel api
import requests;
import time;
import hashlib;
#timestamp
ts = time.time();
ts_str = str(float(ts));
#keys
public_key = 'a3c785ecc50aa21b134fca1391903926';
private_key = 'my_private_key';
#hash and encodings
m_hash = hashlib.md5();
ts_str_byte = bytes(ts_str, 'utf-8');
private_key_byte = bytes(private_key, 'utf-8');
public_key_byte = bytes(public_key, 'utf-8');
m_hash.update(ts_str_byte + private_key_byte + public_key_byte);
m_hash_str = str(m_hash.digest());
#all request parameters
payload = {'ts': ts_str, 'apikey': 'a3c785ecc50aa21b134fca1391903926', 'hash': m_hash_str};
#make request
r = requests.get('https://gateway.marvel.com:443/v1/public/characters', params=payload);
#for debugging
print(r.url);
print(r.json());
Here's the output:
$python3 request.py
https://gateway.marvel.com:443/v1/public/characters...${URL TRUNCATED FOR READABILITY)
{'code': 'InvalidCredentials', 'message': 'That hash, timestamp, and key combination is invalid'}
$
I'm not sure what exactly is causing the combination to be invalid.
I can provide more info on request. Any info would be appreciated. Thank you!
EDIT:
I'm a little new to API calls in general. Are there any resources for understanding more about how to perform them? So far with my limited experience they seem very specific, and getting each one to work takes a while. I'm a college student and whenever I work in hackathons it takes me a long time just to figure out how to perform the API call. I admit I'm not experienced, but in general does figuring out new API's require a large learning curve, even for individuals who have done 10 or so of them?
Again, thanks for your time :)
I've also had similar issues when accessing the Marvel API key. For those that are still struggling, here is my templated code (that I use in a jupyter notebook).
# import dependencies
import hashlib #this is needed for the hashing library
import time #this is needed to produce a time stamp
import json #Marvel provides its information in json format
import requests #This is used to request information from the API
#Constructing the Hash
m = hashlib.md5() #I'm assigning the method to the variable m. Marvel
#requires md5 hashing, but I could also use SHA256 or others for APIS other
#than Marvel's
ts = str(time.time()) #This creates the time stamp as a string
ts_byte = bytes(ts, 'utf-8') #This converts the timestamp into a byte
m.update(ts_byte) # I add the timestamp (in byte format) to the hash
m.update(b"my_private_key") #I add the private key to
#the hash.Notice I added the b in front of the string to convert it to byte
#format, which is required for md5
m.update(b"b2aeb1c91ad82792e4583eb08509f87a") #And now I add my public key to
#the hash
hasht = m.hexdigest() #Marvel requires the string to be in hex; they
#don't say this in their API documentation, unfortunately.
#constructing the query
base_url = "https://gateway.marvel.com" #provided in Marvel API documentation
api_key = "b2aeb1c91ad82792e4583eb08509f87a" #My public key
query = "/v1/public/events" +"?" #My query is for all the events in Marvel Uni
#Building the actual query from the information above
query_url = base_url + query +"ts=" + ts+ "&apikey=" + api_key + "&hash=" +
hasht
print(query_url) #I like to look at the query before I make the request to
#ensure that it's accurate.
#Making the API request and receiving info back as a json
data = requests.get(query_url).json()
print(data) #I like to view the data to make sure I received it correctly
Give credit where credit is due, I relied on this blog a lot. You can go here for more information on the hashlib library. https://docs.python.org/3/library/hashlib.html
I noticed in your terminal your MD5 hash is uppercase. MD5 should output in lowercase. Make sure you convert to that.
That was my issue, I was sending an uppercase hash.
As mentioned above, the solution was that the hash wasn't formatted properly. Needed to be a hexadecimal string and the issue is resolved.
Your final URL should be like this:
http:// gateway.marvel.com/v1/public/characters?apikey=(public_key)&ts=1&hash=(md5_type_hash)
So, you already have public key in developer account. However, how can you produce md5_type_hash?
ts=1 use just 1. So, your pattern should be this:
1 + private_key(ee7) + public_key(aa3). For example: Convert 1ee7aa3 to MD5-Hash = 1ca3591360a252817c30a16b615b0afa (md5_type_hash)
You can create from this website: https://www.md5hashgenerator.com
Done, you can use marvel api now!

Python3 send requests cookies from previous call

I want to resend the initialized cookies from the first call in the second call, so that the session is not changing. This is not working.
Why? And how can I solve it. Sorry, new in python
https_url = "www.google.com"
r = requests.get(https_url)
print(r.cookies.get_dict())
#cookie = {id: abc}
response = requests.get(https_url, cookies=response.cookies.get_dict())
print(response.cookies.get_dict())
#cookie = {id: def}
You aren't necessarily doing it wrong with the way you're passing the cookies from the last response to the next request, except that:
"www.google.com" is not a valid URL.
Even you had used http://www.google.com as the URL, the cookies returned by Google in such a GET request aren't meant to be session cookies and won't be persistent across requests.
You used the variable r to receive the returning value from the first requests.get, and yet you used response.cookies when you make the second requests.get. A possible typo?
If all of the above are due to your trying to mock up your real code, you should really consider using requests.Session to avoid micro-managing session cookies.
Please read requests.Session's documentation for more details.
import requests
with requests.Session() as s:
r = s.get(https_url)
# cookies from the first s.get are automatically passed on to the second s.get
r = s.get(https_url)
...

Resources