Example using poor-smime-sign - python-3.x

I am trying to send SMIME signed email and fail.
I use Python 3, so I chose the poor-smime-sign library.
Using Thunderbird to recieve, I either get an empty email with "undisclosed recipients" and no subject, which is signed or I get the whole thing in plain-text, printing a huge blob of base64 with the message-body - no sign of signing (pun unintended, noticed and left intact).
I have compared the source of a working email (created by Thunderbird from my sent folder) with my franken-mails, but I don't quite see the difference.
Here's the code:
from poor_smime_sign import smime_sign
def signEmail(self, message :str) -> str:
"""Sign the message-body.
The message is encoded in UTF-8, signed and then returned as Unicode.
Check this document on how to generate untrusted example-keys:
https://tools.ietf.org/doc/python-m2crypto/howto.smime.html
Check the settings ('invmail'->'keydir') where to put the keys.
The privateKey is called 'signer_key.pem' and publicKey 'signer.pem'.
"""
import os
privateKey = os.path.join(settings.SF.mail['keydir'], 'signer_key.pem')
publicKey = os.path.join(settings.SF.mail['keydir'], 'signer.pem')
try:
signed = smime_sign(publicKey, privateKey, message.encode('UTF-8'))
except Exception as e:
raise(Exception("Problem during signing: "+str(e)))
return signed.decode('UTF-8')
poor_smime_sign btw is pretty humble about their module, which I quite like. I looked at their code and all they do is call openssl, which is what I would have done, too, as the next step. If you're troubleshooting you can go into the mod and log the command that gets executed and fiddle with that in a shell until you understand why it fails. I e.g. had pub and priv cert the wrong way and got a really stupid error message.
This is what puts the message together in the end. I had copied it from some example somewhere.
body = self.signEmail(body)
emailAr = [
"From: %s" % emailFrom,
"To: %s" % emailTo,
"Subject: %s" % subject,
"", # <- that one
body,
]
message = "\r\n".join(emailAr)
#...
server.sendmail(emailFrom, [emailTo], message.encode('UTF-8'))

TL;DR:
Get the headers right: Signing already creates a header. Just add to that. No empty new lines.
Well, turns out I had no idea how email works. I was indeed wondering about why I use the TO-address twice (once constructing the message and once in sendmail()) but it took a while until I figured it out. I knew, that I needed some sort of header but I did not know where and why.
The solution was twofold:
The text that gets signed needs a header so the email client knows what to do with it. I took one from a working email I had lying around.
Looks like this.
cannedHeader = "\r\n".join([
"Content-Type: text/plain; charset=utf-8; format=flowed",
"Content-Language: de-DE",
"Content-Transfer-Encoding: quoted-printable",
"","" # emtpy new line concludes header
])
body = cannedHeader+bodyInput
body = self.signEmail(body)
The signed message-body already comes with a partial header attached to it. Printing out the result of signing, you see it starts with
MIME-Version: 1.0 Content-Type: multipart/signed;
protocol=3Dpplication/x-pkcs7-signature"; m= icalg=3Dha-256";
boundary=3D---DF20A931579CC3CE98F68AEF4D387131"
This is an S/MIME signed message
------DF20A931579CC3CE98F68AEF4D387131
And I ran that through the initial version of the code, generating two headers in the process. One with recipients and subject suffixed by an empty new line, and then added the signed message-body, which had another header, containing SMIME information, also suffixed by an empty new line. That could not work. So removing the empty line (in the question marked with "that line") fixed this.
As I have not found any example on stackexchange, I thought I'd write this down, so I can find it when I need it the next time.

Related

Bad Request Error when creating a JSON String from a string and a variable

Hello I'm currently sitting on a problem, thats completely baffling me. I am trying to create a REST API for our companies robot and want the user to be able to post an already created mission to it via a GUI with this function:
def post_mission(host, headers):
data = json.dumps({'mission_id': item_selected.variable})
data = data.replace('"',"'")
url= "mission_queue"
post_mission = requests.post(host + url, json = data, headers = headers)
this gives me a #400 Bad Request error...
however when I replace the data = json.dumps({'mission_id': item_selected.variable}) line with data = {'mission_id': '68754b18-bb1f-11e8-954d-94c691173c1e'} (aka don't take the mission_id I sourced via a textbox where the User is able to search for his desired mission but manually insert it) everything works fine and whats even more bizarre if I use print(host + url, data, headers) for both Code versions it spits out the EXACT same text
I hope this is understandable and someone else has an idea where I'm wrong
I guess you don't have to replace double quote with single quote.

Docusign: Verify HMAC key from header response with the secret key

I am working with Docusign connect and planning to use HMAC keys to authenticate the messages. I am referring https://developers.docusign.com/esign-rest-api/guides/connect-hmac#example-hmac-workflow link.
I find few terms confusing in the documentation. Attaching the code snippet from the doc for python.
def ComputeHash(secret, payload):
import hmac
import hashlib
import base64
hashBytes = hmac.new(secret, msg=payload, digestmod=hashlib.sha256).digest()
base64Hash = base64.b64encode(hashBytes)
return base64Hash;
def HashIsValid(secret, payload, verify):
return verify == ComputeHash(secret,payload)
Can you explain what payload(didn't understand exactly what it is), secret (I am guessing the secret key) and verify means from the above code and how do I verify my secret key with X-Docusign-Signature-1 which I get from response header?
My code:
message = request.headers
hashBytes = hmac.new(secret_key.encode('utf-8'), msg=message.encode('utf-8'), digestmod=hashlib.sha256).hexdigest()
base64Hash = base64.b64encode(hashBytes)
[Edited]
I found the solution on my own. Please read the first answer. I have explained it in details.
Sorry for the confusion.
Payload is "The entire body of the POST request is used, including line endings."
This is what you're encoding here using a Hash (HMAC) function.
SHA256 HMAC digest take in an the array of bytes (payload) and a secret (some key to use for encryption) and produces some encrypted version of the payload that can later be verified.
I highly recommend you ensure you first understand how the Connect webhook works without using HAMC encoding. This feature is meant to secure your application and it's a bit more complex. If you first get it working without it - you'll get a better grasp of what's going on (as well as feel a bit better about accomplishing a subtask).
Once you have it working, you can add the HMAC to make it secure and it will be easier at that point.
I found the solution to my problem.
expected_signature = request.headers['X-Docusign-Signature-1']
message = request.data # It is already in bytes. No need to encode it again.
hashBytes = hmac.new(secret_key.encode('utf-8'), msg=message, digestmod=hashlib.sha256).hexdigest()
actual_signature = base64.b64encode(hashBytes)
hmac.compare_digest(actual_signature.decode('utf-8'),expected_signature):

Adding message to gmail error "Payload parts count different from expected"

I am adding a message to a gmail folder using this (example) URL:
https://www.googleapis.com/gmail/v1/users/user#domain.com/messages/import?uploadType=multipart
The body of the request looks like this:
--test_abc123
Content-Type: application/json; charset=UTF-8
{
"labelIds": [ "Label_525" ],
"raw": "RnJvbTogIlNlY3RpZ28gQ2VydGlmaWNh..."
}
--test_abc123--
The raw data is a base64 encoded standard MIME message that looks normal to me. The result of this POST is http error 400 with the error response "Payload parts count different from expected 2. Request payload parts count: 1".
I can supply the original MIME text if that is helpful, but let me emphasize that I have been running this code for several years without problem. I've tried different messages to test this out, but it appears that Google has changed something to break my software.
Is Google objecting to my raw data, or something about the MIME encoding? Any ideas what the problem could be?
---- Addendum ----
I have gotten a few messages to work, they seem to all have image or data attachments. However I really don't see any problem with the messages that are failing - I can import them into Office 365 or Thunderbird or anything else and they render just fine. As a test, I tried importing the message below, which was taken from the MIME RFC. It fails with the same error. I think that Google has changed something to make their MIME parser very fussy, but I don't see how I can fix my input data.
From: Nathaniel Borenstein <nsb#bellcore.com>
To: Ned Freed <ned#innosoft.com>
Subject: Sample message
MIME-Version: 1.0
Content-type: multipart/mixed; boundary="simple boundary"
This is the preamble. It is to be ignored, though it
is a handy place for mail composers to include an
explanatory note to non-MIME compliant readers.
--simple boundary
This is implicitly typed plain ASCII text.
It does NOT end with a linebreak.
--simple boundary
Content-type: text/plain; charset=us-ascii
This is explicitly typed plain ASCII text.
It DOES end with a linebreak.
--simple boundary--
This is the epilogue. It is also to be ignored.
Addendum 2: I tried a simple upload (using content-type header message/rfc822) and it worked, except the message was unlabeled. How
would I specify what label I want applied to a message? I was originally trying to follow the documentation here
link
which tells me to create the json body that I gave above. This allows me to specify the label. But I cannot seem to use
this body in a simple upload. The content type is either invalid, or what Gmail imports is just literally the json body,
it does not parse out the raw data. If you could point me to a specific example showing the URI, message body, http headers
(not java code) that would be very useful to me.
OK never mind, I got it working by adding an empty message/rfc822 part to the body of the multipart upload. That satisfies Google, and the empty part is ignored in favor of the raw data.
You are doing a multipart upload,see here:
The body of the request is formatted as a multipart/related content
type [RFC2387] and contains exactly two parts. The parts are
identified by a boundary string, and the final boundary string is
followed by two hyphens.
This is why it works only for your messages with images or attachments, since your message
--test_abc123
is only one part.
In the past there was no check if this condition is fulfilled, so you might have gotten away with using multipart for 1-part-messages.
But now it's not possible anymore, so if have a single-part message, you should use Simple upload.
If you do not know in advance how many parts your message has, you can always try the multipart first, implementing a try...catch statement, and implement a simple upload request within catch in case of failure.

discord webhook can not send empty message

I have written this small PoC for discord webhooks and i am getting error that Can not send empty string. I tried to google but couldn't find a documentation or an answer
here is my code
import requests
discord_webhook_url = 'https://discordapp.com/api/webhooks/xxxxxxxxxxxxxxxxxx/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
data = {'status': 'success'}
headers = {'Content-Type': 'application/json'}
res = requests.post(discord_webhook_url, data=data, headers=headers)
print(res.content)
I'm late, but I came across this issue recently, and seeing as it has not been answered yet, I thought I document my solution to the problem.
For the most part, it is largely due to the structure of the payload being wrong.
https://birdie0.github.io/discord-webhooks-guide/discord_webhook.html provides an example of a working structure. https://discordapp.com/developers/docs/resources/channel#create-message is the official documentation.
I was also able to get a minimum test case working using: {"content": "Test"}.
If it still fails after that with the same error, the likely causes are:
If using curl, check to make sure there are no accidental escape / backslashes \
If using embeds with fields, ensure there are no empty values
When in doubt, ensure all values are populated, and not "". Through trial-and-error / the process of cancellation, you can figure out exactly what key-value pair is causing an issue, so I suggest playing with the webhook via curl before turning it into a full program.

How can I decode a google OAuth 2.0 JWT (OpenID Connect) in a node app?

I'm having a heck of a time here trying to use google OAuth to authenticate users in my node express app. I can successfully do the OAuth, which returns a response like so:
{
access_token: 'token string',
id_token: 'id.string',
expires_in: 3599,
token_type: "Bearer"
}
This all makes sense, but I can't for the life of me figure out how to decode the JWT. I am a bit inexperienced in all this, so this is all a bit foreign to me.
Following the instructions listed here: https://developers.google.com/accounts/docs/OAuth2Login#validatinganidtoken I am attempting to decode the JWT locally in my node app.
I installed https://github.com/hokaccha/node-jwt-simple in my node environment.
And I'm pretty certain I need to use this certificate (https://www.googleapis.com/oauth2/v1/certs) in all this somehow to decode it, but I am at a bit of a loss here. I don't really understand how I get the certificate into my node app, and after that how to use it with node-jwt-simple. And I also don't really understand how I know when I need to pull a fresh certificate, vs using a cached one.
Anyone out there with some experience in this that can help me out?
Thanks for any help. I'm totally at a loss at this point.
** Update **
So I have made some progress... Kind of.
By calling jwt.decode(id_token, certificate, true); I am able to successfully decode the token. Even if the certificate var is an empty object {}. This leaves me with 3 questions still.
1: What is the best way to get the certificate into my express app using the url from google?
2: How will I know when I need to pull in a fresh version of it?
3: It seems like passing in true for noVerify (3rd arg in jwt.decode) is a terrible idea. How can I get that to work without passing that in?
It looks like perhaps jwt-simple is expecting hs256 and the token is using rs256.
Again, I'm super inexperienced in this, so I may be way off base here.
* UPDATE *
Thanks to the help from Nat, I was able to get this working!
I think I tried every single JWT and JWS node module out there. What I finally landed on is as follows:
I found that none of the modules that I looked at did quite what I wanted out of the box. I created the following jwt decoding helper methods that I am using to decode the id_token, so I can get the kid from the header.
module.exports = {
decodeJwt: function (token) {
var segments = token.split('.');
if (segments.length !== 3) {
throw new Error('Not enough or too many segments');
}
// All segment should be base64
var headerSeg = segments[0];
var payloadSeg = segments[1];
var signatureSeg = segments[2];
// base64 decode and parse JSON
var header = JSON.parse(base64urlDecode(headerSeg));
var payload = JSON.parse(base64urlDecode(payloadSeg));
return {
header: header,
payload: payload,
signature: signatureSeg
}
}
}
function base64urlDecode(str) {
return new Buffer(base64urlUnescape(str), 'base64').toString();
};
function base64urlUnescape(str) {
str += Array(5 - str.length % 4).join('=');
return str.replace(/\-/g, '+').replace(/_/g, '/');
}
I am using this decoding to determine if I need to pull in a new public cert from: https://www.googleapis.com/oauth2/v1/certs
Then I am using that public cert and node-jws (https://github.com/brianloveswords/node-jws) jws.verify(id_token, cert) to verify the signature!
Hooray!
Thanks again for the extra explanation you gave in your response. That went a long way in helping me understand what I was even trying to do. Hope this might help others too.
From the specification point of view, what you are encountering is [OpenID Connect].
id_token is a [JWS] signed [JWT]. In this case, it is a "." separated string with three components. The first portion is the header. The second is the payload. The third is the signature. Each of them are Base64url encoded string.
When you decode the header, you will get something like:
{"alg":"RS256","kid":"43ebb53b0397e7aaf3087d6844e37d55c5fb1b67"}
The "alg" indicates that the signature algorithm is RS256, which is defined in [JWA].
The "kid" indicates the key id of the public key that corresponds to the key used to sign.
Now I am ready to answer some of your questions:
2: How will I know when I need to pull in a fresh version of it?
When the kid of the cached cert file (a [JWK] file) does not match the kid in the header, fetch a new cert file. (BTW, the URL from which you pull the certs are called x5u.)
3: It seems like passing in true for noVerify (3rd arg in jwt.decode)
is a terrible idea. How can I get that to work without passing that
in?
Indeed. Perhaps you might want to look at another library such as kjur.github.io/jsjws/ .
References
[OpenID Connect] openid.bitbucket.org/openid-connect-core-1_0.html
[JWS] tools.ietf.org/html/draft-ietf-jose-json-web-signature
[JWT] tools.ietf.org/html/draft-ietf-oauth-json-web-token‎
[JWK] tools.ietf.org/html/draft-ietf-oauth-json-web-keys
[JWA] tools.ietf.org/html/draft-ietf-jose-json-web-algorithms

Resources