When i try to get subject and content email, i alwasy got this content has been encode, example:
subject: Thư cảm ơn
I got this subject has been encode: Th=C6=B0_c=E1=BA=A3m_=C6=A1n?=
so how to decode this content.
That string looks like it's supposed to be encoded in the format mandated by RFC 2047, except that it is corrupt: RFC 2047 encoded strings always begin with =? end end with ?=.
Assuming you can get a non-corrupt encoded string, you should expect to see not only the Subject but also the display names in To, From, and other headers to be encoded in this way. You might want to look into using a library that does complete MIME parsing for you. Appropriate available libraries will depend on what language you're using.
Related
I need to build a XMPP client. The server PLAIN mechanism to auth. (using zlib compression I think)
I captured trafic from other xmpp system thats use PLAIN mechanism and the text appear to be Base64 (id + token) ADc1Y2M2OWY0MzQwMTUwMjgyOWIwMWY2MDAyN2E0NDE2ADE1YTk0NzM3NTRiYjY2MGExMGYzYTA5MzA5NWQxMmY3 is what the client return. I put that into a Base64 decoder and its give me this : 75cc69f43401502829b01f60027a441615a9473754bb660a10f3a093095d12f7.
When I encode this using a Base64 encoder, Its give me something else than the first Base64 string (NzVjYzY5ZjQzNDAxNTAyODI5YjAxZjYwMDI3YTQ0MTYxNWE5NDczNzU0YmI2NjBhMTBmM2EwOTMwOTVkMTJmNw)
Can someone explain me? I couldn't find anything on google.
Edit:
https://xmpp.org/extensions/xep-0034.html#example-3
the result of your decoding is not correct, in fact the decoded value
contains two binary values that can't be displayed as a character
(here substituted by a �):
�75cc69f43401502829b01f60027a4416�15a9473754bb660a10f3a093095d12f7.
What you encoded then is based on a string in which the two binary
values are not present, so you encoded basically something different
and got of course a different result.
From jps
I know about Data URIs in which base64 encoded data can be used inline such as images. Today I received an email actually an spam one in which there was an animated (gif) icon in its subject:
Here is the icon alone:
So the only thing did cross my mind was all about Data URIs and if Gmail allows some sort of emoticons to be inserted in subject. I saw the full detailed version of email and pointed to subject line at the below picture:
So GIF comes from =?UTF-8?B?876Urg==?= encoded string which is similar to Data URI scheme however I couldn't get the icon out of it. Here is element HTML source:
Long story short, there are lots of emoticons from https://mail.google.com/mail/e/XXX where XXX are hexadecimal numbers. They are documented nowhere or I couldn't find it. If that's about Data URI, so how is it possible to include them in Gmail's email subject? (I forwarded that email to a yahoo email account, seeing [?] instead of icon) and if it's not, then how that encoded string is parsed?
#Short description:
They are referred to internally as goomoji, and they appear to be a non-standard UTF-8 extension. When Gmail encounters one of these characters, it is replaced by the corresponding icon. I wasn't able to find any documentation on them, but I was able to reverse engineer the format.
#What are these icons?
Those icons are actually the icons that appear under the "Insert emoticons" panel.
While I don't see the 52E icon in the list, there are several others that follow the same convention.
B0C
4F4
Note that there are also some icons whose names are prefixed, such as gtalk.03C . I was not able to determine if or how these icons can be used in this manner.
#What is this Data URI thing?
It's not actually a Data URI, though it does share some similarities. It's actually a special syntax for encoding non-ASCII characters in email subjects, defined in RFC 2047. Basically, it works like this.
=?charset?encoding?data?=
So, in our example string, we have the following data.
=?UTF-8?B?876Urg==?=
charset = UTF-8
encoding = B (means base64)
data = 876Urg==
#So, how does it work?
We know that somehow, 876Urg== means the icon 52E, but how?
If we base64 decode 876Urg==, we get 0xf3be94ae. This looks like the following in binary:
11110011 10111110 10010100 10101110
These bits are consistent with a 4-byte UTF-8 encoded character.
11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
So the relevant bits are the following.:
011 111110 010100 101110
Or when aligned:
00001111 11100101 00101110
In hexadecimal, these bytes are the following:
FE52E
As you can see, except for the FE prefix which is presumably to distinguished the goomoji icons from other UTF-8 characters, it matches the 52E in the icon URL. Some testing proves that this holds true for other icons.
#Sounds like a lot of work, is there a converter?:
This can of course be scripted. I created the following Python code for my testing. These functions can convert the base64 encoded string to and from the short hex string found in the URL. Note, this code is written for Python 3, and is not Python 2 compatible.
###Conversion functions:
import base64
def goomoji_decode(code):
#Base64 decode.
binary = base64.b64decode(code)
#UTF-8 decode.
decoded = binary.decode('utf8')
#Get the UTF-8 value.
value = ord(decoded)
#Hex encode, trim the 'FE' prefix, and uppercase.
return format(value, 'x')[2:].upper()
def goomoji_encode(code):
#Add the 'FE' prefix and decode.
value = int('FE' + code, 16)
#Convert to UTF-8 character.
encoded = chr(value)
#Encode UTF-8 to binary.
binary = bytearray(encoded, 'utf8')
#Base64 encode return end return a UTF-8 string.
return base64.b64encode(binary).decode('utf-8')
###Examples:
print(goomoji_decode('876Urg=='))
print(goomoji_encode('52E'))
###Output:
52E
876Urg==
And, of course, finding an icon's URL simply requires creating a new draft in Gmail, inserting the icon you want, and using your browser's DOM inspector.
If you use the correct hex code point (e.g. fe4f4 for 'pile of poo') and If it is correctly encoded within the subject line header, let it be base64 (see #AlexanderOMara) or quoted-printable (=?utf-8?Q?=F3=BE=93=B4?=), then Gmail will automatically parse and replace it with the corresponding emoji.
Here's a Gmail emoji list for copying and pasting into subject lines - or email bodies. Animated emojis, which will grab even more attention in the inbox, are placed on a yellow background:
Many thanks to Alexander O'Mara for such a well-researched answer about the goomoji-tagged HTML images!
I just wanted to add three things:
There are still many many emoji (and other Unicode sequences generating pictures) that spammers and other erstwhile marketers are starting to use in email subject lines and that gmail does not convert to HTML images. In some browsers these show up bold and colored, which is almost as bad as animation. Browsers could also choose to animate these, but I don't know if any do. These Unicode sequences get displayed by the browser as Unicode text, so the exact appearance (color or not, animated or not, ...) depends on what text rendering system the browser is using. The appearance of a given Unicode emoji also depends on any Unicode variation selectors and emoji modifiers that appear near it in the Unicode code point sequence. Unlike the image-based emoji spam, these sequences can be copied-and-pasted out of the browser and into other apps as Unicode text.
I hope the many marketers reading this StackOverflow question will just say no. It is a horrible idea to include these sequences in your email subject lines and it will immediately tarnish you and your brand as lowlife spammers. It is not worth the "attention" your email will get.
Of course the first question coming to everyone's mind is: "how do I get rid of these things?" Fortunately there is this open-source Greasemonkey/Tampermonkey/Violentmonkey userscript:
Gmail Subject Line Emoji Roach Motel
This userscript eliminates both HTML-image (thanks to awesome work of Alexander O'Mara) and pure-Unicode types.
For the latter type, the userscript includes a regular expression designed to capture the Unicode sequences likely to be abused by marketers. The regex looks like this in ES6 Javascript (the userscript translates this to widely-supported pre-ES6 regex using the amazing ES6 Regex Transpiler):
var re = /(\p{Emoji_Modifier_Base}\p{Emoji_Modifier}?|\p{Emoji_Presentation}|\p{Emoji}\uFE0F|[\u{2100}-\u{2BFF}\u{E000}-\u{F8FF}\u{1D000}-\u{1F5FF}\u{1F650}-\u{1FA6F}\u{F0000}-\u{FFFFF}\u{100000}-\u{10FFFF}])\s*/gu
// which includes the Unicode Emoji pattern from
// https://github.com/tc39/proposal-regexp-unicode-property-escapes
// plus also these blocks frequently used for spammy emojis
// (see https://en.wikipedia.org/wiki/Unicode_block ):
// U+2100..U+2BFF Arrows, Dingbats, Box Drawing, ...
// U+E000..U+F8FF Private Use Area (gmail generates them for some emoji)
// U+1D000..U+1F5FF Musical Symbols, Playing Cards (sigh), Pictographs, ...
// U+1F650..U+1FA6F Ornamental Dingbats, Transport and Map symbols, ...
// U+F0000..U+FFFFF Supplementary Private Use Area-A
// U+100000..U+10FFFF Supplementary Private Use Area-B
// plus any space AFTER the discovered emoji spam
I feel a bit of a chicken and egg problem if i write a html meta tag specifying charset as say UTF-16 - like how do we decode the entire HTTP Request in the first place if we didn't know its UTF-16 data ? I believe request header needs to handle this and by the time we try to read metadata like say html tag charset="utf-16" we already know its UTF-16 .
Besides think one level higher about header information like Request Headers - are passed in ASCII as a standard ?
I mean at some level we need to agree upon and you can't set a data that is needed to decode as a metadata information . Can anyone clarify this ?
I am a bit confused on the idea of specifying a data that is needed to interpret the whole data as a metadata information inside the original data .
In general how can any form of encoding work if we don't have a standard agreed upon language/encoding to convey the data about the data itself ?
For example I am informed that Apache default has 8859-1 as the standard . So would all client need to enforce that for HTTP Headers and interpret the real content as UTF-8 if we want UTF-8 for the content-type ?
What character encoding should I use for a HTTP header? is a closely related question
UTF-16 (and other) encodings use a BOM (Byte Order Mark) that is read at the start of the file and that signals which encoding is being used. Only after that, the encoded part of the file begins.
For example, for UTF-16, you'll have the bytes FE FF if big-endian and FF FE if little-endian words are being used.
You also often see UTF-8 BOMs, although they don't need to be used (and may confuse some XML parsers).
I am parsing a MIME mail with LotusScript to get all attachments. But I get issues when it comes to encoded file names in the header. I got one file with the name
"HE336 =?Windows-1251?Q?=CF=E0=EA=E5=F2_=E4=EE=EA=F3=EC=E5=ED=F2=EE=E2.pdf?="
Is there any way to decode it with LotusScript?
The string I get is RFC 2047 header encoding. I found that Notes supports it in MIME headers. The issue I had is when I used MIMEHeader.GetParamVal it always returns the encoded value. However MIMEHeader.GetHeaderVal and GetHeaderValAndParams has an extra parameter
boolean decoded
true decodes any RFC-2047 encodings
false (default) retains any encodings; false is enforced if folded is true
When this is set to true, I get a decoded value.
it's been a while but I once used Julian Robichaux's Base64 classes with Jave and/or LS. You should be able to achieve what you are looking for with these.
Base64Encoding
Hope that helps.
Best wishes - Michael
I am looking for a url encoding method that is most efficient in terms of space. Raw binary (base2) could be represented in base16 which is smaller and is url safe, but base64 is even more efficient. However, the usual base64 encoding isn't url safe....
So what is the smallest encoding method that is also safe for URLS?
This is what the Base64 URL encoding variant is for.
It uses the same standard Base64 Alphabet except that + is changed to - and / is changed to _.
Most modern Base64 implementations will support this alternate encoding. If yours doesn't, it's usually just a matter of doing a search/replace on the Base64 input prior to decoding, or on the output prior to sending it to a browser.
You can use a 62 character representation instead of the usual base 64. This will give you URLs like the youtube ones:
http://www.youtube.com/watch?v=0JD55e5h5JM
You can use the PHP functions provided in this page if you need to map strings to a database numerical ID:
http://bsd-noobz.com/blog/how-to-create-url-shortening-service-using-simple-php
Or this one if you need to directly convert a numerical ID to a short URL string:
http://kevin.vanzonneveld.net/techblog/article/create_short_ids_with_php_like_youtube_or_tinyurl/
"base66" (theoretical, according to spec)
As far as I can tell, the optimal encoding for URLs is a "base66" encoding into the following alphabet:
ABCDEFGHIJKLMNOPQRSTUVWXYZ
abcdefghijklmnopqrstuvwxyz
0123456789-_.~
These are all the "Unreserved characters" according the URI specification RFC 3986 (section 2.3), so they will appear as-is in the URL. Using this "base66" encoding could give a URL like:
https://example.org/articles/.3Ja~jkWe
The question is then if you want . and ~ in your URLs?
On some older servers (ancient by now, I guess) ~joe would mean the "www directory" of the user joe on this server. And thus a user might be confused as to what the ~ character is doing in the middle of your URL.
This is common for academic websites, especially CS professors (e.g. Donald Knuth's website https://www-cs-faculty.stanford.edu/~knuth/)
"base80" (in practice, but not battle-tested)
However, in my own testing the following 14 other symbols also do not get
percent-encoded (in Chrome 95 and Firefox 93):
!$'()*+,:;=#[]
(see also this StackOverflow answer)
leaving a "base80" URL encoding possible. Some of these (notably + and =) would not work in the query string part of the URL, only in the path part. All in all, this ends up giving you beautiful, hyper-compressed URLs like:
https://example.org/articles/1OWG,HmpkySCbBy#RG6_,
https://example.org/articles/21Cq-b6Ud)txMEW$,hc4K
https://example.org/articles/:3Tx**U9X'd;tl~rR]q+
There's a plethora of reasons why you might not want all of those symbols in your URLs. One example is that StackOverflow's own "linkifier" won't include that ending comma in the link it generates (I've manually made it a part of the link here).
Also the percent encoding seems to be quite finicky. In some cases Firefox would initially percent-encode ' and ~] but on later requests would not.