Found an issue with Base64DecoderStream in javamail. Some email content I get are like this:
Content-Type: text/plain; charset=3D"utf-8"
Content-Transfer-Encoding: base64
QmFzZTY0IGlzIGEgZ2VuZXJpYyB0ZXJtIGZvciBhIG51bWJlciBvZiBzaW1pbGFyIGVuY29kaW5=
n
IHNjaGVtZXMgdGhhdCBlbmNvZGUgYmluYXJ5IGRhdGEgYnkgdHJlYXRpbmcgaXQgbnVtZXJpY2F=
s
bHkgYW5kIHRyYW5zbGF0aW5nIGl0IGludG8gYSBiYXNlIDY0IHJlcHJlc2VudGF0aW9uLiBUaGU=
g
QmFzZTY0IHRlcm0gb3JpZ2luYXRlcyBmcm9tIGEgc3BlY2lmaWMgTUlNRSBjb250ZW50IHRyYW5=
z
ZmVyIGVuY29kaW5nLg==
Ideally the = sign should have been replaced with the single character on the following line but gsuite(Gmail) sometimes does like this. This causes Base64DecoderStream to corrupt the message. However, Outlook and many popular online base64 decoders handle this base64 content well. Can this be fixed?
Additional detail was provided privately, which allowed me to determine that the problem is that the message includes an attachment of MIME type message/rfc822 (the original message), and that attachment uses a Content-Transfer-Encoding of quoted-printable. The MIME spec does not allow the use of that encoding for MIME content of that type. This is a violation of the MIME spec that Google really needs to fix. Please provide them this additional information if they haven't figured it out themselves.
RFC 2046, section 5.2.1, says:
No encoding other than "7bit", "8bit", or "binary" is permitted for
the body of a "message/rfc822" entity.
In the mean time, you can set the JavaMail System property mail.mime.allowencodedmessages to "true" to work around this bug in GSuite.
Related
The problem is about getting a direct download link to a Netcloud file.
Documentation :
To obtain a direct link:
POST /ocs/v2.php/apps/dav/api/v1/direct
With the fileId in the body (so fileId=42 for example).
I want to POST this URL with Nodes.js. What is the XML (I guess) format to set "fileId=42" in the body of my request ?
Everything I try returns me
"Invalid query, please check the syntax. API specifications are here:http://www.freedesktop.org/wiki/Specifications/open-collaboration-services." but noway there to get the format.
Samething with curl, no syntax is working.
In a more generic way what is the XML (I guess) format when querying the OCS API?
I answer my question:
At last I suceeded in making it work with curl, single/double quotes in Windows was the problem, fixed using "{" escape character.
This told me this was not about OCS data format itself as data is to be sent as it, ie "fieldId=42"
Switching from NojesJs/https lib to Axios fixed the problem. Probably some content-type header, however I tried 'application/x-www-form-urlencoded' which fits with the data to send.
In a public record request to a city government, I have gotten back a number of records that are .txt format of e-mails with attachments appearing to be base64. The e-mail attachments are jpegs, pdf, png, or doc in base64 a shorter example is below. The government official claim "The records we released to you are in the form that was available, including the file you noted. From what I have been told, it is likely garbled computer coding. We have no other version of those records."
Questions:
Does someone have to intentionally work at saving the e-mail and attachments in this way so that they are unreadable, thus making the information not public (hiding it)?
or is this something that can plausibly happen in saving "garbled computer coding"?
If it is plausible that a computer does it, how?
Is there a way of decoding it?
I have tried a number of online decoding with various settings and have been unsuccessful.
I have done a number of public record requests from this city and department in the past and have never gotten such .txt documents. The public record request is around a city contract that is problematic.
From: "Steinberg, David (DPW)" <david.steinberg#sfdpw.org>
To: "Goldberg, Jonathan (DPW)" <jonathan.goldberg#sfdpw.org>
Sent: Fri, 17 May 2019 20:40:36 +0000
Subject: Re: SOTF - Education, Outreach and Training Committee, May 21, 2019 hearing
----boundary-LibPST-iamunique-566105023_-_-
Content-Type: image/jpeg
Content-Transfer-Encoding: base64
Content-ID: <image002.jpg#01D50CB4.2B093D10>
Content-Disposition: attachment;
filename*=utf-8''image002.jpg;
filename="image002.jpg"
/9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAoHBwkHBgoJCAkLCwoMDxkQDw4ODx4WFxIZJCAmJSMg
IyIoLTkwKCo2KyIjMkQyNjs9QEBAJjBGS0U+Sjk/QD3/2wBDAQsLCw8NDx0QEB09KSMpPT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT3/wAARCABJAFEDASIA
AhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQA
AAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3
ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWm
p6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEA
AwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSEx
BhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElK
U1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3
uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwDKooqa
FY47qIXiSiHcPMVRhtuecZ716ZgRojSOqIpZ2OAo6k0+G2muBIYYZJBEu+Taudi9yfSum1bwlDa6
ENa0q7mlh4lVXGGVD0ORzkcVf8ExwaxHqPnb4bsxeVLLE20SI38RHTcMHnvWTqrl5kUo62OO07T5
9VvY7S0VWmkztDNgcDJ5qB0aN2RhhlJUj3Fdp4N0LT/7YZ5bx/t9nK4FtwOBwGz3GCK5/WmhsfFl
20FvE0UNxkQtyjY6g+xOaaqXk0hWsjJorutb0XS9R8OzaxZReVdpGssiQjCklQeVPbBzkVwtVCak
gasFFKFJViMHb1Hf649KSrEFFFFAE9jMba/t5gMlJFOAAc8+/FaPiO7a5uLdXl8xUQhcyb3UFj8p
OAcgY45rKiDtKgiBMhYbQPWtb+3tRe9ubh4rWedwryu8CvgLjGOwA9qzktboaOsuNXNtbaroscAE
NtpweBm5aQbRkkdB16exrN+H979lt9UXaD8isDjvhv8ACuZe7vvtD30hcvMjAu3IKNlSOe3OPal0
++vNMhmmtSqpIuxmbB6EHgHr1/I1n7P3Wu4+bU6HwPe/a/FiMyhMWhjRe+Bjqe5965vUZHutYu3d
gXedyT0HU80+zm1DSLuNrUtFM5CqBg7j6fqOPzqNGujb3ESR5EkgEz7fmzydpPYZBOPUVajaV0K+
h1eka2n9iw2RCRpeySWZduvESqhPtkj6Zrm9F0oahrFvaXLNDC0pieQDgMATtB6ZOMfjUZurloLP
zI4mggfMSbVXcSRnOOTnHWrdtr2raf5ksMixxGd3aMouwuSM8dTjj6UlFxvbqO99zvNYtLPQ9I8j
SbexivJQY4VlUbpePmAJ6nHr1OBXljI0Z2OrKw4IYYIrS1jUNQ1q+SS+SPziuFVFAwPfnPvzVCcy
mU/aGYyYHLNk4xxz6YopQcVqEncjooorYkltYpZ7uKK3jMkzMAiDua0prTV7RZ7p7UKhA81l2uAM
Y5wScc8n9aj8Nf8AIy6d/wBdhT7ObTtJeWe2uJriZoniWPyPLX5hjLHPIGelZSbvZDRTD3Taa+Fz
aqyxlsfdbkgfU/0pmLhtObAb7Ksm1iDxvI6H8BVyAf8AFI3mOQt7CT7DYwoi48J3BPRr6MA+uI2z
/Onf8wJY7PWLwQ3aQbwQxicsgyDxwCarJJfx6k8AiK3ckmxomTBLEYxj8fz5q5fLpp07R/t5ut/2
T/liExt8x/XvVmYs/jTTJsgxTNbvCec+XwF3Z/i45qFIdjAeR1Cxui74TtDEfMMHp6dalZrk2Ekp
j/0eaUqZP9v7xA54/L8ahu2H2245H+ufv/tGtF/+RPj/AOv9v/RYrTohDLW0v9QY3VvbIVyVeRiE
RyRyCWIBP0qrfQ3FvcmK6h8mRVACY4C44x7e9XfEHXT1X/j1FnGYB26fP+O7Oays5A/SiOuoMKKK
KsRZ037UNSt/sH/H1vHldPvfjxSrpl42pf2etuxu9xTysjOev06VBDM9tPHPEcSRMHU+4Oa9EuYo
7bU7jxVGoMJ04Sx+hlPAH5Y/Ospz5WUlc4rT49UtNVeys4z9rcmKSBgrB8ckEHg9M1M1rrWu3TWq
WxkNqShiiVUjiPfp8ueK68Qpbajd+Kto8l9PWWMjp5jDBH6D86xNRnmtfh9phtHZEuZWa6kQ4LOS
eCfr/IVCnd6IdjPvX1vQoIIL62SOKNSsTSwRyDGScBsH1JxUo0HxNcXceo/ZJZJsrIkhdO33cDPT
2qzo80t54H16O9dpLWFA0LOc7XxnAJ98fnWvrmnxajcaLE2s/YJntkVIwrZc8cgggfnSc7OwWuc4
JtdvbybT/skRuirGSM2sSNjuckD161QtLbUL+xlt7aMyW1sxnk5UBDjGSx9h0ruLW8W9+IrRKrqb
Wza3Z5B8zkEfN+tZus29uPB8kOg3HmW1pcEX2BzKf7xPcA/h+VCnrawWMjRLXXruyK6dai4tAxIW
ZEZA3fbu7/Ss/V1v0vyuqIyXAUDaygYXtgDjH0rcsta0q98PWmlandXWnvbE7JoSdjdeTj61meJd
KuNJ1JEuLo3ayxh4pyTll6DP0q4v3tRPYyKKKK2JCtSTxDeyeH00dtn2ZCCDg7sA5xnPTNZdFJpP
cDUfxDeyeH00djH9mQ5BAO4jOcE56Zo0nxFeaRFJBGIp7WQ5eCddyE+vtWXRS5I2tYd2a2reI7zV
rVbRkhtrRTkQW6bVJ9T61DqWt3WqS2kkwRHtUCRmMEcA5BPPXis+ihQS6Cuzb/4Su9GtnVRDbC5M
XlNhTtYepGetVNK1q50h7gwLHIlyhSWOUEqw+mfc/nWfRS5I7WHdm5YeKrixso7RrKxuIYs+WJos
lec9aoarq11rV59pvGUuF2qqjCovoBVKimoRTuhXYUUUVQH/2Q==
----boundary-LibPST-iamunique-566105023_-_---
I got the image by copying your base64 data to a file and
base64 -d file > image.jpeg
on my Debian/Linux.
RFC 2045 section 6 says binary data are to be encoded in US-ASCII characters
so that it's quite normal images are encoded by base64 although GUI email reader
would not show you raw data.
When you see such raw data, it means somebody might copy and paste other emails blindly, still it less likely happen. (Obviously your example is part of multipart message, but not complete.)
Decode service on web is available, for example, here.
After the fetchmail fetches mails, the new mails are stored in a file like /var/mail/user. We can open the file user by text editor like vim.
How can I create such text-based email files? Say, I want to send an email with contents:
From: sender <sender#xx.com>
To: receiver <receiver#xx.com>
Subject: test subject
Contents: ...
Attached: file1.txt, file2.png, file3.pdf
The problem is how to make these to be a formal text-based email.
Besides, If I have such an email file. How can I extract files(say, subjects, contents, attached files, etc.) by some command line tools. I know I can open it with program like mutt. Can this be done using a command line utility?
There is a bunch of standards you need to understand, but email is fundamentally text.
The file format in /var/spool/mail or /var/mail/user etc is typically Berkeley mbox. This is not formally defined anywhere, but consists of a sequence of RFC5322 (née RFC822) email messages, each preceded by a From_ line, the format of which is basically From %s %C where %s is the sender's email address (what you also see in Return-Path:) and %C is the date when the message arrived. Notice the two spaces between the format strings!
The toplevel email message is RFC5322 but on top of that, you need to understand MIME.
mbox: RFC 4155
Email message format: RFC5322
MIME: RFC2045 RFC2046 RFC2047 RFC2048
You will also stumble over (E)SMTP RFC5321 which is only tangential to your question, but good to know. Notice how 821 and 822 (and later 2821 and 2822, and now 5321 and 5322) have adjacent RFC numbers.
Furthermore, there is a wild, wild West of non-standard headers, some of which are nonetheless significant. Dan Bernstein's reference http://cr.yp.to/immhf.html is a lifesaver. As a general guideline, what spammers typically do is copy/paste headers without understanding them; therefore, an essential practice for deliverability is "don't do that". In other words, if you don't know what a header is for, don't use it.
Any modern programming language will come with libraries to create and manipulate RFC5322 and MIME, and probably mbox too. For creating a message you can send somewhere, you don't need mbox anyway, just something along the lines of (pseudocode)
message = new MIME({'Subject': 'hello', 'From': 'me#example.net',
'To': 'My Friend <you#example.com>'});
message.addbodypart('text/plain', 'Hi Fred.\nHow are you?');
message.addbodypart('image/png', {'file': '/home/you/logo.png'});
smtp = new SMTP('mail.example.net', 587, {'user': 'me', 'pass': 'xyzzy'});
smtp.send(message);
A multipart message looks something like what you describe in your question, except there is no specific header to identify "attachments" and actually conceptually no "attachments", just "body parts". Here is a simple MIME message to show what the message in your question would properly look something like.
From: sender <sender#example.com>
To: receiver <receiver#example.com>
Subject: test subject
MIME-Version: 1.0
Content-type: multipart/mixed; boundary="so_long_eFlop"
This is a MIME multipart message. Nobody actually sees what it says here.
--so_long_eFlop
Content-type: text/plain; charset="utf-8"
Content-disposition: inline
Content-transfer-encoding: 7bit
Many mail clients will display this as the "main part" but MIME does not
define any particular hierarchy. Many mail clients will generate a
text/plain rendering and a text/html rendering of the message you type in,
and the recipient's mail client will decide -- based on user preferences
-- which one to display. Anyway, I will not create an example of that
here. This is just "a text message with a picture attached", or, more
precisely, a MIME message with two body parts.
Oh, the content-disposition: inline is usually just implied for a
text/plain part. Some clients will override or ignore the disposition
set by the sender anyway.
--so_long_eFlop
Content-type: image/png
Content-disposition: attachment
Content-transfer-encoding: base64
Iam+not/attaching+a/real00picture+here/just/a/bunch0of/binary/goo===
--so_long_eFlop--
The file format is called "mbox". There's a good article on Wikipedia (http://en.wikipedia.org/wiki/Mbox), as well as all over the Internet. Like RFC 4155. :)
telnet your.mail.server 25
helo localhost.localdomain
mail from:<sender#address.com>
rcpt to:<recipient#address.com>
data
From:Me
Subject:This is an email via Telnet
Hi,
The first line connects to the server on port 25. Replace "your.mail.server" with the name or address of the MX server for the domain.
Most servers expect the second "HELO" line to begin the session. I have seen servers that don't care, but in general they should throw an error.
You must have a "MAIL FROM:" line with the address you expect a reply to come to.
The mail is going nowhere if you don't specify the "RCPT TO:" address.
The message body begins with "DATA" line. This will usually be met with instruction on how to end the message - a single "." on a line by itself.
The "From:" and "Subject:" headers above are optional. You can add any additional headers here.
.
quit
I am using the XML-INTO op-code to parse a web service request. Every now and then I get errors in the logs
(RNX0351 - "The XML parser detected error code 302").
The help for a 302 is
302 The parser does not support the requested CCSID value or
the first character of the XML document was not '<'
To the best of my knowledge, the first character is "<" and the request is generated from a previous web service call so I would be very suprised if the CCSID has changed.
The error is repeatable, for the specific query so it is almost certainly data related, I am just unsure how I would go about identifying the offending item.
Any thoughts on how to determine the issue, or better yet, how to overcome it?
cheers
CCSID is an AS400/iSeries/Power System attribute, and it applies to the whole IFS.It's like a declaration of what inside the file is, or in other words what its internal encoding "should be".
It's supposed that data content encoding in the file and the file one (the envelope) match, and the box uses this attribute to show and handle corresponding characters.
It sounds like you receive data under one encoding, but CCSID file doesn't match.
Try changing CCSID on your file (only the envelope). E.G.: 37 (american), 500 (latin-1), 819 (utf-8), 850 (dos), 1252 (win) and display file after.You can check first using ls -Sla yourfile in QSH or QP2TERM, or EDTF as well. CHGATTR allows you to change CCSID, as well as setccsid in QSH (again).
This way helped me to find related issues. Remember that although data may be visible in the four hundred, they may not be visible through a share folder in Win. It means that CCSID file, an content encoding don't match.
Hope it helps.
Hi I've seen this error with XML data uploaded to AS400/iSeries/IBM i with FTP and the CCSID 819 (ISO 8859-1 ASCII) and it has some binary garbage in first few positions of file. Changed encoding to CCSID 1208 (UTF-8 with IBM PUA) using FTP "quote type c 1208" and the problem cleared and XML-INTO was successful.
So, suggestion about XML parser error 302 received when using XML-INTO is to look at the file (wrklnk ...) and if first character is not "<" but instead some binary garbage then try CCSID 1208 for utf-8.
Statements in this answer about what 819 is and what ccsid represents utf-8 do not agree with previous answer but are correct, according to IBM documentation:
https://www-01.ibm.com/software/globalization/ccsid/ccsid819.html
https://www-01.ibm.com/software/globalization/ccsid/ccsid1208.html
I'm working on this problem a couple hours,
for me the solution was use option ccsid=UCS2 when you use data structure or variable to store xml.
something like that :
XML-INTO customer %XML( xmlSource : 'ccsid=UCS2');
I have the program running on ccsid = 870, every conversion to ccsid on the xmlSource field don't work,
The strange thing that when I use the file with ccsid = 850, every thing work fine
I mention that becouse this is the first page when you looking about this problem.
Maybe this help someone.
I need to write a program that read the .eml files from IIS's mail drop box, but I can't find a definitive source that tells me the encoding of the .eml files. Is there a specification somewhere that tells me the encoding of the files, or do I just have to guess/assume one?
You need to read the Content-Transfer-Encoding header. This value will tell you how the email is encoded. The most common are 7-Bit (no encoding), Quoted-Printable (where you see a lot of =HEX pairs), and base64 (which is base 64 encoding).
Based upon that header value, you decode the following body part using the specified routine.
I found my answer at en.wikipedia.org/wiki/MIME: "The basic Internet e-mail transmission protocol, SMTP, supports only 7-bit ASCII characters... "
Though it's too late to answer but eml file format nothing but a plaintext MIME (rfc822) file format for storing emails.