Pyzmail/IMAPclient: Can't figure out what key to use - python-3.x

I'm following this guide: https://automatetheboringstuff.com/chapter16/#calibre_link-45
to scrape emails and I am having issues using pyzmail.PyzMessage.factory(). I keep getting a KeyError.
I took the advice from here: Python email bot Pyzmail/IMAPclient error
but I continued to get the same error.
imapObj = imapclient.IMAPClient("imap.gmail.com", ssl = True)
imapObj.login("MY_EMAIL_ADDRESS", "MY_PASSWORD")
imapObj.select_folder("INBOX", readonly=False)
UIDs = imapObj.gmail_search("test1")
print(UIDs)
rawMessages = imapObj.fetch(UIDs, ["BODY[]"])
pprint.pprint(rawMessages)
message = pyzmail.PyzMessage.factory(rawMessages[40041][b'BODY[]'])
I am getting this error:
[7156]
Traceback (most recent call last):
File "C:/Users/Logan/PycharmProjects/email_sending_test/venv/main1.py", line 17, in <module>
message = pyzmail.PyzMessage.factory(rawMessages[0][b'BODY[]'])
KeyError: b'BODY[]'
defaultdict(<class 'dict'>,
{7156: {b'BODY[]': b'MIME-Version: 1.0\r\nDate: Thu, 3 Jan 2019 16:'
b'51:54 -0500\r\nMessage-ID: <CAB4Lt1swQPJvCL3ot'
b'8E7q2Pc9_C26hZxMdUgcZd9LbJUyhZbvw#mail.gmail'
b'.com>\r\nSubject: test1\r\nFrom: Rob Roberts'
b' <swimmingonanarwhal#gmail.com>\r\nTo: Rob Rob'
b'erts <swimmingonanarwhal#gmail.com>\r\nContent'
b'-Type: multipart/alternative; boundary="0000'
b'000000006f5b28057e94c5de"\r\n\r\n--000000000'
b'0006f5b28057e94c5de\r\nContent-Type: text/plai'
b'n; charset="UTF-8"\r\n\r\ntrying this ou'
b't\r\n\r\n--0000000000006f5b28057e94c5de\r\nCon'
b'tent-Type: text/html; charset="UTF-8"\r\n\r'
b'\n<div dir="ltr">trying this out</div>\r\n\r'
b'\n--0000000000006f5b28057e94c5de--',
b'SEQ': 6962}})
Process finished with exit code 1

Related

No MESSAGE-ID and get imap_tools work for imap.mail.yahoo.com

The question is twofold, about getting MESSAGE-ID, and using imap_tools. For an email client ("handmade") in Python I need to lessen the data amount read from the server (presently it takes 2 min to read the whole mbox folder of ~170 msg for yahoo), I believe that having MESSAGE-ID will help me.
imap_tools has IDLE command which is essential to keep the yahoo server connection alive and other features which I believe will simplify the code.
To learn about MESSAGE-ID I started with the following code (file fetch_ssl.py):
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import imaplib
import email
import os
import ssl
import conf
# Why UID==1 has no MESSAGE-ID ?
if __name__ == '__main__':
args = conf.parser.parse_args()
host, port, env_var = conf.config[args.host]
if 0 < args.verbose:
print(host, port, env_var)
with imaplib.IMAP4_SSL(host, port,
ssl_context=ssl.create_default_context()) as mbox:
user, pass_ = os.getenv('USER_NAME_EMAIL'), os.getenv(env_var)
mbox.login(user, pass_)
mbox.select()
typ, data = mbox.search(None, 'ALL')
for num in data[0].split():
typ, data = mbox.fetch(num, '(RFC822)')
msg = email.message_from_bytes(data[0][1])
print(f'num={int(num)}, MESSAGE-ID={msg["MESSAGE-ID"]}')
ans = input('Continue[Y/n]? ')
if ans.upper() in ('', 'Y'):
continue
else:
break
Where conf.py is:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import argparse
HOST = 'imap.mail.yahoo.com'
PORT = 993
config = {'gmail': ('imap.gmail.com', PORT, 'GMAIL_APP_PWD'),
'yahoo': ('imap.mail.yahoo.com', PORT, 'YAHOO_APP_PWD')}
parser = argparse.ArgumentParser(description="""\
Fetch MESSAGE-ID from imap server""")
parser.add_argument('host', choices=config)
parser.add_argument('-verbose', '-v', action='count', default=0)
fetch_ssl.py outputs:
$ python fetch_ssl.py yahoo
num=1, MESSAGE-ID=None
Continue[Y/n]?
num=2, MESSAGE-ID=<83895140.288751#communications.yahoo.com>
Continue[Y/n]? n
I'd like to understand why the message with UID == 1 has no MESSAGE-ID? Does that happen from time to time (I mean there are messages with no MESSAGE-ID)? How to handle these cases? I haven't found such cases for gmail.
Then I attempted to do similar with imap_tools (Version: 0.56.0), (file fetch_tools.py):
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import os
import ssl
from imap_tools import MailBoxTls
import conf
# https://github.com/ikvk/imap_tools/blob/master/examples/tls.py
# advices
# ctx.load_cert_chain(certfile="./one.crt", keyfile="./one.key")
if __name__ == '__main__':
args = conf.parser.parse_args()
host, port, env_var = conf.config[args.host]
if 0 < args.verbose:
print(host, port, env_var)
user, pass_ = os.getenv('USER_NAME_EMAIL'), os.getenv(env_var)
ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
ctx.options &= ~ssl.OP_NO_SSLv3
# imaplib.abort: socket error: EOF
with MailBoxTls(host=host, port=port, ssl_context=ctx) as mbox:
mbox.login(user, pass_, 'INBOX')
for msg in mbox.fetch():
print(msg.subject, msg.date_str)
Command
$python fetch_tools.py yahoo
outputs:
Traceback (most recent call last):
File "/home/vlz/Documents/python-scripts/programming_python/Internet/Email/ymail/imap_tools_lab/fetch_tools.py", line 20, in <module>
with MailBoxTls(host=host, port=port, ssl_context=ctx) as mbox:
File "/home/vlz/Documents/.venv39/lib/python3.9/site-packages/imap_tools/mailbox.py", line 322, in __init__
super().__init__()
File "/home/vlz/Documents/.venv39/lib/python3.9/site-packages/imap_tools/mailbox.py", line 35, in __init__
self.client = self._get_mailbox_client()
File "/home/vlz/Documents/.venv39/lib/python3.9/site-packages/imap_tools/mailbox.py", line 328, in _get_mailbox_client
client = imaplib.IMAP4(self._host, self._port, self._timeout) # noqa
File "/usr/lib/python3.9/imaplib.py", line 205, in __init__
self._connect()
File "/usr/lib/python3.9/imaplib.py", line 247, in _connect
self.welcome = self._get_response()
File "/usr/lib/python3.9/imaplib.py", line 1075, in _get_response
resp = self._get_line()
File "/usr/lib/python3.9/imaplib.py", line 1185, in _get_line
raise self.abort('socket error: EOF')
imaplib.abort: socket error: EOF
Command
$ python fetch_tools.py gmail
Produces identical results. What are my mistakes?
Using Python 3.9.2, Debian GNU/Linux 11 (bullseye), imap_tools
(Version: 0.56.0)
EDIT
Headers from the message with no MESSAGE-ID
X-Apparently-To: vladimir.zolotykh#yahoo.com; Sun, 25 Oct 2015 20:54:21 +0000
Return-Path: <mail#product.communications.yahoo.com>
Received-SPF: fail (domain of product.communications.yahoo.com does not designate 216.39.62.96 as permitted sender)
...
X-Originating-IP: [216.39.62.96]
Authentication-Results: mta1029.mail.bf1.yahoo.com from=product.communications.yahoo.com; domainkeys=neutral (no sig); from=product.communications.yahoo.com; dkim=pass (ok)
Received: from 127.0.0.1 (EHLO n3-vm4.bullet.mail.gq1.yahoo.com) (216.39.62.96)
by mta1029.mail.bf1.yahoo.com with SMTPS; Sun, 25 Oct 2015 20:54:21 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=product.communications.yahoo.com; s=201402-std-mrk-prd; t=1445806460; bh=5PTgF8Jghm92xeMD5mSHp6A3eRVV70PWo1oQ15K7Tfk=; h=Date:From:Reply-To:To:Subject:From:Subject; b=D7ItgOiuLbiexJGHvORgbpRi22X+sYso6gwZKDXVca79DxMMy2R1dUtZTIg7tcft1lovVJUDw/7fC51orDltRidlfnpayeY8lT+94DRlSBwopuxgOqqR9oTTjTBZ0oEvdxUcXl/q54N2GxuBFvmg8UO0OZoCnFPpUVYo9x4arMjt/0TOW1Q5d/yjdmO7iwiued/rliP/Bsq0TaZYcb0oCAT7Q50tb1fB7wcXLYNSC1OCQ1l1LajbUqmU1LWWNse36mUUTBieO2sZT0ERFrHaCTaTNQSXKQG2AxYF7Dd/8i0Iq3xqdcS0bDpjmWE25uoKvCdtXtUbylsuQSChuLFMTw==
Received: from [216.39.60.185] by n3.bullet.mail.gq1.yahoo.com with NNFMP; 25 Oct 2015 20:54:20 -0000
Received: from [98.137.101.84] by t1.bullet.mail.gq1.yahoo.com with NNFMP; 25 Oct 2015 20:54:20 -0000
Date: 25 Oct 2015 20:54:20 +0000
Received: from [127.0.0.1] by nu-repl01.direct.gq1.yahoo.com with NNFMP; 25 Oct 2015 20:54:20 -0000
X-yahoo-newman-expires: 1445810060
From: "Yahoo Mail" <mail#product.communications.yahoo.com>
Reply-To: replies#communications.yahoo.com
To: <ME>#yahoo.com
Subject: Welcome to Yahoo! Vladimir
X-Yahoo-Newman-Property: ydirect
Content-Type: text/html
Content-Length: 25180
I skipped only X-YMailISG.
EDIT II
Of 167 messages 21 have no MESSAGE-ID header.
fetch_ssl.py takes 4m12.342s, and fetch_tools.py -- 3m41.965s
It looks simply like your email without a Message-ID legitimately does not have one; it appears the welcome email Yahoo sent you actually lacks it. Since it's a system generated email, that's not that unexpected. You'd just have to skip over it.
The second problem is that you need to use imap_tools.MailBox.
Looking at the documentation and source at the repo it appears that the relevant classes to use are:
MailBox - for a normal encrypted connection. This is what most email servers use these days, aka IMAPS (imap with SSL/TLS)
MailBoxTls - For a STARTTLS connection: this creates a plaintext connection then upgrades it later by using a STARTTLS command in the protocol. The internet has mostly gone to the "always encrypted" rather than "upgrade" paradigm, so this is not the class to use.
MailBoxUnencrypted - Standard IMAP without SSL/TLS. You should not use this on the public internet.
The naming is a bit confusing. MailBox corresponds to imaplib.IMAP4_SSL; MailBoxTls corresponds to imaplib.IMAP4, then using startls() on the resulting connection; and MailboxUnencrypted corresponds to imaplib.IMAP4 with no security applied. I imagine it's this way so the most common one (Mailbox) is a safe default.

Has anyone encountered anything like this

WARNING:androguard.core.api_specific_resources:Requested API level 29 is larger than maximum we have, returning API level 28 instead.
[INFO] 26/Oct/2021 09:54:16 - Running APKiD 2.1.2
[ERROR] 26/Oct/2021 09:54:16 - Error Performing Static Analysis
Traceback (most recent call last):
File "C:\Users\ACER\OneDrive\Рабочий стол\Mobile-Security-Framework-MobSF\mobsf\StaticAnalyzer\views\android\static_analyzer.py", line 211, in static_analyzer
apkid_results = apkid_analysis(app_dic[
File "C:\Users\ACER\OneDrive\Рабочий стол\Mobile-Security-Framework-MobSF\mobsf\MalwareAnalyzer\views\apkid.py", line 41, in apkid_analysis
rules = options.rules_manager.load()
File "C:\Users\ACER\OneDrive\Рабочий стол\Mobile-Security-Framework-MobSF\venv\lib\site-packages\apkid\rules.py", line 46, in load
self.rules = yara.load(self.rules_path)
yara.Error: could not open file "C:\Users\ACER\OneDrive\Рабочий стол\Mobile-Security-Framework-MobSF\venv\Lib\site-packages\apkid\rules\rules.yarc"
[ERROR] 26/Oct/2021 09:54:16 - could not open file "C:\Users\ACER\OneDrive\Рабочий стол\Mobile-Security-Framework-MobSF\venv\Lib\site-packages\apkid\rules\rules.yarc"
[ERROR] 26/Oct/2021 09:54:16 - Internal Server Error: /static_analyzer/
ERROR:django.request:Internal Server Error: /static_analyzer/
I would be glad to receive any information!!!

ClientError: An error occurred (InvalidTextEncoding) when calling the SelectObjectContent operation: UTF-8 encoding is required. reading gzip file

I am getting the above error in my code. encoding=latin-1 needs to be included as a parameter somewhere in select-object-content. Since I am new to this, I am not sure, where to add it.
Can anyone help me in this?
Code:
client = boto3.client('s3',aws_access_key_id,aws_secret_access_key',region_name)
resp = client.select_object_content(
Bucket='mybucket',
Key='path_to_file/file_name.gz',
ExpressionType='SQL',
Expression=query,
InputSerialization = {'CSV': {"FileHeaderInfo": "Use"}, 'CompressionType': compressionType},
OutputSerialization = {'CSV': {}},
)
Traceback:
ClientError Traceback (most recent call last)
C:\path/3649752754.py in <module>
78 Expression=SQL,
79 InputSerialization = {'CSV': {"FileHeaderInfo": "Use"}, 'CompressionType': compression},
---> 80 OutputSerialization = {'CSV': {}},
81 )
82
ClientError: An error occurred (InvalidTextEncoding) when calling the SelectObjectContent operation: UTF-8 encoding is required. The text encoding error was found near byte 90,112.
You need to save your CSV file with UTF-8 encoding. For example, with Notepad++ or Excel->Save As->Select from the dropdown.

AIOHTTP, can't read multiple parameters from request

I try to read multiple parameters lat and lon from my request using request.rel_url.query["lat"] and request.rel_url.query["lon"], respectively (following this SO post).
When debugging, query has only one element, the second parameter is not read correctly.
The call:
$ curl -i -X GET 127.0.0.1:8000/test?lat=47.659049&lon=9.166003
HTTP/1.1 500 Internal Server Error
Content-Type: text/plain; charset=utf-8
Content-Length: 55
Date: Tue, 22 Sep 2020 06:49:15 GMT
Server: Python/3.8 aiohttp/3.6.2
Connection: close
500 Internal Server Error
Server got itself in trouble
The python error:
Error handling request
Traceback (most recent call last):
File "/home/lorenz/.local/lib/python3.8/site-packages/aiohttp/web_protocol.py", line 418, in start
resp = await task
File "/home/lorenz/.local/lib/python3.8/site-packages/aiohttp/web_app.py", line 458, in _handle
resp = await handler(request)
File "/home/lorenz/Documents/copernicus-service/topography/topography_server.py", line 199, in handle_copernicus
lon = request.rel_url.query["lon"]
KeyError: 'lon'
What's the reason for this, and how can I read the 2nd parameter correctly?

stripe.error.APIConnectionError connected to openssl

I am dealing with the following error
Traceback (most recent call last):
File "once.py", line 1757, in <module>
once()
File "once.py", line 55, in once
stripe.Charge.all()
File "/Library/Python/2.7/site-packages/stripe/resource.py", line 438, in all
return cls.list(*args, **params)
File "/Library/Python/2.7/site-packages/stripe/resource.py", line 450, in list
response, api_key = requestor.request('get', url, params)
File "/Library/Python/2.7/site-packages/stripe/api_requestor.py", line 150, in request
method.lower(), url, params, headers)
File "/Library/Python/2.7/site-packages/stripe/api_requestor.py", line 281, in request_raw
method, abs_url, headers, post_data)
File "/Library/Python/2.7/site-packages/stripe/http_client.py", line 139, in request
self._handle_request_error(e)
File "/Library/Python/2.7/site-packages/stripe/http_client.py", line 159, in _handle_request_error
raise error.APIConnectionError(msg)
stripe.error.APIConnectionError: Unexpected error communicating with Stripe. If this problem persists,
let us know at support#stripe.com.
I get this error when running a simple test program which stripe suggested
import stripe
stripe.api_key = "blah bla"
stripe.api_base = "https://api-tls12.stripe.com"
print "stripe.VERSION = ", stripe.VERSION
if stripe.VERSION in ("1.13.0", "1.14.0", "1.14.1", "1.15.1", "1.16.0", "1.17.0", "1.18.0", "1.19.0"):
print "Bindings update required."
try:
stripe.Charge.all()
print "TLS 1.2 supported, no action required."
except stripe.error.APIConnectionError:
print "TLS 1.2 is not supported. You will need to upgrade your integration."
raise
I do not understand why I get this error, since my stripe version is high enough
stripe.VERSION = 1.55.2
and my openssl version does support TLS?
>>$ openssl version
OpenSSL 1.0.2k 26 Jan 2017
>>$ which openssl
/usr/bin/openssl
any ideas how to debug this further? I am lost...
ok I don't know what exactly caused the problem, but I got it working by changing the client
client = stripe.http_client.PycurlClient()
stripe.default_http_client = client
I think the requests package is the default. pycurl seems to work in my case...

Resources