Error 403 while using exchangelib to access Outlook Exchange server to read emails - python-3.x

I'm trying to read emails from the Microsoft Exchange server using EWS and exchangelib in Python for an email classification problem. But I am unable to connect to the exchange server.
I've tried specifying the version, auth_type, using a certificate (which gives a ssl verify error), using the smtp address in place of the username and it still doesn't connect.
Here is my code:
from exchangelib import Credentials, Account, EWSDateTime, EWSTimeZone, Configuration, DELEGATE, IMPERSONATION, NTLM, ServiceAccount, Version, Build
USER_NAME = 'domain\\user12345'
ACCOUNT_EMAIL = john.doe#ext.companyname.com'
ACCOUNT_PASSWORD = 'John#1234'
ACCOUNT_SERVER = 'oa.company.com'
creds = Credentials(USER_NAME, ACCOUNT_PASSWORD)
config = Configuration(server=ACCOUNT_SERVER, credentials=creds)
account = Account(primary_smtp_address=ACCOUNT_EMAIL, config=config, autodiscover=False, access_type=DELEGATE)
print('connecting ms exchange server account...')
print(type(account))
print(dir(account))
account.root.refresh()
Here is the error I am getting:
TransportError: Unknown failure
Retry: 0
Waited: 10
Timeout: 120
Session: 26271
Thread: 15248
Auth type: <requests_ntlm.requests_ntlm.HttpNtlmAuth object at 0x00000259AA1BD588>
URL: https://oa.company.com/EWS/Exchange.asmx
HTTP adapter: <requests.adapters.HTTPAdapter object at 0x00000259AA0DB7B8>
Allow redirects: False
Streaming: False
Response time: 0.28100000000085856
Status code: 403
Request headers: {'User-Agent': 'python-requests/2.21.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'Keep-Alive', 'Content-Type': 'text/xml; charset=utf-8', 'Content-Length': '469', 'Authorization': 'NTLM TlRMTVNTUAADAAAAGAAYAG0AAAAOAQ4BhQAAAAwADABYAAAACQAJAGQAAAAAAAAAbQAAABAAEACTAQAANoKJ4gYBsR0AAAAP7Pyb+wBnMdrlhr4FKVqPbklDSUNJQkFOS0xURElQUlUzODE5MAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJTLmMBLPHowOJZ46XDs+4ABAQAAAAAAAAZdRefQLNUB+Fc6Z26oxvgAAAAAAgAYAEkAQwBJAEMASQBCAEEATgBLAEwAVABEAAEAEgBIAFkARABFAFgAQwBIADAAOAAEACAAaQBjAGkAYwBpAGIAYQBuAGsAbAB0AGQALgBjAG8AbQADADQASABZAEQARQBYAEMASAAwADgALgBpAGMAaQBjAGkAYgBhAG4AawBsAHQAZAAuAGMAbwBtAAUAIABpAGMAaQBjAGkAYgBhAG4AawBsAHQAZAAuAGMAbwBtAAcACAAGXUXn0CzVAQYABAACAAAACgAQAD9EWlwiiUs304wucsxnkyQAAAAAAAAAALfelDwG05hYOMUqY/e60PY=', 'Cookie': 'ClientId=SINZWMOJKWSKDGEKASFG; expires=Fri, 26-Jun-2020 10:13:02 GMT; path=/; HttpOnly'}
Response headers: {'Cache-Control': 'private', 'Server': 'Microsoft-IIS/8.5', 'request-id': 'ae4dee8d-34e0-471c-8252-b8c1056c8ea0', 'X-CalculatedBETarget': 'pqrexch05.domain.com', 'X-DiagInfo': 'PQREXCH05', 'X-BEServer': 'PQREXCH05', 'X-AspNet-Version': '4.0.30319', 'Set-Cookie': 'exchangecookie=681afc8a0905459182363cce9a98d021; expires=Sat, 27-Jun-2020 10:13:02 GMT; path=/; HttpOnly, X-BackEndCookie=S-1-5-21-1343024091-725345543-504838010-1766210=u56Lnp2ejJqBy87Iysqem5nSy8mbnNLLyZ7H0sfIysbSy5vMz8qdzcvPnpzHgYHNz87G0s/I0s3Iq87Pxc7Mxc/N; expires=Sat, 27-Jul-2019 10:13:02 GMT; path=/EWS; secure; HttpOnly', 'Persistent-Auth': 'true', 'X-Powered-By': 'ASP.NET', 'X-FEServer': 'PQREXCH05', 'Date': 'Thu, 27 Jun 2019 10:13:01 GMT', 'Content-Length': '0'}
Request data: b'<?xml version=\'1.0\' encoding=\'utf-8\'?>\n<s:Envelope xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types"><s:Header><t:RequestServerVersion Version="Exchange2013_SP1"/></s:Header><s:Body><m:ResolveNames ReturnFullContactData="false"><m:UnresolvedEntry>ICICIBANKLTD\\IPRU38190</m:UnresolvedEntry></m:ResolveNames></s:Body></s:Envelope>'
Response data: b''

You might need to configure access policy for EWS using PowerShell.
For example (to allow all apps to use REST and EWS):
Set-OrganizationConfig -EwsApplicationAccessPolicy EnforceBlockList -EwsBlockList $null
Taken from Microsoft docs on Set-OrganizationConfig.
Please search for EwsApplicationAccessPolicy in the above link for more granular access control examples.

Related

Cypress: intercept a network request with compression-type gzip to simulate mapbox

I am currently having trouble with mocking a particular request mapbox-gl is making. When the map is loaded from mapbox pbf-files are being requested and i have not been able to mock this.
My guess is that the core issue is that there seems to be an open bug with cypress issue-16420.
I tried alot of different intercept variants. I tried all kinds of response headers. I gziped, compressed, brd the file that I serve via fixture. I tried different encodings for the fixture. Nothing worked. One of the interceptors looks basically like this
cy.intercept({
method: 'GET',
url: '**/fonts/v1/mapbox/DIN%20Offc%20Pro%20Italic,Arial%20Unicode%20MS%20Regular/0-255.pbf?*',
}, {
fixture: 'fonts/italic.arial.0-255.pbf,binary',
statusCode: 204,
headers: {
'Connection': 'keep-alive',
'Keep-Alive': 'timeout=5',
'Transfer-Encoding': 'chunked',
'access-control-allow-origin': '*',
'access-control-expose-headers': 'Link',
'age': '11631145',
'cache-control': 'max-age=31536000',
'content-encoding': 'compress',
'content-type': 'application/x-protobuf',
'date': 'Sat, 19 Feb 2022 20:46:43 GMT',
'etag': 'W/"b040-+eCb/OHkPqToOcONTDlvpCrjmvs"',
'via': '1.1 4dd80d99fd5d0f6baaaf5179cd921f72.cloudfront.net (CloudFront)',
'x-amz-cf-id': '4uY9rjBgR_R12nkfHFrBMLEpNuWygW9DkmODlMEzwJHABTGCGg8pww==',
'x-amz-cf-pop': 'FRA56-P7',
'x-cache': 'Hit from cloudfront',
'x-origin': 'Mbx-Fonts'
}
}).as('get.0-255.pbf').as('getItalicArial0-255');
Now even if this is a bug there has to be some kind of workaround to serve the file in a cypress test without having an active internet connection. It would be great not having to rely on the network on tests. So all kinds of workarounds and dirty tricks are welcome in making this intercept work.

Making a batch API request to obtain several email bodies in MIME format

Outlook REST API provides an endpoint /me/messages/:message_id:/$value for requesting "raw" emails bodies (old good MIME-encoded email bodies). However, when I try to request bodies of several emails, using the Batch API (https://outlook.office.com/api/v2.0/$batch), I get only a single body in the response (the body of the first email in request I assume). The request body example:
--07edf27e-0552-45db-9a75-add15681b61b
Content-Type: application/http
MIME-Version: 1.0
Content-Transfer-Encoding: binary
GET /api/v2.0/me/messages/AAMkADllYjZhMmZhLWRiMWQtNGY5OS1hM2NiLTA3NDZmODA4NzZiMQBGAAAAAABdR0VqIHc_T6bmj6YLprUhBwDPSIJ3m0xMRZntj-YB8UfpAAAAAAEMAADPSIJ3m0xMRZntj-YB8UfpAAOHtNrzAAA=/$value HTTP/1.1
--07edf27e-0552-45db-9a75-add15681b61b
Content-Type: application/http
MIME-Version: 1.0
Content-Transfer-Encoding: binary
GET /api/v2.0/me/messages/AAMkADllYjZhMmZhLWRiMWQtNGY5OS1hM2NiLTA3NDZmODA4NzZiMQBGAAAAAABdR0VqIHc_T6bmj6YLprUhBwDPSIJ3m0xMRZntj-YB8UfpAAAAAAEMAADPSIJ3m0xMRZntj-YB8UfpAAOHtNryAAA=/$value HTTP/1.1
--07edf27e-0552-45db-9a75-add15681b61b
Content-Type: application/http
MIME-Version: 1.0
Content-Transfer-Encoding: binary
GET /api/v2.0/me/messages/AAMkADllYjZhMmZhLWRiMWQtNGY5OS1hM2NiLTA3NDZmODA4NzZiMQBGAAAAAABdR0VqIHc_T6bmj6YLprUhBwDPSIJ3m0xMRZntj-YB8UfpAAAAAAEMAADPSIJ3m0xMRZntj-YB8UfpAAOE7qpYAAA=/$value HTTP/1.1
--07edf27e-0552-45db-9a75-add15681b61b--
And response is of 'plain\text' Content-Type. Request/Response details:
{'Content-Length': '998', 'Accept-Encoding': 'gzip, deflate', 'Accept': u'application/json', 'User-Agent': u'Nimble', 'Connection': 'keep-alive', 'Content-Type': 'multipart/mixed; boundary="e91fd766-955d-4e87-9204-b28b004d3e5e"', 'Authorization': u'Bearer eyJ0eXAiOiJK..'}
ipdb> resp.request.url
'https://outlook.office.com/api/v2.0/me/$batch'
ipdb> resp.headers
{'X-CalculatedBETarget': 'VI1PR09MB2479.eurprd09.prod.outlook.com', 'Rate-Limit-Reset': '2020-02-06T18:49:26.260Z', 'X-Powered-By': 'ASP.NET', 'Transfer-Encoding': 'chunked', 'X-RUM-Validated': '1', 'X-BackEndHttpStatus': '200, 200', 'X-Proxy-RoutingCorrectness': '1', 'X-BeSku': 'WCS5', 'Date': 'Thu, 06 Feb 2020 18:39:10 GMT', 'Rate-Limit-Limit': '10000', 'Rate-Limit-Remaining': '9999', 'X-AspNet-Version': '4.0.30319', 'Server': 'Microsoft-IIS/10.0', 'X-BEServer': 'VI1PR09MB2479', 'X-DiagInfo': 'VI1PR09MB2479', 'X-FEServer': 'VI1PR0502CA0025, MWHPR10CA0071', 'X-MailboxGuid': '9eb6a2fa-db1d-4f99-a3cb-0746f80876b1', 'X-FEProxyInfo': 'VI1PR0502CA0025.EURPRD05.PROD.OUTLOOK.COM', 'x-ms-appId': 'b27e9f81-dbc4-4d4a-923a-3eacaae45241', 'request-id': 'ff7b1bce-5d41-450f-800a-d2de6f9ae2f0', 'X-Proxy-BackendServerStatus': '200', 'X-CalculatedFETarget': 'VI1PR0502CU001.internal.outlook.com', 'Cache-Control': 'private', **'Content-Type': 'text/plain'**}
If performing the same request but without requesting "raw" bodies, the response becomes of expected content-type: 'Content-Type': 'multipart/mixed; boundary=batchresponse_e4ca25bf-f9e7-4d69-bc09-139bf2776db0'. The request body:
--006b59ee-e662-46d8-be97-a21a24cc9b08
Content-Type: application/http
MIME-Version: 1.0
Content-Transfer-Encoding: binary
GET /api/v2.0/me/messages/AAMkADllYjZhMmZhLWRiMWQtNGY5OS1hM2NiLTA3NDZmODA4NzZiMQBGAAAAAABdR0VqIHc_T6bmj6YLprUhBwDPSIJ3m0xMRZntj-YB8UfpAAAAAAEMAADPSIJ3m0xMRZntj-YB8UfpAAOHtNrzAAA= HTTP/1.1
--006b59ee-e662-46d8-be97-a21a24cc9b08
Content-Type: application/http
MIME-Version: 1.0
Content-Transfer-Encoding: binary
GET /api/v2.0/me/messages/AAMkADllYjZhMmZhLWRiMWQtNGY5OS1hM2NiLTA3NDZmODA4NzZiMQBGAAAAAABdR0VqIHc_T6bmj6YLprUhBwDPSIJ3m0xMRZntj-YB8UfpAAAAAAEMAADPSIJ3m0xMRZntj-YB8UfpAAOHtNryAAA= HTTP/1.1
--006b59ee-e662-46d8-be97-a21a24cc9b08
Content-Type: application/http
MIME-Version: 1.0
Content-Transfer-Encoding: binary
GET /api/v2.0/me/messages/AAMkADllYjZhMmZhLWRiMWQtNGY5OS1hM2NiLTA3NDZmODA4NzZiMQBGAAAAAABdR0VqIHc_T6bmj6YLprUhBwDPSIJ3m0xMRZntj-YB8UfpAAAAAAEMAADPSIJ3m0xMRZntj-YB8UfpAAOE7qpYAAA= HTTP/1.1
--006b59ee-e662-46d8-be97-a21a24cc9b08--
ipdb> resp.headers
{'X-CalculatedBETarget': 'VI1PR09MB2479.eurprd09.prod.outlook.com', 'Rate-Limit-Reset': '2020-02-06T18:49:26.260Z', 'X-Powered-By': 'ASP.NET', 'Transfer-Encoding': 'chunked', 'X-RUM-Validated': '1', 'OData-Version': '4.0', 'X-BackEndHttpStatus': '200, 200', 'X-Proxy-RoutingCorrectness': '1', 'X-BeSku': 'WCS5', 'Date': 'Thu, 06 Feb 2020 18:45:54 GMT', 'Rate-Limit-Limit': '10000', 'Rate-Limit-Remaining': '9984', 'X-AspNet-Version': '4.0.30319', 'Server': 'Microsoft-IIS/10.0', 'X-BEServer': 'VI1PR09MB2479', 'X-DiagInfo': 'VI1PR09MB2479', 'X-FEServer': 'VI1PR03CA0059, MWHPR04CA0069', 'X-MailboxGuid': '9eb6a2fa-db1d-4f99-a3cb-0746f80876b1', 'X-FEProxyInfo': 'VI1PR03CA0059.EURPRD03.PROD.OUTLOOK.COM', 'x-ms-appId': 'b27e9f81-dbc4-4d4a-923a-3eacaae45241', 'request-id': '7f5dcc0f-4109-48a1-9948-39a5ad7471e6', 'X-Proxy-BackendServerStatus': '200', 'X-CalculatedFETarget': 'VI1PR03CU002.internal.outlook.com', 'Cache-Control': 'private', 'Content-Type': 'multipart/mixed; boundary=batchresponse_e4ca25bf-f9e7-4d69-bc09-139bf2776db0'}
ipdb> resp.request.url
'https://outlook.office.com/api/v2.0/me/$batch'
Is it a bug in Outlook REST API? The same request, but to the Graph API works correctly (unfortunately we can't use the Graph API in production).

Python requests text only returning  instead of HTML

I'm trying to scrape the link to a file to download later from a website.
My code:
outage_page = 'https://www.oasis.oati.com/cgi-bin/webplus.dll?script=/woa/woa-planned-outages-report.html&Provider=MISO'
s = requests.Session()
req = s.get(outage_page, stream=True, verify='my cert path is here')
print(req, '\n', req.headers, '\n', req.raw, '\n', req.encoding, '\n', req.content, '\n', req.text)
This is the output I get:
{'Content-Type': 'text/html', 'Content-Encoding': 'gzip', 'Vary': 'Accept-Encoding', 'Server': 'Microsoft-IIS/7.5', 'X-Powered-By': 'ASP.NET', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Date': 'Mon, 26 Aug 2019 15:48:39 GMT', 'Content-Length': '136'}
ISO-8859-1
b'\xef\xbb\xbf\xef\xbb\xbf\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n \r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n'

Process finished with exit code 0
I expected req.text to return the html I could scrape, but it only returns . The other print statements are just for reference here. What am I doing wrong?
I'm going to go ahead and post my solution. So I converted my certificate file from .cer to .pem, included the cert in the session instead of the get and added headers to the request. I changed verify to false because it refers to server side certificate not client side.
# create the connection
s = requests.Session()
s.cert = 'path/to/cert.pem'
head = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36'
}
req = s.get(outage_page, headers=head, verify=False)

After I getting a set-cookie in response, is not saved and transmitted in requests

Basically after an auth, I setting a cookie, but apparently after page refresh on the cookie that was set by cloudflare is saved
And the cookie that I transmitted with set-cookie is not used in after set-cookie requests
# Response headers
HTTP/2.0 200 OK
date: Thu, 18 Jul 2019 10:03:25 GMT
content-type: application/json; charset=utf-8
content-length: 29
set-cookie: __cfduid=d578c7a5e4378dc1b1946964a08ebc4ec1563444205; expires=Fri, 17-Jul-20 10:03:25 GMT; path=/; domain=.doc.io; HttpOnly; Secure
set-cookie: __doc=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlIjoiNzQ2NTczNzQ0MDY2NjE3Mzc0NmQ2MTY5NmMyZTZkNzgiLCJwIjoiNTUzMjQ2NzM2NDQ3NTY2YjU4MzEzODMzNTU0NjUwNTU2ZjRiMzkzMTY3NDUzNDY5NDc3MzM3MzgzOTU5MzczMDUxNjk0ZjQxNjQ0OTM5Nzg0YjZiNzU1Njc3Nzk0NDc0NjE3NDMxNTE0NzcwMzE0YjQxNmY1MjU5MzM3YTZhNDU2NDJiNmU0ZTc0NGE3NTMyNTQ1ODc2NjI1YTczNDc1MTQ1Njc0MjVhNGQ0MTNkM2QiLCJkIjoiMzEzNTM2MzMzNDM0MzQzMjMwMzUzNTM2MzMiLCJpYXQiOjE1NjM0NDQyMDV9.go1jDpc2rBe5FjK2sKX4ybW4PhCPFq1xT1WIX-mSI84; Domain=.doc.io; Path=/; Expires=Thu, 18 Jul 2019 16:03:25 GMT; HttpOnly; Secure
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 4f83a02a3dc36455-FRA
X-Firefox-Spdy: h2
reply
.code(200)
.header('Access-Control-Allow-Origin', '*')
.header('Content-Type', 'application/json; charset=utf-8')
.setCookie('__doc', token, {
domain: '.doc.io',
path: '/',
secure: true,
httpOnly: true,
expires: new Date(new Date().setHours(new Date().getHours() + 6))})
.send({ 'success': 'Sign In success' })
All my websites are https
First I do POST request for an auth on /auth, and you could see response in response headers above and after I do GET on (trying to load page) from /page and get cookies, but with reply.log.info(request.cookies) I see only cookies from cloudflare. Surely I tried to refresh and go to address in different table, there just no any cookies, but from cloudflare.
# Request headers
Host: test.doc.io
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Cookie: __cfduid=d578c7a5e4378dc1b1946964a08ebc4ec1563444205
Upgrade-Insecure-Requests: 1
TE: Trailers
After I tested it with an actual page with js code for a XHR request it works fine. However cookie was only displayed, but not actually saved in the storage when I was sending request directly in Firefox's inspector. Lost a day to figure out that created in inspector cookies seems preventing from installing

Request email audit export fails with status 400 and "Premature end of file."

according to https://developers.google.com/admin-sdk/email-audit/#creating_a_mailbox_for_export I am trying to request the email audit export of an user in G Suite this way:
def requestAuditExport(account):
credentials = getCredentials()
http = credentials.authorize(httplib2.Http())
url = 'https://apps-apis.google.com/a/feeds/compliance/audit/mail/export/helpling.com/'+account
status, response = http.request(url, 'POST', headers={'Content-Type': 'application/atom+xml'})
print(status)
print(response)
And I get the following result:
{'content-length': '22', 'expires': 'Tue, 13 Dec 2016 14:19:37 GMT', 'date': 'Tue, 13 Dec 2016 14:19:37 GMT', 'x-frame-options': 'SAMEORIGIN', 'transfer-encoding': 'chunked', 'x-xss-protection': '1; mode=block', 'content-type': 'text/html; charset=UTF-8', 'x-content-type-options': 'nosniff', '-content-encoding': 'gzip', 'server': 'GSE', 'status': '400', 'cache-control': 'private, max-age=0', 'alt-svc': 'quic=":443"; ma=2592000; v="35,34"'}
b'Premature end of file.'
I cannot see where the problem is, can someone please give me a hint?
Thanks in advance!
Kay
Fix it by going intp the Admin Console, Manage API client access page under Security and add the Client ID, scope needed for the Directory API. For more information, check this document.
Okay, found out what was wrong and fixed it myself. Finally it looks like this:
http = getCredentials().authorize(httplib2.Http())
url = 'https://apps-apis.google.com/a/feeds/compliance/audit/mail/export/helpling.com/'+account
headers = {'Content-Type': 'application/atom+xml'}
xml_data = """<atom:entry xmlns:atom='http://www.w3.org/2005/Atom' xmlns:apps='http://schemas.google.com/apps/2006'> \
<apps:property name='includeDeleted' value='true'/> \
</atom:entry>"""
status, response = http.request(url, 'POST', headers=headers, body=xml_data)
Not sure if it was about the body or the header. It works now and I hope it will help others.
Thanks anyway.

Resources