Apparently there's something I don't know about HEAD requests.
Here's the URL: 'https://theweekinchess.com/assets/files/pgn/eurbli22.pgn', which I'll refer to as <URL> below.
If I curl this, I see a last-modified entry in the headers:
curl --head <URL>
HTTP/2 200
last-modified: Sun, 18 Dec 2022 18:07:16 GMT
accept-ranges: bytes
content-length: 1888745
host-header: c2hhcmVkLmJsdWVob3N0LmNvbQ==
content-type: application/x-chess-pgn
date: Wed, 11 Jan 2023 23:09:14 GMT
server: Apache
But if I make a HEAD request in Node using https, That information is missing:
https.request(<URL>, { method: 'HEAD' }, res => {
console.log([<URL>, res.headers])}).end()
This returns:
[
<URL>
{
date: 'Wed, 11 Jan 2023 23:16:15 GMT',
server: 'Apache',
p3p: 'CP="NOI NID ADMa OUR IND UNI COM NAV"',
'cache-control': 'private, must-revalidate',
'set-cookie': [
'evof3sqa=4b412b5913b38669fc928a0cca9870e4; path=/; secure; HttpOnly'
],
upgrade: 'h2,h2c',
connection: 'Upgrade, Keep-Alive',
'host-header': 'c2hhcmVkLmJsdWVob3N0LmNvbQ==',
'keep-alive': 'timeout=5, max=75',
'content-type': 'text/html; charset=UTF-8'
}
]
I tried axios instead of https:
const response = await axios.head('https://theweekinchess.com/assets/files/pgn/eurbli22.pgn');
console.log({response: response.headers})
And that works (incl. the proper MIME type):
date: 'Thu, 12 Jan 2023 20:00:48 GMT',
server: 'Apache',
upgrade: 'h2,h2c',
connection: 'Upgrade, Keep-Alive',
'last-modified': 'Sun, 18 Dec 2022 18:07:16 GMT',
'accept-ranges': 'bytes',
'content-length': '1888745',
'host-header': 'c2hhcmVkLmJsdWVob3N0LmNvbQ==',
'keep-alive': 'timeout=5, max=75',
'content-type': 'application/x-chess-pgn'
I also tried waiting for re2.on('end', console.log(res.headers)), but same output as before.
I'm going to close this issue and post it instead as a 'bug' on Node's site. I'm sure there's something that needs to be changed in how I'm executing the HEAD request.
I need to download Kaggle's NIH Chest X-rays Dataset programmatically, specifically the Data_Entry_2017.csv.
I want to download it if doesn't exist on system, and redownload it if it's updation date exceeds last download date.
I know I can do this manually, but I'll appreciate if there was a way to do it programmatically.
I have done:
Direct download:
kaggle.api.dataset_download_file(dataset='nih-chest-xrays/data', file_name='Data_Entry_2017.csv', path='data/')
This gives:
File "C:\Program Files\Python311\Lib\site-packages\kaggle\rest.py", line 241, in request
raise ApiException(http_resp=r)
kaggle.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Sun, 18 Dec
2022 03:35:46 GMT', 'Access-Control-Allow-Credentials': 'true', 'Set-Cookie':
'ka_sessionid=bdaa2b71c677d6e48bf93fc71b85a9c6; max-age=2626560; path=/, GCLB=CIeroe2g5sixeA;
path=/; HttpOnly', 'Transfer-Encoding': 'chunked', 'Vary': 'Accept-Encoding', 'Turbolinks-
Location': 'https://www.kaggle.com/api/v1/datasets/download/nih-chest-
xrays/data/Data_Entry_2017.csv', 'X-Kaggle-MillisecondsElapsed': '526', 'X-Kaggle-RequestId':
'03383f8cdac3aa4b6ab83d5dadbc507c', 'X-Kaggle-ApiVersion': '1.5.12', 'X-Frame-Options':
'SAMEORIGIN', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload',
'Content-Security-Policy': "object-src 'none'; script-src 'nonce-gn6A4mfrbofbZRyE9gREnA=='
'report-sample' 'unsafe-inline' 'unsafe-eval' 'strict-dynamic' https: http:; frame-src 'self'
https://www.kaggleusercontent.com https://www.youtube.com/embed/ https://polygraph-cool.github.io
https://www.google.com/recaptcha/ https://form.jotform.com https://submit.jotform.us
https://submit.jotformpro.com https://submit.jotform.com https://www.docdroid.com
https://www.docdroid.net https://kaggle-static.storage.googleapis.com https://kaggle-static-
staging.storage.googleapis.com https://kkb-dev.jupyter-proxy.kaggle.net https://kkb-
staging.jupyter-proxy.kaggle.net https://kkb-production.jupyter-proxy.kaggle.net https://kkb-
dev.firebaseapp.com https://kkb-staging.firebaseapp.com
https://kkb-production.firebaseapp.com https://kaggle-metastore-test.firebaseapp.com
https://kaggle-metastore.firebaseapp.com https://apis.google.com https://content-
sheets.googleapis.com/ https://accounts.google.com/ https://storage.googleapis.com
https://docs.google.com https://drive.google.com https://calendar.google.com/;
base-uri 'none'; report-uri https://csp.withgoogle.com/csp/kaggle/20201130;", 'X-Content-Type-
Options': 'nosniff', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'Via': '1.1 google',
'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
HTTP response body: b'{"code":404,"message":"Not found"}'
Same for any other files in this dataset:
kaggle.api.dataset_download_file(dataset='nih-chest-xrays/data', file_name='test_list.txt', path='data/')
Which again gives
kaggle.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Sun, 18 Dec
2022 03:37:20 GMT', 'Access-Control-Allow-Credentials': 'true', 'Set-Cookie':
'ka_sessionid=b59676e67aa4a36e92657a7fa50e95f3; max-age=2626560; path=/, GCLB=CKOV0bPZ-p6ZSA;
path=/; HttpOnly', 'Transfer-Encoding': 'chunked', 'Vary': 'Accept-Encoding', 'Turbolinks-
Location': 'https://www.kaggle.com/api/v1/datasets/download/nih-chest-xrays/data/test_list.txt',
'X-Kaggle-MillisecondsElapsed': '520', 'X-Kaggle-RequestId': '79fde70e368d73ba7c767a007bacc086',
'X-Kaggle-ApiVersion': '1.5.12', 'X-Frame-Options': 'SAMEORIGIN', 'Strict-Transport-Security':
'max-age=63072000; includeSubDomains; preload', 'Content-Security-Policy': "object-src 'none';
script-src 'nonce-U88j/npGgHfKpSCwewojZg==' 'report-sample' 'unsafe-inline' 'unsafe-eval'
'strict-dynamic' https: http:; frame-src 'self' https://www.kaggleusercontent.com
https://www.youtube.com/embed/ https://polygraph-cool.github.io https://www.google.com/recaptcha/
https://form.jotform.com https://submit.jotform.us https://submit.jotformpro.com
https://submit.jotform.com https://www.docdroid.com https://www.docdroid.net https://kaggle-
static.storage.googleapis.com https://kaggle-static-staging.storage.googleapis.com https://kkb-
dev.jupyter-proxy.kaggle.net https://kkb-staging.jupyter-proxy.kaggle.net https://kkb-
production.jupyter-proxy.kaggle.net https://kkb-dev.firebaseapp.com https://kkb-
staging.firebaseapp.com https://kkb-production.firebaseapp.com https://kaggle-metastore-
test.firebaseapp.com https://kaggle-metastore.firebaseapp.com https://apis.google.com
https://content-sheets.googleapis.com/ https://accounts.google.com/
https://storage.googleapis.com https://docs.google.com https://drive.google.com
https://calendar.google.com/; base-uri 'none'; report-uri
https://csp.withgoogle.com/csp/kaggle/20201130;", 'X-Content-Type-Options': 'nosniff', 'Referrer-
Policy': 'strict-origin-when-cross-origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443";
ma=2592000,h3-29=":443"; ma=2592000'})
HTTP response body: b'{"code":404,"message":"Not found"}'
The strange thing is that I can search this dataset:
dataset_list = kaggle.api.dataset_list(search='nih')
print(dataset_list)
Which gives:
[nih-chest-xrays/data, nih-chest-xrays/sample, allen-institute-for-ai/CORD-19-research-challenge,
kmader/nih-deeplesion-subset, nickuzmenkov/nih-chest-xrays-tfrecords, iarunava/cell-images-for-
detecting-malaria, tunguz/covid19-genomes, nlm-nih/nlm-rxnorm, akhileshdkapse/nih-image-600x600-
data, tunguz/nih-awarded-grant-text, danielmadmon/nih-xray-dataset-tfrec-with-labels,
miracle9to9/files1, nlm-nih/rxnorm-drug-name-conventions, kmader/rsna-bone-age,
redwankarimsony/chestxray8-dataframe, akhileshdkapse/nih-dataframe, luigisaetta/cxr-tfrec256-
may2020, kokyew93/nihdata, ammarali32/startingpointschestx, dannellyz/cancer-incidence-totals-
and-rates-per-us-county]
It's literally the first one on the list.
And checking its metadata:
dataset = vars(dataset_list[0])
print(dataset)
Which gives:
{'subtitleNullable': 'Over 112,000 Chest X-ray images from more than 30,000 unique patients',
'creatorNameNullable': 'Timo Bozsolik', 'creatorUrlNullable': 'timoboz', 'totalBytesNullable':
45096150231, 'urlNullable': 'https://www.kaggle.com/datasets/nih-chest-xrays/data',
'licenseNameNullable': 'CC0: Public Domain', 'descriptionNullable': None, 'ownerNameNullable':
'National Institutes of Health Chest X-Ray Dataset', 'ownerRefNullable': 'nih-chest-xrays',
'titleNullable': 'NIH Chest X-rays', 'currentVersionNumberNullable': 3,
'usabilityRatingNullable': 0.7352941, 'id': 5839, 'ref': 'nih-chest-xrays/data', 'subtitle':
'Over 112,000 Chest X-ray images from more than 30,000 unique patients', 'hasSubtitle': True,
'creatorName': 'Timo Bozsolik', 'hasCreatorName': True, 'creatorUrl': 'timoboz', 'hasCreatorUrl':
True, 'totalBytes': 45096150231, 'hasTotalBytes': True, 'url':
'https://www.kaggle.com/datasets/nih-chest-xrays/data', 'hasUrl': True, 'lastUpdated':
datetime.datetime(2018, 2, 21, 20, 52, 23), 'downloadCount': 65214, 'isPrivate': False,
'isFeatured': False, 'licenseName': 'CC0: Public Domain', 'hasLicenseName': True, 'description':
'', 'hasDescription': False, 'ownerName': 'National Institutes of Health Chest X-Ray Dataset',
'hasOwnerName': True, 'ownerRef': 'nih-chest-xrays', 'hasOwnerRef': True, 'kernelCount': 323,
'title': 'NIH Chest X-rays', 'hasTitle': True, 'topicCount': 0, 'viewCount': 454490, 'voteCount':
967, 'currentVersionNumber': 3, 'hasCurrentVersionNumber': True, 'usabilityRating': 0.7352941,
'hasUsabilityRating': True, 'tags': [biology, health, medicine, computer science, software,
health conditions], 'files': [], 'versions': [], 'size': '42GB'}
I can even list the files for that dataset, and it's visible:
print(kaggle.api.dataset_list_files('nih-chest-xrays/data').files)
Which gives:
[BBox_List_2017.csv, LOG_CHESTXRAY.pdf, ARXIV_V5_CHESTXRAY.pdf, README_CHESTXRAY.pdf,
train_val_list.txt, Data_Entry_2017.csv, FAQ_CHESTXRAY.pdf, test_list.txt]
A problem I noticed was that kaggle.api.dataset_download_file was only fetching from Version 1, where these files don't exist. This was further confirmed when I successfully fetched an image file from Version 1:
res = kaggle.api.dataset_download_file(dataset='nih-chest-xrays/data', file_name='images_001/images/00000001_000.png', path='data/')
print(res)
Which downloads the image and prints True.
My data is Version 3. Is there any way to configure kaggle to use Version 3?
Note that I can download the whole 45GB dataset, but I only require the single file Data_Entry_2017.csv.
Even more strangely, it's also throwing an error when checking its status:
print(kaggle.api.dataset_status(dataset='nih-chest-xrays/data'))
Which gives:
Traceback (most recent call last):
File "C:\Users\jaide\OneDrive\Documents\MScCS2022\Data-Mining-I\chest-datasets\visualise.py",
line 117, in <module>
full_df = load_nih_data()
^^^^^^^^^^^^^^^
File "C:\Users\jaide\OneDrive\Documents\MScCS2022\Data-Mining-I\chest-datasets\visualise.py",
line 89, in load_nih_data
print(kaggle.api.dataset_status(dataset='nih-chest-xrays/data'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\kaggle\api\kaggle_api_extended.py", line
1135, in dataset_status
self.datasets_status_with_http_info(owner_slug=owner_slug,
File "C:\Program Files\Python311\Lib\site-packages\kaggle\api\kaggle_api.py", line 1910, in
datasets_status_with_http_info
return self.api_client.call_api(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\kaggle\api_client.py", line 329, in call_api
return self.__call_api(resource_path, method,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\kaggle\api_client.py", line 161, in
__call_api
response_data = self.request(
^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\kaggle\api_client.py", line 351, in request
return self.rest_client.GET(url,
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\kaggle\rest.py", line 247, in GET
return self.request("GET", url,
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\kaggle\rest.py", line 241, in request
raise ApiException(http_resp=r)
kaggle.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Sun, 18 Dec
2022 04:14:32 GMT', 'Access-Control-Allow-Credentials': 'true', 'Set-Cookie':
'ka_sessionid=1ab6e66561c7d9ae43ead26df25beea3; max-age=2626560; path=/, GCLB=CNS4vKLYyPuCpgE;
path=/; HttpOnly', 'Transfer-Encoding': 'chunked', 'Vary':
'Accept-Encoding', 'Turbolinks-Location': 'https://www.kaggle.com/api/v1/datasets/status/nih-
chest-xrays/data', 'X-Kaggle-MillisecondsElapsed': '46', 'X-Kaggle-RequestId':
'66473cc55f62c39d2deb0aa4cd2953a7', 'X-Kaggle-ApiVersion': '1.5.12', 'X-Frame-Options':
'SAMEORIGIN', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload',
'Content-Security-Policy': "object-src 'none'; script-src 'nonce-5AJVxKPJRg4Gsi/iT3D5Aw=='
'report-sample' 'unsafe-inline' 'unsafe-eval' 'strict-dynamic' https: http:; frame-src 'self'
https://www.kaggleusercontent.com https://www.youtube.com/embed/ https://polygraph-cool.github.io
https://www.google.com/recaptcha/ https://form.jotform.com https://submit.jotform.us
https://submit.jotformpro.com https://submit.jotform.com https://www.docdroid.com
https://www.docdroid.net https://kaggle-static.storage.googleapis.com https://kaggle-static-
staging.storage.googleapis.com https://kkb-dev.jupyter-proxy.kaggle.net https://kkb-
staging.jupyter-proxy.kaggle.net https://kkb-production.jupyter-proxy.kaggle.net https://kkb-
dev.firebaseapp.com https://kkb-staging.firebaseapp.com https://kkb-production.firebaseapp.com
https://kaggle-metastore-test.firebaseapp.com https://kaggle-metastore.firebaseapp.com
https://apis.google.com https://content-sheets.googleapis.com/ https://accounts.google.com/
https://storage.googleapis.com https://docs.google.com https://drive.google.com
https://calendar.google.com/; base-uri 'none'; report-uri
https://csp.withgoogle.com/csp/kaggle/20201130;", 'X-Content-Type-Options': 'nosniff', 'Referrer-
Policy': 'strict-origin-when-cross-origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443";
ma=2592000,h3-29=":443"; ma=2592000'})
HTTP response body: {"code":404,"message":"Not found"}
CLI download
Check if dataset exists:
C:\Users\jaide\OneDrive\Documents\MScCS2022\Data-Mining-I\chest-datasets>kaggle datasets list -s "nih-chest-xrays/data"
ref title size lastUpdated downloadCount voteCount usabilityRating
------------------------------------------------------------- ----------------------------------------------- ----- ------------------- ------------- --------- ---------------
nickuzmenkov/nih-chest-xrays-tfrecords NIH Chest X-rays TFRecords 11GB 2021-03-09 04:49:23 2121 119 0.9411765
nih-chest-xrays/data NIH Chest X-rays 42GB 2018-02-21 20:52:23 65221 967 0.7352941
nih-chest-xrays/sample Random Sample of NIH Chest X-ray Dataset 4GB 2017-11-23 02:58:24 14930 241 0.7647059
ammarali32/startingpointschestx StartingPoints-ChestX 667MB 2021-02-27 17:35:19 668 68 0.6875
redwankarimsony/chestxray8-dataframe ChestX-ray8_DataFrame 51MB 2020-07-12 02:28:33 416 10 0.64705884
amanullahasraf/covid19-pneumonia-normal-chest-xray-pa-dataset COVID19_Pneumonia_Normal_Chest_Xray_PA_Dataset 2GB 2020-07-13 05:54:22 1482 14 0.8125
harshsoni/nih-chest-xray-tfrecords NIH Chest X-ray TFRecords 42GB 2020-09-26 18:48:04 52 6 0.5294118
amanullahasraf/covid19-pneumonia-normal-chest-xraypa-dataset COVID19_Pneumonia_Normal_Chest_Xray(PA)_Dataset 1GB 2020-06-29 08:28:01 161 3 0.5882353
roderikmogot/nih-chest-x-ray-models nih chest x ray models 2GB 2022-08-17 01:32:33 66 4 0.47058824
ericspod/project-monai-2020-bootcamp-challenge-dataset Project MONAI 2020 Bootcamp Challenge Dataset 481MB 2021-01-25 01:11:42 22 2 0.6875
sunghyunjun/nih-chest-xrays-600-jpg-dataset NIH Chest X-rays 600 JPG Dataset 7GB 2021-03-15 07:53:12 35 3 0.64705884
zhuangjw/chest-xray-cleaned chest_xray_cleaned 2GB 2019-11-30 19:05:24 13 1 0.375
kambarakun/nih-chest-xrays-trained-models NIH Chest X-rays: Trained Models 3GB 2020-04-30 11:52:56 10 2 0.3125
jessevent/all-kaggle-datasets Complete Kaggle Datasets Collection 390KB 2018-01-16 12:32:58 2256 118 0.8235294
vzaguskin/nih-chest-xrays-tfrecords-756 NIH Chest X-rays tfrecords 756 11GB 2021-02-25 14:25:12 3 1 0.29411766
Yes, it is the second one. Now does it have my required file?
C:\Users\jaide\OneDrive\Documents\MScCS2022\Data-Mining-I\chest-datasets>kaggle datasets files "nih-chest-xrays/data"
name size creationDate
---------------------- ----- -------------------
ARXIV_V5_CHESTXRAY.pdf 9MB 2018-02-21 20:52:23
FAQ_CHESTXRAY.pdf 71KB 2018-02-21 20:52:23
Data_Entry_2017.csv 7MB 2018-02-21 20:52:23
BBox_List_2017.csv 90KB 2018-02-21 20:52:23
README_CHESTXRAY.pdf 827KB 2018-02-21 20:52:23
LOG_CHESTXRAY.pdf 4KB 2018-02-21 20:52:23
train_val_list.txt 1MB 2018-02-21 20:52:23
test_list.txt 425KB 2018-02-21 20:52:23
Yes it does. Now download it.
C:\Users\jaide\OneDrive\Documents\MScCS2022\Data-Mining-I\chest-datasets>kaggle datasets download -f "Data_Entry_2017.csv" -p "data/" "nih-chest-xrays/data"
404 - Not Found
Nope, error 404.
How do I download Kaggle's NIH Chest X-rays Dataset programmatically, specifically the Data_Entry_2017.csv?
Any help is appreciated!
I have checked:
I checked out Kaggle Python API, but it doesn't appear to have any way to configure Version.
I checked out Kaggle API on github, but it also doesn't appear to have any ways to configure Version.
Related Questions Read:
Kaggle Dataset Download - is irrelevant.
Download a Kaggle Dataset - is downloading competition files.
How to download data set into Colab? Stuck with a problem, it says "401 - Unauthorized"? - They had api key issues
How to load just one chosen file of a way too large Kaggle dataset from Kaggle into Colab - Is related to kaggle and jupyter notebooks, I am currently just using a python script.
Trouble turning comorbidity data into a table using Python and Pandas - Are again using Kernels (Kaggle Notebooks).
How to download kaggle dataset? - Downloads the whole dataset.
I am currently having trouble with mocking a particular request mapbox-gl is making. When the map is loaded from mapbox pbf-files are being requested and i have not been able to mock this.
My guess is that the core issue is that there seems to be an open bug with cypress issue-16420.
I tried alot of different intercept variants. I tried all kinds of response headers. I gziped, compressed, brd the file that I serve via fixture. I tried different encodings for the fixture. Nothing worked. One of the interceptors looks basically like this
cy.intercept({
method: 'GET',
url: '**/fonts/v1/mapbox/DIN%20Offc%20Pro%20Italic,Arial%20Unicode%20MS%20Regular/0-255.pbf?*',
}, {
fixture: 'fonts/italic.arial.0-255.pbf,binary',
statusCode: 204,
headers: {
'Connection': 'keep-alive',
'Keep-Alive': 'timeout=5',
'Transfer-Encoding': 'chunked',
'access-control-allow-origin': '*',
'access-control-expose-headers': 'Link',
'age': '11631145',
'cache-control': 'max-age=31536000',
'content-encoding': 'compress',
'content-type': 'application/x-protobuf',
'date': 'Sat, 19 Feb 2022 20:46:43 GMT',
'etag': 'W/"b040-+eCb/OHkPqToOcONTDlvpCrjmvs"',
'via': '1.1 4dd80d99fd5d0f6baaaf5179cd921f72.cloudfront.net (CloudFront)',
'x-amz-cf-id': '4uY9rjBgR_R12nkfHFrBMLEpNuWygW9DkmODlMEzwJHABTGCGg8pww==',
'x-amz-cf-pop': 'FRA56-P7',
'x-cache': 'Hit from cloudfront',
'x-origin': 'Mbx-Fonts'
}
}).as('get.0-255.pbf').as('getItalicArial0-255');
Now even if this is a bug there has to be some kind of workaround to serve the file in a cypress test without having an active internet connection. It would be great not having to rely on the network on tests. So all kinds of workarounds and dirty tricks are welcome in making this intercept work.
I'm trying to read emails from the Microsoft Exchange server using EWS and exchangelib in Python for an email classification problem. But I am unable to connect to the exchange server.
I've tried specifying the version, auth_type, using a certificate (which gives a ssl verify error), using the smtp address in place of the username and it still doesn't connect.
Here is my code:
from exchangelib import Credentials, Account, EWSDateTime, EWSTimeZone, Configuration, DELEGATE, IMPERSONATION, NTLM, ServiceAccount, Version, Build
USER_NAME = 'domain\\user12345'
ACCOUNT_EMAIL = john.doe#ext.companyname.com'
ACCOUNT_PASSWORD = 'John#1234'
ACCOUNT_SERVER = 'oa.company.com'
creds = Credentials(USER_NAME, ACCOUNT_PASSWORD)
config = Configuration(server=ACCOUNT_SERVER, credentials=creds)
account = Account(primary_smtp_address=ACCOUNT_EMAIL, config=config, autodiscover=False, access_type=DELEGATE)
print('connecting ms exchange server account...')
print(type(account))
print(dir(account))
account.root.refresh()
Here is the error I am getting:
TransportError: Unknown failure
Retry: 0
Waited: 10
Timeout: 120
Session: 26271
Thread: 15248
Auth type: <requests_ntlm.requests_ntlm.HttpNtlmAuth object at 0x00000259AA1BD588>
URL: https://oa.company.com/EWS/Exchange.asmx
HTTP adapter: <requests.adapters.HTTPAdapter object at 0x00000259AA0DB7B8>
Allow redirects: False
Streaming: False
Response time: 0.28100000000085856
Status code: 403
Request headers: {'User-Agent': 'python-requests/2.21.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'Keep-Alive', 'Content-Type': 'text/xml; charset=utf-8', 'Content-Length': '469', 'Authorization': 'NTLM TlRMTVNTUAADAAAAGAAYAG0AAAAOAQ4BhQAAAAwADABYAAAACQAJAGQAAAAAAAAAbQAAABAAEACTAQAANoKJ4gYBsR0AAAAP7Pyb+wBnMdrlhr4FKVqPbklDSUNJQkFOS0xURElQUlUzODE5MAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJTLmMBLPHowOJZ46XDs+4ABAQAAAAAAAAZdRefQLNUB+Fc6Z26oxvgAAAAAAgAYAEkAQwBJAEMASQBCAEEATgBLAEwAVABEAAEAEgBIAFkARABFAFgAQwBIADAAOAAEACAAaQBjAGkAYwBpAGIAYQBuAGsAbAB0AGQALgBjAG8AbQADADQASABZAEQARQBYAEMASAAwADgALgBpAGMAaQBjAGkAYgBhAG4AawBsAHQAZAAuAGMAbwBtAAUAIABpAGMAaQBjAGkAYgBhAG4AawBsAHQAZAAuAGMAbwBtAAcACAAGXUXn0CzVAQYABAACAAAACgAQAD9EWlwiiUs304wucsxnkyQAAAAAAAAAALfelDwG05hYOMUqY/e60PY=', 'Cookie': 'ClientId=SINZWMOJKWSKDGEKASFG; expires=Fri, 26-Jun-2020 10:13:02 GMT; path=/; HttpOnly'}
Response headers: {'Cache-Control': 'private', 'Server': 'Microsoft-IIS/8.5', 'request-id': 'ae4dee8d-34e0-471c-8252-b8c1056c8ea0', 'X-CalculatedBETarget': 'pqrexch05.domain.com', 'X-DiagInfo': 'PQREXCH05', 'X-BEServer': 'PQREXCH05', 'X-AspNet-Version': '4.0.30319', 'Set-Cookie': 'exchangecookie=681afc8a0905459182363cce9a98d021; expires=Sat, 27-Jun-2020 10:13:02 GMT; path=/; HttpOnly, X-BackEndCookie=S-1-5-21-1343024091-725345543-504838010-1766210=u56Lnp2ejJqBy87Iysqem5nSy8mbnNLLyZ7H0sfIysbSy5vMz8qdzcvPnpzHgYHNz87G0s/I0s3Iq87Pxc7Mxc/N; expires=Sat, 27-Jul-2019 10:13:02 GMT; path=/EWS; secure; HttpOnly', 'Persistent-Auth': 'true', 'X-Powered-By': 'ASP.NET', 'X-FEServer': 'PQREXCH05', 'Date': 'Thu, 27 Jun 2019 10:13:01 GMT', 'Content-Length': '0'}
Request data: b'<?xml version=\'1.0\' encoding=\'utf-8\'?>\n<s:Envelope xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types"><s:Header><t:RequestServerVersion Version="Exchange2013_SP1"/></s:Header><s:Body><m:ResolveNames ReturnFullContactData="false"><m:UnresolvedEntry>ICICIBANKLTD\\IPRU38190</m:UnresolvedEntry></m:ResolveNames></s:Body></s:Envelope>'
Response data: b''
You might need to configure access policy for EWS using PowerShell.
For example (to allow all apps to use REST and EWS):
Set-OrganizationConfig -EwsApplicationAccessPolicy EnforceBlockList -EwsBlockList $null
Taken from Microsoft docs on Set-OrganizationConfig.
Please search for EwsApplicationAccessPolicy in the above link for more granular access control examples.
I created a new Azure Website, using Umbraco as the CMS. I got a page up and going, and I already have a .co.nz domain with www.domains4less.com. There's a whole lot of stuff on the internet about pointing URLs to Azure, but that seems to be more of a redirection service than anything (i.e. my URLs still use azurewebsites.net once I land on my site).
Has anybody had any luck getting it to go?
Here's the error I get when I try adding the DNS entry to Azure (I'm in reserved mode, reemdairy is the name of the website):
There was an error processing your request. Please try again in a few
moments.
Browser: 5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like
Gecko) Chrome/19.0.1084.56 Safari/536.5 User language: undefined
Portal Version: 6.0.6002.18488 (rd_auxportal_stable.120609-0259)
Subscriptions: 3aabe358-d178-4790-a97b-ffba902b2851 User email
address: touchmaniacs#live.com
Last 10 Requests
message: Failure: Ajax call to: Websites/UpdateConfig. failed with
status: error (500) in 2.57 seconds. x-ms-client-request-id was:
38834edf-c9f3-46bb-a1f7-b2839c692bcf-2012-06-12 22:25:14Z dateTime:
Wed Jun 13 2012 10:25:17 GMT+1200 (New Zealand Standard Time)
durationSeconds: 2.57 url: Websites/UpdateConfig status: 500
textStatus: error clientMsRequestId:
38834edf-c9f3-46bb-a1f7-b2839c692bcf-2012-06-12 22:25:14Z sessionId:
09c72263-6ce7-422b-84d7-4c21acded759 referrer:
https://manage.windowsazure.com/#Workspaces/WebsiteExtension/Website/reemdairy/configure
host: manage.windowsazure.com response: {"message":"Try again. Contact
support if the problem persists.","ErrorMessage":"Try again. Contact
support if the problem
persists.","httpStatusCode":"InternalServerError","operationTrackingId":"","stackTrace":null}
message: Complete: Ajax call to: Websites/GetConfig. completed with
status: success (200) in 1.021 seconds. x-ms-client-request-id was:
a0cdcced-13d0-44e2-866d-e0b061b9461b-2012-06-12 22:24:43Z dateTime:
Wed Jun 13 2012 10:24:44 GMT+1200 (New Zealand Standard Time)
durationSeconds: 1.021 url: Websites/GetConfig status: 200 textStatus:
success clientMsRequestId:
a0cdcced-13d0-44e2-866d-e0b061b9461b-2012-06-12 22:24:43Z sessionId:
09c72263-6ce7-422b-84d7-4c21acded759 referrer:
https://manage.windowsazure.com/#Workspaces/WebsiteExtension/Website/reemdairy/configure
host: manage.windowsazure.com
message: Complete: Ajax call to:
https://manage.windowsazure.com/Service/OperationTracking?subscriptionId=3aabe358-d178-4790-a97b-ffba902b2851.
completed with status: success (200) in 1.887 seconds.
x-ms-client-request-id was:
a7689fe9-b9f9-4d6c-8926-734ec9a0b515-2012-06-12 22:24:40Z dateTime:
Wed Jun 13 2012 10:24:42 GMT+1200 (New Zealand Standard Time)
durationSeconds: 1.887 url:
https://manage.windowsazure.com/Service/OperationTracking?subscriptionId=3aabe358-d178-4790-a97b-ffba902b2851
status: 200 textStatus: success clientMsRequestId:
a7689fe9-b9f9-4d6c-8926-734ec9a0b515-2012-06-12 22:24:40Z sessionId:
09c72263-6ce7-422b-84d7-4c21acded759 referrer:
https://manage.windowsazure.com/#Workspaces/WebsiteExtension/Website/reemdairy/configure
host: manage.windowsazure.com
message: Complete: Ajax call to: /Service/GetUserSettings. completed
with status: success (200) in 0.941 seconds. x-ms-client-request-id
was: 805e554d-1e2e-4214-afd5-be87c0f255d1-2012-06-12 22:24:40Z
dateTime: Wed Jun 13 2012 10:24:40 GMT+1200 (New Zealand Standard
Time) durationSeconds: 0.941 url: /Service/GetUserSettings status: 200
textStatus: success clientMsRequestId:
805e554d-1e2e-4214-afd5-be87c0f255d1-2012-06-12 22:24:40Z sessionId:
09c72263-6ce7-422b-84d7-4c21acded759 referrer:
https://manage.windowsazure.com/#Workspaces/WebsiteExtension/Website/reemdairy/configure
host: manage.windowsazure.com
message: Complete: Ajax call to:
Extensions/ApplicationsExtension/SqlAzure/ClusterSuffix. completed
with status: success (200) in 0.483 seconds. x-ms-client-request-id
was: 85157ceb-c538-40ca-8c1e-5cc07c57240f-2012-06-12 22:24:39Z
dateTime: Wed Jun 13 2012 10:24:40 GMT+1200 (New Zealand Standard
Time) durationSeconds: 0.483 url:
Extensions/ApplicationsExtension/SqlAzure/ClusterSuffix status: 200
textStatus: success clientMsRequestId:
85157ceb-c538-40ca-8c1e-5cc07c57240f-2012-06-12 22:24:39Z sessionId:
09c72263-6ce7-422b-84d7-4c21acded759 referrer:
https://manage.windowsazure.com/#Workspaces/WebsiteExtension/Website/reemdairy/configure
host: manage.windowsazure.com
message: Complete: Ajax call to:
Extensions/ApplicationsExtension/SqlAzure/GetClientIp. completed with
status: success (200) in 0.309 seconds. x-ms-client-request-id was:
2eb194b6-66ca-49e2-9016-e0f89164314c-2012-06-12 22:24:39Z dateTime:
Wed Jun 13 2012 10:24:40 GMT+1200 (New Zealand Standard Time)
durationSeconds: 0.309 url:
Extensions/ApplicationsExtension/SqlAzure/GetClientIp status: 200
textStatus: success clientMsRequestId:
2eb194b6-66ca-49e2-9016-e0f89164314c-2012-06-12 22:24:39Z sessionId:
09c72263-6ce7-422b-84d7-4c21acded759 referrer:
https://manage.windowsazure.com/#Workspaces/WebsiteExtension/Website/reemdairy/configure
host: manage.windowsazure.com
message: Complete: Ajax call to:
Extensions/ApplicationsExtension/SqlAzure/DefaultServerLocation.
completed with status: success (200) in 0.309 seconds.
x-ms-client-request-id was:
1bc165ef-2081-48f2-baed-16c6edf8ea67-2012-06-12 22:24:39Z dateTime:
Wed Jun 13 2012 10:24:40 GMT+1200 (New Zealand Standard Time)
durationSeconds: 0.309 url:
Extensions/ApplicationsExtension/SqlAzure/DefaultServerLocation
status: 200 textStatus: success clientMsRequestId:
1bc165ef-2081-48f2-baed-16c6edf8ea67-2012-06-12 22:24:39Z sessionId:
09c72263-6ce7-422b-84d7-4c21acded759 referrer:
https://manage.windowsazure.com/#Workspaces/WebsiteExtension/Website/reemdairy/configure
host: manage.windowsazure.com
message: Complete: Ajax call to:
Extensions/ApplicationsExtension/SqlAzure/ServerLocations. completed
with status: success (200) in 0.309 seconds. x-ms-client-request-id
was: e1fba7df-6a12-47f8-9434-bf17ca7d93f4-2012-06-12 22:24:39Z
dateTime: Wed Jun 13 2012 10:24:40 GMT+1200 (New Zealand Standard
Time) durationSeconds: 0.309 url:
Extensions/ApplicationsExtension/SqlAzure/ServerLocations status: 200
textStatus: success clientMsRequestId:
e1fba7df-6a12-47f8-9434-bf17ca7d93f4-2012-06-12 22:24:39Z sessionId:
09c72263-6ce7-422b-84d7-4c21acded759 referrer:
https://manage.windowsazure.com/#Workspaces/WebsiteExtension/Website/reemdairy/configure
host: manage.windowsazure.com
Its good that you have Windows Azure Website (Reserve) because only Reserve websites can have CNAME or A record point to your_site.azurewebsites.net.
May I ask what are you setting, CNAME or A Record?
Here is the step by step direction on setting either CNAME (which is suggested for websites and webroles) so please follow and suggest at what step you have an issue:
Configuring a Custom Domain Name in Windows Azure
I've successfully done this with the exact same setup (Umbraco, running on Azure with a domains4less.co.nz domain). DNS setup with Domains4less can be obtuse - but here's how to setup a CNAME to your app:
From 'Manage Domains', hit 'Setup' on the domain you want. Next page, 'Advanced Control Panel' and login. Expand 'Services', open 'Domain Templates' and click 'Add a domain template'
Give your template a name (e.g 'yourdomain.co.nz'), and scroll down to the CNAME Records section. Click 'Add a CNAME record', Prefix field: www, Destination Address: YOURAPPNAME.azurewebsites.net.. Save.
Open 'Domains' in the left-side menu and click '[edit]' on the domain you want to setup, open the 'Advanced Mode' tab and choose your template from the Template drop-down list. Save, done.
Phew.