Generating signed XPI via jpm failed - linux

There was a problem signing an Add-On via jpm: The command
jpm -v sign --api-key 'user:xxxxxxxx:xxx' --api-secret xxxxxxxxxxxxxxxxxxxxxxxxx
failed with the error message
Error: Received bad response from the server while requesting https://addons.mozilla.org/api/v3/addons/%40addonname/versions/0.x.y/
Signing via the web interface worked.
How can this be fixed?
The full verbose output is
JPM [info] binary set to /usr/bin/firefox
JPM [info] verbose set
JPM [info] Checking compatability bootstrap.js and install.rdf for xpi
Validating the manifest
JPM [info] Creating fallbacks if they are necessary..
Creating XPI
JPM [info] Creating XPI...
JPM [info] .jpmignore found
JPM [info] Adding: /path/to/file
[...]
JPM [info] Adding: /path/to/another_file
JPM [info] XPI created at /tmp/tmp-unsigned-xpi-25600yuqzFN3Alxa0/#addonname-0.x.y.xpi (121ms)
JPM [info] Removing fallbacks if they were necessary..
JPM [info] Creating updateRDF...
Created XPI at /tmp/tmp-unsigned-xpi-25600yuqzFN3Alxa0/#addonname-0.x.y.xpi
JPM [info] Created XPI for signing: /tmp/tmp-unsigned-xpi-25600yuqzFN3Alxa0/#addonname-0.x.y.xpi
JPM [debug] [API] -{ url: 'https://addons.mozilla.org/api/v3/addons/%40addonname/versions/0.x.y/',
formData:
{ upload:
{ _readableState: [Object],
readable: true,
domain: null,
_events: [Object],
_eventsCount: 1,
_maxListeners: undefined,
path: '/tmp/tmp-unsigned-xpi-25600yuqzFN3Alxa0/#addonname-0.x.y.xpi',
fd: null,
flags: 'r',
mode: 438,
start: undefined,
end: undefined,
autoClose: true,
pos: undefined } },
headers: { Authorization: '<REDACTED>', Accept: 'application/json' } }
JPM [debug] [API] <- { headers:
{ allow: 'GET, PUT, HEAD, OPTIONS',
'content-type': 'application/json',
date: 'Thu, 17 Dec 2015 10:17:23 GMT',
server: 'nginx',
'set-cookie': '<REDACTED>',
'strict-transport-security': 'max-age=31536000',
vary: 'Accept, X-Mobile, User-Agent',
'www-authenticate': 'JWT realm="api"',
'x-frame-options': 'DENY',
'content-length': '51',
connection: 'Close' },
response: { detail: 'Incorrect authentication credentials.' } }
JPM [error] FAIL
Error: Received bad response from the server while requesting https://addons.mozilla.org/api/v3/addons/%40addonname/versions/0.x.y/
status: 401
response: {"detail":"Incorrect authentication credentials."}
headers: {"allow":"GET, PUT, HEAD, OPTIONS","content-type":"application/json","date":"Thu, 17 Dec 2015 10:17:23 GMT","server":"nginx","set-cookie":["multidb_pin_writes=y; expires=Thu, 17-Dec-2015 10:17:38 GMT; Max-Age=15; Path=/"],"strict-transport-security":"max-age=31536000","vary":"Accept, X-Mobile, User-Agent","www-authenticate":"JWT realm=\"api\"","x-frame-options":"DENY","content-length":"51","connection":"Close"}
at /usr/local/lib/node_modules/jpm/lib/amo-client.js:85:13
at tryCatchReject (/usr/local/lib/node_modules/jpm/node_modules/when/lib/makePromise.js:845:30)
at runContinuation1 (/usr/local/lib/node_modules/jpm/node_modules/when/lib/makePromise.js:804:4)
at Fulfilled.when (/usr/local/lib/node_modules/jpm/node_modules/when/lib/makePromise.js:592:4)
at Pending.run (/usr/local/lib/node_modules/jpm/node_modules/when/lib/makePromise.js:483:13)
at Scheduler._drain (/usr/local/lib/node_modules/jpm/node_modules/when/lib/Scheduler.js:62:19)
at Scheduler.drain (/usr/local/lib/node_modules/jpm/node_modules/when/lib/Scheduler.js:27:9)
at doNTCallback0 (node.js:417:9)
at process._tickCallback (node.js:346:13)

These posts (1), (2) show that this error can occur if your local clock is off by more than 60 seconds with the Mozilla server.
This is also documented in the Olympia 3.0 Authentication API (see exp parameter):
iat
This is a standard JWT claim indicating the issued at time. It should be a Unix epoch timestamp and must be in UTC time.
exp
This is a standard JWT claim indicating the expiration time. It should be a Unix epoch timestamp in UTC time and must be no longer
than 60 seconds past the issued at time.
Note:
If you’re having trouble authenticating, make sure your system clock
is correct and consider synchronizing it with something like NTP
(Network Time Protocol).

Related

Curl HEAD request returns a last-modified header but Node https.request(..., {method: HEAD},...) does not

Apparently there's something I don't know about HEAD requests.
Here's the URL: 'https://theweekinchess.com/assets/files/pgn/eurbli22.pgn', which I'll refer to as <URL> below.
If I curl this, I see a last-modified entry in the headers:
curl --head <URL>
HTTP/2 200
last-modified: Sun, 18 Dec 2022 18:07:16 GMT
accept-ranges: bytes
content-length: 1888745
host-header: c2hhcmVkLmJsdWVob3N0LmNvbQ==
content-type: application/x-chess-pgn
date: Wed, 11 Jan 2023 23:09:14 GMT
server: Apache
But if I make a HEAD request in Node using https, That information is missing:
https.request(<URL>, { method: 'HEAD' }, res => {
console.log([<URL>, res.headers])}).end()
This returns:
[
<URL>
{
date: 'Wed, 11 Jan 2023 23:16:15 GMT',
server: 'Apache',
p3p: 'CP="NOI NID ADMa OUR IND UNI COM NAV"',
'cache-control': 'private, must-revalidate',
'set-cookie': [
'evof3sqa=4b412b5913b38669fc928a0cca9870e4; path=/; secure; HttpOnly'
],
upgrade: 'h2,h2c',
connection: 'Upgrade, Keep-Alive',
'host-header': 'c2hhcmVkLmJsdWVob3N0LmNvbQ==',
'keep-alive': 'timeout=5, max=75',
'content-type': 'text/html; charset=UTF-8'
}
]
I tried axios instead of https:
const response = await axios.head('https://theweekinchess.com/assets/files/pgn/eurbli22.pgn');
console.log({response: response.headers})
And that works (incl. the proper MIME type):
date: 'Thu, 12 Jan 2023 20:00:48 GMT',
server: 'Apache',
upgrade: 'h2,h2c',
connection: 'Upgrade, Keep-Alive',
'last-modified': 'Sun, 18 Dec 2022 18:07:16 GMT',
'accept-ranges': 'bytes',
'content-length': '1888745',
'host-header': 'c2hhcmVkLmJsdWVob3N0LmNvbQ==',
'keep-alive': 'timeout=5, max=75',
'content-type': 'application/x-chess-pgn'
I also tried waiting for re2.on('end', console.log(res.headers)), but same output as before.
I'm going to close this issue and post it instead as a 'bug' on Node's site. I'm sure there's something that needs to be changed in how I'm executing the HEAD request.

Unable to programmatically download Kaggle dataset "nih-chest-xrays/data" specific file Version 3

I need to download Kaggle's NIH Chest X-rays Dataset programmatically, specifically the Data_Entry_2017.csv.
I want to download it if doesn't exist on system, and redownload it if it's updation date exceeds last download date.
I know I can do this manually, but I'll appreciate if there was a way to do it programmatically.
I have done:
Direct download:
kaggle.api.dataset_download_file(dataset='nih-chest-xrays/data', file_name='Data_Entry_2017.csv', path='data/')
This gives:
File "C:\Program Files\Python311\Lib\site-packages\kaggle\rest.py", line 241, in request
raise ApiException(http_resp=r)
kaggle.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Sun, 18 Dec
2022 03:35:46 GMT', 'Access-Control-Allow-Credentials': 'true', 'Set-Cookie':
'ka_sessionid=bdaa2b71c677d6e48bf93fc71b85a9c6; max-age=2626560; path=/, GCLB=CIeroe2g5sixeA;
path=/; HttpOnly', 'Transfer-Encoding': 'chunked', 'Vary': 'Accept-Encoding', 'Turbolinks-
Location': 'https://www.kaggle.com/api/v1/datasets/download/nih-chest-
xrays/data/Data_Entry_2017.csv', 'X-Kaggle-MillisecondsElapsed': '526', 'X-Kaggle-RequestId':
'03383f8cdac3aa4b6ab83d5dadbc507c', 'X-Kaggle-ApiVersion': '1.5.12', 'X-Frame-Options':
'SAMEORIGIN', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload',
'Content-Security-Policy': "object-src 'none'; script-src 'nonce-gn6A4mfrbofbZRyE9gREnA=='
'report-sample' 'unsafe-inline' 'unsafe-eval' 'strict-dynamic' https: http:; frame-src 'self'
https://www.kaggleusercontent.com https://www.youtube.com/embed/ https://polygraph-cool.github.io
https://www.google.com/recaptcha/ https://form.jotform.com https://submit.jotform.us
https://submit.jotformpro.com https://submit.jotform.com https://www.docdroid.com
https://www.docdroid.net https://kaggle-static.storage.googleapis.com https://kaggle-static-
staging.storage.googleapis.com https://kkb-dev.jupyter-proxy.kaggle.net https://kkb-
staging.jupyter-proxy.kaggle.net https://kkb-production.jupyter-proxy.kaggle.net https://kkb-
dev.firebaseapp.com https://kkb-staging.firebaseapp.com
https://kkb-production.firebaseapp.com https://kaggle-metastore-test.firebaseapp.com
https://kaggle-metastore.firebaseapp.com https://apis.google.com https://content-
sheets.googleapis.com/ https://accounts.google.com/ https://storage.googleapis.com
https://docs.google.com https://drive.google.com https://calendar.google.com/;
base-uri 'none'; report-uri https://csp.withgoogle.com/csp/kaggle/20201130;", 'X-Content-Type-
Options': 'nosniff', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'Via': '1.1 google',
'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
HTTP response body: b'{"code":404,"message":"Not found"}'
Same for any other files in this dataset:
kaggle.api.dataset_download_file(dataset='nih-chest-xrays/data', file_name='test_list.txt', path='data/')
Which again gives
kaggle.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Sun, 18 Dec
2022 03:37:20 GMT', 'Access-Control-Allow-Credentials': 'true', 'Set-Cookie':
'ka_sessionid=b59676e67aa4a36e92657a7fa50e95f3; max-age=2626560; path=/, GCLB=CKOV0bPZ-p6ZSA;
path=/; HttpOnly', 'Transfer-Encoding': 'chunked', 'Vary': 'Accept-Encoding', 'Turbolinks-
Location': 'https://www.kaggle.com/api/v1/datasets/download/nih-chest-xrays/data/test_list.txt',
'X-Kaggle-MillisecondsElapsed': '520', 'X-Kaggle-RequestId': '79fde70e368d73ba7c767a007bacc086',
'X-Kaggle-ApiVersion': '1.5.12', 'X-Frame-Options': 'SAMEORIGIN', 'Strict-Transport-Security':
'max-age=63072000; includeSubDomains; preload', 'Content-Security-Policy': "object-src 'none';
script-src 'nonce-U88j/npGgHfKpSCwewojZg==' 'report-sample' 'unsafe-inline' 'unsafe-eval'
'strict-dynamic' https: http:; frame-src 'self' https://www.kaggleusercontent.com
https://www.youtube.com/embed/ https://polygraph-cool.github.io https://www.google.com/recaptcha/
https://form.jotform.com https://submit.jotform.us https://submit.jotformpro.com
https://submit.jotform.com https://www.docdroid.com https://www.docdroid.net https://kaggle-
static.storage.googleapis.com https://kaggle-static-staging.storage.googleapis.com https://kkb-
dev.jupyter-proxy.kaggle.net https://kkb-staging.jupyter-proxy.kaggle.net https://kkb-
production.jupyter-proxy.kaggle.net https://kkb-dev.firebaseapp.com https://kkb-
staging.firebaseapp.com https://kkb-production.firebaseapp.com https://kaggle-metastore-
test.firebaseapp.com https://kaggle-metastore.firebaseapp.com https://apis.google.com
https://content-sheets.googleapis.com/ https://accounts.google.com/
https://storage.googleapis.com https://docs.google.com https://drive.google.com
https://calendar.google.com/; base-uri 'none'; report-uri
https://csp.withgoogle.com/csp/kaggle/20201130;", 'X-Content-Type-Options': 'nosniff', 'Referrer-
Policy': 'strict-origin-when-cross-origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443";
ma=2592000,h3-29=":443"; ma=2592000'})
HTTP response body: b'{"code":404,"message":"Not found"}'
The strange thing is that I can search this dataset:
dataset_list = kaggle.api.dataset_list(search='nih')
print(dataset_list)
Which gives:
[nih-chest-xrays/data, nih-chest-xrays/sample, allen-institute-for-ai/CORD-19-research-challenge,
kmader/nih-deeplesion-subset, nickuzmenkov/nih-chest-xrays-tfrecords, iarunava/cell-images-for-
detecting-malaria, tunguz/covid19-genomes, nlm-nih/nlm-rxnorm, akhileshdkapse/nih-image-600x600-
data, tunguz/nih-awarded-grant-text, danielmadmon/nih-xray-dataset-tfrec-with-labels,
miracle9to9/files1, nlm-nih/rxnorm-drug-name-conventions, kmader/rsna-bone-age,
redwankarimsony/chestxray8-dataframe, akhileshdkapse/nih-dataframe, luigisaetta/cxr-tfrec256-
may2020, kokyew93/nihdata, ammarali32/startingpointschestx, dannellyz/cancer-incidence-totals-
and-rates-per-us-county]
It's literally the first one on the list.
And checking its metadata:
dataset = vars(dataset_list[0])
print(dataset)
Which gives:
{'subtitleNullable': 'Over 112,000 Chest X-ray images from more than 30,000 unique patients',
'creatorNameNullable': 'Timo Bozsolik', 'creatorUrlNullable': 'timoboz', 'totalBytesNullable':
45096150231, 'urlNullable': 'https://www.kaggle.com/datasets/nih-chest-xrays/data',
'licenseNameNullable': 'CC0: Public Domain', 'descriptionNullable': None, 'ownerNameNullable':
'National Institutes of Health Chest X-Ray Dataset', 'ownerRefNullable': 'nih-chest-xrays',
'titleNullable': 'NIH Chest X-rays', 'currentVersionNumberNullable': 3,
'usabilityRatingNullable': 0.7352941, 'id': 5839, 'ref': 'nih-chest-xrays/data', 'subtitle':
'Over 112,000 Chest X-ray images from more than 30,000 unique patients', 'hasSubtitle': True,
'creatorName': 'Timo Bozsolik', 'hasCreatorName': True, 'creatorUrl': 'timoboz', 'hasCreatorUrl':
True, 'totalBytes': 45096150231, 'hasTotalBytes': True, 'url':
'https://www.kaggle.com/datasets/nih-chest-xrays/data', 'hasUrl': True, 'lastUpdated':
datetime.datetime(2018, 2, 21, 20, 52, 23), 'downloadCount': 65214, 'isPrivate': False,
'isFeatured': False, 'licenseName': 'CC0: Public Domain', 'hasLicenseName': True, 'description':
'', 'hasDescription': False, 'ownerName': 'National Institutes of Health Chest X-Ray Dataset',
'hasOwnerName': True, 'ownerRef': 'nih-chest-xrays', 'hasOwnerRef': True, 'kernelCount': 323,
'title': 'NIH Chest X-rays', 'hasTitle': True, 'topicCount': 0, 'viewCount': 454490, 'voteCount':
967, 'currentVersionNumber': 3, 'hasCurrentVersionNumber': True, 'usabilityRating': 0.7352941,
'hasUsabilityRating': True, 'tags': [biology, health, medicine, computer science, software,
health conditions], 'files': [], 'versions': [], 'size': '42GB'}
I can even list the files for that dataset, and it's visible:
print(kaggle.api.dataset_list_files('nih-chest-xrays/data').files)
Which gives:
[BBox_List_2017.csv, LOG_CHESTXRAY.pdf, ARXIV_V5_CHESTXRAY.pdf, README_CHESTXRAY.pdf,
train_val_list.txt, Data_Entry_2017.csv, FAQ_CHESTXRAY.pdf, test_list.txt]
A problem I noticed was that kaggle.api.dataset_download_file was only fetching from Version 1, where these files don't exist. This was further confirmed when I successfully fetched an image file from Version 1:
res = kaggle.api.dataset_download_file(dataset='nih-chest-xrays/data', file_name='images_001/images/00000001_000.png', path='data/')
print(res)
Which downloads the image and prints True.
My data is Version 3. Is there any way to configure kaggle to use Version 3?
Note that I can download the whole 45GB dataset, but I only require the single file Data_Entry_2017.csv.
Even more strangely, it's also throwing an error when checking its status:
print(kaggle.api.dataset_status(dataset='nih-chest-xrays/data'))
Which gives:
Traceback (most recent call last):
File "C:\Users\jaide\OneDrive\Documents\MScCS2022\Data-Mining-I\chest-datasets\visualise.py",
line 117, in <module>
full_df = load_nih_data()
^^^^^^^^^^^^^^^
File "C:\Users\jaide\OneDrive\Documents\MScCS2022\Data-Mining-I\chest-datasets\visualise.py",
line 89, in load_nih_data
print(kaggle.api.dataset_status(dataset='nih-chest-xrays/data'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\kaggle\api\kaggle_api_extended.py", line
1135, in dataset_status
self.datasets_status_with_http_info(owner_slug=owner_slug,
File "C:\Program Files\Python311\Lib\site-packages\kaggle\api\kaggle_api.py", line 1910, in
datasets_status_with_http_info
return self.api_client.call_api(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\kaggle\api_client.py", line 329, in call_api
return self.__call_api(resource_path, method,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\kaggle\api_client.py", line 161, in
__call_api
response_data = self.request(
^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\kaggle\api_client.py", line 351, in request
return self.rest_client.GET(url,
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\kaggle\rest.py", line 247, in GET
return self.request("GET", url,
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\kaggle\rest.py", line 241, in request
raise ApiException(http_resp=r)
kaggle.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Sun, 18 Dec
2022 04:14:32 GMT', 'Access-Control-Allow-Credentials': 'true', 'Set-Cookie':
'ka_sessionid=1ab6e66561c7d9ae43ead26df25beea3; max-age=2626560; path=/, GCLB=CNS4vKLYyPuCpgE;
path=/; HttpOnly', 'Transfer-Encoding': 'chunked', 'Vary':
'Accept-Encoding', 'Turbolinks-Location': 'https://www.kaggle.com/api/v1/datasets/status/nih-
chest-xrays/data', 'X-Kaggle-MillisecondsElapsed': '46', 'X-Kaggle-RequestId':
'66473cc55f62c39d2deb0aa4cd2953a7', 'X-Kaggle-ApiVersion': '1.5.12', 'X-Frame-Options':
'SAMEORIGIN', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload',
'Content-Security-Policy': "object-src 'none'; script-src 'nonce-5AJVxKPJRg4Gsi/iT3D5Aw=='
'report-sample' 'unsafe-inline' 'unsafe-eval' 'strict-dynamic' https: http:; frame-src 'self'
https://www.kaggleusercontent.com https://www.youtube.com/embed/ https://polygraph-cool.github.io
https://www.google.com/recaptcha/ https://form.jotform.com https://submit.jotform.us
https://submit.jotformpro.com https://submit.jotform.com https://www.docdroid.com
https://www.docdroid.net https://kaggle-static.storage.googleapis.com https://kaggle-static-
staging.storage.googleapis.com https://kkb-dev.jupyter-proxy.kaggle.net https://kkb-
staging.jupyter-proxy.kaggle.net https://kkb-production.jupyter-proxy.kaggle.net https://kkb-
dev.firebaseapp.com https://kkb-staging.firebaseapp.com https://kkb-production.firebaseapp.com
https://kaggle-metastore-test.firebaseapp.com https://kaggle-metastore.firebaseapp.com
https://apis.google.com https://content-sheets.googleapis.com/ https://accounts.google.com/
https://storage.googleapis.com https://docs.google.com https://drive.google.com
https://calendar.google.com/; base-uri 'none'; report-uri
https://csp.withgoogle.com/csp/kaggle/20201130;", 'X-Content-Type-Options': 'nosniff', 'Referrer-
Policy': 'strict-origin-when-cross-origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443";
ma=2592000,h3-29=":443"; ma=2592000'})
HTTP response body: {"code":404,"message":"Not found"}
CLI download
Check if dataset exists:
C:\Users\jaide\OneDrive\Documents\MScCS2022\Data-Mining-I\chest-datasets>kaggle datasets list -s "nih-chest-xrays/data"
ref title size lastUpdated downloadCount voteCount usabilityRating
------------------------------------------------------------- ----------------------------------------------- ----- ------------------- ------------- --------- ---------------
nickuzmenkov/nih-chest-xrays-tfrecords NIH Chest X-rays TFRecords 11GB 2021-03-09 04:49:23 2121 119 0.9411765
nih-chest-xrays/data NIH Chest X-rays 42GB 2018-02-21 20:52:23 65221 967 0.7352941
nih-chest-xrays/sample Random Sample of NIH Chest X-ray Dataset 4GB 2017-11-23 02:58:24 14930 241 0.7647059
ammarali32/startingpointschestx StartingPoints-ChestX 667MB 2021-02-27 17:35:19 668 68 0.6875
redwankarimsony/chestxray8-dataframe ChestX-ray8_DataFrame 51MB 2020-07-12 02:28:33 416 10 0.64705884
amanullahasraf/covid19-pneumonia-normal-chest-xray-pa-dataset COVID19_Pneumonia_Normal_Chest_Xray_PA_Dataset 2GB 2020-07-13 05:54:22 1482 14 0.8125
harshsoni/nih-chest-xray-tfrecords NIH Chest X-ray TFRecords 42GB 2020-09-26 18:48:04 52 6 0.5294118
amanullahasraf/covid19-pneumonia-normal-chest-xraypa-dataset COVID19_Pneumonia_Normal_Chest_Xray(PA)_Dataset 1GB 2020-06-29 08:28:01 161 3 0.5882353
roderikmogot/nih-chest-x-ray-models nih chest x ray models 2GB 2022-08-17 01:32:33 66 4 0.47058824
ericspod/project-monai-2020-bootcamp-challenge-dataset Project MONAI 2020 Bootcamp Challenge Dataset 481MB 2021-01-25 01:11:42 22 2 0.6875
sunghyunjun/nih-chest-xrays-600-jpg-dataset NIH Chest X-rays 600 JPG Dataset 7GB 2021-03-15 07:53:12 35 3 0.64705884
zhuangjw/chest-xray-cleaned chest_xray_cleaned 2GB 2019-11-30 19:05:24 13 1 0.375
kambarakun/nih-chest-xrays-trained-models NIH Chest X-rays: Trained Models 3GB 2020-04-30 11:52:56 10 2 0.3125
jessevent/all-kaggle-datasets Complete Kaggle Datasets Collection 390KB 2018-01-16 12:32:58 2256 118 0.8235294
vzaguskin/nih-chest-xrays-tfrecords-756 NIH Chest X-rays tfrecords 756 11GB 2021-02-25 14:25:12 3 1 0.29411766
Yes, it is the second one. Now does it have my required file?
C:\Users\jaide\OneDrive\Documents\MScCS2022\Data-Mining-I\chest-datasets>kaggle datasets files "nih-chest-xrays/data"
name size creationDate
---------------------- ----- -------------------
ARXIV_V5_CHESTXRAY.pdf 9MB 2018-02-21 20:52:23
FAQ_CHESTXRAY.pdf 71KB 2018-02-21 20:52:23
Data_Entry_2017.csv 7MB 2018-02-21 20:52:23
BBox_List_2017.csv 90KB 2018-02-21 20:52:23
README_CHESTXRAY.pdf 827KB 2018-02-21 20:52:23
LOG_CHESTXRAY.pdf 4KB 2018-02-21 20:52:23
train_val_list.txt 1MB 2018-02-21 20:52:23
test_list.txt 425KB 2018-02-21 20:52:23
Yes it does. Now download it.
C:\Users\jaide\OneDrive\Documents\MScCS2022\Data-Mining-I\chest-datasets>kaggle datasets download -f "Data_Entry_2017.csv" -p "data/" "nih-chest-xrays/data"
404 - Not Found
Nope, error 404.
How do I download Kaggle's NIH Chest X-rays Dataset programmatically, specifically the Data_Entry_2017.csv?
Any help is appreciated!
I have checked:
I checked out Kaggle Python API, but it doesn't appear to have any way to configure Version.
I checked out Kaggle API on github, but it also doesn't appear to have any ways to configure Version.
Related Questions Read:
Kaggle Dataset Download - is irrelevant.
Download a Kaggle Dataset - is downloading competition files.
How to download data set into Colab? Stuck with a problem, it says "401 - Unauthorized"? - They had api key issues
How to load just one chosen file of a way too large Kaggle dataset from Kaggle into Colab - Is related to kaggle and jupyter notebooks, I am currently just using a python script.
Trouble turning comorbidity data into a table using Python and Pandas - Are again using Kernels (Kaggle Notebooks).
How to download kaggle dataset? - Downloads the whole dataset.

Cypress: intercept a network request with compression-type gzip to simulate mapbox

I am currently having trouble with mocking a particular request mapbox-gl is making. When the map is loaded from mapbox pbf-files are being requested and i have not been able to mock this.
My guess is that the core issue is that there seems to be an open bug with cypress issue-16420.
I tried alot of different intercept variants. I tried all kinds of response headers. I gziped, compressed, brd the file that I serve via fixture. I tried different encodings for the fixture. Nothing worked. One of the interceptors looks basically like this
cy.intercept({
method: 'GET',
url: '**/fonts/v1/mapbox/DIN%20Offc%20Pro%20Italic,Arial%20Unicode%20MS%20Regular/0-255.pbf?*',
}, {
fixture: 'fonts/italic.arial.0-255.pbf,binary',
statusCode: 204,
headers: {
'Connection': 'keep-alive',
'Keep-Alive': 'timeout=5',
'Transfer-Encoding': 'chunked',
'access-control-allow-origin': '*',
'access-control-expose-headers': 'Link',
'age': '11631145',
'cache-control': 'max-age=31536000',
'content-encoding': 'compress',
'content-type': 'application/x-protobuf',
'date': 'Sat, 19 Feb 2022 20:46:43 GMT',
'etag': 'W/"b040-+eCb/OHkPqToOcONTDlvpCrjmvs"',
'via': '1.1 4dd80d99fd5d0f6baaaf5179cd921f72.cloudfront.net (CloudFront)',
'x-amz-cf-id': '4uY9rjBgR_R12nkfHFrBMLEpNuWygW9DkmODlMEzwJHABTGCGg8pww==',
'x-amz-cf-pop': 'FRA56-P7',
'x-cache': 'Hit from cloudfront',
'x-origin': 'Mbx-Fonts'
}
}).as('get.0-255.pbf').as('getItalicArial0-255');
Now even if this is a bug there has to be some kind of workaround to serve the file in a cypress test without having an active internet connection. It would be great not having to rely on the network on tests. So all kinds of workarounds and dirty tricks are welcome in making this intercept work.

Error 403 while using exchangelib to access Outlook Exchange server to read emails

I'm trying to read emails from the Microsoft Exchange server using EWS and exchangelib in Python for an email classification problem. But I am unable to connect to the exchange server.
I've tried specifying the version, auth_type, using a certificate (which gives a ssl verify error), using the smtp address in place of the username and it still doesn't connect.
Here is my code:
from exchangelib import Credentials, Account, EWSDateTime, EWSTimeZone, Configuration, DELEGATE, IMPERSONATION, NTLM, ServiceAccount, Version, Build
USER_NAME = 'domain\\user12345'
ACCOUNT_EMAIL = john.doe#ext.companyname.com'
ACCOUNT_PASSWORD = 'John#1234'
ACCOUNT_SERVER = 'oa.company.com'
creds = Credentials(USER_NAME, ACCOUNT_PASSWORD)
config = Configuration(server=ACCOUNT_SERVER, credentials=creds)
account = Account(primary_smtp_address=ACCOUNT_EMAIL, config=config, autodiscover=False, access_type=DELEGATE)
print('connecting ms exchange server account...')
print(type(account))
print(dir(account))
account.root.refresh()
Here is the error I am getting:
TransportError: Unknown failure
Retry: 0
Waited: 10
Timeout: 120
Session: 26271
Thread: 15248
Auth type: <requests_ntlm.requests_ntlm.HttpNtlmAuth object at 0x00000259AA1BD588>
URL: https://oa.company.com/EWS/Exchange.asmx
HTTP adapter: <requests.adapters.HTTPAdapter object at 0x00000259AA0DB7B8>
Allow redirects: False
Streaming: False
Response time: 0.28100000000085856
Status code: 403
Request headers: {'User-Agent': 'python-requests/2.21.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'Keep-Alive', 'Content-Type': 'text/xml; charset=utf-8', 'Content-Length': '469', 'Authorization': 'NTLM TlRMTVNTUAADAAAAGAAYAG0AAAAOAQ4BhQAAAAwADABYAAAACQAJAGQAAAAAAAAAbQAAABAAEACTAQAANoKJ4gYBsR0AAAAP7Pyb+wBnMdrlhr4FKVqPbklDSUNJQkFOS0xURElQUlUzODE5MAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJTLmMBLPHowOJZ46XDs+4ABAQAAAAAAAAZdRefQLNUB+Fc6Z26oxvgAAAAAAgAYAEkAQwBJAEMASQBCAEEATgBLAEwAVABEAAEAEgBIAFkARABFAFgAQwBIADAAOAAEACAAaQBjAGkAYwBpAGIAYQBuAGsAbAB0AGQALgBjAG8AbQADADQASABZAEQARQBYAEMASAAwADgALgBpAGMAaQBjAGkAYgBhAG4AawBsAHQAZAAuAGMAbwBtAAUAIABpAGMAaQBjAGkAYgBhAG4AawBsAHQAZAAuAGMAbwBtAAcACAAGXUXn0CzVAQYABAACAAAACgAQAD9EWlwiiUs304wucsxnkyQAAAAAAAAAALfelDwG05hYOMUqY/e60PY=', 'Cookie': 'ClientId=SINZWMOJKWSKDGEKASFG; expires=Fri, 26-Jun-2020 10:13:02 GMT; path=/; HttpOnly'}
Response headers: {'Cache-Control': 'private', 'Server': 'Microsoft-IIS/8.5', 'request-id': 'ae4dee8d-34e0-471c-8252-b8c1056c8ea0', 'X-CalculatedBETarget': 'pqrexch05.domain.com', 'X-DiagInfo': 'PQREXCH05', 'X-BEServer': 'PQREXCH05', 'X-AspNet-Version': '4.0.30319', 'Set-Cookie': 'exchangecookie=681afc8a0905459182363cce9a98d021; expires=Sat, 27-Jun-2020 10:13:02 GMT; path=/; HttpOnly, X-BackEndCookie=S-1-5-21-1343024091-725345543-504838010-1766210=u56Lnp2ejJqBy87Iysqem5nSy8mbnNLLyZ7H0sfIysbSy5vMz8qdzcvPnpzHgYHNz87G0s/I0s3Iq87Pxc7Mxc/N; expires=Sat, 27-Jul-2019 10:13:02 GMT; path=/EWS; secure; HttpOnly', 'Persistent-Auth': 'true', 'X-Powered-By': 'ASP.NET', 'X-FEServer': 'PQREXCH05', 'Date': 'Thu, 27 Jun 2019 10:13:01 GMT', 'Content-Length': '0'}
Request data: b'<?xml version=\'1.0\' encoding=\'utf-8\'?>\n<s:Envelope xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types"><s:Header><t:RequestServerVersion Version="Exchange2013_SP1"/></s:Header><s:Body><m:ResolveNames ReturnFullContactData="false"><m:UnresolvedEntry>ICICIBANKLTD\\IPRU38190</m:UnresolvedEntry></m:ResolveNames></s:Body></s:Envelope>'
Response data: b''
You might need to configure access policy for EWS using PowerShell.
For example (to allow all apps to use REST and EWS):
Set-OrganizationConfig -EwsApplicationAccessPolicy EnforceBlockList -EwsBlockList $null
Taken from Microsoft docs on Set-OrganizationConfig.
Please search for EwsApplicationAccessPolicy in the above link for more granular access control examples.

How to setup custom DNS with Azure Websites Preview?

I created a new Azure Website, using Umbraco as the CMS. I got a page up and going, and I already have a .co.nz domain with www.domains4less.com. There's a whole lot of stuff on the internet about pointing URLs to Azure, but that seems to be more of a redirection service than anything (i.e. my URLs still use azurewebsites.net once I land on my site).
Has anybody had any luck getting it to go?
Here's the error I get when I try adding the DNS entry to Azure (I'm in reserved mode, reemdairy is the name of the website):
There was an error processing your request. Please try again in a few
moments.
Browser: 5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like
Gecko) Chrome/19.0.1084.56 Safari/536.5 User language: undefined
Portal Version: 6.0.6002.18488 (rd_auxportal_stable.120609-0259)
Subscriptions: 3aabe358-d178-4790-a97b-ffba902b2851 User email
address: touchmaniacs#live.com
Last 10 Requests
message: Failure: Ajax call to: Websites/UpdateConfig. failed with
status: error (500) in 2.57 seconds. x-ms-client-request-id was:
38834edf-c9f3-46bb-a1f7-b2839c692bcf-2012-06-12 22:25:14Z dateTime:
Wed Jun 13 2012 10:25:17 GMT+1200 (New Zealand Standard Time)
durationSeconds: 2.57 url: Websites/UpdateConfig status: 500
textStatus: error clientMsRequestId:
38834edf-c9f3-46bb-a1f7-b2839c692bcf-2012-06-12 22:25:14Z sessionId:
09c72263-6ce7-422b-84d7-4c21acded759 referrer:
https://manage.windowsazure.com/#Workspaces/WebsiteExtension/Website/reemdairy/configure
host: manage.windowsazure.com response: {"message":"Try again. Contact
support if the problem persists.","ErrorMessage":"Try again. Contact
support if the problem
persists.","httpStatusCode":"InternalServerError","operationTrackingId":"","stackTrace":null}
message: Complete: Ajax call to: Websites/GetConfig. completed with
status: success (200) in 1.021 seconds. x-ms-client-request-id was:
a0cdcced-13d0-44e2-866d-e0b061b9461b-2012-06-12 22:24:43Z dateTime:
Wed Jun 13 2012 10:24:44 GMT+1200 (New Zealand Standard Time)
durationSeconds: 1.021 url: Websites/GetConfig status: 200 textStatus:
success clientMsRequestId:
a0cdcced-13d0-44e2-866d-e0b061b9461b-2012-06-12 22:24:43Z sessionId:
09c72263-6ce7-422b-84d7-4c21acded759 referrer:
https://manage.windowsazure.com/#Workspaces/WebsiteExtension/Website/reemdairy/configure
host: manage.windowsazure.com
message: Complete: Ajax call to:
https://manage.windowsazure.com/Service/OperationTracking?subscriptionId=3aabe358-d178-4790-a97b-ffba902b2851.
completed with status: success (200) in 1.887 seconds.
x-ms-client-request-id was:
a7689fe9-b9f9-4d6c-8926-734ec9a0b515-2012-06-12 22:24:40Z dateTime:
Wed Jun 13 2012 10:24:42 GMT+1200 (New Zealand Standard Time)
durationSeconds: 1.887 url:
https://manage.windowsazure.com/Service/OperationTracking?subscriptionId=3aabe358-d178-4790-a97b-ffba902b2851
status: 200 textStatus: success clientMsRequestId:
a7689fe9-b9f9-4d6c-8926-734ec9a0b515-2012-06-12 22:24:40Z sessionId:
09c72263-6ce7-422b-84d7-4c21acded759 referrer:
https://manage.windowsazure.com/#Workspaces/WebsiteExtension/Website/reemdairy/configure
host: manage.windowsazure.com
message: Complete: Ajax call to: /Service/GetUserSettings. completed
with status: success (200) in 0.941 seconds. x-ms-client-request-id
was: 805e554d-1e2e-4214-afd5-be87c0f255d1-2012-06-12 22:24:40Z
dateTime: Wed Jun 13 2012 10:24:40 GMT+1200 (New Zealand Standard
Time) durationSeconds: 0.941 url: /Service/GetUserSettings status: 200
textStatus: success clientMsRequestId:
805e554d-1e2e-4214-afd5-be87c0f255d1-2012-06-12 22:24:40Z sessionId:
09c72263-6ce7-422b-84d7-4c21acded759 referrer:
https://manage.windowsazure.com/#Workspaces/WebsiteExtension/Website/reemdairy/configure
host: manage.windowsazure.com
message: Complete: Ajax call to:
Extensions/ApplicationsExtension/SqlAzure/ClusterSuffix. completed
with status: success (200) in 0.483 seconds. x-ms-client-request-id
was: 85157ceb-c538-40ca-8c1e-5cc07c57240f-2012-06-12 22:24:39Z
dateTime: Wed Jun 13 2012 10:24:40 GMT+1200 (New Zealand Standard
Time) durationSeconds: 0.483 url:
Extensions/ApplicationsExtension/SqlAzure/ClusterSuffix status: 200
textStatus: success clientMsRequestId:
85157ceb-c538-40ca-8c1e-5cc07c57240f-2012-06-12 22:24:39Z sessionId:
09c72263-6ce7-422b-84d7-4c21acded759 referrer:
https://manage.windowsazure.com/#Workspaces/WebsiteExtension/Website/reemdairy/configure
host: manage.windowsazure.com
message: Complete: Ajax call to:
Extensions/ApplicationsExtension/SqlAzure/GetClientIp. completed with
status: success (200) in 0.309 seconds. x-ms-client-request-id was:
2eb194b6-66ca-49e2-9016-e0f89164314c-2012-06-12 22:24:39Z dateTime:
Wed Jun 13 2012 10:24:40 GMT+1200 (New Zealand Standard Time)
durationSeconds: 0.309 url:
Extensions/ApplicationsExtension/SqlAzure/GetClientIp status: 200
textStatus: success clientMsRequestId:
2eb194b6-66ca-49e2-9016-e0f89164314c-2012-06-12 22:24:39Z sessionId:
09c72263-6ce7-422b-84d7-4c21acded759 referrer:
https://manage.windowsazure.com/#Workspaces/WebsiteExtension/Website/reemdairy/configure
host: manage.windowsazure.com
message: Complete: Ajax call to:
Extensions/ApplicationsExtension/SqlAzure/DefaultServerLocation.
completed with status: success (200) in 0.309 seconds.
x-ms-client-request-id was:
1bc165ef-2081-48f2-baed-16c6edf8ea67-2012-06-12 22:24:39Z dateTime:
Wed Jun 13 2012 10:24:40 GMT+1200 (New Zealand Standard Time)
durationSeconds: 0.309 url:
Extensions/ApplicationsExtension/SqlAzure/DefaultServerLocation
status: 200 textStatus: success clientMsRequestId:
1bc165ef-2081-48f2-baed-16c6edf8ea67-2012-06-12 22:24:39Z sessionId:
09c72263-6ce7-422b-84d7-4c21acded759 referrer:
https://manage.windowsazure.com/#Workspaces/WebsiteExtension/Website/reemdairy/configure
host: manage.windowsazure.com
message: Complete: Ajax call to:
Extensions/ApplicationsExtension/SqlAzure/ServerLocations. completed
with status: success (200) in 0.309 seconds. x-ms-client-request-id
was: e1fba7df-6a12-47f8-9434-bf17ca7d93f4-2012-06-12 22:24:39Z
dateTime: Wed Jun 13 2012 10:24:40 GMT+1200 (New Zealand Standard
Time) durationSeconds: 0.309 url:
Extensions/ApplicationsExtension/SqlAzure/ServerLocations status: 200
textStatus: success clientMsRequestId:
e1fba7df-6a12-47f8-9434-bf17ca7d93f4-2012-06-12 22:24:39Z sessionId:
09c72263-6ce7-422b-84d7-4c21acded759 referrer:
https://manage.windowsazure.com/#Workspaces/WebsiteExtension/Website/reemdairy/configure
host: manage.windowsazure.com
Its good that you have Windows Azure Website (Reserve) because only Reserve websites can have CNAME or A record point to your_site.azurewebsites.net.
May I ask what are you setting, CNAME or A Record?
Here is the step by step direction on setting either CNAME (which is suggested for websites and webroles) so please follow and suggest at what step you have an issue:
Configuring a Custom Domain Name in Windows Azure
I've successfully done this with the exact same setup (Umbraco, running on Azure with a domains4less.co.nz domain). DNS setup with Domains4less can be obtuse - but here's how to setup a CNAME to your app:
From 'Manage Domains', hit 'Setup' on the domain you want. Next page, 'Advanced Control Panel' and login. Expand 'Services', open 'Domain Templates' and click 'Add a domain template'
Give your template a name (e.g 'yourdomain.co.nz'), and scroll down to the CNAME Records section. Click 'Add a CNAME record', Prefix field: www, Destination Address: YOURAPPNAME.azurewebsites.net.. Save.
Open 'Domains' in the left-side menu and click '[edit]' on the domain you want to setup, open the 'Advanced Mode' tab and choose your template from the Template drop-down list. Save, done.
Phew.

Resources