Unable to programmatically download Kaggle dataset "nih-chest-xrays/data" specific file Version 3 - python-3.x

Question

I need to download Kaggle's NIH Chest X-rays Dataset programmatically, specifically the Data_Entry_2017.csv.
I want to download it if doesn't exist on system, and redownload it if it's updation date exceeds last download date.
I know I can do this manually, but I'll appreciate if there was a way to do it programmatically.
I have done:
Direct download:
kaggle.api.dataset_download_file(dataset='nih-chest-xrays/data', file_name='Data_Entry_2017.csv', path='data/')
This gives:
File "C:\Program Files\Python311\Lib\site-packages\kaggle\rest.py", line 241, in request
raise ApiException(http_resp=r)
kaggle.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Sun, 18 Dec
2022 03:35:46 GMT', 'Access-Control-Allow-Credentials': 'true', 'Set-Cookie':
'ka_sessionid=bdaa2b71c677d6e48bf93fc71b85a9c6; max-age=2626560; path=/, GCLB=CIeroe2g5sixeA;
path=/; HttpOnly', 'Transfer-Encoding': 'chunked', 'Vary': 'Accept-Encoding', 'Turbolinks-
Location': 'https://www.kaggle.com/api/v1/datasets/download/nih-chest-
xrays/data/Data_Entry_2017.csv', 'X-Kaggle-MillisecondsElapsed': '526', 'X-Kaggle-RequestId':
'03383f8cdac3aa4b6ab83d5dadbc507c', 'X-Kaggle-ApiVersion': '1.5.12', 'X-Frame-Options':
'SAMEORIGIN', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload',
'Content-Security-Policy': "object-src 'none'; script-src 'nonce-gn6A4mfrbofbZRyE9gREnA=='
'report-sample' 'unsafe-inline' 'unsafe-eval' 'strict-dynamic' https: http:; frame-src 'self'
https://www.kaggleusercontent.com https://www.youtube.com/embed/ https://polygraph-cool.github.io
https://www.google.com/recaptcha/ https://form.jotform.com https://submit.jotform.us
https://submit.jotformpro.com https://submit.jotform.com https://www.docdroid.com
https://www.docdroid.net https://kaggle-static.storage.googleapis.com https://kaggle-static-
staging.storage.googleapis.com https://kkb-dev.jupyter-proxy.kaggle.net https://kkb-
staging.jupyter-proxy.kaggle.net https://kkb-production.jupyter-proxy.kaggle.net https://kkb-
dev.firebaseapp.com https://kkb-staging.firebaseapp.com
https://kkb-production.firebaseapp.com https://kaggle-metastore-test.firebaseapp.com
https://kaggle-metastore.firebaseapp.com https://apis.google.com https://content-
sheets.googleapis.com/ https://accounts.google.com/ https://storage.googleapis.com
https://docs.google.com https://drive.google.com https://calendar.google.com/;
base-uri 'none'; report-uri https://csp.withgoogle.com/csp/kaggle/20201130;", 'X-Content-Type-
Options': 'nosniff', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'Via': '1.1 google',
'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
HTTP response body: b'{"code":404,"message":"Not found"}'
Same for any other files in this dataset:
kaggle.api.dataset_download_file(dataset='nih-chest-xrays/data', file_name='test_list.txt', path='data/')
Which again gives
kaggle.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Sun, 18 Dec
2022 03:37:20 GMT', 'Access-Control-Allow-Credentials': 'true', 'Set-Cookie':
'ka_sessionid=b59676e67aa4a36e92657a7fa50e95f3; max-age=2626560; path=/, GCLB=CKOV0bPZ-p6ZSA;
path=/; HttpOnly', 'Transfer-Encoding': 'chunked', 'Vary': 'Accept-Encoding', 'Turbolinks-
Location': 'https://www.kaggle.com/api/v1/datasets/download/nih-chest-xrays/data/test_list.txt',
'X-Kaggle-MillisecondsElapsed': '520', 'X-Kaggle-RequestId': '79fde70e368d73ba7c767a007bacc086',
'X-Kaggle-ApiVersion': '1.5.12', 'X-Frame-Options': 'SAMEORIGIN', 'Strict-Transport-Security':
'max-age=63072000; includeSubDomains; preload', 'Content-Security-Policy': "object-src 'none';
script-src 'nonce-U88j/npGgHfKpSCwewojZg==' 'report-sample' 'unsafe-inline' 'unsafe-eval'
'strict-dynamic' https: http:; frame-src 'self' https://www.kaggleusercontent.com
https://www.youtube.com/embed/ https://polygraph-cool.github.io https://www.google.com/recaptcha/
https://form.jotform.com https://submit.jotform.us https://submit.jotformpro.com
https://submit.jotform.com https://www.docdroid.com https://www.docdroid.net https://kaggle-
static.storage.googleapis.com https://kaggle-static-staging.storage.googleapis.com https://kkb-
dev.jupyter-proxy.kaggle.net https://kkb-staging.jupyter-proxy.kaggle.net https://kkb-
production.jupyter-proxy.kaggle.net https://kkb-dev.firebaseapp.com https://kkb-
staging.firebaseapp.com https://kkb-production.firebaseapp.com https://kaggle-metastore-
test.firebaseapp.com https://kaggle-metastore.firebaseapp.com https://apis.google.com
https://content-sheets.googleapis.com/ https://accounts.google.com/
https://storage.googleapis.com https://docs.google.com https://drive.google.com
https://calendar.google.com/; base-uri 'none'; report-uri
https://csp.withgoogle.com/csp/kaggle/20201130;", 'X-Content-Type-Options': 'nosniff', 'Referrer-
Policy': 'strict-origin-when-cross-origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443";
ma=2592000,h3-29=":443"; ma=2592000'})
HTTP response body: b'{"code":404,"message":"Not found"}'
The strange thing is that I can search this dataset:
dataset_list = kaggle.api.dataset_list(search='nih')
print(dataset_list)
Which gives:
[nih-chest-xrays/data, nih-chest-xrays/sample, allen-institute-for-ai/CORD-19-research-challenge,
kmader/nih-deeplesion-subset, nickuzmenkov/nih-chest-xrays-tfrecords, iarunava/cell-images-for-
detecting-malaria, tunguz/covid19-genomes, nlm-nih/nlm-rxnorm, akhileshdkapse/nih-image-600x600-
data, tunguz/nih-awarded-grant-text, danielmadmon/nih-xray-dataset-tfrec-with-labels,
miracle9to9/files1, nlm-nih/rxnorm-drug-name-conventions, kmader/rsna-bone-age,
redwankarimsony/chestxray8-dataframe, akhileshdkapse/nih-dataframe, luigisaetta/cxr-tfrec256-
may2020, kokyew93/nihdata, ammarali32/startingpointschestx, dannellyz/cancer-incidence-totals-
and-rates-per-us-county]
It's literally the first one on the list.
And checking its metadata:
dataset = vars(dataset_list[0])
print(dataset)
Which gives:
{'subtitleNullable': 'Over 112,000 Chest X-ray images from more than 30,000 unique patients',
'creatorNameNullable': 'Timo Bozsolik', 'creatorUrlNullable': 'timoboz', 'totalBytesNullable':
45096150231, 'urlNullable': 'https://www.kaggle.com/datasets/nih-chest-xrays/data',
'licenseNameNullable': 'CC0: Public Domain', 'descriptionNullable': None, 'ownerNameNullable':
'National Institutes of Health Chest X-Ray Dataset', 'ownerRefNullable': 'nih-chest-xrays',
'titleNullable': 'NIH Chest X-rays', 'currentVersionNumberNullable': 3,
'usabilityRatingNullable': 0.7352941, 'id': 5839, 'ref': 'nih-chest-xrays/data', 'subtitle':
'Over 112,000 Chest X-ray images from more than 30,000 unique patients', 'hasSubtitle': True,
'creatorName': 'Timo Bozsolik', 'hasCreatorName': True, 'creatorUrl': 'timoboz', 'hasCreatorUrl':
True, 'totalBytes': 45096150231, 'hasTotalBytes': True, 'url':
'https://www.kaggle.com/datasets/nih-chest-xrays/data', 'hasUrl': True, 'lastUpdated':
datetime.datetime(2018, 2, 21, 20, 52, 23), 'downloadCount': 65214, 'isPrivate': False,
'isFeatured': False, 'licenseName': 'CC0: Public Domain', 'hasLicenseName': True, 'description':
'', 'hasDescription': False, 'ownerName': 'National Institutes of Health Chest X-Ray Dataset',
'hasOwnerName': True, 'ownerRef': 'nih-chest-xrays', 'hasOwnerRef': True, 'kernelCount': 323,
'title': 'NIH Chest X-rays', 'hasTitle': True, 'topicCount': 0, 'viewCount': 454490, 'voteCount':
967, 'currentVersionNumber': 3, 'hasCurrentVersionNumber': True, 'usabilityRating': 0.7352941,
'hasUsabilityRating': True, 'tags': [biology, health, medicine, computer science, software,
health conditions], 'files': [], 'versions': [], 'size': '42GB'}
I can even list the files for that dataset, and it's visible:
print(kaggle.api.dataset_list_files('nih-chest-xrays/data').files)
Which gives:
[BBox_List_2017.csv, LOG_CHESTXRAY.pdf, ARXIV_V5_CHESTXRAY.pdf, README_CHESTXRAY.pdf,
train_val_list.txt, Data_Entry_2017.csv, FAQ_CHESTXRAY.pdf, test_list.txt]
A problem I noticed was that kaggle.api.dataset_download_file was only fetching from Version 1, where these files don't exist. This was further confirmed when I successfully fetched an image file from Version 1:
res = kaggle.api.dataset_download_file(dataset='nih-chest-xrays/data', file_name='images_001/images/00000001_000.png', path='data/')
print(res)
Which downloads the image and prints True.
My data is Version 3. Is there any way to configure kaggle to use Version 3?
Note that I can download the whole 45GB dataset, but I only require the single file Data_Entry_2017.csv.
Even more strangely, it's also throwing an error when checking its status:
print(kaggle.api.dataset_status(dataset='nih-chest-xrays/data'))
Which gives:
Traceback (most recent call last):
File "C:\Users\jaide\OneDrive\Documents\MScCS2022\Data-Mining-I\chest-datasets\visualise.py",
line 117, in <module>
full_df = load_nih_data()
^^^^^^^^^^^^^^^
File "C:\Users\jaide\OneDrive\Documents\MScCS2022\Data-Mining-I\chest-datasets\visualise.py",
line 89, in load_nih_data
print(kaggle.api.dataset_status(dataset='nih-chest-xrays/data'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\kaggle\api\kaggle_api_extended.py", line
1135, in dataset_status
self.datasets_status_with_http_info(owner_slug=owner_slug,
File "C:\Program Files\Python311\Lib\site-packages\kaggle\api\kaggle_api.py", line 1910, in
datasets_status_with_http_info
return self.api_client.call_api(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\kaggle\api_client.py", line 329, in call_api
return self.__call_api(resource_path, method,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\kaggle\api_client.py", line 161, in
__call_api
response_data = self.request(
^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\kaggle\api_client.py", line 351, in request
return self.rest_client.GET(url,
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\kaggle\rest.py", line 247, in GET
return self.request("GET", url,
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\kaggle\rest.py", line 241, in request
raise ApiException(http_resp=r)
kaggle.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Sun, 18 Dec
2022 04:14:32 GMT', 'Access-Control-Allow-Credentials': 'true', 'Set-Cookie':
'ka_sessionid=1ab6e66561c7d9ae43ead26df25beea3; max-age=2626560; path=/, GCLB=CNS4vKLYyPuCpgE;
path=/; HttpOnly', 'Transfer-Encoding': 'chunked', 'Vary':
'Accept-Encoding', 'Turbolinks-Location': 'https://www.kaggle.com/api/v1/datasets/status/nih-
chest-xrays/data', 'X-Kaggle-MillisecondsElapsed': '46', 'X-Kaggle-RequestId':
'66473cc55f62c39d2deb0aa4cd2953a7', 'X-Kaggle-ApiVersion': '1.5.12', 'X-Frame-Options':
'SAMEORIGIN', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload',
'Content-Security-Policy': "object-src 'none'; script-src 'nonce-5AJVxKPJRg4Gsi/iT3D5Aw=='
'report-sample' 'unsafe-inline' 'unsafe-eval' 'strict-dynamic' https: http:; frame-src 'self'
https://www.kaggleusercontent.com https://www.youtube.com/embed/ https://polygraph-cool.github.io
https://www.google.com/recaptcha/ https://form.jotform.com https://submit.jotform.us
https://submit.jotformpro.com https://submit.jotform.com https://www.docdroid.com
https://www.docdroid.net https://kaggle-static.storage.googleapis.com https://kaggle-static-
staging.storage.googleapis.com https://kkb-dev.jupyter-proxy.kaggle.net https://kkb-
staging.jupyter-proxy.kaggle.net https://kkb-production.jupyter-proxy.kaggle.net https://kkb-
dev.firebaseapp.com https://kkb-staging.firebaseapp.com https://kkb-production.firebaseapp.com
https://kaggle-metastore-test.firebaseapp.com https://kaggle-metastore.firebaseapp.com
https://apis.google.com https://content-sheets.googleapis.com/ https://accounts.google.com/
https://storage.googleapis.com https://docs.google.com https://drive.google.com
https://calendar.google.com/; base-uri 'none'; report-uri
https://csp.withgoogle.com/csp/kaggle/20201130;", 'X-Content-Type-Options': 'nosniff', 'Referrer-
Policy': 'strict-origin-when-cross-origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443";
ma=2592000,h3-29=":443"; ma=2592000'})
HTTP response body: {"code":404,"message":"Not found"}
CLI download
Check if dataset exists:
C:\Users\jaide\OneDrive\Documents\MScCS2022\Data-Mining-I\chest-datasets>kaggle datasets list -s "nih-chest-xrays/data"
ref title size lastUpdated downloadCount voteCount usabilityRating
------------------------------------------------------------- ----------------------------------------------- ----- ------------------- ------------- --------- ---------------
nickuzmenkov/nih-chest-xrays-tfrecords NIH Chest X-rays TFRecords 11GB 2021-03-09 04:49:23 2121 119 0.9411765
nih-chest-xrays/data NIH Chest X-rays 42GB 2018-02-21 20:52:23 65221 967 0.7352941
nih-chest-xrays/sample Random Sample of NIH Chest X-ray Dataset 4GB 2017-11-23 02:58:24 14930 241 0.7647059
ammarali32/startingpointschestx StartingPoints-ChestX 667MB 2021-02-27 17:35:19 668 68 0.6875
redwankarimsony/chestxray8-dataframe ChestX-ray8_DataFrame 51MB 2020-07-12 02:28:33 416 10 0.64705884
amanullahasraf/covid19-pneumonia-normal-chest-xray-pa-dataset COVID19_Pneumonia_Normal_Chest_Xray_PA_Dataset 2GB 2020-07-13 05:54:22 1482 14 0.8125
harshsoni/nih-chest-xray-tfrecords NIH Chest X-ray TFRecords 42GB 2020-09-26 18:48:04 52 6 0.5294118
amanullahasraf/covid19-pneumonia-normal-chest-xraypa-dataset COVID19_Pneumonia_Normal_Chest_Xray(PA)_Dataset 1GB 2020-06-29 08:28:01 161 3 0.5882353
roderikmogot/nih-chest-x-ray-models nih chest x ray models 2GB 2022-08-17 01:32:33 66 4 0.47058824
ericspod/project-monai-2020-bootcamp-challenge-dataset Project MONAI 2020 Bootcamp Challenge Dataset 481MB 2021-01-25 01:11:42 22 2 0.6875
sunghyunjun/nih-chest-xrays-600-jpg-dataset NIH Chest X-rays 600 JPG Dataset 7GB 2021-03-15 07:53:12 35 3 0.64705884
zhuangjw/chest-xray-cleaned chest_xray_cleaned 2GB 2019-11-30 19:05:24 13 1 0.375
kambarakun/nih-chest-xrays-trained-models NIH Chest X-rays: Trained Models 3GB 2020-04-30 11:52:56 10 2 0.3125
jessevent/all-kaggle-datasets Complete Kaggle Datasets Collection 390KB 2018-01-16 12:32:58 2256 118 0.8235294
vzaguskin/nih-chest-xrays-tfrecords-756 NIH Chest X-rays tfrecords 756 11GB 2021-02-25 14:25:12 3 1 0.29411766
Yes, it is the second one. Now does it have my required file?
C:\Users\jaide\OneDrive\Documents\MScCS2022\Data-Mining-I\chest-datasets>kaggle datasets files "nih-chest-xrays/data"
name size creationDate
---------------------- ----- -------------------
ARXIV_V5_CHESTXRAY.pdf 9MB 2018-02-21 20:52:23
FAQ_CHESTXRAY.pdf 71KB 2018-02-21 20:52:23
Data_Entry_2017.csv 7MB 2018-02-21 20:52:23
BBox_List_2017.csv 90KB 2018-02-21 20:52:23
README_CHESTXRAY.pdf 827KB 2018-02-21 20:52:23
LOG_CHESTXRAY.pdf 4KB 2018-02-21 20:52:23
train_val_list.txt 1MB 2018-02-21 20:52:23
test_list.txt 425KB 2018-02-21 20:52:23
Yes it does. Now download it.
C:\Users\jaide\OneDrive\Documents\MScCS2022\Data-Mining-I\chest-datasets>kaggle datasets download -f "Data_Entry_2017.csv" -p "data/" "nih-chest-xrays/data"
404 - Not Found
Nope, error 404.
How do I download Kaggle's NIH Chest X-rays Dataset programmatically, specifically the Data_Entry_2017.csv?
Any help is appreciated!
I have checked:
I checked out Kaggle Python API, but it doesn't appear to have any way to configure Version.
I checked out Kaggle API on github, but it also doesn't appear to have any ways to configure Version.
Related Questions Read:
Kaggle Dataset Download - is irrelevant.
Download a Kaggle Dataset - is downloading competition files.
How to download data set into Colab? Stuck with a problem, it says "401 - Unauthorized"? - They had api key issues
How to load just one chosen file of a way too large Kaggle dataset from Kaggle into Colab - Is related to kaggle and jupyter notebooks, I am currently just using a python script.
Trouble turning comorbidity data into a table using Python and Pandas - Are again using Kernels (Kaggle Notebooks).
How to download kaggle dataset? - Downloads the whole dataset.