Formatting json data to convert to xslx - python-3.x

Question

I am downloading data in json format in Python 3.7 and trying to display it as an Excel spreadsheet. I tried using pandas, but I think that there is a problem in getting the data into a dataframe. I also tried a csv approach but that was not successful.
I am new to learning python, so maybe there is something obvious I am missing.
import json, requests,urllib,csv
import pandas as pd
url = urllib.request.urlopen('https://library.brown.edu/search/solr_pub/iip/?start=0&rows=100&indent=on&wt=json&q=*')
str_response = url.read().decode('utf-8')
jsondata=json.loads(str_response)
df = pd.DataFrame(jsondata)
I was hoping to get a number of rows for each item, e.g., zoor0353, with the columns for each of the keys associated with it (e.g., region, date_desc, etc. -there are quite a few). Instead, it seemed only to take the first section, returning:
responseHeader \
QTime 1
docs NaN
numFound NaN
params {'q': '*', 'indent': 'on', 'start': '0', 'rows...
start NaN
status 0
response
QTime NaN
docs [{'inscription_id': 'zoor0353', 'metadata': ['...
numFound 4356
params NaN
start 0
status NaN
I tried this with normalization method, but did no better. Ultimately, I would like to use a dataset made of an appended file of many calls to this api, and am also wondering if I will need to manipulate data, and how, to get it to work with pandas.

Answer 1

Since I'm not sure what exactly you need to get, I'll show you
how to use pandas.io.json.json_normalize
how to pick extract key/values
json_normalize
import requests
import pandas as pd
from pandas.io.json import json_normalize
url = 'https://library.brown.edu/search/solr_pub/iip/?start=0&rows=100&indent=on&wt=json&q=*'
r = requests.get(url).json()
df = json_normalize(r['response']['docs']).T
This is a sample output; each column is a separate item of r['response']['docs'] containing values, and rows are the keys for said values.
Data are all messed up, and also some of the keys you probably don't want to use. That's why I think it's better to extract specific keys and values.
df = pd.DataFrame()
query = r['response']['docs']
for i in range(len(query)):
records = [
query[i]['inscription_id'], query[i]['city'], query[i]['_version_'],
query[i]['type'][0], query[i]['language_display'][0], query[i]['physical_type'][0]
]
# recs.append(records)
data = pd.DataFrame.from_records([records], columns=['inscription_id', 'city', '_version_', 'type', 'language_display', 'physical_type'])
df = df.append(data)
# sample data
inscription_id city _version_ type language_display physical_type
0 unkn0103 Unknown 1613822149755142144 prayer Hebrew other_object
0 zoor0391 Zoora 1613822173978296320 funerary.epitaph Greek tombstone
0 zoor0369 Zoora 1613822168079007744 funerary.epitaph Greek tomb
0 zoor0378 Zoora 1613822170509606912 funerary.epitaph Greek tombstone
0 zoor0393 Zoora 1613822174648336384 funerary.epitaph Greek tombstone
edit
query=r[][] - what is it and how can I learn more about it?
r = requests.get(url).json()
by using .json() at the end of requests.get method, I converted the response object (data from the API) to a dict type. Hence, r is now a dictionary containing keys:
r.keys()
# dict_keys(['responseHeader', 'response'])
to access values of a specific key, I use a format: dictionary[key][key_inside_of_another_key]:
r['response'].keys()
# dict_keys(['numFound', 'start', 'docs'])
r['response']['docs'][0]
# sample output
{'inscription_id': 'zoor0353',
'metadata': ['zoor0353',
'Negev',
'Zoora',
'Zoora, Negev. cemetery in An Naq. \nNegev.\n Zoora. Found by local inhabitants in the northwest corner of the Bronze\n Age, Byzantine and Islamic cemetery in the An Naq neighborhood south of\n the Wadi al-Hasa, probably in secondary use in later graves. \n',
'funerary.epitaph',
...
...
...
# [0] here is an item inside of a key - in the case of your data,
# there were 100 such items: [0, 1, .. 99] inside of r['response']['docs']
# to access item #4, you'd use ['response']['docs'][4]
That's how you navigate through a dictionary. Now, to access specific key, say inscription_id or _version_:
r['response']['docs'][0]['inscription_id']
# 'zoor0353'
r['response']['docs'][99]['inscription_id']
# 'mger0001'
r['response']['docs'][33]['_version_']
# 1613822151126679552
Lastly, to iterate through all rows, or items of data, I used a for loop: i here is a substitude for a range of numbers representing each item of your data -- from r['response']['docs'][1] to r['response']['docs'][99].