inserting into jsonb type column - python-3.x

Question

import psycopg2, json, requests, hidden
# Load secrets
secrets = hidden.secrets()
conn = psycopg2.connect(host=secrets['host'],port=secrets['port'],
....,connect_timeout=3)
cur = conn.cursor()
defaulturl = 'https://pokeapi.co/api/v2/pokemon?limit=100&offset=0'
sql = '''
CREATE TABLE IF NOT EXISTS pokeapi
(id INTEGER, body JSONB);
'''
print(sql)
cur.execute(sql)
response = requests.get(defaulturl)
js = json.loads(response.text)
# js is a library and i'm interested in the values of 'results' key.
results = js['results']
# 'results' is a list of libraries and i want to loop through each element of the list
# and extract the value of 'url' key
# I NEED TO INSERT EACH VALUE INTO pokeapi (body), note that 'body' is of type JSONB
for x in range(len(results)):
body = requests.get(results[x]['url'])
js_body = json.loads(body.text)
sql = f"INSERT INTO pokeapi (body) VALUES ('{js_body}')::JSONB";
cur.execute(sql, (defaulturl))
print('Closing database connection...')
conn.commit()
cur.close()
This script keeps throwing back an error:
CREATE TABLE IF NOT EXISTS pokeapi (id INTEGER, body text); Traceback
(most recent call last): File "pokeapi.py", line 45, in
cur.execute(sql, (defaulturl)) psycopg2.errors.SyntaxError: syntax error at or near "{" LINE 1: INSERT INTO pokeapi (body) VALUES
{'abilities': [{'ability':...
I have tried to insert into pokeapi (body) without casting to jsonb but I keep getting the same error back. Is there a fundamental that I'm missing?

Answer 1

You should pass the JSON string normally and not parse it, and without quotes and casting:
js_body = body.text
sql = "INSERT INTO pokeapi (body) VALUES (%s)";
cur.execute(sql, [js_body])
IMPORTANT: DO NOT USE format on random internet data! Always use
psycopg2's built-in parameter handling. It will correctly handle SQL
injection risks for you.
Currently you aren't using defaulturl, if you want to insert it then you need a column to insert it into. Also, you need to make the id auto increment:
sql = '''
CREATE TABLE IF NOT EXISTS pokeapi
("id" int8 NOT NULL GENERATED BY DEFAULT AS IDENTITY, body JSONB);
'''
If not you will have to supply an id with the body.
Finally, you should generally avoid trying to execute once every loop. If you have the memory for it, you should just loop over the payloads then use execute_values(): https://www.psycopg.org/docs/extras.html
rows = list()
for result in results:
response = requests.get(result['url'])
rows.append([response.text])
sql = "INSERT INTO pokeapi (body) VALUES %s";
sql_template = "(%s)"
execute_values(cur, sql, rows, sql_template)
(Also, for future reference, the requests library has a .json() method on the responses which can load the json string into python primitives for you. That said you don't need to parse the json in this case. https://docs.python-requests.org/en/master/user/quickstart/#json-response-content)

Answer 2

Here is the solution I ended coming up with. What I learnt was the need to understand deserialization of 'response' to proper python dictionary and then serialization of python dictionary before casting it to JSONB type.
import psycopg2, json, requests, hidden
# Load secrets
secrets = hidden.secrets()
conn = psycopg2.connect(host=secrets['host'],port=secrets['port'],
....,connect_timeout=3)
cur = conn.cursor()
defaulturl = 'https://pokeapi.co/api/v2/pokemon?limit=100&offset=0'
sql = '''
CREATE TABLE IF NOT EXISTS pokeapi
(id SERIAL, body JSONB); # <== CREATING id OF SERIAL TYPE HELPS AUTO-
# GENERATE ids of INTEGER TYPE.
'''
print(sql)
cur.execute(sql)
response = requests.get(defaulturl)
js = response.json() # <== THIS IS ONE OF THE CORRECTIONS, I NEEDED TO DE-
# SERIALIZE THE RESPONSE SO THAT IT'S A PROPER
# PYTHON DICTIONERY
# js is a library and i'm interested in the values of 'results' key.
results = js['results']
# 'results' is a list of libraries and i want to loop through each element of the list
# and extract the value of 'url' key
# I NEED TO INSERT EACH VALUE INTO pokeapi (body), note that 'body' is of type JSONB
for x in range(len(results)):
body = requests.get(results[x]['url'])
js_body = json.dumps(body) # <== 2ND MAJOR CORRECTION, I HAVE TO
# SERIALIZE THE PYTHON DICTIONERY/LIST
# TO BE ABLE TO CAST IT TO JSONB BELLOW
sql = f"INSERT INTO pokeapi (body) VALUES ('{js_body}'::JSONB)";
cur.execute(sql, (defaulturl))
print('Closing database connection...')
conn.commit()
cur.close()