I'm writing a script in python and am accessing an API. I can get some information I need but where I'm stuck is with nested queries. In my code below, first_name needs to equal what would essentially be result[customer->firstname] but I cannot figure out how to get that.
What is the proper syntax to get a nested query like that?
Orders-> customer -> firstname
for result in results['orders']:
order_status_info= self_api.which_api('order_statuses/%d' % result['order_status_id'])
for customer_blocked_reason in customer_blocked_reasons:
if customer_blocked_reason in order_status_info['name']:
customer_is_blocked = True
order_id = 0
order_date = result['ordered_at']
first_name = result [??????]
Is result a dict?
You can access nested dictionaries like so:
>>> d = {"level1": {"level2": "somedata"}}
>>> d["level1"]["level2"]
'somedata'
Related
Is there a way to aggregate values in a column in sqlite in one-to-many relationship into array?
For example, I have 2 tables like this:
Artists:
ArtistId name
1 AC/DC
2 Accept
Albums:
AlbumId ArtistId Title
1 1 For Those About To Rock We Salute You
2 1 Let There Be Rock
3 2 Balls to the Wall
4 2 Restless and Wild
When I just do a query with a join:
SELECT
Name,
Title
FROM
artists
JOIN albums USING(ArtistId)
WHERE artists.ArtistId = 1;
I get:
I found out that I can do group_concat:
SELECT
Name,
GROUP_CONCAT(Title)
FROM
artists
JOIN albums USING(ArtistId)
WHERE artists.ArtistId = 1;
To concatenate all values together:
But I still have to parse the coma-separated string with titles: For Those About To Rock We Salute You,Let There Be Rock in the code to get the array of titles for each artist.
I use Python and I'd prefer to get something like a tuple for each row:
(name, titlesArray)
A much easier way in this case for me would be to use json.loads and json.dumps functions to save all the "many" array members into the same row in the same table, instead of using the recommended way for databases to save values in different tables and then use joins to retrieve them: the "many" values is an array on the object, and it's just much easier to save and get them using just 2 functions: json.loads and json.dumps, compared to manually saving the "many" values into a separate table, create binding to the "one" value, then use group_concat to concat them into a string, and then parse it more to actually get my array back.
Is it possible to get an array of values, or do I have to do group_concat and parse the string?
You might not be able to receive an array from sqlite straight away, but you can achieve the result with a very little edit on your query and a single line in python.
group_concat supports a custom delimiter that you can use later to split the entries.
Let's assume you have something like this:
from typing import Typle
import sqlite3
def connect(file: str = None) -> sqlite3.Connection:
connection = None
try:
connection = sqlite3.connect(file)
except sqlite3.Error:
raise
return connection
def select(connection: sqlite3.Connection) -> Tuple(str, str)):
entry = None
try:
sql = """
SELECT
Name,
GROUP_CONCAT(Title)
FROM artists
JOIN albums USING(ArtistId)
WHERE artists.ArtistId = 1;
"""
cursor.execute(sql, parameters)
reply = cursor.fetchone()
if reply is not None:
entry = reply
except sqlite3.Error:
raise
finally:
cursor.close()
return entry
that you can use to connect to the database and select from it like so:
connection = connect(r"/path/to/file.sqlite3")
if connection is not None:
entry = select(connection)
connection.close()
It is not important if your query is inside a function or not, the important concept is that you are using python to do this query, and you can add some code to manipulate the values.
As you can see here group_concat accepts a separator that you can use to arbitrarily separate values.
Your new select function could be something like:
def select(connection: sqlite3.Connection) -> Tuple(str, Tuple(str, ...)):
entry = None
separator = r"|"
try:
sql = f"""
SELECT
Name,
GROUP_CONCAT(Title, {separator})
FROM artists
JOIN albums USING(ArtistId)
WHERE artists.ArtistId = 1;
"""
cursor.execute(sql, parameters)
reply = cursor.fetchone()
if reply is not None:
reply[1] = reply[1].split(separator)
entry = reply
except sqlite3.Error:
raise
finally:
cursor.close()
return entry
Without changing how you use this function, you would now have a tuple with all the titles.
Another idea you'd like to consider is to do a more specific select query, like:
select albums.Title
from albums
where albums.ArtistId = 1;
In this case, you can have a list of titles using: cursor.fetchall().
Of course the band name should be asked separately in this case.
Is there a way to put my cypher query into a python variable? I am able to put the non-parameter of the cypher into a variable but getting the error [![enter image description here][1]][1]
[1]: https://i.stack.imgur.com/mfRrp.png on trying to put the parameter part into a variable.
I am using pandas and py2neo to load a CSV into neo4j. df_data is my pandas dataframe with 2 columns and 3 rows.
for index,row in df_data.iterrows():
tx.evaluate('''MERGE (x:PATIENT_ID {property:$PATIENT_ID})
MERGE(y:GLB_ID {property:$GLB_ID})
MERGE (x)-[r:R_TYPE]->(y) ''', parameters = {'PATIENT_ID': int(row['PATIENT_ID']), 'GLB_ID': int(row['GLB_ID'])})
My code runs without issues if I run the following code storing non-parameter part into cq:
for index,row in df_data.iterrows():
tx.evaluate(cq, parameters = {'PATIENT_ID': int(row['PATIENT_ID']), 'GLB_ID': int(row['GLB_ID'])})
I am looking for a way to store the parameter part into a python variable
I used dictionary to get this resolved. I used dictionary over the cypher query to get it resolved:
for index,row in df_data.iterrows():
t_dict = {}
for k in p_dict.keys():
try:
t_dict[k] = int(row[k])
except TypeError:
t_dict[k] = row[k]
tx.evaluate(cq,t_dict)
Tools: Peewee 3, SQLite, Python 3
Official documentation for Peewee 3 recursive common table expression (cte):
http://docs.peewee-orm.com/en/latest/peewee/querying.html#common-table-expressions
I am storing a family tree in a simple self-referencing table called Person.
Structure (see below): id, name, parent, custom_order
Notes:
- parent field equals null if this person is an ancestor / root item, otherwise equals to id of parent record if this person is a child
- custom_order is a float number (score to determine who is the user's favourite person)
Objective:
I would like to retrieve the whole family tree and ORDER the results FIRST by parent and SECOND by custom_order.
Issue:
I managed to get the results list but the ORDER is wrong.
DB model
class Person(Model):
name = CharField()
parent = ForeignKeyField('self', backref='children', null = True)
custom_order = FloatField()
Note: if parent field is null then it's a root item
Query code
# Define the base case of our recursive CTE. This will be people that have a null parent foreign-key.
Base = Person.alias()
base_case = (Base
.select(Base)
.where(Base.parent.is_null())
.cte('base', recursive=True))
# Define the recursive terms.
RTerm = Person.alias()
recursive = (RTerm
.select(RTerm)
.join(base_case, on=(RTerm.parent == base_case.c.id)))
# The recursive CTE is created by taking the base case and UNION ALL with the recursive term.
cte = base_case.union_all(recursive)
# We will now query from the CTE to get the people
query = cte.select_from(cte.c.id, cte.c.name, cte.c.parent_id, cte.c.custom_order).order_by(cte.c.parent_id, cte.c.custom_order)
print(query.sql())
Printed query syntax
('WITH RECURSIVE "base" AS
(
SELECT "t1"."id", "t1"."name", "t1"."parent_id", "t1"."custom_order" FROM "person" AS "t1" WHERE ("t1"."parent_id" IS ?)
UNION ALL
SELECT "t2"."id", "t2"."name", "t2"."parent_id", "t2"."custom_order" FROM "person" AS "t2" INNER JOIN "base" ON ("t2"."parent_id" = "base"."id")
)
SELECT "base"."id", "base"."name", "base"."parent_id" FROM "base"
ORDER BY "base"."parent_id", "base"."custom_order"',
[None])
Root of the problem
The code posted in the question works correctly. I verified it by printing the query results to the console:
query = cte.select_from(cte.c.id, cte.c.name, cte.c.parent_id, cte.c.custom_order).order_by(cte.c.parent_id, cte.c.custom_order).dicts()
print(json.dumps(list(query), indent=4))
The problem originated from the fact that I was passing the query results to a nested python Dictionary before printing them to the console, BUT the Python Dictionary is unordered. So, no wonder the printed results were in a different order than the database results.
Solution
Use a Python Ordered Dictionary if you want to store the query results in a fixed order:
import collections
treeDictionary = collections.OrderedDict()
I've scraped some websites and stored the html info in a sqlite database. Now, I want to extract and store the email addresses. I'm able to successfully extract and print the id and emails. But, I keep getting TypeError: "'NoneType' object is not subscriptable" and "sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type" when I try to update the database with these new email addresses.
I've verified that the data types I'm using in the update statement are the same as my database (id is class int and email is str). I've googled a bunch of different examples and mucked around with the syntax alot.
I also tried removing the Where Clause in the update statement but got the same errors.
import sqlite3
import re
conn = sqlite3.connect('spider.sqlite')
cur = conn.cursor()
x = cur.execute('SELECT id, html FROM Pages WHERE html is NOT NULL and email is NULL ORDER BY RANDOM()').fetchone()
#print(x)#for testing purposes
for row in x:
row = cur.fetchone()
id = row[0]
html = row[1]
email = re.findall(b'[a-z0-9\.\-+_]+#[a-z0-9\.\-+_]+\.[a-z]+', html)
#print(email)#testing purposes
if not email:
email = 'no email found'
print(id, email)
cur.execute('''UPDATE pages SET email = ? WHERE id = ? ''', (email, id))
conn.commit
I want the update statement to update the database with the extracted email addresses for the appropriate row.
There are a few things going on here.
First off, you don't want to do this:
for row in x:
row = cur.fetchone()
If you want to iterate over the results returned by the query, you should consider something like this:
for row in cur.fetchall():
id = row[0]
html = row[1]
# ...
To understand the rest of the errors you are seeing, let's take a look at them step by step.
TypeError: "'NoneType' object is not subscriptable":
This is likely generated here:
row = cur.fetchone()
id = row[0]
Cursor.fetchone returns None if the executed query doesn't match any rows or if there are no rows left in the result set. The next line, then, is trying to do None[0] which would raise the error in question.
sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type:
re.findall returns a list of non-overlapping matches, not an individual match. There's no support for binding a Python list to a sqlite3 text column type. To fix this, you'll need to get the first element from the matched list (if it exists) and then pass that as your email parameter in the UPDATE.
.findall() returns a list.
You want to iterate over that list:
for email in re.findall(..., str(html)):
print(id, email)
cur.execute(...)
Not sure what's going on with that b'[a-z...' expression.
Recommend you use a raw string instead: r'[a-z...'.
It handles regex \ backwhacks nicely.
What is the correct method to have the list (countryList) be available via %s in the SQL statement?
# using psycopg2
countryList=['UK','France']
sql='SELECT * from countries WHERE country IN (%s)'
data=[countryList]
cur.execute(sql,data)
As it is now, it errors out after trying to run "WHERE country in (ARRAY[...])". Is there a way to do this other than through string manipulation?
Thanks
For the IN operator, you want a tuple instead of list, and remove parentheses from the SQL string.
# using psycopg2
data=('UK','France')
sql='SELECT * from countries WHERE country IN %s'
cur.execute(sql,(data,))
During debugging you can check that the SQL is built correctly with
cur.mogrify(sql, (data,))
To expland on the answer a little and to address named parameters, and converting lists to tuples:
countryList = ['UK', 'France']
sql = 'SELECT * from countries WHERE country IN %(countryList)s'
cur.execute(sql, { # You can pass a dict for named parameters rather than a tuple. Makes debugging hella easier.
'countryList': tuple(countryList), # Converts the list to a tuple.
})
You could use a python list directly as below. It acts like the IN operator in SQL and also handles a blank list without throwing any error.
data=['UK','France']
sql='SELECT * from countries WHERE country = ANY (%s)'
cur.execute(sql,(data,))
source:
http://initd.org/psycopg/docs/usage.html#lists-adaptation
Since the psycopg3 question was marked as a duplicate, I'll add the answer to that here too.
In psycopg3, you can not use in %s with a tuple, like you could in psycopg2. Instead you have to use ANY() and wrap your list inside another list:
conn.execute("SELECT * FROM foo WHERE id = ANY(%s)", [[10,20,30]])
Docs: https://www.psycopg.org/psycopg3/docs/basic/from_pg2.html#you-cannot-use-in-s-with-a-tuple