What is the correct method to have the list (countryList) be available via %s in the SQL statement?
# using psycopg2
countryList=['UK','France']
sql='SELECT * from countries WHERE country IN (%s)'
data=[countryList]
cur.execute(sql,data)
As it is now, it errors out after trying to run "WHERE country in (ARRAY[...])". Is there a way to do this other than through string manipulation?
Thanks
For the IN operator, you want a tuple instead of list, and remove parentheses from the SQL string.
# using psycopg2
data=('UK','France')
sql='SELECT * from countries WHERE country IN %s'
cur.execute(sql,(data,))
During debugging you can check that the SQL is built correctly with
cur.mogrify(sql, (data,))
To expland on the answer a little and to address named parameters, and converting lists to tuples:
countryList = ['UK', 'France']
sql = 'SELECT * from countries WHERE country IN %(countryList)s'
cur.execute(sql, { # You can pass a dict for named parameters rather than a tuple. Makes debugging hella easier.
'countryList': tuple(countryList), # Converts the list to a tuple.
})
You could use a python list directly as below. It acts like the IN operator in SQL and also handles a blank list without throwing any error.
data=['UK','France']
sql='SELECT * from countries WHERE country = ANY (%s)'
cur.execute(sql,(data,))
source:
http://initd.org/psycopg/docs/usage.html#lists-adaptation
Since the psycopg3 question was marked as a duplicate, I'll add the answer to that here too.
In psycopg3, you can not use in %s with a tuple, like you could in psycopg2. Instead you have to use ANY() and wrap your list inside another list:
conn.execute("SELECT * FROM foo WHERE id = ANY(%s)", [[10,20,30]])
Docs: https://www.psycopg.org/psycopg3/docs/basic/from_pg2.html#you-cannot-use-in-s-with-a-tuple
Related
how to make a Multidimensional Dictionary with multiple keys and value and how to print its keys and values?
from this format:
main_dictionary= { Mainkey: {keyA: value
keyB: value
keyC: value
}}
I tried to do it but it gives me an error in the manufacturer. here is my code
car_dict[manufacturer] [type]= [( sedan, hatchback, sports)]
Here is my error:
File "E:/Programming Study/testupdate.py", line 19, in campany
car_dict[manufacturer] [type]= [( sedan, hatchback, sports)]
KeyError: 'Nissan'
And my printing code is:
for manufacuted_by, type,sedan,hatchback, sports in cabuyao_dict[bgy]:
print("Manufacturer Name:", manufacuted_by)
print('-' * 120)
print("Car type:", type)
print("Sedan:", sedan)
print("Hatchback:", hatchback)
print("Sports:", sports)
Thank you! I'm new in Python.
I think you have a slight misunderstanding of how a dict works, and how to "call back" the values inside of it.
Let's make two examples for how to create your data-structure:
car_dict = {}
car_dict["Nissan"] = {"types": ["sedan", "hatchback", "sports"]}
print(car_dict) # Output: {'Nissan': {'types': ['sedan', 'hatchback', 'sports']}}
from collections import defaultdict
car_dict2 = defaultdict(dict)
car_dict2["Nissan"]["types"] = ["sedan", "hatchback", "sports"]
print(car_dict2) # Output: defaultdict(<class 'dict'>, {'Nissan': {'types': ['sedan', 'hatchback', 'sports']}})
In both examples above, I first create a dictionary, and then on the row after I add the values I want it to contain. In the first example, I give car_dict the key "Nissan" and set it's values to a new dictionary containing some values.
In the second example I use defaultdict(dict) which basically has the logic of "if i am not given a value for key then use the factory (dict) to create a value for it.
Can you see the difference of how to initiate the values inside of both of the different methods?
When you called car_dict[manufacturer][type] in your code, you hadn't yet initiated car_dict["Nissan"] = value, so when you tried to retrieve it, car_dict returned a KeyError.
As for printing out the values, you can do something like this:
for key in car_dict:
manufacturer = key
car_types = car_dict[key]["types"]
print(f"The manufacturer '{manufacturer}' has the following types:")
for t in car_types:
print(t)
Output:
The manufacturer 'Nissan' has the following types:
sedan
hatchback
sports
When you loop through a dict, you are looping through only the keys that are contained in it by default. That means that we have to retrieve the values of key inside of the loop itself to be able to print them correctly.
Also as a side note: You should try to avoid using Built-in's names such as type as variable names, because you then overwrite that functions namespace, and you can have some problems in the future when you have to do comparisons of types of variables.
I am trying to compare a list of links to the liks stored in an sqlite database.
assuming the links in the database are:
link.com\page1
link.com\page2
link.com\page3
I have written the following code to chick if a given link exists in the database and adds it if it did not exist.
links = ['link.com\page2', 'link.com\page4']
c.execute('SELECT link FROM ads')
previouslinks = c.fetchall()
for l in links:
if l not in previouslinks:
c.execute('''INSERT INTO ads(link) VALUES(?)''', (l))
conn.commit()
else:
pass
the problem is even though the link is in the database, the script does not recognise it!
when I try to print previouslinks variable, results look something like this:
[('link.com\page1',), ('link.com\page2',), ('link.com\page3',)]
I think the problem is with the extra parentheses and commas, but I am not exactly sure.
fetchall() returns a list of rows, where each row is a tuple containing all column values. A tuple containing a string is not the same as the string.
You have to extract the values from the rows (and you don't need fetchall() when iterating over a cursor):
previouslinks = [row[0] for row in c]
I'm trying to create the Mysql Insert query like this for inserting million of records:
INSERT INTO mytable (fee, fi) VALUES
('data1',96)
,('data2',33)
,('boot',17)
My values is stored as tuple in the list:
datatuplst = [("data1",96), ("data2", 33),("data3", 17)]
My code:
c3 = con.cursor()
c3.execute("INSERT INTO `bigdataloadin` (`block_key`, `id`) VALUES %s" %','.join(datatuplst))
This is not working and I'm getting error:
TypeError: sequence item 0: expected str instance, tuple found
Need help on how to create the dynamic query with values stored in tuples list.
You can generate the string that you need. The error is explanatory enough. When it is ','.join(datatuplst) the interpreter is forced to join tuples. So using list comprehension you can say this instead:
','.join([str(el) for el in datatuplst])
The output for this statement is going to be: "('data1', 96),('data2', 33),('data3', 17)"
Then your actual INSERT statement will be interpreted as follows:
"INSERT INTO `bigdataloadin` (`block_key`, `id`) VALUES ('data1', 96),
('data2', 33),('data3', 17)"
Good luck!
I have a List of Lists that looks like this (Python3):
myLOL = ["['1466279297', '703.0']", "['1466279287', '702.0']", "['1466279278', '702.0']", "['1466279268', '706.0']", "['1466279258', '713.0']"]
I'm trying to use a list comprehension to convert the first item of each inner list to an int and the second item to a float so that I end up with this:
newLOL = [[1466279297, 703.0], [1466279287, 702.0], [1466279278, 702.0], [1466279268, 706.0], [1466279258, 713.0]]
I'm learning list comprehensions, can somebody please help me with this syntax?
Thank you!
[edit - to explain why I asked this question]
This question is a means to an end - the syntax requested is needed for testing. I'm collecting sensor data on a ZigBee network, and I'm using an Arduino to format the sensor messages in JSON. These messages are published to an MQTT broker (Mosquitto) running on a Raspberry Pi. A Redis server (also running on the Pi) serves as an in-memory message store. I'm writing a service (python-MQTT client) to parse the JSON and send a LoL (a sample of the data you see in my question) to Redis. Finally, I have a dashboard running on Apache on the Pi. The dashboard utilizes Highcharts to plot the sensor data dynamically (via a web socket connection between the MQTT broker and the browser). Upon loading the page, I pull historical chart data from my Redis LoL to "very quickly" populate the charts on my dashboard (before any realtime data is added dynamically). I realize I can probably format the sensor data the way I want in the Redis store, but that is a problem I haven't worked out yet. Right now, I'm trying to get my historical data to plot correctly in Highcharts. With the data properly formatted, I can get this piece working.
Well, you could use ast.literal_eval:
from ast import literal_eval
myLOL = ["['1466279297', '703.0']", "['1466279287', '702.0']", "['1466279278', '702.0']", "['1466279268', '706.0']", "['1466279258', '713.0']"]
items = [[int(literal_eval(i)[0]), float(literal_eval(i)[1])] for i in myLOL]
Try:
import json
newLOL = [[int(a[0]), float(a[1])] for a in (json.loads(s.replace("'", '"')) for s in myLOL)]
Here I'm considering each element of the list as a JSON, but since it's using ' instead of " for the strings, I have to replace it first (it only works because you said there will be only numbers).
This may work? I wish I was more clever.
newLOL = []
for listObj in myLOL:
listObj = listObj.replace('[', '').replace(']', '').replace("'", '').split(',')
newListObj = [int(listObj[0]), float(listObj[1])]
newLOL.append(newListObj)
Iterates through your current list, peels the string apart into a list by replace un-wanted string chracters and utilizing a split on the comma. Then we take the modified list object and create another new list object with the values being the respective ints and floats. We then append the prepared newListObj to the newLOL list. Considering you want an actual set of lists within your list. Your previously documented input list actually contains strings, which look like lists.
This is a very strange format and the best solution is likely to change the code which generates that.
That being said, you can use ast.literal_eval to safely evaluate the elements of the list as Python tokens:
>>> lit = ast.literal_eval
>>> [[lit(str_val) for str_val in lit(str_list)] for str_list in myLOL]
[[1466279297, 703.0], [1466279287, 702.0], [1466279278, 702.0], [1466279268, 706.0], [1466279258, 713.0]]
We need to do it twice - once to turn the string into a list containing two strings, and then once per resulting string to convert it into a number.
Note that this will succeed even if the strings contain other valid tokens. If you want to validate the format too, you'd want to do something like:
>>> def process_str_list(str_list):
... l = ast.literal_eval(str_list)
... if not isinstance(l, list):
... raise TypeError("Expected list")
... str_int, str_float = l
... return [int(str_int), float(str_float)]
...
>>> [process_str_list(str_list) for str_list in myLOL]
[[1466279297, 703.0], [1466279287, 702.0], [1466279278, 702.0], [1466279268, 706.0], [1466279258, 713.0]]
Your input consists of a list of strings, where each string is the string representation of a list. The first task is to convert the strings back into lists:
import ast
lol2 = map(ast.literal_eval, mylol) # [['1466279297', '703.0'], ...]
Now, you can simply get int and float values from lol2:
newlol = [[int(a[0]), float(a[1])] for a in lol2]
I've got the following code on a Controller
def db = new Sql(dataSource)
def rawLines = db.rows("SELECT name FROM LINES")
def lines = []
/*(db.rows returns the values as [NAME:value] */
rawLines.each {
lines.add(it.name)
}
/*Then, use lines */
I can't keep away the impression that there is probably some way to do this in a more elegant way, something similar to a list comprehension in Python:
lines = [ l.name for l in db.rows("SELECT name FROM LINES") ]
Having to declare an empty list and then populate it doesn't seem the best way of doing things...
Is it possible to do something like this, or Groovy doesn't allow it?
Can't you just use the spread operator, and do:
lines = rawLines*.name
(see http://docs.groovy-lang.org/latest/html/documentation/index.html#_spread_operator)
tim_yates' answer is a clean solution when you want to call a method (or property) on each element of list in order to transform it e.g.:
[1,2,3]*.multiply(5)
but if you want to call a method from another object or do something more complex you can use collect:
[1, 2, 3].collect {Math.cos(it * Math.PI)}
Well, If you are using grails, why aren't you simply using the a model class together with the findAll method?
Using plain raw SQL should be done on exceptional cases.