I am trying to convert my sql query output into a list to look a certain way.
Here is my code:
def get_sf_metadata():
import sqlite3
#Tables I want to be dynamically created
table_names=['AcceptedEventRelation','Asset', 'Book']
#SQLIte Connection
conn = sqlite3.connect('aaa_test.db')
c = conn.cursor()
#select the metadata table records
c.execute("select name, type from sf_field_metadata1 limit 10 ")
print(list(c))
get_sf_metadata()
Here is my output:
[('Id', 'id'), ('RelationId', 'reference'), ('EventId', 'reference')]
Is there any way to make the output looks like this:
[Id id, RelationId reference, EventId reference]
You can try
print(["{} {}".format(i[0], i[1]) for i in list(c)])
That will print you
['Id id', 'RelationId reference', 'EventId reference']
Related
I'm using psycopg2 to connect to postgre DB, and to export the data into CSV file.
This is how I made the export DB to csv:
def export_table_to_csv(self, table, csv_path):
sql = "COPY (SELECT * FROM %s) TO STDOUT WITH CSV DELIMITER ','" % table
self.cur.execute(sql)
with open(csv_path, "w") as file:
self.cur.copy_expert(sql, file)
But the data is just the rows - without the column names.
How can I export the data with the column names?
P.S. I am able to print the column names:
sql = '''SELECT * FROM test'''
self.cur.execute(sql)
column_names = [desc[0] for desc in self.cur.description]
for i in column_names:
print(i)
I want the cleanest way to do export the DB with columns name (i.e. I prefer to do this in one method, and not rename columns In retrospect).
As I said in my comment, you can add HEADER to the WITH clause of your SQL:
sql = "COPY (SELECT * FROM export_test) TO STDOUT WITH CSV HEADER"
By default, comma delimiters are used with CSV option so you don't need to specify.
For future Questions, you should submit a minimal reproducible example. That is, code we can directly copy and paste and run. I was curious if this would work so I made one and tried it:
import psycopg2
conn = psycopg2.connect('host=<host> dbname=<dbname> user=<user>')
cur = conn.cursor()
# create test table
cur.execute('DROP TABLE IF EXISTS export_test')
sql = '''CREATE TABLE export_test
(
id integer,
uname text,
fruit1 text,
fruit2 text,
fruit3 text
)'''
cur.execute(sql)
# insert data into table
sql = '''BEGIN;
insert into export_test
(id, uname, fruit1, fruit2, fruit3)
values(1, 'tom jones', 'apple', 'banana', 'pear');
insert into export_test
(id, uname, fruit1, fruit2, fruit3)
values(2, 'billy idol', 'orange', 'cherry', 'strawberry');
COMMIT;'''
cur.execute(sql)
# export to csv
fid = open('export_test.csv', 'w')
sql = "COPY (SELECT * FROM export_test) TO STDOUT WITH CSV HEADER"
cur.copy_expert(sql, fid)
fid.close()
And the resultant file is:
id,uname,fruit1,fruit2,fruit3
1,tom jones,apple,banana,pear
2,billy idol,orange,cherry,strawberry
I have just started learning SQLite and was creating a project which has a .sqlite file in which there are multiple tables. I want to ask the user to input the table_name and then the program will fetch the columns present in that particular table.
So far I have done this.
app_database.py
def column_names(table_name):
conn = sqlite3.connect('northwind_small.sqlite')
c = conn.cursor()
c.execute("PRAGMA table_info(table_name)")
columns = c.fetchall()
for c in columns :
print(c[1])
conn.commit()
conn.close()
our-app.py
import app_database
table_name = input("Enter the table name = ")
app_database.column_names(table_name)
when I run our-app.py I don't get anything.
C:\Users\database-project>python our-app.py
Enter the table name = Employee
C:\Users\database-project>
Can anyone tell me how should I proceed?
I am trying to read a large table (10-15M rows) from a database into pandas dataframe and I'm using the following code:
def read_sql_tmpfile(query, db_engine):
with tempfile.TemporaryFile() as tmpfile:
copy_sql = "COPY ({query}) TO STDOUT WITH CSV {head}".format(
query=query, head="HEADER"
)
conn = db_engine.raw_connection()
cur = conn.cursor()
cur.copy_expert(copy_sql, tmpfile)
tmpfile.seek(0)
df = pandas.read_csv(tmpfile)
return df
I can use this if I have a simple query like this and I pass this into above func:
'''SELECT * from hourly_data'''
But what if I want to pass some variable into this query i.e.
'''SELECT * from hourly_data where starttime >= %s '''
Now where do I pass the parameter?
You cannot use parameters with COPY. Unfortunately that extends to the query you use inside COPY, even if you could use parameters with the query itself.
You will have to construct a query string including the parameter (beware of SQL injection) and use that with COPY.
Can't execute a psycopg2 insert query (Postgres db), which uses the best practice %s sign for inserting and contains a LIKE statement which includes a % sign.
% sign in LIKE statement is interpreted as an insert placeholder.
'IndexError: tuple index out of range' is thrown.
Tried escaping % with backslash, didn't work out.
with psycopg2.connect(some_url) as conn:
with conn.cursor() as cur:
query = """
SELECT id
FROM users
WHERE surname IN %s AND named LIKE '%john'
"""
cur.execute(query, (tuple(["smith", "mcnamara"]),))
data = cur.fetchall()
Try using a placeholder also for the LIKE expression, and then bind a literal with a wildcard to it:
query = """
SELECT id
FROM users
WHERE surname IN %s AND named LIKE %s"""
cur.execute(query, (tuple(["smith", "mcnamara"]), "%John",))
data = cur.fetchall()
try this one:
with psycopg2.connect(some_url) as conn:
with conn.cursor() as cur:
query = """
SELECT id
FROM users
WHERE surname IN %s AND named LIKE '%sjohn'
"""
cur.execute(query, (tuple(["smith", "mcnamara"]), '%'))
data = cur.fetchall()
I have to code on python sqlite3 a function to count rows of a table.
The thing is that the user should input the name of that table once the function is executed.
So far I have the following. However, I don't know how to "connect" the variable (table) with the function, once it's executed.
Any help would be great.
Thanks
def RT():
import sqlite3
conn= sqlite3.connect ("MyDB.db")
table=input("enter table name: ")
cur = conn.cursor()
cur.execute("Select count(*) from ?", [table])
for row in cur:
print str(row[0])
conn.close()
Columns and Tables Can't be Parameterized
As explained in this SO answer, Columns and tables can't be parameterized. A fact that might not be documented by any authoritative source (I couldn't find one, so if you you know of one please edit this answer and/or the one linked above), but instead has been learned through people trying exactly what was attempted in the question.
The only way to dynamically insert a column or table name is through standard python string formatting:
cur.execute("Select count(*) from {0}".format(table))
Unfortunately This opens you up to the possibility of SQL injection
Whitelist Acceptable Column/Table Names
This SO answer explains that you should use a whitelist to check against acceptable table names. This is what it would look like for you:
import sqlite3
def RT():
conn = sqlite3.connect ("MyDB.db")
table = input("enter table name: ")
cur = conn.cursor()
if table not in ['user', 'blog', 'comment', ...]:
raise ... #Include your own error here
execute("Select count(*) from {0}".format(table))
for row in cur:
print str(row[0])
conn.close()
The same SO answer cautions accepting submitted names directly "because the validation and the actual table could go out of sync, or you could forget the check." Meaning, you should only derive the name of the table yourself. You could do this by making a clear distinction between accepting user input and the actual query. Here is an example of what you might do.
import sqlite3
acceptable_table_names = ['user', 'blog', 'comment', ...]
def RT():
"""
Client side logic: Prompt the user to enter table name.
You could also give a list of names that you associate with ids
"""
table = input("enter table name: ")
if table in acceptable_table_names:
table_index = table_names.index(table)
RT_index(table_index)
def RT_index(table_index):
"""
Backend logic: Accept table index instead of querying user for
table name.
"""
conn = sqlite3.connect ("MyDB.db")
cur = conn.cursor()
table = acceptable_table_names[table_index]
execute("Select count(*) from {0}".format(table))
for row in cur:
print str(row[0])
conn.close()
This may seem frivolous, but this keeps the original interface while addressing the potential problem of forgetting to check against a whitelist. The validation and the actual table could still go out of sync; you'll need to write tests to fight against that.