psycopg2 export DB to csv with column names - python-3.x

I'm using psycopg2 to connect to postgre DB, and to export the data into CSV file.
This is how I made the export DB to csv:
def export_table_to_csv(self, table, csv_path):
sql = "COPY (SELECT * FROM %s) TO STDOUT WITH CSV DELIMITER ','" % table
self.cur.execute(sql)
with open(csv_path, "w") as file:
self.cur.copy_expert(sql, file)
But the data is just the rows - without the column names.
How can I export the data with the column names?
P.S. I am able to print the column names:
sql = '''SELECT * FROM test'''
self.cur.execute(sql)
column_names = [desc[0] for desc in self.cur.description]
for i in column_names:
print(i)
I want the cleanest way to do export the DB with columns name (i.e. I prefer to do this in one method, and not rename columns In retrospect).

As I said in my comment, you can add HEADER to the WITH clause of your SQL:
sql = "COPY (SELECT * FROM export_test) TO STDOUT WITH CSV HEADER"
By default, comma delimiters are used with CSV option so you don't need to specify.
For future Questions, you should submit a minimal reproducible example. That is, code we can directly copy and paste and run. I was curious if this would work so I made one and tried it:
import psycopg2
conn = psycopg2.connect('host=<host> dbname=<dbname> user=<user>')
cur = conn.cursor()
# create test table
cur.execute('DROP TABLE IF EXISTS export_test')
sql = '''CREATE TABLE export_test
(
id integer,
uname text,
fruit1 text,
fruit2 text,
fruit3 text
)'''
cur.execute(sql)
# insert data into table
sql = '''BEGIN;
insert into export_test
(id, uname, fruit1, fruit2, fruit3)
values(1, 'tom jones', 'apple', 'banana', 'pear');
insert into export_test
(id, uname, fruit1, fruit2, fruit3)
values(2, 'billy idol', 'orange', 'cherry', 'strawberry');
COMMIT;'''
cur.execute(sql)
# export to csv
fid = open('export_test.csv', 'w')
sql = "COPY (SELECT * FROM export_test) TO STDOUT WITH CSV HEADER"
cur.copy_expert(sql, fid)
fid.close()
And the resultant file is:
id,uname,fruit1,fruit2,fruit3
1,tom jones,apple,banana,pear
2,billy idol,orange,cherry,strawberry

Related

How to dynamically input the table name and fetch the results in SQLite?

I have just started learning SQLite and was creating a project which has a .sqlite file in which there are multiple tables. I want to ask the user to input the table_name and then the program will fetch the columns present in that particular table.
So far I have done this.
app_database.py
def column_names(table_name):
conn = sqlite3.connect('northwind_small.sqlite')
c = conn.cursor()
c.execute("PRAGMA table_info(table_name)")
columns = c.fetchall()
for c in columns :
print(c[1])
conn.commit()
conn.close()
our-app.py
import app_database
table_name = input("Enter the table name = ")
app_database.column_names(table_name)
when I run our-app.py I don't get anything.
C:\Users\database-project>python our-app.py
Enter the table name = Employee
C:\Users\database-project>
Can anyone tell me how should I proceed?

python, SQLite3 showing existing Tables and data

I have been given a .db file, that has already been populated with both Tables and Data. However, no description of the content of the database has been made available.
Is there a way for me to retrieve individual lists listing the different tables, and their respective sets of columns using SQLite3 and python?
This code help you to show tables with keys , when you get tables and their keys you can get data.
import sqlite3
def readDb():
connection = sqlite3.connect('data.db')
connection.row_factory = sqlite3.Row
cursor = connection.cursor()
cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
rows = cursor.fetchall()
tabs=[]
for row in rows:
for r in row:
tabs.append(r)
d={}
for tab in tabs:
cursor.execute("SELECT * FROM "+tab+";")
rows = cursor.fetchone()
t=[]
for row in rows.keys():
t.append(row)
d[tab]=t
connection.commit()
return d
print(readDb())

How to convert sql query to list?

I am trying to convert my sql query output into a list to look a certain way.
Here is my code:
def get_sf_metadata():
import sqlite3
#Tables I want to be dynamically created
table_names=['AcceptedEventRelation','Asset', 'Book']
#SQLIte Connection
conn = sqlite3.connect('aaa_test.db')
c = conn.cursor()
#select the metadata table records
c.execute("select name, type from sf_field_metadata1 limit 10 ")
print(list(c))
get_sf_metadata()
Here is my output:
[('Id', 'id'), ('RelationId', 'reference'), ('EventId', 'reference')]
Is there any way to make the output looks like this:
[Id id, RelationId reference, EventId reference]
You can try
print(["{} {}".format(i[0], i[1]) for i in list(c)])
That will print you
['Id id', 'RelationId reference', 'EventId reference']

How to chain multiple statements within psycopg2?

I can copy data from a s3 bucket into a redshift table using psycopg2:
import psycopg2
sql = """ copy table1 from 's3://bucket/myfile.csv'
access_key_id 'xxxx'
secret_access_key 'xxx' DELIMITER '\t'
timeformat 'auto'
maxerror as 250 GZIP IGNOREHEADER 1 """
cur.execute(sql)
How do I string multiple redshift statements to do these three things:
create another table (table2) from table1 after data has moved from s3
move data over from table1 to table2
drop table1
I tried the following:
sql = """ copy table1 from 's3://bucket/myfile.csv'
access_key_id 'xxxx'
secret_access_key 'xxx' DELIMITER '\t'
timeformat 'auto'
maxerror as 250 GZIP IGNOREHEADER 1
create table table2 as table1
drop table table1"""
I don't get back any error, but the table is not created, only the copy is working from above. What am I doing wrong in my sql?
Following code does Copy from Table1 to Table2 by creating a duplicate Copy. Then, it deletes Table1.
import psycopg2
def redshift():
conn = psycopg2.connect(dbname='***', host='******.redshift.amazonaws.com', port='5439', user='****', password='*****')
cur = conn.cursor();
cur.execute("create table table2 as select * from table1;")
cur.execute(" drop table table1;")
print("Copy executed fine!")
redshift()

interator should return strings, not bytes

I found the following code on Stackoverflow.The coding makes the table "student" but nothing is populated. I get the following error:
"ERROR:iterator should return strings, not bytes(did you open the file in text mode?)
Here is the code:
import csv
import sqlite3
# Create the database
connection = sqlite3.connect('attendance5.db')
cursor = connection.cursor()
# Create the table
cursor.execute ('DROP TABLE IF EXISTS student')
cursor.execute ('CREATE TABLE student (StuID REAL, LastName TEXT, FirstName TEXT, Grade REAL, Address TEXT, Phone TEXT, Parent TEXT)')
connection.commit()
# Load the CSV file into CSV reader
csvfile = open ('attendance.csv','rb')
creader = csv.reader(csvfile, delimiter=',', quotechar='|')
# Iterate through the CSV reader, inserting values into the database
for t in creader
cursor.execute ('INSERT INTO student VALUES (?,?,?,?,?,?,?)'t)
# Close the csv file, commit changes, and close the connection
csvfile.close()
connection.commit()
connection.close()
Does it have anything to do with the t in creader?
What am I doing wrong? Is it the creader t value? Thank you.

Resources