I'm aware that this one has been asked several times - particularly in this question, but I have not managed to solve my problem. Both snippets below have cx_Oracle and sqlalchemy installed
import cx_Oracle
from sqlachemy import create_engine
I'm trying to connect/write a pandas dataframe to an Oracle Database. I've managed to write to the database using the following code snippet:
ora_con = cx_Oracle.connect("{}/{}#{}".format(schema_name, password, name_of_service))
cur = ora_con.cursor()
statement='CREATE TABLE '+schema_Name+'.History (Name VARCHAR2(15), Entity VARCHAR2(15), Status VARCHAR2(25), Type VARCHAR2(25), Owner VARCHAR2(15), User_521 VARCHAR2(15), Manufacturer VARCHAR2(15), Model VARCHAR2(15), FusInv_Last_inventory DATE, Serial_Number VARCHAR2(25), ID VARCHAR2(15), Version_of_OS VARCHAR2(15), OS VARCHAR2(15), Date_of_Report DATE)'
cur.execute(statement)
That works.
When I try:
con_str = """oracle+cx_oracle://schema_name:password#Host_address:port/?service_name=name_of_service"""
engine = create_engine(con_str, echo=False)
pandasDataframe.to_sql('History', engine , index = False) # Insert the values from the INSERT QUERY into the table 'History'
The pandas .to_sql command fails with a cx_Oracle error of:
cx_Oracle.DatabaseError: ORA-12569: TNS:packet checksum failure
Googling the error indicates a networking error (network is fine) or a listener error (Port number, but that is also fine)
I can connect, write and read the database in SQL Developer.
Any thoughts anyone? Thanks in advance...
Your plain cx_Oracle connect string is different from the one for sqlalchemy. Note that sqlalchemy uses cx_Oracle.makedsn(). So if you have this connect syntax with plain cx_Oracle:
cx_Oracle.connect('myuser/mypassword#myhost:myport/myservice')
you would need this syntax for sqlalchemy:
con_str = 'oracle+cx_oracle://myuser:mypassword#myhost:myport/?service_name=myservice'
Related
Hello, I need your help for this error
I don’t have experience In Python, I am new on it
I need to export a query from MS Access to excel
I create the connection with my database and the connection seems OK
However, when I call the dataframe from Pandas I have this message:
Empty DataFrame
Columns: [Cofor6, TypePce, Ste, CompteDeNfact]
Index: []
This is my code:
connection_string = (
r"DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};"
r"DBQ=C:\xxx\xx\Python_Projects\xxx\xx\xx\mydb.mdb;"
r"ExtendedAnsiSQL=1;")`
connection_url = sa.engine.URL.create("access+pyodbc",query={"odbc_connect": connection_string})
engine = sa.create_engine(connection_url)
print ('connected')
RAFAELquery= '''SELECT SAP_CONSO.Cofor6,SAP_CONSO.TypePce,SAP_CONSO.Ste,Count(SAP_CONSO.Nfact) AS CompteDeNfact FROM SAP_CONSO GROUP BY SAP_CONSO.Cofor6,SAP_CONSO.TypePce,SAP_CONSO.Ste,SAP_CONSO.SaiParFlux HAVING (((SAP_CONSO.TypePce) Not Like 'y\*') AND ((SAP_CONSO.SaiParFlux) Like '\*E')) ORDER BY SAP_CONSO.TypePce DESC;'''
df=pd.read_sql_query(RAFAELquery,engine)
print(df)
Anyone can help me? Thanks a lot
I am trying to replace an existing table in MySQL database. I used the below piece of code to convert the data frame called frame to a database table:
import pandas as pd
import sqlalchemy
from sqlalchemy.types import VARCHAR
database_username = 'root'
database_password = '1234'
database_ip = 'localhost'
database_name = 'my_new_database'
database_connection = sqlalchemy.create_engine('mysql+mysqlconnector://{0}:{1}#{2}/{3}'.format(database_username, database_password, database_ip, database_name),pool_size=3,pool_recycle=3600)
frame.to_sql(schema=database_name,con=database_connection,name='table1',if_exists='replace',chunksize=1000,dtype={'Enrollment No': VARCHAR(frame.index.get_level_values('Enrollment No').str.len().max())})
table1 gets created successfully. But when I rerun the last line of the above code i.e. frame.to_sql(), it throws the below error:
InvalidRequestError: Could not reflect: requested table(s) not available in Engine(mysql+mysqlconnector://root:***#localhost/my_new_database) schema 'my_new_database': (table1)
I want to know why this error is thrown when the table already exists, even though I've used if_exists='replace' and why it works correctly only when creating the table for the first time. What must be done to avoid getting this error?
N.B.: Answers to similar questions only suggest using the table name in lowercase, which I'm following by naming the table as 'table1'.
I'm trying to retrieve the SQL that makes up a stored query inside an Access database.
I'm using a combination of UcanAccess 4.0.2, and jaydebeapi and the ucanaccess console. The ultimate goal is to be able to do the following from a python script with no user intervention.
When UCanAccess loads, it successfully loads the query:
Please, enter the full path to the access file (.mdb or .accdb): /Users/.../SnohomishRiverEstuaryHydrology_RAW.accdb
Loaded Tables:
Sensor Data, Sensor Details, Site Details
Loaded Queries:
Jeff_Test
Loaded Procedures:
Loaded Indexes:
Primary Key on Sensor Data Columns: (ID)
, Primary Key on Sensor Details Columns: (ID)
, Primary Key on Site Details Columns: (ID)
, Index on Sensor Details Columns: (SiteID)
, Index on Site Details Columns: (SiteID)
UCanAccess>
When I run, from the UCanAccess console a query like
SELECT * FROM JEFF_TEST;
I get the expected results of the query.
I tried things including this monstrous query from inside a python script even using the sysSchema=True option (from here: http://www.sqlquery.com/Microsoft_Access_useful_queries.html):
SELECT DISTINCT MSysObjects.Name,
IIf([Flags]=0,"Select",IIf([Flags]=16,"Crosstab",IIf([Flags]=32,"Delete",IIf
([Flags]=48,"Update",IIf([flags]=64,"Append",IIf([flags]=128,"Union",
[Flags])))))) AS Type
FROM MSysObjects INNER JOIN MSysQueries ON MSysObjects.Id =
MSysQueries.ObjectId;
But get an object not found or insufficient privileges error.
At this point, I've tried mdbtools and can successfully retrieve metadata, and data from access. I just need to get the queries out too.
If anyone can point me in the right direction, I'd appreciate it. Windows is not a viable option.
Cheers, Seth
***********************************
* SOLUTION
***********************************
from jpype import *
startJVM(getDefaultJVMPath(), "-ea", "-Djava.class.path=/Users/seth.urion/local/access/UCanAccess-4.0.2-bin/ucanaccess-4.0.2.jar:/Users/seth.urion/local/access/UCanAccess-4.0.2-bin/lib/commons-lang-2.6.jar:/Users/seth.urion/local/access/UCanAccess-4.0.2-bin/lib/commons-logging-1.1.1.jar:/Users/seth.urion/local/access/UCanAccess-4.0.2-bin/lib/hsqldb.jar:/Users/seth.urion/local/access/UCanAccess-4.0.2-bin/lib/jackcess-2.1.6.jar")
conn = java.sql.DriverManager.getConnection("jdbc:ucanaccess:///Users/seth.urion/PycharmProjects/pyAccess/FE_Hall_2010_2016_SnohomishRiverEstuaryHydrology_RAW.accdb")
for query in conn.getDbIO().getQueries():
print(query.getName())
print(query.toSQLString())
If you can find a satisfactory way to call Java methods from within Python then you could use the Jackcess Query#toSQLString() method to extract the SQL for a saved query. For example, I just got this to work under Jython:
from java.sql import DriverManager
def get_query_sql(conn, query_name):
sql = ''
for query in conn.getDbIO().getQueries():
if query.getName() == query_name:
sql = query.toSQLString()
break
return sql
# usage example
if __name__ == '__main__':
conn = DriverManager.getConnection("jdbc:ucanaccess:///home/gord/UCanAccessTest.accdb")
query_name = 'Jeff_Test'
query_sql = get_query_sql(conn, query_name)
if query_sql == '':
print '(Query not found.)'
else:
print 'SQL for query [%s]:' % (query_name)
print
print query_sql
conn.close()
producing
SQL for query [Jeff_Test]:
SELECT Invoice.InvoiceNumber, Invoice.InvoiceDate
FROM Invoice
WHERE (((Invoice.InvoiceNumber)>1));
i have two database #1 is tripplanner.db and the #2 is trip.db .I want to add trip.db table 'restaurants' data to db # 1 tripplanner.db table 'restaurants'(which is empty now). I am using sqlite which is builtin in python.
Help me please.And tell me how i can execute this in python.
import sqlite3
import os
conn = sqlite3.connect('trip.db')
c = conn.cursor()
c.execute("DROP TABLE IF EXISTS things")
c.execute("ATTACH DATABASE ? AS db2", (os.path.join('data', 'db', 'trip_tripplanner.db'),))
c.execute("SELECT things FROM db2.sqlite_master WHERE type='table' AND name='things'")
c.execute(c.fetchone()[0])
c.execute("INSERT INTO trip.things SELECT * FROM db2.things")
conn.commit()
conn.close()
This code is what i have tried so far by seeing posts in stackoverflow.but it is giving me error because i don't know what is 'data' in os.path.join('data').
I am trying to hook the cqlengine CQL 3 object mapper with my web application running on CherryPy. Athough the documentation is very clear about querying, I am still not aware how to make queries on an existing table(and an existing keyspace) in my cassandra database. For instance I already have this table Movies containing the fields Title, rating, Year. I want to make the CQL query
SELECT * FROM Movies
How do I go ahead with the query after establishing the connection with
from cqlengine import connection
connection.setup(['127.0.0.1:9160'])
The KEYSPACE is called "TEST1".
Abhiroop Sarkar,
I highly suggest that you read through all of the documentation at:
Current Object Mapper Documentation
Legacy CQLEngine Documentation
Installation: pip install cassandra-driver
And take a look at this example project by the creator of CQLEngine, rustyrazorblade:
Example Project - Meat bot
Keep in mind, CQLEngine has been merged into the DataStax Cassandra-driver:
Official Python Cassandra Driver Documentation
You'll want to do something like this:
CQLEngine <= 0.21.0:
from cqlengine.connection import setup
setup(['127.0.0.1'], 'keyspace_name', retry_connect=True)
If you need to create the keyspace still:
from cqlengine.management import create_keyspace
create_keyspace(
'keyspace_name',
replication_factor=1,
strategy_class='SimpleStrategy'
)
Setup your Cassandra Data Model
You can do this in the same .py or in your models.py:
import datetime
import uuid
from cqlengine import columns, Model
class YourModel(Model):
__key_space__ = 'keyspace_name' # Not Required
__table_name__ = 'columnfamily_name' # Not Required
some_int = columns.Integer(
primary_key=True,
partition_key=True
)
time = columns.TimeUUID(
primary_key=True,
clustering_order='DESC',
default=uuid.uuid1,
)
some_uuid = columns.UUID(primary_key=True, default=uuid.uuid4)
created = columns.DateTime(default=datetime.datetime.utcnow)
some_text = columns.Text(required=True)
def __str__(self):
return self.some_text
def to_dict(self):
data = {
'text': self.some_text,
'created': self.created,
'some_int': self.some_int,
}
return data
Sync your Cassandra ColumnFamilies
from cqlengine.management import sync_table
from .models import YourModel
sync_table(YourModel)
Considering everything above, you can put all of the connection and syncing together, as many examples have outlined, say this is connection.py in our project:
from cqlengine.connection import setup
from cqlengine.management import sync_table
from .models import YourTable
def cass_connect():
setup(['127.0.0.1'], 'keyspace_name', retry_connect=True)
sync_table(YourTable)
Actually Using the Model and Data
from __future__ import print_function
from .connection import cass_connect
from .models import YourTable
def add_data():
cass_connect()
YourTable.create(
some_int=5,
some_text='Test0'
)
YourTable.create(
some_int=6,
some_text='Test1'
)
YourTable.create(
some_int=5,
some_text='Test2'
)
def query_data():
cass_connect()
query = YourTable.objects.filter(some_int=5)
# This will output each YourTable entry where some_int = 5
for item in query:
print(item)
Feel free to let ask for further clarification, if necessary.
The most straightforward way to achieve this is to make model classes which mirror the schema of your existing cql tables, then run queries on them
cqlengine is primarily an Object Mapper for Cassandra. It does not interrogate an existing database in order to create objects for existing tables. Rather it is usually intended to be used in the opposite direction (i.e. create tables from python classes). If you want to query an existing table using cqlengine you will need to create python models that exactly correspond to your existing tables.
For example, if your current Movies table had 3 columns, id, title, and release_date you would need to create a cqlengine model that had those three columns. Additionally, you would need to ensure that the table_name attribute on the class was exactly the same as the table name in the database.
from cqlengine import columns, Model
class Movie(Model):
__table_name__ = "movies"
id = columns.UUID(primary_key=True)
title = columns.Text()
release_date = columns.Date()
The key thing is to make sure that model exactly mirrors the existing table. If there are small differences you may be able to use sync_table(MyModel) to update the table to match your model.