I am trying to load the pandas dataframe into Redshift, but it keeps giving me an error. Please guide me on the same. Need help on correcting the cluster configuration to make it work successfully.
Below is my code and error traceback:
from sqlalchemy import create_engine
import pyodbc
import psycopg2
username = "#####"
host = "redshift-cluster-****.*****.ap-south-1.redshift.amazonaws.com"
driver = "Amazon Redshift (x64)"
port = 5439
pwd = "******"
db = "dev"
table = "tablename"
rs_engine = create_engine(f"postgresql://{username}:{pwd}#{host}:{port}/{db}")
df.to_sql(table, con=rs_engine, if_exists='replace',index=False)
Traceback:
OperationalError: (psycopg2.OperationalError) connection to server at "redshift-cluster-****.****.ap-south-1.redshift.amazonaws.com" (3.109.77.136), port 5439 failed: Connection timed out (0x0000274C/10060)
Is the server running on that host and accepting TCP/IP connections?
Even tried the below options, but getting the same error,
rs_engine = create_engine(f"redshift+psycopg2://{username}#{host}:{port}/{db}")
rs_engine = create_engine(f"postgresql+psycopg2://{username}:{pwd}#{host}:{port}/{db}")
rs_engine = redshift_connector.connect(
host='redshift-cluster-####.****.ap-south-1.redshift.amazonaws.com',
database='dev',
user='****',
password='#####'
)
Also, have the Public accessible setting Enabled in Redshift cluster. Still unable to connect and load the data.
UPDATE:
Also tried using ODBC Driver, but getting the same error,
import pyodbc
cnxn = pyodbc.connect(Driver=driver,
Server=host,
Database=db,
UID=username,PWD=pwd,Port=port)
When tried to setup using ODBC Datasources app, getting the same error,
Related
I tried to put in requirements.txt for MWAA Airflow with pyodbc=4.0.30 and in code, made connection string like
dbconnection = pyodbc.connect("Driver={ODBC Driver 17 for SQL Server};Server="+Server+";Database="+Database+";UID="+UserID+";PWD="+Password, autocommit=True)
Now the error is Broken DAG: [/usr/local/airflow/dags/test.py] No module named 'pyodbc'
Version of Airflow: 1.10.12
There is hardly any documentation on SQL Server/ Postgres based connection on MWAA AWS documentation, especially for pyodbc connection, I earlier got this issue with lambda functions and figured it out with lambda layers, but not sure how MWAA works, any suggestions appreciated.
Please don't recommend any other technology like EC2 to host Airflow as the company is very rigid to use MWAA Airflow.
import pymssql
conn = pymssql.connect(
server=server,
user=username,
password=password,
database=database
)
query ="select IDpk,name,Remarks from TestTable"
df = pd.read_sql(query,conn)
How do I connect to a MySQL database using Python's peewee library while specifying the tls-versions ['TLSv1.1', 'TLSv1.2'] as shown here and here in the MySQL documentation?
I can successfully connect to the MySQL database using mysql.connector as shown here:
import mysql.connector
config = {'database': 'dbname',
'user': 'username_so',
'password': 'psswrd',
'host': 'maria####-##-###-##.###.####.com',
'port': 3306,
'tls_versions': ['TLSv1.1', 'TLSv1.2']}
cnx = mysql.connector.connect(**config)
cnx.close()
However, I am unable to pass the 'tls_versions' parameter to peewee when establishing a connection. As a result, I get an error message:
ImproperlyConfigured: MySQL driver not installed!
I am pretty sure that the problem is with specifying the tls versions in peewee because I was getting the same error message with mysql.connector before I added in the additional 'tls_versions' parameter.
Here is the code I am using in peewee that is failing:
db = MySQLDatabase(**config)
db.get_tables() # This function and any other that connects to the db gets the same error message specified above
My Setup:
Linux
Python 3.7
peewee==3.13.3
mysql-connector-python==8.0.21
As I responded to your github issue:
You need to use playhouse.mysql_ext.MySQLConnectorDatabase to connect using the mysql-connector driver. That will resolve your issue.
import pyodbc
import numpy
import pandas as pd
import pypyodbc
def sql_conn():
conn = pyodbc.connect("Driver={ODBC Driver 17 for SQL Server};"
"Server=;"
"Database=db_name;"
"uid=xxxx;pwd=xxxx;")
cursor = conn.cursor()
cursor.execute('SELECT * FROM dbo.ImageDB')
for row in cursor:
print(row)
It's running on a Windows Server 2012 R2.
Whenever I run the python script I keep getting the message
Process finished with exit code 0
I know the connection wasn't made. How do I get the server name? Is it a combination of the server IP and the SQL Server name? Do I need to provide a PORT number? I tried a bunch of combinations for the server name but all give me the same output.
Also which is better pyodbc or pypyodbc?
I am sorry if this sounds like a stupid question but I am really new to this and any help would be appreciated.
Thanks.
I'm trying to connect to a oracle DB with SQLAlchemy however I get the following error:
ORA-12545: Connect failed because target host or object does not exist
Note that the code running this is on a docker container that is located on a vm in GCP.
I tried using tools like telnet, curl, nmap, etc and they all are able to connect/say open. So I don't see why connecting through python would all of a sudden make it not visible.
Here is the code that is used to try to connect.
from sqlalchemy.orm.session import sessionmaker
from framework.db import BuildOracleConnection
Creds_Oracle = {
'userName': 'urname',
'password': 'pass',
'host': '10.10.10.10',
'port': '1521',
'serviceName': 'svcName'
}
Conn_Oracle = BuildOracleConnection(Creds_Oracle)
metaConn = sessionmaker(bind=Conn_Oracle)
metaSession = metaConn()
sql = 'select * from table'
sql = sql.replace('\n', ' ')
sourceExtract = metaSession.execute(sql)
The part that throws the error is the last line.
I expect to be able to connect but instead I get the following error:
ORA-12545: Connect failed because target host or object does not exist.
For some reason I wasn't able to connect directly to the loadbalancer, instead I had to connect to the nodes themselves.
I want to connect to MSSQL server2012 from Python using pyodbc on a Debian8 host. The current code already works for Azure SQL server but not for local SQL server.
The connection string looks like this:
import pyodbc
server = 'IP'
database = 'DB_Name'
username = 'user'
password = 'pass'
driver = '{ODBC Driver 17 for SQL Server}'
cnxn = pyodbc.connect('DRIVER='+driver+';SERVER='+server+';PORT=1433;DATABASE='+database+';UID='+username+';PWD='+ password)
I had to use IP address because I could not reach SQL server via name. I can connect to the server with SSMS on a windows machine. I have tried the following combinations as well:
server='sqlservername\instancename'
server='IP, port'
server='IP\instancename'
driver='SQL Server'
Integrated_Security=false;
Trusted_Connection=no;
but none of them worked. I got either Login timeout or error in registering user. I run the same python code on windows machine without any problem.
Finally i found the correct combination. Besides thanks to #GordThompson the TCP/IP was enabled but on another port.
import pyodbc
server = 'IP\InstanceName, port'
database = 'DB_Name'
username = 'user'
password = 'pass'
driver = '{ODBC Driver 17 for SQL Server}'
cnxn = pyodbc.connect('DRIVER='+driver+';SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)