I am setting a fastAPI, and I have a postgresql database I want to connect it to. I am following a this guide to do so, and I am finding some errors when trying to set the DATABASE_URL.
My end goal is to perform a SQL query to such database, and then later, deploy the application to Heroku where I should connect it to the postgresql database stored there.
host_server = os.environ.get('host_server', 'localhost')
db_server_port = urllib.parse.quote_plus(str(os.environ.get('db_server_port', '5432')))
database_name = os.environ.get('nameofmydatabase', 'fastapi')
db_username = urllib.parse.quote_plus(str(os.environ.get('myusername', 'postgres')))
db_password = urllib.parse.quote_plus(str(os.environ.get('mypassword123', 'secret')))
ssl_mode = urllib.parse.quote_plus(str(os.environ.get('ssl_mode','prefer')))
DATABASE_URL = 'postgresql://{}:{}#{}:{}/{}?sslmode={}'.format(db_username, db_password, host_server, db_server_port, database_name, ssl_mode)
With that configuration I am getting
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: role "postgres" does not exist
Related
I'm running MySQL in a Docker container, and in my jupyter lab I have the following cell:
# Importing module
import mysql.connector
# Creating connection object
mydb = mysql.connector.connect(
host = "localhost",
port = 3307,
user = "bob",
password = "1234",
database = 'testDB'
)
# Printing the connection object
print(mydb)
Which prints: <mysql.connector.connection_cext.CMySQLConnection object at 0x7fc633769dd0>
But I want to connect using ipython-sql, so I load the extension with %load_ext sql, but when I try:
%sql mysql://bob:1234#localhost:3307/testDB
I get the error:
Connection info needed in SQLAlchemy format, example:
postgresql://username:password#hostname/dbname
or an existing connection: dict_keys([])
No module named 'MySQLdb'
Connection info needed in SQLAlchemy format, example:
postgresql://username:password#hostname/dbname
or an existing connection: dict_keys([])
I am not able to connect to my azure database in python using pyodbc library.
When I pass the connection string in the below format:
*
cnxn = pyodbc.connect(
'DRIVER={SQL Server};'
'SERVER= servername;'
'DATABASE=dbname;'
'username=username;'
'password=password'
)
I get the following error:
line 4, in
cnxn = pyodbc.connect(
pyodbc.OperationalError: ('08001', '[08001] [Microsoft][ODBC SQL Server Driver][DBNETLIB]SQL Server does not exist or access denied.
When I pass the connection string in the below format:
*
cnxn = pyodbc.connect(
driver = '{SQL Server}',
server = 'servername',
database = 'dbname',
username = 'username',
password = 'password'
)
I get the following error:
line 4, in
cnxn = pyodbc.connect(
pyodbc.InterfaceError: ('28000', "[28000] [Microsoft][ODBC SQL Server Driver][SQL Server]Login failed for user
In order to give you full answer please add where are trying to connect to the server from: your computer? a server?
Did you make sure that you IP is whitelisted ?
did you tied to connect to the DB with SSMS(sql server management studio) so you know for sure it is not a user/password issue?
A possible solution if you are trying from your windows base computer is to
to install ODBC Driver 13&17 and then change your code:
driver = '{ODBC Driver 17 for SQL Server}'
Below is the correct format of the connection string:
cnxn = pyodbc.connect(
'DRIVER={SQL Server};'
'SERVER= servername;'
'DATABASE=dbname;'
'UID=username;'
'PWD=password'
)
I have an existing postgres table in RDS with a database name my-rds-table-name
I've connected to it using pgAdmin4 with the following configs of a read-only user:
host_name = "my-rds-table-name.123456.us-east-1.rds.amazonaws.com"
user_name = "my_user_name"
password = "abc123def345"
I have verified that I can query against the table.
However, I cannot connect to it using python:
SQLAlchemy==1.2.16
psycopg2-binary==2.7.6.1
mysqlclient==1.4.1
With:
import psycopg2
engine = psycopg2.connect(
database="my-rds-table-name",
user="my_user_name",
password="abc123def345",
host="my-rds-table-name.123456.us-east-1.rds.amazonaws.com",
port='5432'
)
It fails with
psycopg2.OperationalError: FATAL: database "my-rds-table-name" does not exist
Similarly, if I try to connect to it with sqlalchemy:
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: database "my-rds-table-name" does not exist
What am I missing?
Thank's John Rotenstein for your comment.
As he pointed out, my-rds-table-name is the database instance name, not the database name, the default database name is postgres.
import psycopg2
engine = psycopg2.connect(
database="postgres",
user="my_user_name",
password="abc123def345",
host="my-rds-table-name.123456.us-east-1.rds.amazonaws.com",
port='5432'
)
Using sqlalchemy you can do the following:
engine = create_engine('postgresql://postgres:postgres#<AWS_RDS_end-point>:5432/postgres')
Then you can update your database.
For example:
df = pd.DataFrame({'A': [1,2], 'B':[3,4]})
df.to_sql('tablename', engine, schema='public', if_exists='append', index=False)
I want to store my aws iot mqtt messages into my postgresql. To do so, I have already connected my local posrtgresql to the amazon RDS instance. Now, I need to create a connection between amazon lambda calculus and then send the data to the postgresql database. But whenever, I am testing my lambda calculus, it was giving me "name 'conn' is not defined: NameError" error. Here, is my python code in aws lambda. I have also included the psycopg2 library to my project.
import sys
import logging
import rds_config
import psycopg2
#rds settings
rds_host = "myhost"
name = "username"
password = "username_password"
db_name = "dbname"
logger = logging.getLogger()
logger.setLevel(logging.INFO)
try:
conn = psycopg2.connect(host=rds_host, user=name, password=password,
dbname=db_name, connect_timeout=5)
except:
logger.error("ERROR: Unexpected error: Could not connect to postgreSQL
instance.")
logger.info("SUCCESS: Connection to RDS postgreSQL instance succeeded")
def handler(event, context):
"""
This function fetches content from postgreSQL RDS instance
"""
item_count = 0
with conn.cursor() as cur:
cur.execute('insert into awsiotdata (serialnumber, dateandtime, clicktype, batteryvoltage) values(serialNumber, datetime.datetime.utcnow(), clickType, batteryVoltage)')
conn.commit()
cur.execute("select * from awsiotdata")
for row in cur:
item_count += 1
logger.info(row)
#print(row)
return "Added %d items from RDS PostgreSQL table" %(item_count)
You are hiding a true error message. Exception handling patter for Python looks like this:
try:
conn = psycopg2.connect(host=rds_host,
user=name,
password=password,
database=db_name)
except Exception as e:
print(e)
This way you will see the real error message:
invalid dsn: invalid connection option "passwd"
Edit #1:
"Timeout" means that lambda can't connect because of "Security group rules" for RDS instance. Please keep in mind that even public RDS instance by default have inbound restriction by IP (i.e. it is posible to connect from PC but it is imposible to connect from AWS Lambda).
How do you connect to a remote server via IP address in the manner that TOAD, SqlDeveloper, are able to connect to databases with just the ip address, username, SID and password?
Whenever I try to specify and IP address, it seems to be taking it locally.
In other words, how should the string for cx_Oracle.connect() be formatted to a non local database?
There was a previous post which listed as an answer connecting to Oracle via cx_Oracle module with the following code:
#!/usr/bin/python
import cx_Oracle
connstr='scott/tiger'
conn = cx_Oracle.connect(connstr)
curs = conn.cursor()
curs.execute('select * from emp')
print curs.description
for row in curs:
print row
conn.close()
I like to do it this way:
ip = '192.168.0.1'
port = 1521
SID = 'YOURSIDHERE'
dsn_tns = cx_Oracle.makedsn(ip, port, SID)
db = cx_Oracle.connect('username', 'password', dsn_tns)
One of the main reasons I like this method is that I usually have a TNSNAMES.ORA file lying around someplace, and I can check that the dsn_tns object will do the right thing by doing:
print dsn_tns
and comparing the output to my TNSNAMES.ORA
You can specify the server in the connection string, e.g.:
import cx_Oracle
connstr = 'scott/tiger#server:1521/orcl'
conn = cx_Oracle.connect(connstr)
"server" is the server, or the IP address if you want.
"1521" is the port that the database is listening on.
"orcl" is the name of the instance (or database service).
import cx_Oracle
CONN_INFO = {
'host': 'xxx.xx.xxx.x',
'port': 12345,
'user': 'user_name',
'psw': 'your_password',
'service': 'abc.xyz.com',
}
CONN_STR = '{user}/{psw}#{host}:{port}/{service}'.format(**CONN_INFO)
connection = cx_Oracle.connect(CONN_STR)
Instead of specifying the SID, you can create a dsn and connect via service_name like:
import cx_Oracle
ip = '192.168.0.1'
port = 1521
service_name = 'my_service'
dsn = cx_Oracle.makedsn(ip, port, service_name=service_name)
db = cx_Oracle.connect('user', 'password', dsn)
The benefit of using the service name instead of the specific instance identifier (SID), is that it will work in a RAC environment as well (using a SID won't). This parameter is available as of cx_Oracle version 5.1.1 (Aug 28, 2011)
import cx_Oracle
dsn = cx_Oracle.makedsn(host='127.0.0.1', port=1521, sid='your_sid')
conn = cx_Oracle.connect(user='your_username', password='your_password', dsn=dsn)
conn.close()
import cx_Oracle
ip = '172.30.1.234'
port = 1524
SID = 'dev3'
dsn_tns = cx_Oracle.makedsn(ip, port, SID)
conn = cx_Oracle.connect('dbmylike', 'pass', dsn_tns)
print conn.version
conn.close()