I'm trying to query from an online Postgres database (slave) via SQLAlchemy. I'm able to create an SSH-tunnel and creating engine and session (see code below). However when trying to setup the cursor (cursor = session.connection().connection.cursor()), we get a connection refused error.
Do you know what we should change in order to connect?
Some more considerations:
When I'm connection to this databases via DBeaver with same settings and credentials, it doesn't throw any errors
A colleague of mine is able to connect with same script (see below) and same SSH-key without any errors.
PostgreSQL 9.6.18 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11), 64-bit
The database is managed by external party, so I'm not able to make changes directly, however on request we can.
The error:
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
(Background on this error at: http://sqlalche.me/e/13/e3q8)
my_script.py:
from connection import create_connection
db_name = 'myDB'
session, engine = create_connection(
db_name, ssh=True)
cursor = session.connection().connection.cursor()
cursor.execute(''' SELECT * from file limit 10 ''')
data = cursor.fetchall()
print(data[0])
session.close()
cursor.close()
connection.py:
from sqlalchemy.orm import sessionmaker
from sqlalchemy import create_engine
import config
def create_connection(db_name, ssh=False):
connection_id_db = config.CONNECTION_IDS[db_name]['DB_CONNECTION']
if ssh:
connection_id_ssh = config.CONNECTION_IDS[db_name]['SSH_CONNECTION']
with sshtunnel.SSHTunnelForwarder(
# Remote server IP and SSH port
connection_id_ssh['REMOTE_SERVER'],
ssh_username=connection_id_ssh['SSH_USERNAME'],
ssh_private_key=connection_id_ssh['SSH_PRIVATE_KEY'],
remote_bind_address=connection_id_ssh['REMOTE_BIND_ADDRESS'],
local_bind_address=connection_id_ssh['LOCAL_BIND_ADDRESS']
) as server: # PostgreSQL server IP and sever port on remote machine
server.start()
# connect to PostgreSQL
engine = create_engine('postgresql://' +
connection_id_db['DB_USERNAME'] + ':' +
connection_id_db['DB_PASSWORD'] + '#' +
connection_id_db['DB_LOCAL_ADDRESS'] + ':' +
connection_id_db['DB_LOCAL_PORT'] + '/' +
connection_id_db['DB_NAME'], use_batch_mode=True)
Session = sessionmaker(bind=engine)
session = Session()
print('Database session created')
return session, engine
And the config.py:
CONNECTION_IDS = {
'myDB': {
'SSH_CONNECTION': {
'REMOTE_SERVER': ('XXX.XXX.nl', 22),
'SSH_USERNAME': 'ubuntu',
'SSH_PRIVATE_KEY': '/location/to/.ssh/My_ssh_key.pem',
'REMOTE_BIND_ADDRESS': ('XXX.XXX.eu-west-1.rds.amazonaws.com', 5432),
'LOCAL_BIND_ADDRESS': ('localhost', 1234)
},
'DB_CONNECTION': {
'DB_USERNAME': '<username>',
'DB_PASSWORD': '<password>',
'DB_LOCAL_ADDRESS': 'localhost',
'DB_LOCAL_PORT': '5432',
'DB_NAME': 'myDB'
}
},
}
The SSH tunnel is set up to accept connections on port 1234 locally, but the connection is attempted at port 5432.
If you don't have a PostgreSQL instance running locally (according to the error message, you don't), the port 5432 is available, so change in config.py:
'LOCAL_BIND_ADDRESS': ('localhost', 1234)
into:
'LOCAL_BIND_ADDRESS': ('localhost', 5432)
Related
I'm running MySQL in a Docker container, and in my jupyter lab I have the following cell:
# Importing module
import mysql.connector
# Creating connection object
mydb = mysql.connector.connect(
host = "localhost",
port = 3307,
user = "bob",
password = "1234",
database = 'testDB'
)
# Printing the connection object
print(mydb)
Which prints: <mysql.connector.connection_cext.CMySQLConnection object at 0x7fc633769dd0>
But I want to connect using ipython-sql, so I load the extension with %load_ext sql, but when I try:
%sql mysql://bob:1234#localhost:3307/testDB
I get the error:
Connection info needed in SQLAlchemy format, example:
postgresql://username:password#hostname/dbname
or an existing connection: dict_keys([])
No module named 'MySQLdb'
Connection info needed in SQLAlchemy format, example:
postgresql://username:password#hostname/dbname
or an existing connection: dict_keys([])
My Postgres database only accepts requests from localhost. Requests from a remote computer rely on SSH tunnel.
with SSHTunnelForwarder(
('my.db.ip.address', 22),
ssh_username='ssh_user_name',
ssh_private_key='id_rsa.pem',
remote_bind_address=('127.0.0.1', 5432)) as server:
conn = psycopg2.connect(database="my_db", port=server.local_bind_port, user=id, password=pwd)
curs = conn.cursor()
sql = "select t.id from public.my_table t"
This code works from Windows machine. On Ubuntu it fails with
File "/home/me/projects/compliance/response_generator.py", line 22, in get_rows
conn = psycopg2.connect(database="my_db", port=server.local_bind_port, user=id,password=pwd)
File "/home/me/anaconda3/envs/my_env/lib/python3.8/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server on socket "/tmp/.s.PGSQL.38945" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
What could be the reason and how to fix it?
----Update-----
tunnel = SSHTunnelForwarder(
('my.db.ip.address', 22),
ssh_username='ssh_user_name',
ssh_private_key='exdtras.dat',
remote_bind_address=('localhost', 5432),
local_bind_address=('localhost',6543), # could be any available port
Returns raise ValueError('No password or public key available!') ValueError: No password or public key available!
Any idea why this code works on Windows, but fails on Ubuntu?
I set up my server on MacBook air and my client is connecting from a Windows 10 machine. I turned off all security and firewalls on both machines (because that's what you do when you've lost all hope) and tried to connect from my windows machine multiple times, but to no avail. I'm on the same network btw (not using a VM).
Error:
TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
Server:
import socket, threading
import multiprocessing
from time import sleep
# tells us the bytes of the message
HEADER = 64
PORT = 5430
# or you could do socket.gethostbyname(socket.gethostname())
SERVER = socket.gethostbyname(socket.gethostname())
ADDR = (SERVER, PORT)
FORMAT = 'utf-8'
DISCONNECT_MESSAGE = '!END'
# defines the type of connection
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(ADDR)
def start():
server.listen()
print(f'[LISTENING] Server is listening on {SERVER}')
while True:
conn, addr = server.accept()
print('[ACTIVE CONNECTIONS] 1')
start()
Client:
from time import sleep
import socket, subprocess
HEADER = 64
PORT = 5430
FORMAT = 'utf-8'
DISCONNECT_MESSAGE = '!END'
SERVER = '192.32.322.3'
ADDR = (SERVER, PORT)
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(ADDR)
I have an object that initiate ssh tunnel connection to a remote mongoDB. The same script works perfectly fine in other scraper running in a virtual machine that write data into the MongoDB.
However, in object in my GUI application keep giving me could not establish connection to ssh gateway error, while the scraper with the same code works fine.
Version:
sshtunnel version = 0.1.5
pymongo version = 3.8.0
python version = 3.6.7
def initMongo(self):
for attempt in range(5):
try:
MONGO_HOST = "xxx.xxx.xxx.xx"
REMOTE_PORT = 22
MONGO_USER = "USER"
MONGO_PASS = "PASSWORD"
#I have tried this but doesnt work
#self.Mclient = MongoClient('mongodb://{}:{}#{}:{}/'.format(MONGO_USER,MONGO_PASS,MONGO_HOST,REMOTE_PORT),connect=False)
sshtunnel.SSH_TIMEOUT = 60.0
sshtunnel.TUNNEL_TIMEOUT = 60.0
self.remoteServer = sshtunnel.SSHTunnelForwarder((MONGO_HOST,22),
ssh_username=MONGO_USER,
ssh_password=MONGO_PASS,
remote_bind_address=('127.0.0.1',27017))
self.remoteServer.start()
self.client = MongoClient('127.0.0.1',self.remoteServer.local_bind_port)
self.db = self.client.db_I_want
return
except Exception as e:
print("From Object :{}".format(e))
#time.sleep(5)
self.initMongo()
2019-08-25 17:54:11,266| ERROR | Could not connect to gateway xxx.xxx.xxx.xxx:xx : xxx
From Object :Could not establish session to SSH gateway
2019-08-25 17:54:11,290| ERROR | Could not connect to gateway xxx.xxx.xxx.xxx:xx : xxx
From Object :Could not establish session to SSH gateway
2019-08-25 17:54:11,311| ERROR | Could not connect to gateway xxx.xxx.xxx.xxx:xx : xxx
From Object :Could not establish session to SSH gateway
2019-08-25 17:54:11,336| ERROR | Could not connect to gateway xxx.xxx.xxx.xxx:xx : xxx
Fixed it by increasing the remote database's max connection.
https://www.howtogeek.com/240809/is-it-possible-to-have-multiple-ssh-connections-to-the-same-system/
How do you connect to a remote server via IP address in the manner that TOAD, SqlDeveloper, are able to connect to databases with just the ip address, username, SID and password?
Whenever I try to specify and IP address, it seems to be taking it locally.
In other words, how should the string for cx_Oracle.connect() be formatted to a non local database?
There was a previous post which listed as an answer connecting to Oracle via cx_Oracle module with the following code:
#!/usr/bin/python
import cx_Oracle
connstr='scott/tiger'
conn = cx_Oracle.connect(connstr)
curs = conn.cursor()
curs.execute('select * from emp')
print curs.description
for row in curs:
print row
conn.close()
I like to do it this way:
ip = '192.168.0.1'
port = 1521
SID = 'YOURSIDHERE'
dsn_tns = cx_Oracle.makedsn(ip, port, SID)
db = cx_Oracle.connect('username', 'password', dsn_tns)
One of the main reasons I like this method is that I usually have a TNSNAMES.ORA file lying around someplace, and I can check that the dsn_tns object will do the right thing by doing:
print dsn_tns
and comparing the output to my TNSNAMES.ORA
You can specify the server in the connection string, e.g.:
import cx_Oracle
connstr = 'scott/tiger#server:1521/orcl'
conn = cx_Oracle.connect(connstr)
"server" is the server, or the IP address if you want.
"1521" is the port that the database is listening on.
"orcl" is the name of the instance (or database service).
import cx_Oracle
CONN_INFO = {
'host': 'xxx.xx.xxx.x',
'port': 12345,
'user': 'user_name',
'psw': 'your_password',
'service': 'abc.xyz.com',
}
CONN_STR = '{user}/{psw}#{host}:{port}/{service}'.format(**CONN_INFO)
connection = cx_Oracle.connect(CONN_STR)
Instead of specifying the SID, you can create a dsn and connect via service_name like:
import cx_Oracle
ip = '192.168.0.1'
port = 1521
service_name = 'my_service'
dsn = cx_Oracle.makedsn(ip, port, service_name=service_name)
db = cx_Oracle.connect('user', 'password', dsn)
The benefit of using the service name instead of the specific instance identifier (SID), is that it will work in a RAC environment as well (using a SID won't). This parameter is available as of cx_Oracle version 5.1.1 (Aug 28, 2011)
import cx_Oracle
dsn = cx_Oracle.makedsn(host='127.0.0.1', port=1521, sid='your_sid')
conn = cx_Oracle.connect(user='your_username', password='your_password', dsn=dsn)
conn.close()
import cx_Oracle
ip = '172.30.1.234'
port = 1524
SID = 'dev3'
dsn_tns = cx_Oracle.makedsn(ip, port, SID)
conn = cx_Oracle.connect('dbmylike', 'pass', dsn_tns)
print conn.version
conn.close()