I'm setting up a cron job, where it fetches some data from MongoDB atlas in Python3 through Pymongo in Cpanel. I always get a Error 111 Connnection refused.
I using python3.6 and pymongo-3.9.0, Cloud MongoDB-4.0.2
I have tried via SSHtunnel forwarder, but not sure how to give host_ip_addres, where MongoDB is in cluster
class DbConnection():
def __init__(self):
self.connectionServer = "mongodb+srv://"
self.userName = "name"
self.userPass = "pass"
self.connectionCluster = "#temp-cluster0-lt2rb.mongodb.net"
self.connectionDb = "developmentDB"
def db_connect(self):
''' This function is used to connect to remote db with authentication
Return type --> returns the url string of the db
parameters--> self
'''
try:
connectionUrl = self.connectionServer + self.userName + ":" + self.userPass + self.connectionCluster + "/test?retryWrites=true&w=majority"
print(connectionUrl)
myClient = pymongo.MongoClient(connectionUrl, port=12312)
db = myClient.test
print(myClient.test)
I'm expecting it to connect to the MongoDB cluster DB and read/Write the documents through it.
So the comments for the question solved the issue, so i'll just put this here for any future reference
The error can be resulted for must people because of a miss-configuration when moving from a self hosted MongoDB to a Remoted host service (Like MongoDB Atlas)
So, the +srv is DNS Seed List Connection Format.
When switching from port based connection to DNS seed based connection, we should remove any port configuration from our connection string, i.e:
class DbConnection():
def __init__(self):
self.connectionServer = "mongodb+srv://"
self.userName = "name"
self.userPass = "pass"
self.connectionCluster = "#temp-cluster0-lt2rb.mongodb.net"
self.connectionDb = "developmentDB"
def db_connect(self):
''' This function is used to connect to remote db with authentication
Return type --> returns the url string of the db
parameters--> self
'''
try:
connectionUrl = self.connectionServer + self.userName + ":" + self.userPass + self.connectionCluster + "/test?retryWrites=true&w=majority"
print(connectionUrl)
// myClient = pymongo.MongoClient(connectionUrl, port=12312)
// We remove the PORT
myClient = pymongo.MongoClient(connectionUrl)
db = myClient.test
print(myClient.test)
Related
I have an issue running a query using MySQLdb. I have the following error message
': 'utf-8' codec can't decode byte 0x92 in position 2: invalid start byte
I have been able to run this query in the past without issue of encoding and therefore I don't get where my issue comes from.
The only change I have done is downloading Postgres on my laptop and removing (accidentally) a db.sqlite3 file. But I don't understand why this will impact the encoding of my query especially since I don't have any special characters. Running a more simple query works fine and running this query from my colleague laptop also works.
The query is as following:
SLEEPERS_QUERY = """
select * from candidates"""
import MySQLdb as db
with SSHTunnelForwarder(
ssh_address_or_host = host,
ssh_port = 22,
ssh_username = ssh_username,
ssh_pkey = ssh_private_key,
remote_bind_address = (rds, 3306), # 3306 = mysql port
) as server:
server.start()
print('Connected to the SSH server')
while True:
try:
conn = db.connect(
host = localhost,
port = server.local_bind_port,
user = user,
passwd = password,
db = database
)
print('Connected to the database server')
break
except: pass
df = pd.read_sql_query(query,conn)
return df
I found a solution: my default encoding changed to from latin1 to utf8.
I had to add charset = 'latin1'
conn = db.connect(
host = localhost,
port = server.local_bind_port,
user = user,
passwd = password,
db = database,
charset = 'latin1'
)
I have an object that initiate ssh tunnel connection to a remote mongoDB. The same script works perfectly fine in other scraper running in a virtual machine that write data into the MongoDB.
However, in object in my GUI application keep giving me could not establish connection to ssh gateway error, while the scraper with the same code works fine.
Version:
sshtunnel version = 0.1.5
pymongo version = 3.8.0
python version = 3.6.7
def initMongo(self):
for attempt in range(5):
try:
MONGO_HOST = "xxx.xxx.xxx.xx"
REMOTE_PORT = 22
MONGO_USER = "USER"
MONGO_PASS = "PASSWORD"
#I have tried this but doesnt work
#self.Mclient = MongoClient('mongodb://{}:{}#{}:{}/'.format(MONGO_USER,MONGO_PASS,MONGO_HOST,REMOTE_PORT),connect=False)
sshtunnel.SSH_TIMEOUT = 60.0
sshtunnel.TUNNEL_TIMEOUT = 60.0
self.remoteServer = sshtunnel.SSHTunnelForwarder((MONGO_HOST,22),
ssh_username=MONGO_USER,
ssh_password=MONGO_PASS,
remote_bind_address=('127.0.0.1',27017))
self.remoteServer.start()
self.client = MongoClient('127.0.0.1',self.remoteServer.local_bind_port)
self.db = self.client.db_I_want
return
except Exception as e:
print("From Object :{}".format(e))
#time.sleep(5)
self.initMongo()
2019-08-25 17:54:11,266| ERROR | Could not connect to gateway xxx.xxx.xxx.xxx:xx : xxx
From Object :Could not establish session to SSH gateway
2019-08-25 17:54:11,290| ERROR | Could not connect to gateway xxx.xxx.xxx.xxx:xx : xxx
From Object :Could not establish session to SSH gateway
2019-08-25 17:54:11,311| ERROR | Could not connect to gateway xxx.xxx.xxx.xxx:xx : xxx
From Object :Could not establish session to SSH gateway
2019-08-25 17:54:11,336| ERROR | Could not connect to gateway xxx.xxx.xxx.xxx:xx : xxx
Fixed it by increasing the remote database's max connection.
https://www.howtogeek.com/240809/is-it-possible-to-have-multiple-ssh-connections-to-the-same-system/
I am using Python SMPP lib to send a SMS. When I try to connect to SmppServer with a longer credentials the username and password are getting truncated and authorisation is failing.
Auth failing case:
- Username/password longer than 16 char length
Passing case:
- Username/password not longer than 16 char
So because of the above observation I am sure there is no issue with the SMMPP gateway. The gateway I am trying to communicate itself accepts of username/password of any length.
The following is my code which wraps the smpplib in to a custom class:
import smpplib
import smpplib.gsm
from smpplib.client import Client
class SmppClientConfig(object, ):
def __init__(self, host, port, username, password, source, target, on_delivered_cb, on_sent_cb):
self.HOST = host
self.PORT = port
self.USERNAME = username
self.PASSWORD = password
self.SOURCE = source
self.TARGET = target
self.ON_RECEIVED_CALLBACK = on_sent_cb
self.ON_DELIVERED_CALLBACK = on_delivered_cb
class SmppSenderClient(object):
def __init__(self, config: SmppClientConfig):
print('Creating SMPP client config with host: ' + config.HOST + ' port: ' + str(config.PORT))
self._config = config
self._client = Client(config.HOST, config.PORT)
self._on_delivered_callback = config.ON_DELIVERED_CALLBACK
self._on_received_callback = config.ON_RECEIVED_CALLBACK
self._init_client()
def _init_client(self):
print('Initializing SmppSender client with username: ' + self._config.USERNAME)
self._register_events()
self._client.connect()
self._client.bind_transmitter(system_id=self._config.USERNAME, password=self._config.PASSWORD)
def _register_events(self):
print('Registering Smpp events')
self._client.set_message_sent_handler(self._config.ON_DELIVERED_CALLBACK)
self._client.set_message_received_handler(self._config.ON_RECEIVED_CALLBACK)
def send_message(self, message: str):
print('Sending SMS message to target: ' + self._config.TARGET)
parts, encoding_flag, msg_type_flag = smpplib.gsm.make_parts(message)
for part in parts:
self._client.send_message(
source_addr_ton=smpplib.consts.SMPP_TON_INTL,
source_addr_npi=smpplib.consts.SMPP_NPI_ISDN,
source_addr=self._config.SOURCE,
dest_addr_ton=smpplib.consts.SMPP_TON_INTL,
dest_addr_npi=smpplib.consts.SMPP_NPI_ISDN,
destination_addr=self._config.TARGET,
short_message=part,
data_coding=encoding_flag,
esm_class=msg_type_flag,
registered_delivery=True,
)
I am not sure if it's an expected behaviour of the library or a limitation. I have tried the find the documentation for this lib but could not find anything other than this.
Please advise if you experience a similar issue any work around possible or if this is expected behaviour in SMPP protocol (which very unlikely).
Update:
I found the limitation in the source code:
class BindTransmitter(Command):
"""Bind as a transmitter command"""
params = {
'system_id': Param(type=str, max=16),
'password': Param(type=str, max=9),
'system_type': Param(type=str, max=13),
'interface_version': Param(type=int, size=1),
'addr_ton': Param(type=int, size=1),
'addr_npi': Param(type=int, size=1),
'address_range': Param(type=str, max=41),
}
# Order is important, but params dictionary is unordered
params_order = (
'system_id', 'password', 'system_type',
'interface_version', 'addr_ton', 'addr_npi', 'address_range',
)
def __init__(self, command, **kwargs):
super(BindTransmitter, self).__init__(command, need_sequence=False, **kwargs)
self._set_vars(**(dict.fromkeys(self.params)))
self.interface_version = consts.SMPP_VERSION_34
As you can see the BindTransmitter contructor (__init__) truncates the system_id to a max length of 16 and passsword to 9. Not sure why this was done this way.
I checked the official SMPP protocol spec. According to the spec the system_id is supposed to max 16 and password is supposed to be 9 for connection type: transmitter.
The following is the screenshot of this spec:
This is the link to the spec if anyone is interested.
So in conclusion, the library implementation of the spec is correct.
I want to store my aws iot mqtt messages into my postgresql. To do so, I have already connected my local posrtgresql to the amazon RDS instance. Now, I need to create a connection between amazon lambda calculus and then send the data to the postgresql database. But whenever, I am testing my lambda calculus, it was giving me "name 'conn' is not defined: NameError" error. Here, is my python code in aws lambda. I have also included the psycopg2 library to my project.
import sys
import logging
import rds_config
import psycopg2
#rds settings
rds_host = "myhost"
name = "username"
password = "username_password"
db_name = "dbname"
logger = logging.getLogger()
logger.setLevel(logging.INFO)
try:
conn = psycopg2.connect(host=rds_host, user=name, password=password,
dbname=db_name, connect_timeout=5)
except:
logger.error("ERROR: Unexpected error: Could not connect to postgreSQL
instance.")
logger.info("SUCCESS: Connection to RDS postgreSQL instance succeeded")
def handler(event, context):
"""
This function fetches content from postgreSQL RDS instance
"""
item_count = 0
with conn.cursor() as cur:
cur.execute('insert into awsiotdata (serialnumber, dateandtime, clicktype, batteryvoltage) values(serialNumber, datetime.datetime.utcnow(), clickType, batteryVoltage)')
conn.commit()
cur.execute("select * from awsiotdata")
for row in cur:
item_count += 1
logger.info(row)
#print(row)
return "Added %d items from RDS PostgreSQL table" %(item_count)
You are hiding a true error message. Exception handling patter for Python looks like this:
try:
conn = psycopg2.connect(host=rds_host,
user=name,
password=password,
database=db_name)
except Exception as e:
print(e)
This way you will see the real error message:
invalid dsn: invalid connection option "passwd"
Edit #1:
"Timeout" means that lambda can't connect because of "Security group rules" for RDS instance. Please keep in mind that even public RDS instance by default have inbound restriction by IP (i.e. it is posible to connect from PC but it is imposible to connect from AWS Lambda).
How do you connect to a remote server via IP address in the manner that TOAD, SqlDeveloper, are able to connect to databases with just the ip address, username, SID and password?
Whenever I try to specify and IP address, it seems to be taking it locally.
In other words, how should the string for cx_Oracle.connect() be formatted to a non local database?
There was a previous post which listed as an answer connecting to Oracle via cx_Oracle module with the following code:
#!/usr/bin/python
import cx_Oracle
connstr='scott/tiger'
conn = cx_Oracle.connect(connstr)
curs = conn.cursor()
curs.execute('select * from emp')
print curs.description
for row in curs:
print row
conn.close()
I like to do it this way:
ip = '192.168.0.1'
port = 1521
SID = 'YOURSIDHERE'
dsn_tns = cx_Oracle.makedsn(ip, port, SID)
db = cx_Oracle.connect('username', 'password', dsn_tns)
One of the main reasons I like this method is that I usually have a TNSNAMES.ORA file lying around someplace, and I can check that the dsn_tns object will do the right thing by doing:
print dsn_tns
and comparing the output to my TNSNAMES.ORA
You can specify the server in the connection string, e.g.:
import cx_Oracle
connstr = 'scott/tiger#server:1521/orcl'
conn = cx_Oracle.connect(connstr)
"server" is the server, or the IP address if you want.
"1521" is the port that the database is listening on.
"orcl" is the name of the instance (or database service).
import cx_Oracle
CONN_INFO = {
'host': 'xxx.xx.xxx.x',
'port': 12345,
'user': 'user_name',
'psw': 'your_password',
'service': 'abc.xyz.com',
}
CONN_STR = '{user}/{psw}#{host}:{port}/{service}'.format(**CONN_INFO)
connection = cx_Oracle.connect(CONN_STR)
Instead of specifying the SID, you can create a dsn and connect via service_name like:
import cx_Oracle
ip = '192.168.0.1'
port = 1521
service_name = 'my_service'
dsn = cx_Oracle.makedsn(ip, port, service_name=service_name)
db = cx_Oracle.connect('user', 'password', dsn)
The benefit of using the service name instead of the specific instance identifier (SID), is that it will work in a RAC environment as well (using a SID won't). This parameter is available as of cx_Oracle version 5.1.1 (Aug 28, 2011)
import cx_Oracle
dsn = cx_Oracle.makedsn(host='127.0.0.1', port=1521, sid='your_sid')
conn = cx_Oracle.connect(user='your_username', password='your_password', dsn=dsn)
conn.close()
import cx_Oracle
ip = '172.30.1.234'
port = 1524
SID = 'dev3'
dsn_tns = cx_Oracle.makedsn(ip, port, SID)
conn = cx_Oracle.connect('dbmylike', 'pass', dsn_tns)
print conn.version
conn.close()