I have an issue running a query using MySQLdb. I have the following error message
': 'utf-8' codec can't decode byte 0x92 in position 2: invalid start byte
I have been able to run this query in the past without issue of encoding and therefore I don't get where my issue comes from.
The only change I have done is downloading Postgres on my laptop and removing (accidentally) a db.sqlite3 file. But I don't understand why this will impact the encoding of my query especially since I don't have any special characters. Running a more simple query works fine and running this query from my colleague laptop also works.
The query is as following:
SLEEPERS_QUERY = """
select * from candidates"""
import MySQLdb as db
with SSHTunnelForwarder(
ssh_address_or_host = host,
ssh_port = 22,
ssh_username = ssh_username,
ssh_pkey = ssh_private_key,
remote_bind_address = (rds, 3306), # 3306 = mysql port
) as server:
server.start()
print('Connected to the SSH server')
while True:
try:
conn = db.connect(
host = localhost,
port = server.local_bind_port,
user = user,
passwd = password,
db = database
)
print('Connected to the database server')
break
except: pass
df = pd.read_sql_query(query,conn)
return df
I found a solution: my default encoding changed to from latin1 to utf8.
I had to add charset = 'latin1'
conn = db.connect(
host = localhost,
port = server.local_bind_port,
user = user,
passwd = password,
db = database,
charset = 'latin1'
)
Related
Some information about my relevant-to-the-thread background:
Client-Server connection domain knowledge: Beginner
Python-programmer: Mid to advanced level
Problem I am facing:
connecting a Client to a Server using the OPCUA protocol (tool I am using: opcua python-library and VS-code editor).
How did the problem start?
I was attempting to use the node.get_value() method (where node = client.get_node("ns=2;i=2")). Then the error is: "None type does not have a method called 'send_request'". This made me think that the socket connection is not established. Therefore, I tried to establish a socket connection first, by using the UAClient.connect_socket() method, which lead me to the origin of the problem, which is the last command line in the following code:
My code:
from opcua import Server
from opcua import Client
from opcua import ua
from ua_client import *
from traceback import print_exc
from opcua import Client
name = "OPC_SIMULATION_SERVER"
addspace = server.register_namespace(name)
node = server.get_objects_node()
Param = node.add_object(addspace, "Parameters")
server = Server()
url = 'opc.tcp://127.0.0.1:4840' # opc server URL (connection works, checked in in command line with "ping")
server.start()
print("Server started at {}".format(url))
client = Client(url) # create client object
ref = client.uaclient # create UAClient object
# establish socket (to my understanding)
ref._uasocket = UASocketClient(ref._timeout, security_policy=ref.security_policy)
host = client.server_url.hostname
port = client.server_url.port
refsock = ref._uasocket
sock = socket.create_connection((host, port), timeout=refsock.timeout)
sock.settimeout(None) # for blocking (doesn’t cause the error)
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
refsock._socket = ua.utils.SocketWrapper(sock)
# ERRATIC LINE: The following command gets stuck without an error/warning message
chunk = refsock._socket.socket.recv(8)
# \==================================================
I should note that initially I had no problem with getting the value from the node.
I am trying to connect the socket via the following code
try:
# create an INET, STREAMing socket
self.client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# now connect to the web server on port 80 - the normal http port
self.client.connect((host, port))
user = ['User.loginByToken', access_token]
self.client.sendall(user)
# self.client._locust_environment = self.environment
except Exception as e:
color_print("\n[!] Check Server Address or Port", color="red", underline=True)
it throws an error memoryview: a bytes-like object is required, not 'list'. what can I do to solve it?
You need to convert user var to bytes in order to send thru a socket:
You can try this:
import json
user = json.dumps({'User.loginByToken': access_token})
user = user.encode("utf-8")
I'm setting up a cron job, where it fetches some data from MongoDB atlas in Python3 through Pymongo in Cpanel. I always get a Error 111 Connnection refused.
I using python3.6 and pymongo-3.9.0, Cloud MongoDB-4.0.2
I have tried via SSHtunnel forwarder, but not sure how to give host_ip_addres, where MongoDB is in cluster
class DbConnection():
def __init__(self):
self.connectionServer = "mongodb+srv://"
self.userName = "name"
self.userPass = "pass"
self.connectionCluster = "#temp-cluster0-lt2rb.mongodb.net"
self.connectionDb = "developmentDB"
def db_connect(self):
''' This function is used to connect to remote db with authentication
Return type --> returns the url string of the db
parameters--> self
'''
try:
connectionUrl = self.connectionServer + self.userName + ":" + self.userPass + self.connectionCluster + "/test?retryWrites=true&w=majority"
print(connectionUrl)
myClient = pymongo.MongoClient(connectionUrl, port=12312)
db = myClient.test
print(myClient.test)
I'm expecting it to connect to the MongoDB cluster DB and read/Write the documents through it.
So the comments for the question solved the issue, so i'll just put this here for any future reference
The error can be resulted for must people because of a miss-configuration when moving from a self hosted MongoDB to a Remoted host service (Like MongoDB Atlas)
So, the +srv is DNS Seed List Connection Format.
When switching from port based connection to DNS seed based connection, we should remove any port configuration from our connection string, i.e:
class DbConnection():
def __init__(self):
self.connectionServer = "mongodb+srv://"
self.userName = "name"
self.userPass = "pass"
self.connectionCluster = "#temp-cluster0-lt2rb.mongodb.net"
self.connectionDb = "developmentDB"
def db_connect(self):
''' This function is used to connect to remote db with authentication
Return type --> returns the url string of the db
parameters--> self
'''
try:
connectionUrl = self.connectionServer + self.userName + ":" + self.userPass + self.connectionCluster + "/test?retryWrites=true&w=majority"
print(connectionUrl)
// myClient = pymongo.MongoClient(connectionUrl, port=12312)
// We remove the PORT
myClient = pymongo.MongoClient(connectionUrl)
db = myClient.test
print(myClient.test)
I try to develop a simple server and client program, I run the code with python 3.4.2.( on Debian 8.6 ) . The server run well, the client program connect's to the server but when I pass a text in terminal to send to server and send back with time stamp, I get this error in the client terminal window
Traceback (most recent call last):
File "tcp_client", line 15, in
tcpCliSock.send(data)
TypeError: 'str' does not support the buffer interface
this is the server code
from socket import *
from time import ctime
HOST = '192.168.0.141'
PORT = 21577
BUFSIZ = 1024
ADDR = (HOST, PORT)
tcpSerSock = socket(AF_INET, SOCK_STREAM)
tcpSerSock.bind(ADDR)
tcpSerSock.listen(5)
while True:
print('waiting for connection...')
tcpCliSock, addr = tcpSerSock.accept()
print('....connected from :', addr)
while True:
data = tcpCliSock.recv(BUFSIZ)
if not data:
break
tcpCliSock.send('[%s] %s' % (bytes(ctime(), 'utf-8'), data))
tcpCliSock.close()
tcpSerSock.close()
and this it the client code
from socket import *
HOST = '192.168.0.141'
PORT = 21577
BUFSIZ = 1024
ADDR = (HOST, PORT)
tcpCliSock = socket(AF_INET, SOCK_STREAM)
tcpCliSock.connect(ADDR)
while True:
data = input('> ')
if not data:
break
tcpCliSock.send(data)
data = tcpCliSock.recv(BUFSIZ)
if not data:
break
print(data.decode('utf-8'))
tcpCliSock.close()
How do you connect to a remote server via IP address in the manner that TOAD, SqlDeveloper, are able to connect to databases with just the ip address, username, SID and password?
Whenever I try to specify and IP address, it seems to be taking it locally.
In other words, how should the string for cx_Oracle.connect() be formatted to a non local database?
There was a previous post which listed as an answer connecting to Oracle via cx_Oracle module with the following code:
#!/usr/bin/python
import cx_Oracle
connstr='scott/tiger'
conn = cx_Oracle.connect(connstr)
curs = conn.cursor()
curs.execute('select * from emp')
print curs.description
for row in curs:
print row
conn.close()
I like to do it this way:
ip = '192.168.0.1'
port = 1521
SID = 'YOURSIDHERE'
dsn_tns = cx_Oracle.makedsn(ip, port, SID)
db = cx_Oracle.connect('username', 'password', dsn_tns)
One of the main reasons I like this method is that I usually have a TNSNAMES.ORA file lying around someplace, and I can check that the dsn_tns object will do the right thing by doing:
print dsn_tns
and comparing the output to my TNSNAMES.ORA
You can specify the server in the connection string, e.g.:
import cx_Oracle
connstr = 'scott/tiger#server:1521/orcl'
conn = cx_Oracle.connect(connstr)
"server" is the server, or the IP address if you want.
"1521" is the port that the database is listening on.
"orcl" is the name of the instance (or database service).
import cx_Oracle
CONN_INFO = {
'host': 'xxx.xx.xxx.x',
'port': 12345,
'user': 'user_name',
'psw': 'your_password',
'service': 'abc.xyz.com',
}
CONN_STR = '{user}/{psw}#{host}:{port}/{service}'.format(**CONN_INFO)
connection = cx_Oracle.connect(CONN_STR)
Instead of specifying the SID, you can create a dsn and connect via service_name like:
import cx_Oracle
ip = '192.168.0.1'
port = 1521
service_name = 'my_service'
dsn = cx_Oracle.makedsn(ip, port, service_name=service_name)
db = cx_Oracle.connect('user', 'password', dsn)
The benefit of using the service name instead of the specific instance identifier (SID), is that it will work in a RAC environment as well (using a SID won't). This parameter is available as of cx_Oracle version 5.1.1 (Aug 28, 2011)
import cx_Oracle
dsn = cx_Oracle.makedsn(host='127.0.0.1', port=1521, sid='your_sid')
conn = cx_Oracle.connect(user='your_username', password='your_password', dsn=dsn)
conn.close()
import cx_Oracle
ip = '172.30.1.234'
port = 1524
SID = 'dev3'
dsn_tns = cx_Oracle.makedsn(ip, port, SID)
conn = cx_Oracle.connect('dbmylike', 'pass', dsn_tns)
print conn.version
conn.close()