I have a serverless function that's trying to send some data to kafka. Sometimes it works and sometimes the connection just drop and the data is lost.
The reason for this is that the library for kafka is not raising the exception but it's adding error logs instead. So i cannot add my piece of code in a try:except.
Here is the error that i am often getting in my logs:
<BrokerConnection node_id=11 host=... port=9092>: Error receiving network data closing socket
Traceback (most recent call last):
File "/var/task/kafka/conn.py", line 745, in _recv
data = self._sock.recv(SOCK_CHUNK_BYTES)
ConnectionResetError: [Errno 104] Connection reset by peer
and the function _recv mentioned above has definition as follows:
I am still looking for a solution but adding the code within try:except doesn't work.
def _recv(self):
responses = []
SOCK_CHUNK_BYTES = 4096
while True:
try:
data = self._sock.recv(SOCK_CHUNK_BYTES)
# We expect socket.recv to raise an exception if there is not
# enough data to read the full bytes_to_read
# but if the socket is disconnected, we will get empty data
# without an exception raised
if not data:
log.error('%s: socket disconnected', self)
self.close(error=Errors.ConnectionError('socket disconnected'))
break
else:
responses.extend(self.receive_bytes(data))
if len(data) < SOCK_CHUNK_BYTES:
break
except SSLWantReadError:
break
except ConnectionError as e:
if six.PY2 and e.errno == errno.EWOULDBLOCK:
break
log.exception('%s: Error receiving network data'
' closing socket', self)
self.close(error=Errors.ConnectionError(e))
break
except BlockingIOError:
if six.PY3:
break
raise
return responses
I expect when the connection is lost, an exception is raised and the application breaks but instead only there are only error logs and the application continues even if the kafka part fails.
Related
I modified the code above from what Torxed posted in the URL: Sending string via socket (python)
I used this as a base as want a multi-threading python3 server. I'm still learning Python so am getting a little stuck.
I've got 2 threads working where the 1st (a background) thread sends a UDP broadcast to inform clients that the source host (the python server) is waiting for client connections.
The client detects the UDP broadcast (on port 1300) & initiates a new TCP session to port 1200. I have got the python server to immediately send some bytes (bytes as not ascii) to the client which 'asks' the client to identity itself upon connection (the client does not initiate the conversation other than establishing the TCP session). In this case the client host is a solar inverter.
The client responds with some basics such as it's serial number, software version, etc which I want to store. Currently I'm not able to store the reply in a variable as it's byte sequence, not ascii so I'm currently writing the datagram to a local file until I figure out how to unpack the byte sequence & save in variables associated to the client session. This will be used to aid identifying which data-set is associated to which client.
Every 15 seconds after that, I want the python server to send some more bytes (again bytes not ascii) which will inform the client to reply with some PV (solar) data.
The client will reply with the PV data which I want to store associated to the client session.
Lastly if an event occurs on the client (inverter such as AC loss power) it will send to the server another datagram in the established TCP session a specifically formatted byte sequence so the server needs to handle an independent response from the client which will be to initiate sending an email (to me) containing the event type. Is it better to use a single recv datagram handler with something like a case statement to identify the response then process it accordingly (but the server must first receive the client's id).
I want the TCP session to remain open indefinitely but also support a 2nd inverter to connect too & allow them should connectivity is temporally interpreted to re-connection without issue.
Any help appreciated as I'm stuck with being unable to read the response from the client.
Here is what I currently have:
#!/usr/bin/env python3
# need threading support to run broadcast UDP 1300 thread
from threading import *
from socket import *
# import time for sleep/delay timers
import time
# Now we can create socket object
serversocket = socket(AF_INET, SOCK_STREAM)
#serversocket.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
#serversocket.setsockopt(SOL_SOCKET, SO_BROADCAST, 1)
# Set the IP address and port to listen on
HOST = "192.168.1.11"
PORT = 1200
# Bind the socket on the interface/port
serversocket.bind((HOST, PORT))
# This is the client thread allowing multiple clients to connect
class client(Thread):
# This section is what will run in the background
def background():
while True:
cs.sendto(b'\x55\xaa\x00\x40\x02\x00\x0b\x49\x20\x41\x4d\x20\x53\x45\x52\x56\x45\x52\x04\x3a', ("192.168.1.255", 1301))
time.sleep(5)
# This section is what will run in the foreground
def __init__(self, socket, address):
Thread.__init__(self)
self.sock = socket
self.addr = address
self.start()
def run(self):
while True:
msg = (b'\x55\xaa\x01\x03\x02\x00\x00\x01\x05')
self.sock.send(msg)
#Open recv.txt file in write mode
recvdatafile1 = open("recv1.txt", "wb")
print("Destination file name will be recv1.txt on the server\n")
# Receive intiate data from client side
# RecvData = self.sock.recv(1024)
while True:
recvdatafile1.write(RecvData)
# recvdatafile1.write(self.sock.recv(1024))
RecvData = self.sock.recv(1024)
# Request initial configuration information (replies with S/N, model number, etc
#msg = (b'\x55\xaa\x01\x03\x02\x00\x00\x01\x05')
#conn.send(msg)
# Close the file opened at server side once copy is completed
recvdatafile1.close()
print("\n File has been copied successfully \n")
#Open recv.txt file in write mode
recvdatafile2 = open("recv2.txt", "wb")
print("Destination file name will be recv2.txt on the server\n")
# Request PV data
msg = (b'\x55\xaa\x01\x02\x02\x00\x00\x01\x04')
conn.send(msg)
# Receive any data from client side
RecvData = conn.recv(1024)
while RecvData:
recvdatafile2.write(RecvData)
RecvData = conn.recv(1024)
# Close the file opened at server side once copy is completed
recvdatafile2.close()
print("\n File has been copied successfully \n")
# Close connection with client
conn.close()
print("\n Server closed the connection \n")
time.sleep(3)
# Come out from the infinite while loop as the file has been copied from client.
#break
serversocket.listen(5)
print("Server is listing on port:", PORT, "\n")
while 1:
clientsocket, address = serversocket.accept()
client(clientsocket, address)
print ("connection found!")
data = clientsocket.recv(1024).decode()
print (data)
b = threading.Thread(name='background', target=background)
f = threading.Thread(name='foreground', target=foreground)
b.start()
f.start()
Server Output:
user#server:~# python3 ./Samil_Solar_River_Monitor_Server.py
Server is listing on port: 1200
connection found!
Destination file name will be recv1.txt on the server
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "./Samil_Solar_River_Monitor_Server.py", line 62, in run
recvdatafile1.write(RecvData)
UnboundLocalError: local variable 'RecvData' referenced before assignment
Traceback (most recent call last):
File "./Samil_Solar_River_Monitor_Server.py", line 109, in <module>
data = clientsocket.recv(1024).decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xaa in position 1: invalid start byte
user#server:~#
Client Output:
user#client:~$ python3 test-solar-client.py 192.168.1.11
################## Below rcved from srv ##################
b'U\xaa\x01\x03\x02\x00\x00\x01\x05' <-- This is good/correct
################## Below rcved from srv ##################
b'' <-- this & below appears when traceback on server occurs
Traceback (most recent call last):
File "test-solar-client.py", line 47, in <module>
s.send(SendData)
BrokenPipeError: [Errno 32] Broken pipe
user#client:~$
I had using pika to make a connection to RabbitMQ and consume message, once I start the script on ubuntu prod environment it is working as expected but is opening mysql connection and never closes them and ends up in Too many connection on mysql server.
Will appreciate any recommendation on the code below, as well can not understand what is going wrong. Thanking you in advance.
The flow is the following
Starting pika on Python3
Subscribe to a channel and waiting for messages
In callback i do various validation and save or update data inside MySql
The result that is showing the problem is the at the end of question a screenshot from ubuntu htop, that is showing new connection on MySql and keep adding them on the top
Pika Verion = 0.13.0
For MySql I use pymysql.
Pika Script
def main():
credentials = pika.PlainCredentials(tunnel['queue']['name'], tunnel['queue']['password'])
while True:
try:
cp = pika.ConnectionParameters(
host=tunnel['queue']['host'],
port=tunnel['queue']['port'],
credentials=credentials,
ssl=tunnel['queue']['ssl'],
heartbeat=600,
blocked_connection_timeout=300
)
connection = pika.BlockingConnection(cp)
channel = connection.channel()
def callback(ch, method, properties, body):
if 'messageType' in properties.headers:
message_type = properties.headers['messageType']
if events.get(message_type):
result = Descriptors._reflection.ParseMessage(events[message_type]['decode'], body)
if result:
result = protobuf_to_dict(result)
model.write_response(external_response=result, message_type=message_type)
else:
app_log.warning('Message type not in allowed list = ' + str(message_type))
app_log.warning('continue listening...')
channel.basic_consume(callback, queue=tunnel['queue']['name'], no_ack=True)
try:
channel.start_consuming()
except KeyboardInterrupt:
channel.stop_consuming()
connection.close()
break
except pika.connection.exceptions.ConnectionClosed as e:
app_log.error('ConnectionClosed :: %s' % str(e))
continue
except pika.connection.exceptions.AMQPChannelError as e:
app_log.error('AMQPChannelError :: %s' % str(e))
continue
except Exception as e:
app_log.error('Connection was closed, retrying... %s' % str(e))
continue
if __name__ == '__main__':
main()
Inside the script i have a model that doing inserts or updated in the database, code below
def write_response(self, external_response, message_type):
table_name = events[message_type]['table_name']
original_response = external_response[events[message_type]['response']]
if isinstance(original_response, list):
external_response = []
for o in original_response:
record = self.map_keys(o, message_type, events[message_type].get('values_fix', {}))
external_response.append(self.validate_fields(record))
else:
external_response = self.map_keys(original_response, message_type, events[message_type].get('values_fix', {}))
external_response = self.validate_fields(external_response)
if not self.mysql.open:
self.mysql.ping(reconnect=True)
with self.mysql.cursor() as cursor:
if isinstance(original_response, list):
for e in external_response:
id_name = events[message_type]['id_name']
filters = {id_name: e[id_name]}
self.event(
cursor=cursor,
table_name=table_name,
filters=filters,
external_response=e,
message_type=message_type,
event_id=e[id_name],
original_response=e # not required here
)
else:
id_name = events[message_type]['id_name']
filters = {id_name: external_response[id_name]}
self.event(
cursor=cursor,
table_name=table_name,
filters=filters,
external_response=external_response,
message_type=message_type,
event_id=external_response[id_name],
original_response=original_response
)
cursor.close()
self.mysql.close()
return
On ubuntu i use systemd to run the script and restart in case something goes wrong, below is systemd file
[Unit]
Description=Pika Script
Requires=stunnel4.service
Requires=mysql.service
Requires=mongod.service
[Service]
User=user
Group=group
WorkingDirectory=/home/pika_script
ExecStart=/home/user/venv/bin/python pika_script.py
Restart=always
[Install]
WantedBy=multi-user.target
Image from ubuntu htop, how the MySql keeps adding in the list and never close it
Error
tornado_mysql.err.OperationalError: (1040, 'Too many connections')
i have found the issue, posting if will help somebody else.
the problem was that mysqld went into infinite loop trying to create indexing to a specific database, after found to which database was trying to create the indexes and never succeed and was trying again and again.
solution was to remove the database and recreate it, and the mysqld process went back to normal. and the infinite loop to create indexes dissapeared as well.
I would say increasing connection may solve your problem temperately.
1st find out why the application is not closing the connection after completion of task.
2nd Any slow queries/calls on the DB and fix them if any.
3rd considering no slow queries/calls on DB and also application is closing the connection/thread after immediately completing the task, then consider playing with "wait_timeout" on mysql side.
According to this answer, if you have MySQL 5.7 and 5.8 :
It is worth knowing that if you run out of usable disc space on your
server partition or drive, that this will also cause MySQL to return
this error. If you're sure it's not the actual number of users
connected then the next step is to check that you have free space on
your MySQL server drive/partition.
From the same thread. You can inspect and increase number of MySQL connections.
I am searching for a method to test if the camera is on for PTG camera.
In PyCapture2 the below code works but the presumed PySpin cam.DeviceConnectionStatus() will not work because the function seems not to be present.
PySpin Camera library version: 1.23.0.27
The Error:
Error: Spinnaker: GenICam::AccessException= Feature not present (reference not valid) : AccessException thrown (file 'IEnumerationT.h', line 341) [-2006]
(False, SpinnakerException("Spinnaker: GenICam::AccessException= Feature not present (reference not valid) : AccessException thrown (file 'IEnumerationT.h', line 341) [-2006]"))
I've tried also PySpin.Camera.DeviceConnectionStatus() but it gives the following error whether prior or after cam.Init():
Traceback (most recent call last):
File "X.py", line 82, in YZ
print (PySpin.Camera.DeviceConnectionStatus())
TypeError: 'property' object is not callable
Working PyCapture2 code:
def cameraOn(self, cam):
# Power on the Camera
cameraPower = 0x610
powerVal = 0x80000000
cam.writeRegister(cameraPower, powerVal)
# Waiting for camera to power up
retries = 10
timeToSleep = 0.1 #seconds
for i in range(retries):
sleep(timeToSleep)
try:
regVal = cam.readRegister(cameraPower)
except PyCapture2.Fc2error: # Camera might not respond to register reads during powerup.
pass
awake = True
if regVal == powerVal:
break
awake = False
if not awake:
print ("Could not wake Camera. Exiting...")
exit()
As it seems there is an IsValid() function available from the CameraBase() class in the PySpin/Spinnaker library. This function returns either the bool True once a connection could be made, communication was successful and the camera is still valid for use or a "False" respectively. However, this function does not turn the camera ON or OFF. Nor does it power from a sleep/wake state.
For unknown reasings the IsValid() function does not report back tracebacks for logging or debug purpose. So keep in mind to implement try/except for certain methods.
try:
... your code ...
except PySpin.SpinnakerException as error:
print('Error: %s' % error)
return False, error
I'm trying to create a simple echo server that can handle multiple connections, I am using select and need to use non blocking sockets but I get an error saying 'BlockingIOError: [Errno 35] Resource temporarily unavailable' on the line where I receive the data.
Here's the server and client code
inputs = [server]
while True:
inready, outready, excready = select.select(inputs, [], [])
for s in inready:
if s == server:
client, address = server.accept()
client.setblocking(0)
print(address)
inready.append(client)
else:
data = s.recv(1024);
if data:
s.send(data)
else:
inputs.remove(s)
s.close()
This little error took me also a while to find:
inready.append(client)
Correct is, of course:
inputs.append(client)
Since inready was accidentally modified, the else block was executed when it shouldn't have been.
I'm porting some code from perl (log4perl) and java (slf4j). All is fine except for logging.critical() does not dump stacktrace and die like it does in the other frameworks, need to add a lot of extra code, logger.exception() also only writes error.
Today I do:
try:
errmsg = "--id={} not found on --host={}".format(args.siteid, args.host)
raise GX8Exception(errmsg)
except GX8Exception as e:
log.exception(e)
sys.exit(-1)
This produces:
2018-01-10 10:09:56,814 [ERROR ] root --id=7A4A7845-7559-4F89-B678-8ADFECF5F7C3 not found on --host=welfare-qa
Traceback (most recent call last):
File "./gx8-controller.py", line 85, in <module>
raise GX8Exception(errmsg)
GX8Exception: --id=7A4A7845-7559-4F89-B678-8ADFECF5F7C3 not found on --host=welfare-qa
Is there a way to config pythonmodule logger to do this, or any other framework to do the same:
log.critical("--id={} not found on --host={}".format(args.siteid, args.host))
One approach would be to create a custom Handler that does nothing but pass log messages on to its super and then exit if the log level is high enough:
import logging
class ExitOnExceptionHandler(logging.StreamHandler):
def emit(self, record):
super().emit(record)
if record.levelno in (logging.ERROR, logging.CRITICAL):
raise SystemExit(-1)
logging.basicConfig(handlers=[ExitOnExceptionHandler()], level=logging.DEBUG)
logger = logging.getLogger('MYTHING')
def causeAProblem():
try:
raise ValueError("Oh no!")
except Exception as e:
logger.exception(e)
logger.warning('Going to try something risky...')
causeAProblem()
print("This won't get printed")
Output:
rat#pandion:~$ python test.py
ERROR:root:Oh no!
Traceback (most recent call last):
File "test.py", line 14, in causeAProblem
raise ValueError("Oh no!")
ValueError: Oh no!
rat#pandion:~$ echo $?
255
However, this could cause unexpected behavior for users of your code. It would be much more straightfoward, if you want to log an exception and exit, to simply leave the exception uncaught. If you want to log a traceback and exit wherever the code is currently calling logging.critical, change it to raise an exception instead.
I inherited some code where I could not change the handler class. I resorted to run time patching of the handler which is a variation on the solution by #nathan-vērzemnieks:
import types
def patch_logging_handler(logger):
def custom_emit(self, record):
self.orig_emit(record)
if record.levelno == logging.FATAL:
raise SystemExit(-1)
handler = logger.handlers[0]
setattr(handler, 'orig_emit', handler.emit)
setattr(handler, 'emit', types.MethodType(custom_emit, handler))
Nathans anwser is great! Been looking for this for a long time,
will just add that you can also do:
if record.levelno >= logging.ERROR:
instead of
if record.levelno in (logging.ERROR, logging.CRITICAL):
to set the minimum level that would cause an exit.