I have a two-way datachannel setup that takes a heartbeat from a browser client and keeps the session alive as long as the heartbeat stays. The heartbeat is the 'main' communication for WebRTC, but I have other bits of into (Such as coordinates) I need to send constantly.
To do this when a webrtc offer is given, it takes that HTTP request:
Creates a new event loop 'rtcloop'
Set's that as the main event loop.
Then run 'rtcloop' until complete, calling my webRtcStart function and passing through the session info.
Then run a new thread with the target being 'rtcloop', run it forever and start.
Inside the new thread I set the loop with 'get_event_loop' and later define ' #webRtcPeer.on("datachannel")' so when we get a Datachannel message, we run code around that. Depending on the situation, I attempt to do the following:
ptzcoords = 'Supported' #PTZ Coords will be part of WebRTC Communication, send every 0.5 seconds.
ptzloop = asyncio.new_event_loop()
ptzloop.run_until_complete(updatePTZReadOut(webRtcPeer, cameraName, loop))
ptzUpdateThread = Thread(target=ptzloop.run_forever)
ptzUpdateThread.start()
The constant error I get no matter how I structure things is "coroutine 'updatePTZReadOut' was never awaited"
With updatePTZReadOut being:
async def updatePTZReadOut(rtcPeer, cameraName, eventLoop):
# Get Camera Info
# THE CURRENT ISSUE I am having is with the event loops, because this get's called to run in another thread, but it still needs
# to be awaitable,
# Current Warning Is: /usr/lib/python3.10/threading.py:953: RuntimeWarning: coroutine 'updatePTZReadOut' was never awaited
# Ref Article: https://xinhuang.github.io/posts/2017-07-31-common-mistakes-using-python3-asyncio.html
# https://lucumr.pocoo.org/2016/10/30/i-dont-understand-asyncio/
# Get current loop
# try:
loop = asyncio.set_event_loop(eventLoop)
# loop.run_until_complete()
# except RuntimeError:
# loop = asyncio.new_event_loop()
# asyncio.set_event_loop(loop)
# Getting Current COORDS from camera
myCursor.execute("Select * from localcameras where name = '{0}' ".format(cameraName))
camtuple = myCursor.fetchall()
camdata = camtuple[0]
# Create channel object
channel_local = rtcPeer.createDataChannel("chat")
while True:
ptzcoords = readPTZCoords(camdata[1], camdata[3], cryptocode.decrypt(str(camdata[4]), passwordRandomKey))
print("Updating Coords to {0}".format(ptzcoords))
# Publish Here
await channel_local.send("TTTT")
asyncio.sleep(0.5)
Any help here?
updatePTZReadOut is async function. You need to add await whenever you call this function.
Related
I'm trying to simulate processing in threads by using asyncio.Queue. However, I'm struggling to turn a threaded processing simulation part to asynchronous loop.
So what my script does in brief: 1) receive processing requests over a websocket, 2) assign the request to the requested queue (which simulates a thread), 3) runs processing queues, which put responses into one shared response queue, and then 4) the websocket takes out the responses from the shared queue one by one and sends them out to the server.
Simplified version of my code:
# Initialize empty processing queues for the number of threads I want to simulate
processing_queues = [asyncio.Queue() for i in range(n_queues)
# Initialize shared response queue
response_q = asyncio.Queue()
# Set up a websocket context manager
async with websockets.connect(f"ws://{host}:{port}") as websocket:
while True:
# Read incoming requests
message = await websocket.recv()
# Parse mssg -> get request data and on which thread / queue to process it
request_data, queue_no = parse_message(message)
# Put the request data to the requested queue (imitating thread)
await processing_queues[queue_no].put(request_data)
# THIS IS WHERE I THINK ASYNCHRONY BREAKS (AND I NEED HELP)
# Do processing in each imitated processing thread
for proc_q in processing_queues:
if not proc_q.empty():
request_data = await proc_q.get()
# do the processing
response = process_data(request_data)
# Add the response to the response queue
await response_q.put(response)
# Send responses back to the server
if not response_q.empty():
response_data = response_q.get()
await websocket.send(response_data)
From the output of the script, I deduced that 1) I seem to receive requests and send out responses asynchronously; 2) processing in queues does not happen asynchronously. Correct me if I'm wrong.
I was reading about create_task() in asyncio. Maybe that could be a way to solve my problem?
I'm open to any solution (even hacky).
P.S. I would just use threads from threading library, but I need asyncio for websockets library.
P.P.S. Threaded version of my idea.
class ProcessingImitationThread(threading.Thread):
def __init__(self, thread_id, request_q, response_q):
threading.Thread.__init__(self)
self.thread_id = thread_id
self.request_q = request_q
self.response_q = response_q
def run(self):
while True:
try:
(x, request_id) = self.request_q.get()
except Empty:
time.sleep(0.2)
else:
if x == -1:
# EXIT CONDITION
break
else:
sleep_time_for_x = count_imitation(x, state)
time.sleep(sleep_time_for_x)
self.response_q.put(request_id)
print(f"request {request_id} executed")
# Set up
processing_qs = [queue.Queue() for i in range(n_processes_simulated)]
response_q = queue.Queue()
processing_thread_handlers = []
for i in n_processes_simulated:
# create thread
t = ProcessingImitationThread(i, processing_qs[i], response_q)
processing_thread_handlers.append(t)
# Main loop
while True:
# receive requests and assign to requested queue (so that thread picks up)
if new_request:
requested_process, x, request_id = parse(new_request)
processing_qs[requested_process].put((x, request_id))
...
# if there are any new responses, sent them out to the server
if response_q.q_size() > 0:
request_id = response_q.get()
# Networking: send to server
...
# Close down
...
EDIT: fixes small typos.
Your intuition that you need create_task is correct, as create_task is the closest async equivalent of Thread.start: it creates a task that runs in parallel (in an async sense) to whatever you are doing now.
You need separate coroutines that drain the respective queues running in parallel; something like this:
async def main():
processing_qs = [asyncio.Queue() for i in range(n_queues)]
response_q = asyncio.Queue()
async with websockets.connect(f"ws://{host}:{port}") as websocket:
processing_tasks = [
asyncio.create_task(processing(processing_q, response_q))
for processing_q in processing_qs
]
response_task = asyncio.create_task(
send_responses(websocket, response_q))
while True:
message = await websocket.recv()
requested_process, x, request_id = parse(message)
await processing_qs[requested_process].put((x, request_id))
async def processing(processing_q, response_q):
while True:
x, request_id = await processing_q.get()
... create response ...
await response_q.put(response)
async def send_responses(websocket, response_q):
while True:
msg = await response_q.get()
await websocket.send(msg)
I am trying to develop a Python 3.6 script which uses pika and threading modules.
I have a problem which I think is caused by my A) being very new to Python and coding in general, and B) my not understanding how to pass variables between functions when they are run in separate threads and already being passed a parameter in parentheses at the end of the receiving function name.
The reason I think this, is because when I do not use threading, I can pass a variable between functions simply by calling the receiving function name, and supplying the variable to be passed, in parentheses, a basic example is shown below:
def send_variable():
body = "this is a text string"
receive_variable(body)
def receive_variable(body):
print(body)
This when run, prints:
this is a text string
A working version of the code I need to to get working with threading is shown below - this uses straight functions (no threading) and I am using pika to receive messages from a (RabbitMQ) queue via the pika callback function, I then pass the body of the message received in the 'callback' function to the 'processing function' :
import pika
...mq connection variables set here...
# defines username and password credentials as variables set at the top of this script
credentials = pika.PlainCredentials(mq_user_name, mq_pass_word)
# defines mq server host, port and user credentials and creates a connection
connection = pika.BlockingConnection(pika.ConnectionParameters(host=mq_host, port=mq_port, credentials=credentials))
# creates a channel connection instance using the above settings
channel = connection.channel()
# defines the queue name to be used with the above channel connection instance
channel.queue_declare(queue=mq_queue)
def callback(ch, method, properties, body):
# passes (body) to processing function
body_processing(body)
# sets channel consume type, also sets queue name/message acknowledge settings based on variables set at top of script
channel.basic_consume(callback, queue=mq_queue, no_ack=mq_no_ack)
# tells the callback function to start consuming
channel.start_consuming()
# calls the callback function to start receiving messages from mq server
callback()
# above deals with pika connection and the main callback function
def body_processing(body):
...code to send a pika message every time a 'body' message is received...
This works fine however I want to translate this to run within a script that uses threading. When I do this I have to supply the parameter 'channel' to the function name that runs in its own thread - when I then try to include the 'body' parameter so that the 'processing_function' looks as per the below:
def processing_function(channel, body):
I get an error saying:
[function_name] is missing 1 positional argument: 'body'
I know that when using threading there is more code needed and I have included the actual code that I use for threading below so that you can see what I am doing:
...imports and mq variables and pika connection details are set here...
def get_heartbeats(channel):
channel.queue_declare(queue=queue1)
#print (' [*] Waiting for messages. To exit press CTRL+C')
def callback(ch, method, properties, body):
process_body(body)
#print (" Received %s" % (body))
channel.basic_consume(callback, queue=queue1, no_ack=no_ack)
channel.start_consuming()
def process_body(channel, body):
channel.queue_declare(queue=queue2)
#print (' [*] Waiting for Tick messages. To exit press CTRL+C')
# sets the mq host which pika client will use to send a message to
connection = pika.BlockingConnection(pika.ConnectionParameters(host=mq_host))
# create a channel connection instance
channel = connection.channel()
# declare a queue to be used by the channel connection instance
channel.queue_declare(queue=order_send_queue)
# send a message via the above channel connection settings
channel.basic_publish(exchange='', routing_key=send_queue, body='Test Message')
# send a message via the above channel settings
# close the channel connection instance
connection.close()
def manager():
# Channel 1 Connection Details - =======================================================================================
credentials = pika.PlainCredentials(mq_user_name, mq_password)
connection1 = pika.BlockingConnection(pika.ConnectionParameters(host=mq_host, credentials=credentials))
channel1 = connection1.channel()
# Channel 1 thread =====================================================================================================
t1 = threading.Thread(target=get_heartbeats, args=(channel1,))
t1.daemon = True
threads.append(t1)
# as this is thread 1 call to start threading is made at start threading section
# Channel 2 Connection Details - =======================================================================================
credentials = pika.PlainCredentials(mq_user_name, mq_password)
connection2 = pika.BlockingConnection(pika.ConnectionParameters(host=mq_host, credentials=credentials))
channel2 = connection2.channel()
# Channel 2 thread ====================================================================================================
t2 = threading.Thread(target=process_body, args=(channel2, body))
t2.daemon = True
threads.append(t2)
t2.start() # as this is thread 2 - we need to start the thread here
# Start threading
t1.start() # start the first thread - other threads will self start as they call t1.start() in their code block
for t in threads: # for all the threads defined
t.join() # join defined threads
manager() # run the manager module which starts threads that call each module
This when run produces the error
process_body() missing 1 required positional argument: (body)
and I do not understand why this is or how to fix it.
Thank you for taking the time to read this question and any help or advice you can supply is much appreciated.
Please keep in mind that I am new to python and coding so may need things spelled out rather than being able to understand more cryptic replies.
Thanks!
On further looking in to this and playing with the code it seems that if I edit the lines:
def process_body(channel, body):
to read
def process_body(body):
and
t2 = threading.Thread(target=process_body, args=(channel2, body))
so that it reads:
t2 = threading.Thread(target=process_body)
then the code seems to work as needed - I also see multiple script processes in htop so it appears that threading is working - I have left the script processing for 24 hours + and did not receive any errors...
I have an mqtt client app that subscribes to topics based on a configuration file. Something like:
def connectMQTT():
global Connection
Connection = Client()
Connection.on_message = handleQuery
for clientid in clientids.allIDs(): # clientids.allIDs() reads files to get this
topic = '{}/{}/Q/+'.format(Basename, clientid)
print('subscription:', topic)
Connection.subscribe(topic)
I have been using it with a simple invocation like:
def main():
connectMQTT()
Connection.loop_forever()
The loop_forever will block forever. But I'd like to notice when the information read by clientids.allIDs() is out of date and I should reconnect forcing it to subscribe afresh.
I can detect a change in the files with pyinotify:
def filesChanged():
# NOT SURE WHAT TO DO HERE
def watchForChanges():
watchManager = pyinotify.WatchManager()
notifier = pyinotify.ThreadedNotifier(watchManager, FileEventHandler(eventCallback))
notifier.start()
watchManager.add_watch('/etc/my/config/dir', pyinotify.IN_CLOSE_WRITE | pyinotify.IN_DELETE)
Basically, I need loop_forever (or some other paho mqtt mechanism) to run until some signal comes from the pyinotify machinery. I'm not sure how to weld those two together though. In pseudo code, I thing I want something like
def main():
signal = setup_directory_change_signal()
while True:
connectMQTT()
Connection.loop(until=signal)
Connection.disconnect()
I'm not sure how to effect that though.
I finally circled around to the following solution which seems to work. Whereas I was trying to run the notifier in another thread and the mqtt loop in the main thread, the trick seemed to be invert that setup:
def restartMQTT():
if Connection:
Connection.loop_stop()
connectMQTT()
Connection.loop_start()
class FileEventHandler(pyinotify.ProcessEvent):
def process_IN_CREATE(self, fileEvent):
restartMQTT()
def process_IN_DELETE(self, fileEvent):
restartMQTT()
def main():
restartMQTT()
watchManager = pyinotify.WatchManager()
notifier = pyinotify.Notifier(watchManager, FileEventHandler())
watchManager.add_watch('/etc/my/config_directory', pyinotify.IN_CREATE | pyinotify.IN_DELETE)
notifier.loop()
Where connectMQTT stores a newly connected and configured MQTT client in the Connection global.
I've just started learning python, but I have problem with my code:
import pifacecad
# listener initialization
cad = pifacecad.PiFaceCAD()
listener = pifacecad.SwitchEventListener(chip=cad)
listener.register(4, pifacecad.IODIR_ON, blowMyMind)
listener.activate()
def blowMyMind(event):
print('some prints...')
time.sleep(4)
print('and the end.')
blowMyMind() will be fired as many times as listener it tells to. That is okay.
My goal is to deactivate listener UNTIL blowMyMind ends. Pifacecad suggest Barrier() to achieve that, at least I think that it was here for that reason(correct me if I'm wrong).
Now it's working as many times as I activate listener event, but It's not like pushing function 99 times at once, but queues it and runs one by one.
With Barriers I think it should look like this:
# Barrier
global end_barrier
end_barrier = Barrier(1)
# listener initialization
listener = pifacecad.SwitchEventListener(chip=cad)
listener.register(4, pifacecad.IODIR_ON, blowMyMind)
listener.activate()
def blowMyMind(event):
global end_barrier
test = end_barrier.wait()
print(test) # returns 0, which should not in about 5 seconds
print('some prints...')
time.sleep(4)
print('and the end.')
The funny part is when I change parties in Barrier initialization it is causing BrokenBarrierError at first listener event.
Actually I think that I completely misunderstood Barrier() I think the problem with it is that all listener events are in one thread instead of their own threads.
It's making me even more confused when I'm reading:
parties The number of threads required to pass the barrier.
from here: https://docs.python.org/3/library/threading.html
My conclusion: when initializing Barrier(X) it would be realeased when there will be X('or less', 'or more'?) number of threads. That sounds VERY stupid :D
I tried to make it that way with no luck:
# listener initialization
global busy
busy = 0
cad = pifacecad.PiFaceCAD()
listener = pifacecad.SwitchEventListener(chip=cad)
listener.register(4, pifacecad.IODIR_ON, blowMyMind)
listener.activate()
def blowMyMind(event):
global busy
if busy == 0:
busy = 1
print('some prints...')
time.sleep(4)
print('and the end.')
busy = 0
else:
return None
This question is two-fold.
1. So I need to run code for a socket server that's all defined and created in another.py, Clicking run on PyCharm works just fine, but if you exec() the file it just runs the bottom part of the code.
There are a few answers here but they are conflicting and for Python 2.
From what I can gather there are three ways:
- Execfile(), Which I think is Python 2 code.
- os.system() (But I've seen it be said that it's not correct to pass to the OS for this)
- And subprocess.Popen (unsure how to use this either)
I need this to run in the background, it is used to create threads for sockets for the recv portion of the overall program and listen on those ports so I can input commands to a router.
This is the complete code in question:
import sys
import socket
import threading
import time
QUIT = False
class ClientThread(threading.Thread): # Class that implements the client threads in this server
def __init__(self, client_sock): # Initialize the object, save the socket that this thread will use.
threading.Thread.__init__(self)
self.client = client_sock
def run(self): # Thread's main loop. Once this function returns, the thread is finished and dies.
global QUIT # Need to declare QUIT as global, since the method can change it
done = False
cmd = self.readline() # Read data from the socket and process it
while not done:
if 'quit' == cmd:
self.writeline('Ok, bye. Server shut down')
QUIT = True
done = True
elif 'bye' == cmd:
self.writeline('Ok, bye. Thread closed')
done = True
else:
self.writeline(self.name)
cmd = self.readline()
self.client.close() # Make sure socket is closed when we're done with it
return
def readline(self): # Helper function, read up to 1024 chars from the socket, and returns them as a string
result = self.client.recv(1024)
if result is not None: # All letters in lower case and without and end of line markers
result = result.strip().lower().decode('ascii')
return result
def writeline(self, text): # Helper func, writes the given string to the socket with and end of line marker at end
self.client.send(text.strip().encode("ascii") + b'\n')
class Server: # Server class. Opens up a socket and listens for incoming connections.
def __init__(self): # Every time a new connection arrives, new thread object is created and
self.sock = None # defers the processing of the connection to it
self.thread_list = []
def run(self): # Server main loop: Creates the server (incoming) socket, listens > creates thread to handle it
all_good = False
try_count = 0 # Attempt to open the socket
while not all_good:
if 3 < try_count: # Tried more than 3 times without success, maybe post is in use by another program
sys.exit(1)
try:
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # Create the socket
port = 80
self.sock.bind(('127.0.0.1', port)) # Bind to the interface and port we want to listen on
self.sock.listen(5)
all_good = True
break
except socket.error:
print('Socket connection error... Waiting 10 seconds to retry.')
del self.sock
time.sleep(10)
try_count += 1
print('Server is listening for incoming connections.')
print('Try to connect through the command line with:')
print('telnet localhost 80')
print('and then type whatever you want.')
print()
print("typing 'bye' finishes the thread. but not the server",)
print("eg. you can quit telnet, run it again and get a different ",)
print("thread name")
print("typing 'quit' finishes the server")
try:
while not QUIT:
try:
self.sock.settimeout(0.500)
client = self.sock.accept()[0]
except socket.timeout:
time.sleep(1)
if QUIT:
print('Received quit command. Shutting down...')
break
continue
new_thread = ClientThread(client)
print('Incoming Connection. Started thread ',)
print(new_thread.getName())
self.thread_list.append(new_thread)
new_thread.start()
for thread in self.thread_list:
if not thread.isAlive():
self.thread_list.remove(thread)
thread.join()
except KeyboardInterrupt:
print('Ctrl+C pressed... Shutting Down')
except Exception as err:
print('Exception caught: %s\nClosing...' % err)
for thread in self.thread_list:
thread.join(1.0)
self.sock.close()
if "__main__" == __name__:
server = Server()
server.run()
print('Terminated')
Notes:
This is created in Python 3.4
I use Pycharm as my IDE.
One part of a whole.
2. So I'm creating a lightning detection system and this is how I expect it to be done:
- Listen to the port on the router forever
The above is done, but the issue with this is described in question 1.
- Pull numbers from a text file for sending text message
Completed this also.
- Send http get / post to port on the router
The issue with this is that i'm unsure how the router will act if I send this in binary form, I suspect it wont matter, the input commands for sending over GSM are specific. Some clarification may be needed at some point.
- Recieve reply from router and exception manage
- Listen for relay trip for alarm on severe or close strike warning.
- If tripped, send messages to phones in storage from text file
This would be the http get / post that's sent.
- Wait for reply from router to indicate messages have been sent, exception handle if it's not the case
- Go back to start
There are a few issues I'd like some background knowledge on that is proving hard to find via the old Google and here on the answers in stack.
How do I grab the receive data from the router from another process running in another file? I guess I can write into a text file and call that data but i'd rather not.
How to multi-process and which method to use.
How to send http get / post to socket on router, post needed occording to the router manual is as follows: e.g. "http://192.168.1.1/cgi-bin/sms_send?number=0037061212345&text=test"
Notes: Using Sockets, threading, sys and time on Python 3.4/Pycharm IDE.
Lightning detector used is LD-250 with RLO Relay attached.
RUT500 Teltonica router used.
Any direction/comments, errors spotted, anything i'm drastically missing would be greatly appreciated! Thank you very much in advance :D constructive criticism is greatly encouraged!
Okay so for the first part none of those suggested in the OP were my answer. Running the script as is from os.system(), exec() without declaring a new socket object just ran from __name__, this essentially just printed out "terminated", to get around this was simple. As everything was put into a classes already, all I had to do is create a new thread. This is how it was done:
import Socketthread2
new_thread = Socketthread2.Server() # Effectively declaring a new server class object.
new_thread.run()
This allowed the script to run from the beginning by initialising the code from the start in Socket, which is also a class of Clientthread, so that was also run too. Running this at the start of the parent program allowed this to run in the background, then continue with the new code in parent while the rest of the script was continuously active.