I am trying to implement multiple database triggers for my MongoDB (connected through Pymongo to my Python code).
I am able to successfully implement a database trigger but not able to extend this to multiple ones.
The code for the single database trigger can be found below:
try:
resume_token = None
pipeline = [{"$match": {"operationType": "insert"}}]
with db.collection.watch(pipeline) as stream:
for insert_change in stream:
print("postprocessing logic goes here")
except pymongo.errors.PyMongoError:
logging.error("Oops")
The problem is that once a single database is implemented, the post processing code waits to receive incoming requests for that collection and am able to include other collection watches in the same module.
Any help appreciated
Create separate functions for each watch and launch them in separate threads with something like:
import queue
import threading
main_q = queue.Queue()
thread1 = threading.Thread(target=function1, args=(main_q, None))
thread1.daemon = True
thread1.start()
thread2 = threading.Thread(target=function2, args=(main_q,))
thread2.daemon = True
thread2.start()
Related
I have an issue related to queue using a background task that never ends (continuously run to grab real-time data.
What I want to achieve:
Starting server via flask-socketio (eventlet),
monkey_patch(),
Using start_background_task, run a function from another file that grabs data in real time,
While this background task is running (indefinitely), storing incoming data in a queue via queue.put(),
Always while this task is running, from the main program watching for new data in the queue and processing them, meaning here socketio.emit().
What works: my program works well if, in the background task file, the while loop ends (while count < 100: for instance). In this case, I can access the queue from the main file and emit data.
What doesn't work: if this while loop is now while True:, the program blocks somewhere, I can't access the queue from the main program as it seems that it waits until the background task returns or stops.
So I guess I'm missing something here... so if you guys can help me with that, or give me some clues, that would be awesome.
Here some relevant parts of the code:
main.py
from threading import Thread
from threading import Lock
from queue import Queue
from get_raw_program import get_raw_data
from flask import Flask, send_from_directory, Response, jsonify, request, abort
from flask_socketio import SocketIO
import eventlet
eventlet.patcher.monkey_patch(select=True, socket=True)
app = Flask(__name__, static_folder=static_folder, static_url_path='')
app.config['SECRET_KEY'] = 'secret_key'
socketio = SocketIO(app, binary=True, async_mode="eventlet", logger=True, engineio_logger=True)
thread = None
thread_lock = Lock()
data_queue = Queue()
[...]
#socketio.on('WebSocket_On')
def grab_raw_data(test):
global thread
with thread_lock:
if thread is None:
socketio.emit('raw_data', {'msg': 'Thread is None:'})
socketio.emit('raw_data', {'msg': 'Starting Thread... '})
thread = socketio.start_background_task(target=get_raw_data(data_queue, test['mode']))
while True:
if not data_queue.empty():
data = data_queue.get(block=True, timeout=0.05)
socketio.emit('raw_data', {'msg': data})
socketio.sleep(0.0001)
get_raw_program.py (which works, can access queue from main.py)
def get_raw_data(data_queue, test):
count = 0
while count < 100:
data.put(b'\xe5\xce\x04\x00\xfe\xd2\x04\x00')
time.sleep(0.001)
count += 1
get_raw_program.py (which DOESN'T work, can't access queue from main.py)
def get_raw_data(data_queue, test):
count = 0
while True:
data.put(b'\xe5\xce\x04\x00\xfe\xd2\x04\x00')
time.sleep(0.001)
count += 1
I tried with regular Thread instead of start_background_task, and it works well. Thanks again for your help, greatly appreciated :-)
Below is the function called for scheduling a job on server start.
But somehow the scheduled job is getting called again and again, and this is causing too many calls to that respective function.
Either this is happening because of multiple function calls or something else? Suggestions please.
def redis_schedule():
with current_app.app_context():
redis_url = current_app.config["REDIS_URL"]
with Connection(redis.from_url(redis_url)):
q = Queue("notification")
from ..tasks.notification import send_notifs
task = q.enqueue_in(timedelta(minutes=5), send_notifs)
Refer - https://python-rq.org/docs/job_registries/
Needed to read scheduled_job_registry and retrieve jobids.
Currently below logic works for me as I only have a single scheduled_job.
But in case of multiple jobs, I will need to loop these jobids to find the right job exists or not.
def redis_schedule():
with current_app.app_context():
redis_url = current_app.config["REDIS_URL"]
with Connection(redis.from_url(redis_url)):
q = Queue("notification")
if len(q.scheduled_job_registry.get_job_ids()) == 0:
from ..tasks.notification import send_notifs
task = q.enqueue_in(timedelta(seconds=30), send_notifs)
I'm writing a Python program to interact with a device based on a CAN Bus. I'm using the python-can module successfully for this purpose. I'm also using asyncio to react to asynchronous events. I have written a "CanBusManager" class that is used by the "CanBusSequencer" class. The "CanBusManager" class takes care of generating/sending/receiving messages, and the CanBusSequencer drives the sequence of messages to be sent.
At some point in the sequence I want to wait until a specific message is received to "unlock" the remaining messages to be sent in the sequence. Overview in code:
main.py
async def main():
event = asyncio.Event()
sequencer = CanBusSequencer(event)
task = asyncio.create_task(sequencer.doSequence())
await task
asyncio.run(main(), debug=True)
canBusSequencer.py
from canBusManager import CanBusManager
class CanBusSequencer:
def __init__(self, event)
self.event = event
self.canManager = CanBusManager(event)
async def doSequence(self):
for index, row in self.df_sequence.iterrows():
if:...
self.canManager.sendMsg(...)
else:
self.canManager.sendMsg(...)
await self.event.wait()
self.event.clear()
canBusManager.py
import can
class CanBusManager():
def __init__(self, event):
self.event = event
self.startListening()
**EDIT**
def startListening(self):
self.msgNotifier = can.Notifier(self.canBus, self.receivedMsgCallback)
**EDIT**
def receivedMsgCallback(self, msg):
if(msg == ...):
self.event.set()
For now my program stays by the await self.event.wait(), even though the relevant message is received and the self.event.set() is executed. Running the program with debug = True reveals an
RuntimeError: Non-thread-safe operation invoked on an event loop other than the current one
that I don't really get. It has to do with the asyncio event loop, somehow not properly defined/managed. I'm coming from the C++ world and I'm currently writing my first large program with Python. Any guidance would be really appreciated:)
Your question doesn't explain how you arrange for receivedMsgCallback to be invoked.
If it is invoked by a classic "async" API which uses threads behind the scenes, then it will be invoked from outside the thread that runs the event loop. According to the documentation, asyncio primitives are not thread-safe, so invoking event.set() from another thread doesn't properly synchronize with the running event loop, which is why your program doesn't wake up when it should.
If you want to do anything asyncio-related, such as invoke Event.set, from outside the event loop thread, you need to use call_soon_threadsafe or equivalent. For example:
def receivedMsgCallback(self, msg):
if msg == ...:
self.loop.call_soon_threadsafe(self.event.set)
The event loop object should be made available to the CanBusManager object, perhaps by passing it to its constructor and assigning it to self.loop.
On a side note, if you are creating a task only to await it immediately, you don't need a task in the first place. In other words, you can replace task = asyncio.create_task(sequencer.doSequence()); await task with the simpler await sequencer.doSequence().
I need to get some prices for an external API, one by one for dozen of objects and it's like 2 or 3 seconds for each request so it can become pretty long.
I (kind of) knew how to do multithread in python, i implemented it and it works fine and it's pretty fast.
Then i've recently discovered asyncio and it seems it could be useful in my situation instead of opening several thread.
So i tried to "convert" my multithread code to a code using asyncio, as you can see below, after reading some examples.
But when testOne doesn't work and the error is Task exception was never retrieved.
I cleaned the code for better understanding (let me know if you need more informations).
from threading import Thread
import asyncio
### ASYNC MULTI THREAD ####
def prixMulti(client, symbol, prix):
prix[symbol] = # API price request using client
def testMulti(client, sql):
prix = {}
objects = # Database request using sql
listeThread = []
for object in objects:
listeThread.append(Thread(target=prixMulti, args=(client, object['name'], prix)))
for t in listeThread:
t.start()
for t in listeThread:
t.join()
print(prix)
#### ASYNC ONE THREAD ####
async def prixOne(client, symbol):
return #same API price request using client
async def prixOneWait(client, symbol, prix):
prix[symbol] = await prixOne(client, symbol)
def testOne(client, sql):
prix = {}
objects = # Database request using sql
tasks = []
loop = asyncio.get_event_loop()
for object in objects:
tasks.append(loop.create_task(prixOneWait(client, prix, object['nom'] )))
loop.run_until_complete(asyncio.wait(tasks))
loop.close()
print(prix)
# Some code to initialise client and sql
testMulti(client, sql)
testOne(client, sql)
I read messages from RabbitMQ with the pika python library. Reading messages in the loop is done by
connection = rpc.connect()
channel = connection.channel()
channel.basic_consume(rpc.consumeCallback, queue=FromQueue, no_ack=Ack)
channel.start_consuming()
This works fine.
But I also have the need to read one single message, which I do with:
method, properties, body = channel.basic_get(queue=FromQueue)
rpc.consumeCallback(Channel=channel,Method=method, Properties=properties,Body=body)
But when there is no message in the queue, the script is craching. How do I implement the get_empty() method described here ?
I solved it temporarily with a check on the response like:
method, properties, body = channel.basic_get(queue=FromQueue)
if(method == None):
## queue is empty
you can check empty in body like this:
def callback(ch, method, properties, body):
decodeBodyInfo = body.decode('utf-8')
if decodeBodyInfo != '':
cacheResult = decodeBodyInfo
ch.stop_consuming()
That so simple and easy to use :D
In case you're using the channel.consume generator in a for loop, you can set the inactivity_timeout parameter.
From the pika docs,
:param float inactivity_timeout: if a number is given (in seconds), will cause the
method to yield (None, None, None) after the given period of inactivity; this
permits for pseudo-regular maintenance activities to be carried out by the user
while waiting for messages to arrive. If None is given (default), then the method
blocks until the next event arrives. NOTE that timing granularity is limited by the
timer resolution of the underlying implementation.NEW in pika 0.10.0.
so changing your code to something like this might help
for method_frame, properties, body in channel.consume(queue, inactivity_timeout=120):
# break of the loop after 2 min of inactivity (no new item fetched)
if method_frame is None
break
Don't forget to properly handle the channel and the connection after exiting the loop