I am currently working with Simpy, and I want to model a parallel queueing system. There are 4 buffers, each storing products. A product can be taken out, but the buffer must then wait 15 minutes for the next product to be taken out. Each buffer has a capacity of 1, but products can be stored while these 15 minutes go by. I have tried solving this using env.timeout(15), but this times the entire system out for 15 minutes. How do I block only the buffer, not the full system?
Are you calling your timeout with a yield?
I did this by using a queue of queues (queueQueue) and then breaking the get process into three steps, pull the queue from queueQueue, then pull the obj from the queue. the queue will not get any more requests until it is returned to queueQueue. I wait until the obj get request is filled (can be a delay if the queue is empty and needs to wait for a obj to be added), then I make a async call (no yeild) to a process to wait 15 minuets and then put the queue back into queueQueue. Calling this process without a yield keeps the main process from getting blocked.
take a look and see if this works for you
"""
example where given a pool of queues, once a queue has a get request, block the queue from getting
furture get requests until 15 minutes after the request is filled. Note the queue could be empty
and therefore not fill the request immeditly. The queue can still get put requests
The two features is a queueQueue which has queue that are ready to process requests.
if a queue is not in the queueQueue it does not get any requests
Second, the process of putting a queue back in the queueQueue is wrapped in a process that can
be called async (no yield) so it does not block the main process
Programmer: Michael R. Gibbs
"""
import simpy
import random
class Obj():
"""
dummy object to queue
has unique IDs
"""
# id counter
id = 0
def __init__(self):
# assign unique id
Obj.id += 1
self.id = Obj.id
def getObj(env, queueQueue):
"""
gets a queue, then gets a obj
if the queue does not have a obj then will wait until a obj is added to the queue
once the obj request if filled by the queue the queue is sent to the putQueue process
async (no yield) which waits 15 mins before putting the queue in the queueQueue
"""
# wait for a queue to become available
queue = yield queueQueue.get()
print(f'{env.now} queue {queue.id} has been pulled')
# wait for obj from queue
obj = yield queue.get()
print(f'{env.now} obj {obj.id} has been pulled from queue {queue.id}')
# do not block while watting 15 min (do not use yield here)
env.process(putQueue( env, queueQueue, queue))
return obj
def putQueue(env, queueQueue, queue):
"""
waits 15 minutes and puts a queuue in a queue of queues
"""
yield env.timeout(15)
yield queueQueue.put(queue)
print(f'{env.now} queue {queue.id} has been queued')
def genObj(env, queueList):
"""
load the queues with objects.
uses round robin to select queue to add to
"""
while True:
# create a obj for the queue and give it a tracking id
obj = Obj()
# get the next queue and add obj (queues are filled round robin)
q = obj.id % 4
yield queueList[q].put(obj)
print(f'{env.now} added obj {obj.id} to queue {queueList[q].id}')
# queue fill rate
yield env.timeout(random.triangular(10,20,15))
def consume(env, id, queueQueue):
"""
consumer that make queue get requests from the pool of queues
"""
while True:
# comsuption rate
yield env.timeout(random.triangular(10,20,14))
# wraps getting a queue, get a obj from the queue, and queueing the queue
obj = yield env.process(getObj(env,queueQueue))
print(f'{env.now} consumer: {id} pulled obj {obj.id}')
env = simpy.Environment()
# the queue of queues
queueQueue = simpy.Store(env, capacity=4)
# the pool of queues with queued obj
queueList = []
for i in range(4):
queue = simpy.Store(env)
queue.id = i+1
queueList.append(queue)
queueQueue.put(queue)
print('queue',queue.id)
# start queuing obj (update code to change queue rate)
env.process(genObj(env,queueList))
# start consumming obj (can add more of these)
env.process(consume(env, 1, queueQueue))
env.run(200)
print('done')
Related
I have a rabbitMQ server, which has two queues [hello, other].
If I push anything to one of the queues it will be consumed as desired.
The problem occurs when the first queue does not exist (lets say [fail, other]). The second queue is now not consumed even though it exists. It does not throw any errors either, the procces keeps running, but when pushing new items to the queue it wont show up in the consumer.
I have tried queue_declare with passive=True, but this just freezes everything (if the queue does not exists)
import pika
class QueueConsumer:
def __init__(
self,
host: str,
queues: list,
callback,
):
self.host = host
self.queues = queues
self.callback = callback
def on_open(self, connection):
connection.channel(on_open_callback=self.on_channel_open)
def on_channel_open(self, channel):
for queue in self.queues:
channel.basic_consume(queue=queue, on_message_callback=self.callback, auto_ack=True)
def start_listening(self):
connection_params = pika.ConnectionParameters(
host=self.host,
port=5672,
)
connection = pika.SelectConnection(parameters=connection_params,
on_open_callback=self.on_open)
connection.ioloop.start()
def callback(channel, method, properties, body):
print(body)
consumer = QueueConsumer('test','test',callback=callback,queues=['hello','other'], host="localhost")
consumer.start_listening()
I would like to check if the queue exists. If not ignore it, otherwise consume it
Two coroutintes in code below, running in different threads, cannot communicate with each other by asyncio.Queue. After the producer inserts a new item in asyncio.Queue, the consumer cannot get this item from that asyncio.Queue, it gets blocked in method await self.n_queue.get().
I try to print the ids of asyncio.Queue in both consumer and producer, and I find that they are same.
import asyncio
import threading
import time
class Consumer:
def __init__(self):
self.n_queue = None
self._event = None
def run(self, loop):
loop.run_until_complete(asyncio.run(self.main()))
async def consume(self):
while True:
print("id of n_queue in consumer:", id(self.n_queue))
data = await self.n_queue.get()
print("get data ", data)
self.n_queue.task_done()
async def main(self):
loop = asyncio.get_running_loop()
self.n_queue = asyncio.Queue(loop=loop)
task = asyncio.create_task(self.consume())
await asyncio.gather(task)
async def produce(self):
print("id of queue in producer ", id(self.n_queue))
await self.n_queue.put("This is a notification from server")
class Producer:
def __init__(self, consumer, loop):
self._consumer = consumer
self._loop = loop
def start(self):
while True:
time.sleep(2)
self._loop.run_until_complete(self._consumer.produce())
if __name__ == '__main__':
loop = asyncio.get_event_loop()
print(id(loop))
consumer = Consumer()
threading.Thread(target=consumer.run, args=(loop,)).start()
producer = Producer(consumer, loop)
producer.start()
id of n_queue in consumer: 2255377743176
id of queue in producer 2255377743176
id of queue in producer 2255377743176
id of queue in producer 2255377743176
I try to debug step by step in asyncio.Queue, and I find after the method self._getters.append(getter) is invoked in asyncio.Queue, the item is inserted in queue self._getters. The following snippets are all from asyncio.Queue.
async def get(self):
"""Remove and return an item from the queue.
If queue is empty, wait until an item is available.
"""
while self.empty():
getter = self._loop.create_future()
self._getters.append(getter)
try:
await getter
except:
# ...
raise
return self.get_nowait()
When a new item is inserted into asycio.Queue in producer, the methods below would be invoked. The variable self._getters has no items although it has same id in methods put() and set().
def put_nowait(self, item):
"""Put an item into the queue without blocking.
If no free slot is immediately available, raise QueueFull.
"""
if self.full():
raise QueueFull
self._put(item)
self._unfinished_tasks += 1
self._finished.clear()
self._wakeup_next(self._getters)
def _wakeup_next(self, waiters):
# Wake up the next waiter (if any) that isn't cancelled.
while waiters:
waiter = waiters.popleft()
if not waiter.done():
waiter.set_result(None)
break
Does anyone know what's wrong with the demo code above? If the two coroutines are running in different threads, how could they communicate with each other by asyncio.Queue?
Short answer: no!
Because the asyncio.Queue needs to share the same event loop, but
An event loop runs in a thread (typically the main thread) and executes all callbacks and Tasks in its thread. While a Task is running in the event loop, no other Tasks can run in the same thread. When a Task executes an await expression, the running Task gets suspended, and the event loop executes the next Task.
see
https://docs.python.org/3/library/asyncio-dev.html#asyncio-multithreading
Even though you can pass the event loop to threads, it might be dangerous to mix the different concurrency concepts. Still note, that passing the loop just means that you can add tasks to the loop from different threads, but they will still be executed in the main thread. However, adding tasks from threads can lead to race conditions in the event loop, because
Almost all asyncio objects are not thread safe, which is typically not a problem unless there is code that works with them from outside of a Task or a callback. If there’s a need for such code to call a low-level asyncio API, the loop.call_soon_threadsafe() method should be used
see
https://docs.python.org/3/library/asyncio-dev.html#asyncio-multithreading
Typically, you should not need to run async functions in different threads, because they should be IO bound and therefore a single thread should be sufficient to handle the work load. If you still have some CPU bound tasks, you are able to dispatch them to different threads and make the result awaitable using asyncio.to_thread, see https://docs.python.org/3/library/asyncio-task.html#running-in-threads.
There are many questions already about this topic, see e.g. Send asyncio tasks to loop running in other thread or How to combine python asyncio with threads?
If you want to learn more about the concurrency concepts, I recommend to read https://medium.com/analytics-vidhya/asyncio-threading-and-multiprocessing-in-python-4f5ff6ca75e8
Well I am looking into python documentation for study for my work. I am new to python and also programming, I also do not understand concepts of programming like async operations very well.
I usign Fedora 29 with Python 3.7.3 for try examples of queue and the lib asyncio.
Follow the example of queue and async operations below:
import asyncio
import random
import time
async def worker(name, queue):
while True:
# Get a "work item" out of the queue.
sleep_for = await queue.get()
# Sleep for the "sleep_for" seconds.
await asyncio.sleep(sleep_for)
# Notify the queue that the "work item" has been processed.
queue.task_done()
print(f'{name} has slept for {sleep_for:.2f} seconds')
async def main():
# Create a queue that we will use to store our "workload".
queue = asyncio.Queue()
# Generate random timings and put them into the queue.
total_sleep_time = 0
for _ in range(20):
sleep_for = random.uniform(0.05, 1.0)
total_sleep_time += sleep_for
queue.put_nowait(sleep_for)
# Create three worker tasks to process the queue concurrently.
tasks = []
for i in range(3):
task = asyncio.create_task(worker(f'worker-{i}', queue))
tasks.append(task)
# Wait until the queue is fully processed.
started_at = time.monotonic()
await queue.join()
total_slept_for = time.monotonic() - started_at
# Cancel our worker tasks.
for task in tasks:
task.cancel()
# Wait until all worker tasks are cancelled.
await asyncio.gather(*tasks, return_exceptions=True)
print('====')
print(f'3 workers slept in parallel for {total_slept_for:.2f} seconds')
print(f'total expected sleep time: {total_sleep_time:.2f} seconds')
asyncio.run(main())
Why in this example I need cancel the tasks? Why I can exclude this part of code
# Cancel our worker tasks.
for task in tasks:
task.cancel()
# Wait until all worker tasks are cancelled.
await asyncio.gather(*tasks, return_exceptions=True)
and the example work fine?
Why in this example i need cancel the tasks?
Because they will otherwise remain hanging indefinitely, waiting for a new item in the queue that will never arrive. In that particular example you are exiting the event loop anyway, so there's no harm from them "hanging", but if you did that as part of a utility function, you would create a coroutine leak.
In other words, canceling the workers tells them to exit because their services are no longer necessary, and is needed to ensure that resources associated with them get freed.
I have this script
import asyncio
import random
q = asyncio.Queue()
async def producer(num):
while True:
await q.put(num + random.random())
await asyncio.sleep(random.random())
async def consumer(num):
while True:
value = await q.get()
print('Consumed', num, value)
loop = asyncio.get_event_loop()
for i in range(6):
loop.create_task(producer(i))
for i in range(3):
loop.create_task(consumer(i))
loop.run_forever()
that uses asyncio.Queue()
I am running my script forever and produces tasks randomly and adding them to the queue. In the event that there are no tasks to consume in the queue, will this be using cpu for nothing or is it harmless and produces no error?
In the event that there are no tasks to consume in the queue, will this produce an error?
Since the consumer calls get() to consume the next queued item, if the queue is empty, it will simply wait for the next item to arrive. No CPU will be wasted on the wait, and no error reported - the consumer coroutine will just be suspended until an item is produced, after which it will be immediately woken up. During the wait the event loop is free to run other coroutines, if any.
I have one producer and 3 consumers. Each consumer waits to acquire a global lock before it can proceed. The program runs and doesnt finish and come out of the while loop. Could you tell me where it is going wrong?
import asyncio
import random
async def produce(queue, n):
for x in range(1, n + 1):
# produce an item
print('producing {}/{}'.format(x, n))
# simulate i/o operation using sleep
await asyncio.sleep(random.random())
item = str(x)
# put the item in the queue
await queue.put(item)
# indicate the producer is done
await queue.put(None)
async def consume(queue, lock):
while True:
item = await queue.get()
if item is None:
# the producer emits None to indicate that it is done
break
# wait for an item from the producer
async with lock:
# process the item
print('consuming item {}...'.format(item))
# simulate i/o operation using sleep
await asyncio.sleep(0.3)
loop = asyncio.get_event_loop()
lock = asyncio.Lock()
queue = asyncio.Queue(loop=loop)
producer_coro = produce(queue, 10)
consumers = []
for _ in range(3):
consumers.append(consume(queue, lock))
all_coroutines = []
all_coroutines.append(producer_coro)
all_coroutines.extend(consumers)
loop.run_until_complete(asyncio.wait(all_coroutines))
loop.close()
The problem is in the consumer:
if item is None:
# the producer emits None to indicate that it is done
break
The None sentinel is only picked up by a single consumer, and the rest are left waiting for the next value to arrive through the queue. A simple fix is to return the sentinel value back to the queue:
if item is None:
# The producer emits None to indicate that it is done.
# Propagate it to other consumers and quit.
await queue.put(None)
break
Alternatively, produce could enqueue as many None sentinels as there are consumers - but that would require the producer to know how many consumers there are, which is not always desirable.
Adding to the answer #user4815162342 provided, try:
if item is None and queue.qsize() == 0:
await queue.put(None)
break
I had an issue where the consumer also had to queue.put() to the same queue to rerun the function but it hung at the end without the both conditions.