All existing examples of threading.Condition and threading.Event present in internet are to solve producer consumer issue. But this can be done using either of them. Only advantage of Condition I have found is - it supports two types of notify function (notify and notifyAll)
Here I have easily replicated an existing example of Condition to Event. the Condition example is taken from here
Example using Condition:
def consumer(cv):
logging.debug('Consumer thread started ...')
with cv:
logging.debug('Consumer waiting ...')
cv.wait()
logging.debug('Consumer consumed the resource')
def producer(cv):
logging.debug('Producer thread started ...')
with cv:
logging.debug('Making resource available')
logging.debug('Notifying to all consumers')
cv.notifyAll()
if __name__ == '__main__':
condition = threading.Condition()
cs1 = threading.Thread(name='consumer1', target=consumer, args=(condition,))
cs2 = threading.Thread(name='consumer2', target=consumer, args=(condition,))
pd = threading.Thread(name='producer', target=producer, args=(condition,))
cs1.start()
time.sleep(2)
cs2.start()
time.sleep(2)
pd.start()
Example using Event:
def consumer(ev):
logging.debug('Consumer thread started ...')
logging.debug('Consumer waiting ...')
ev.wait()
logging.debug('Consumer consumed the resource')
def producer(ev):
logging.debug('Producer thread started ...')
logging.debug('Making resource available')
logging.debug('Notifying to all consumers')
ev.set()
if __name__ == '__main__':
event = threading.Event()
cs1 = threading.Thread(name='consumer1', target=consumer, args=(event,))
cs2 = threading.Thread(name='consumer2', target=consumer, args=(event,))
pd = threading.Thread(name='producer', target=producer, args=(event,))
cs1.start()
time.sleep(2)
cs2.start()
time.sleep(2)
pd.start()
Output:
(consumer1) Consumer thread started ...
(consumer1) Consumer waiting ...
(consumer2) Consumer thread started ...
(consumer2) Consumer waiting ...
(producer ) Producer thread started ...
(producer ) Making resource available
(producer ) Notifying to all consumers
(consumer1) Consumer consumed the resource
(consumer2) Consumer consumed the resource
Can someone give an example which is possible by using Condition but not by Event?
Event has state: You can call e.is_set() to find out whether e has been set() or clear()ed.
A Condition has no state. If some thread happens to call c.notify() or c.notify_all() when other threads are waiting in c.wait(), then one or more of those other threads will be awakened. But if the notification comes at a moment when no other thread is waiting, then c.notify() and c.notify_all() do absolutely nothing at all.
In short: An Event is a complete communication mechanism that can be use to transmit information from one thread to another. But a Condition cannot, by itself, be reliably used for communication. It has to be incorporated into some larger scheme.
IMO: Both classes are poorly named. A better name for Event would be Bit because it stores one bit of information.
A better name for Condition would be Event. Notifying a condition is an "event" it the true sense of the word. If you happen to be watching for it at the moment when it happens, then you learn something. But if you don't happen to be watching, then you miss it altogether.
Related
I have multiple producer and one consumer processes. Each process is launched by MPIPoolExecutor class. First I launch the consumer process and then starts launching producer processes using starmap method. Consumer accepts receives data and save it to hard drive. Each producer process will create a buffer with the same size as the data needs to be sent and send it using the blocker method bsend. I am expecting each producer process to dump the data into the buffer and exit. However, I am noticing a delay where it looks like each producer process waits for the data to be consumed by the consumer process. What am I missing? My code goes like this:
def consumer(args...):
comm = MPI.COMM_WORLD
file = tb.open_file(file_name, 'w')
filters = tb.Filters(complevel=5, complib='blosc')
array = file.create_carray(file.root, 'data', tb.Float32Atom(), shape=(n_, n_), filters=filters)
for i in range(num_tasks):
t = time.time()
idxs, data = MPI.Comm.recv()
print("time for waiting --consumer ", time.time() - t)
array[idxs,:] = data
def producer(args...):
comm = MPI.COMM_WORLD
#adding 1000 just to be in the safe side.
mem = MPI.Alloc_mem(data.nbytes + idxs.nbytes + 1000)
MPI.Attach_buffer(mem)
#Since consumer is launched first, it guarantees to get a rank of 1.
MPI.Comm.bsend([idxs, data], 1)
MPI.Detach_buffer()
....
with MPIPoolExecutor() as executor:
executor.starmap(consumer, [(args)])
executor.starmap(produces, list_of_args)
If the consumer is launched first, it gets rank zero, not 1. Also: you're misunderstanding buffered communication. If you want the producer to return, use MPI_Isend. The buffer detach call blocks until all messages in it have been completed.
I have code like the one below :
def expensive(self,c,v):
.....
def inner_loop(self,c,collector):
self.db.query('SELECT ...',(c,))
for v in self.db.cursor.fetchall() :
collector.append( self.expensive(c,v) )
def method(self):
# create a Pool
#join the Pool ??
self.db.query('SELECT ...')
for c in self.db.cursor.fetchall() :
collector = []
#RUN the whole cycle in parallel in separate processes
self.inner_loop(c, collector)
#do stuff with the collector
#! close the pool ?
both the Outer and the Inner loop are thousands of steps ...
I think I understand how to run a Pool of couple of processes.
All the examples I found show that more or less.
But in my case I need to lunch a persistent Pool and then feed the data (c-value). Once a inner-loop process has finished I have to supply the next-available-c-value.
And keep the processes running and collect the results.
How do I do that ?
A clunky idea I have is :
def method(self):
ws = 4
with Pool(processes=ws) as pool :
cs = []
for i,c in enumerate(..) :
cs.append(c)
if i % ws == 0 :
res = [pool.apply(self.inner_loop, (c)) for i in range(ws)]
cs = []
collector.append(res)
will this keep the same pool running !! i.e. not lunch new process every time ?i
Do I need 'if i % ws == 0' part or I can use imap(), map_async() and the Pool obj will block the loop when available workers are exhausted and continue when some are freed ?
Yes, the way that multiprocessing.Pool works is:
Worker processes within a Pool typically live for the complete duration of the Pool’s work queue.
So simply submitting all your work to the pool via imap should be sufficient:
with Pool(processes=4) as pool:
initial_results = db.fetchall("SELECT c FROM outer")
results = [pool.imap(self.inner_loop, (c,)) for c in initial_results]
That said, if you really are doing this to fetch things from the DB, it may make more sense to move more processing down into that layer (bring the computation to the data rather than bringing the data to the computation).
I'm working with an API that can only access its objects on the main thread, so I need to create a new thread to be used for my GUI and then swap back to the original thread for any lengthy calculations involving the API.
So far I have the following code:
[<EntryPoint; STAThread>]
let main _ =
Debug.WriteLine($"[{Thread.CurrentThread.ManagedThreadId}] - Inital thread")
let initCtx = SynchronizationContext.Current
let uiThread = new Thread(fun () ->
let guiCtx = SynchronizationContext.Current
Debug.WriteLine($"[{Thread.CurrentThread.ManagedThreadId}] - New UI thread")
async{
do! Async.SwitchToContext initCtx
Debug.WriteLine($"[{Thread.CurrentThread.ManagedThreadId}] - Back to initial thread")
// Lengthy API calculation here
do! Async.SwitchToContext guiCtx
Debug.WriteLine($"[{Thread.CurrentThread.ManagedThreadId}] - Back to UI thread")
} |> Async.RunSynchronously
)
uiThread.SetApartmentState(ApartmentState.STA)
uiThread.Start()
1
However when I run this I get the output:
[1] - Inital thread
[4] - New UI thread
[5] - Back to initial thread
[5] - Back to UI thread
So it doesn't seem to be switching contexts the way I would expect. How can I switch back to the original thread after creating a new thread this way?
I have tried calling
SynchronizationContext.SetSynchronizationContext(new DispatcherSynchronizationContext(Dispatcher.CurrentDispatcher)) first to ensure that the original thread has a valid SynchronizationContext but that causes the program to exit at the Async.SwitchToContext lines without throwing any exception.
I have also tried using Async.StartImmediate instead of RunSynchronously with the same result.
If I try both of these at the same time then the program just freezes up at the Async.SwitchToContext lines instead of exiting out.
I'm trying to simulate processing in threads by using asyncio.Queue. However, I'm struggling to turn a threaded processing simulation part to asynchronous loop.
So what my script does in brief: 1) receive processing requests over a websocket, 2) assign the request to the requested queue (which simulates a thread), 3) runs processing queues, which put responses into one shared response queue, and then 4) the websocket takes out the responses from the shared queue one by one and sends them out to the server.
Simplified version of my code:
# Initialize empty processing queues for the number of threads I want to simulate
processing_queues = [asyncio.Queue() for i in range(n_queues)
# Initialize shared response queue
response_q = asyncio.Queue()
# Set up a websocket context manager
async with websockets.connect(f"ws://{host}:{port}") as websocket:
while True:
# Read incoming requests
message = await websocket.recv()
# Parse mssg -> get request data and on which thread / queue to process it
request_data, queue_no = parse_message(message)
# Put the request data to the requested queue (imitating thread)
await processing_queues[queue_no].put(request_data)
# THIS IS WHERE I THINK ASYNCHRONY BREAKS (AND I NEED HELP)
# Do processing in each imitated processing thread
for proc_q in processing_queues:
if not proc_q.empty():
request_data = await proc_q.get()
# do the processing
response = process_data(request_data)
# Add the response to the response queue
await response_q.put(response)
# Send responses back to the server
if not response_q.empty():
response_data = response_q.get()
await websocket.send(response_data)
From the output of the script, I deduced that 1) I seem to receive requests and send out responses asynchronously; 2) processing in queues does not happen asynchronously. Correct me if I'm wrong.
I was reading about create_task() in asyncio. Maybe that could be a way to solve my problem?
I'm open to any solution (even hacky).
P.S. I would just use threads from threading library, but I need asyncio for websockets library.
P.P.S. Threaded version of my idea.
class ProcessingImitationThread(threading.Thread):
def __init__(self, thread_id, request_q, response_q):
threading.Thread.__init__(self)
self.thread_id = thread_id
self.request_q = request_q
self.response_q = response_q
def run(self):
while True:
try:
(x, request_id) = self.request_q.get()
except Empty:
time.sleep(0.2)
else:
if x == -1:
# EXIT CONDITION
break
else:
sleep_time_for_x = count_imitation(x, state)
time.sleep(sleep_time_for_x)
self.response_q.put(request_id)
print(f"request {request_id} executed")
# Set up
processing_qs = [queue.Queue() for i in range(n_processes_simulated)]
response_q = queue.Queue()
processing_thread_handlers = []
for i in n_processes_simulated:
# create thread
t = ProcessingImitationThread(i, processing_qs[i], response_q)
processing_thread_handlers.append(t)
# Main loop
while True:
# receive requests and assign to requested queue (so that thread picks up)
if new_request:
requested_process, x, request_id = parse(new_request)
processing_qs[requested_process].put((x, request_id))
...
# if there are any new responses, sent them out to the server
if response_q.q_size() > 0:
request_id = response_q.get()
# Networking: send to server
...
# Close down
...
EDIT: fixes small typos.
Your intuition that you need create_task is correct, as create_task is the closest async equivalent of Thread.start: it creates a task that runs in parallel (in an async sense) to whatever you are doing now.
You need separate coroutines that drain the respective queues running in parallel; something like this:
async def main():
processing_qs = [asyncio.Queue() for i in range(n_queues)]
response_q = asyncio.Queue()
async with websockets.connect(f"ws://{host}:{port}") as websocket:
processing_tasks = [
asyncio.create_task(processing(processing_q, response_q))
for processing_q in processing_qs
]
response_task = asyncio.create_task(
send_responses(websocket, response_q))
while True:
message = await websocket.recv()
requested_process, x, request_id = parse(message)
await processing_qs[requested_process].put((x, request_id))
async def processing(processing_q, response_q):
while True:
x, request_id = await processing_q.get()
... create response ...
await response_q.put(response)
async def send_responses(websocket, response_q):
while True:
msg = await response_q.get()
await websocket.send(msg)
I've just started learning python, but I have problem with my code:
import pifacecad
# listener initialization
cad = pifacecad.PiFaceCAD()
listener = pifacecad.SwitchEventListener(chip=cad)
listener.register(4, pifacecad.IODIR_ON, blowMyMind)
listener.activate()
def blowMyMind(event):
print('some prints...')
time.sleep(4)
print('and the end.')
blowMyMind() will be fired as many times as listener it tells to. That is okay.
My goal is to deactivate listener UNTIL blowMyMind ends. Pifacecad suggest Barrier() to achieve that, at least I think that it was here for that reason(correct me if I'm wrong).
Now it's working as many times as I activate listener event, but It's not like pushing function 99 times at once, but queues it and runs one by one.
With Barriers I think it should look like this:
# Barrier
global end_barrier
end_barrier = Barrier(1)
# listener initialization
listener = pifacecad.SwitchEventListener(chip=cad)
listener.register(4, pifacecad.IODIR_ON, blowMyMind)
listener.activate()
def blowMyMind(event):
global end_barrier
test = end_barrier.wait()
print(test) # returns 0, which should not in about 5 seconds
print('some prints...')
time.sleep(4)
print('and the end.')
The funny part is when I change parties in Barrier initialization it is causing BrokenBarrierError at first listener event.
Actually I think that I completely misunderstood Barrier() I think the problem with it is that all listener events are in one thread instead of their own threads.
It's making me even more confused when I'm reading:
parties The number of threads required to pass the barrier.
from here: https://docs.python.org/3/library/threading.html
My conclusion: when initializing Barrier(X) it would be realeased when there will be X('or less', 'or more'?) number of threads. That sounds VERY stupid :D
I tried to make it that way with no luck:
# listener initialization
global busy
busy = 0
cad = pifacecad.PiFaceCAD()
listener = pifacecad.SwitchEventListener(chip=cad)
listener.register(4, pifacecad.IODIR_ON, blowMyMind)
listener.activate()
def blowMyMind(event):
global busy
if busy == 0:
busy = 1
print('some prints...')
time.sleep(4)
print('and the end.')
busy = 0
else:
return None