I've just started learning python, but I have problem with my code:
import pifacecad
# listener initialization
cad = pifacecad.PiFaceCAD()
listener = pifacecad.SwitchEventListener(chip=cad)
listener.register(4, pifacecad.IODIR_ON, blowMyMind)
listener.activate()
def blowMyMind(event):
print('some prints...')
time.sleep(4)
print('and the end.')
blowMyMind() will be fired as many times as listener it tells to. That is okay.
My goal is to deactivate listener UNTIL blowMyMind ends. Pifacecad suggest Barrier() to achieve that, at least I think that it was here for that reason(correct me if I'm wrong).
Now it's working as many times as I activate listener event, but It's not like pushing function 99 times at once, but queues it and runs one by one.
With Barriers I think it should look like this:
# Barrier
global end_barrier
end_barrier = Barrier(1)
# listener initialization
listener = pifacecad.SwitchEventListener(chip=cad)
listener.register(4, pifacecad.IODIR_ON, blowMyMind)
listener.activate()
def blowMyMind(event):
global end_barrier
test = end_barrier.wait()
print(test) # returns 0, which should not in about 5 seconds
print('some prints...')
time.sleep(4)
print('and the end.')
The funny part is when I change parties in Barrier initialization it is causing BrokenBarrierError at first listener event.
Actually I think that I completely misunderstood Barrier() I think the problem with it is that all listener events are in one thread instead of their own threads.
It's making me even more confused when I'm reading:
parties The number of threads required to pass the barrier.
from here: https://docs.python.org/3/library/threading.html
My conclusion: when initializing Barrier(X) it would be realeased when there will be X('or less', 'or more'?) number of threads. That sounds VERY stupid :D
I tried to make it that way with no luck:
# listener initialization
global busy
busy = 0
cad = pifacecad.PiFaceCAD()
listener = pifacecad.SwitchEventListener(chip=cad)
listener.register(4, pifacecad.IODIR_ON, blowMyMind)
listener.activate()
def blowMyMind(event):
global busy
if busy == 0:
busy = 1
print('some prints...')
time.sleep(4)
print('and the end.')
busy = 0
else:
return None
Related
I have a two-way datachannel setup that takes a heartbeat from a browser client and keeps the session alive as long as the heartbeat stays. The heartbeat is the 'main' communication for WebRTC, but I have other bits of into (Such as coordinates) I need to send constantly.
To do this when a webrtc offer is given, it takes that HTTP request:
Creates a new event loop 'rtcloop'
Set's that as the main event loop.
Then run 'rtcloop' until complete, calling my webRtcStart function and passing through the session info.
Then run a new thread with the target being 'rtcloop', run it forever and start.
Inside the new thread I set the loop with 'get_event_loop' and later define ' #webRtcPeer.on("datachannel")' so when we get a Datachannel message, we run code around that. Depending on the situation, I attempt to do the following:
ptzcoords = 'Supported' #PTZ Coords will be part of WebRTC Communication, send every 0.5 seconds.
ptzloop = asyncio.new_event_loop()
ptzloop.run_until_complete(updatePTZReadOut(webRtcPeer, cameraName, loop))
ptzUpdateThread = Thread(target=ptzloop.run_forever)
ptzUpdateThread.start()
The constant error I get no matter how I structure things is "coroutine 'updatePTZReadOut' was never awaited"
With updatePTZReadOut being:
async def updatePTZReadOut(rtcPeer, cameraName, eventLoop):
# Get Camera Info
# THE CURRENT ISSUE I am having is with the event loops, because this get's called to run in another thread, but it still needs
# to be awaitable,
# Current Warning Is: /usr/lib/python3.10/threading.py:953: RuntimeWarning: coroutine 'updatePTZReadOut' was never awaited
# Ref Article: https://xinhuang.github.io/posts/2017-07-31-common-mistakes-using-python3-asyncio.html
# https://lucumr.pocoo.org/2016/10/30/i-dont-understand-asyncio/
# Get current loop
# try:
loop = asyncio.set_event_loop(eventLoop)
# loop.run_until_complete()
# except RuntimeError:
# loop = asyncio.new_event_loop()
# asyncio.set_event_loop(loop)
# Getting Current COORDS from camera
myCursor.execute("Select * from localcameras where name = '{0}' ".format(cameraName))
camtuple = myCursor.fetchall()
camdata = camtuple[0]
# Create channel object
channel_local = rtcPeer.createDataChannel("chat")
while True:
ptzcoords = readPTZCoords(camdata[1], camdata[3], cryptocode.decrypt(str(camdata[4]), passwordRandomKey))
print("Updating Coords to {0}".format(ptzcoords))
# Publish Here
await channel_local.send("TTTT")
asyncio.sleep(0.5)
Any help here?
updatePTZReadOut is async function. You need to add await whenever you call this function.
All existing examples of threading.Condition and threading.Event present in internet are to solve producer consumer issue. But this can be done using either of them. Only advantage of Condition I have found is - it supports two types of notify function (notify and notifyAll)
Here I have easily replicated an existing example of Condition to Event. the Condition example is taken from here
Example using Condition:
def consumer(cv):
logging.debug('Consumer thread started ...')
with cv:
logging.debug('Consumer waiting ...')
cv.wait()
logging.debug('Consumer consumed the resource')
def producer(cv):
logging.debug('Producer thread started ...')
with cv:
logging.debug('Making resource available')
logging.debug('Notifying to all consumers')
cv.notifyAll()
if __name__ == '__main__':
condition = threading.Condition()
cs1 = threading.Thread(name='consumer1', target=consumer, args=(condition,))
cs2 = threading.Thread(name='consumer2', target=consumer, args=(condition,))
pd = threading.Thread(name='producer', target=producer, args=(condition,))
cs1.start()
time.sleep(2)
cs2.start()
time.sleep(2)
pd.start()
Example using Event:
def consumer(ev):
logging.debug('Consumer thread started ...')
logging.debug('Consumer waiting ...')
ev.wait()
logging.debug('Consumer consumed the resource')
def producer(ev):
logging.debug('Producer thread started ...')
logging.debug('Making resource available')
logging.debug('Notifying to all consumers')
ev.set()
if __name__ == '__main__':
event = threading.Event()
cs1 = threading.Thread(name='consumer1', target=consumer, args=(event,))
cs2 = threading.Thread(name='consumer2', target=consumer, args=(event,))
pd = threading.Thread(name='producer', target=producer, args=(event,))
cs1.start()
time.sleep(2)
cs2.start()
time.sleep(2)
pd.start()
Output:
(consumer1) Consumer thread started ...
(consumer1) Consumer waiting ...
(consumer2) Consumer thread started ...
(consumer2) Consumer waiting ...
(producer ) Producer thread started ...
(producer ) Making resource available
(producer ) Notifying to all consumers
(consumer1) Consumer consumed the resource
(consumer2) Consumer consumed the resource
Can someone give an example which is possible by using Condition but not by Event?
Event has state: You can call e.is_set() to find out whether e has been set() or clear()ed.
A Condition has no state. If some thread happens to call c.notify() or c.notify_all() when other threads are waiting in c.wait(), then one or more of those other threads will be awakened. But if the notification comes at a moment when no other thread is waiting, then c.notify() and c.notify_all() do absolutely nothing at all.
In short: An Event is a complete communication mechanism that can be use to transmit information from one thread to another. But a Condition cannot, by itself, be reliably used for communication. It has to be incorporated into some larger scheme.
IMO: Both classes are poorly named. A better name for Event would be Bit because it stores one bit of information.
A better name for Condition would be Event. Notifying a condition is an "event" it the true sense of the word. If you happen to be watching for it at the moment when it happens, then you learn something. But if you don't happen to be watching, then you miss it altogether.
I am writing a job scheduler where I schedule M jobs across N co-routines (N < M). As soon as one job finishes, I add a new job so that it can start immediately and run in parallel with the other jobs. Additionally, I would like to ensure that no single job takes more than a certain fixed amount of time. Any jobs that take too long should be cancelled. I have something pretty close, like this:
def update_run_set(waiting, running, max_concurrency):
number_to_add = min(len(waiting), max_concurrency - len(running))
for i in range(0, number_to_add):
next_one = waiting.pop()
running.add(next_one)
async def _run_test_invocations_asynchronously(jobs:List[MyJob], max_concurrency:int, timeout_seconds:int):
running = set() # These tasks are actively being run
waiting = set() # These tasks have not yet started
waiting = {_run_job_coroutine(job) for job in jobs}
update_run_set(waiting, running, max_concurrency)
while len(running) > 0:
done, running = await asyncio.wait(running, timeout=timeout_seconds,
return_when=asyncio.FIRST_COMPLETED)
if not done:
timeout_count = len(running)
[r.cancel() for r in running] # Start cancelling the timed out jobs
done, running = await asyncio.wait(running) # Wait for cancellation to finish
assert(len(done) == timeout_count)
assert(len(running) == 0)
else:
for d in done:
job_return_code = await d
if len(waiting) > 0:
update_run_set(waiting, running, max_concurrency)
assert(len(running) > 0)
The problem here is that say my timeout is 5 seconds, and I'm scheduling 3 jobs across 4 cores. Job A takes 2 seconds, Job B takes 6 seconds and job C takes 7 seconds.
We have something like this:
t=0 t=1 t=2 t=3 t=4 t=5 t=6 t=7
-------|-------|-------|-------|-------|-------|-------|-------|
AAAAAAAAAAAAAAA
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
However, at t=2 the asyncio.await() call returns because A completed. It then loops back up to the top and runs again. At this point B has already been running for 2 seconds, but since it starts the countdown over, and only has 4 seconds remaining until it completes, B will appear to be successful. So after 4 seconds we return again, B is successful, then we start the loop over and now C completes.
How do I make it so that B and C both fail? I somehow need the time to be preserved across calls to asyncio.wait().
One idea that I had is to do my own bookkeeping of how much time each job is allowed to continue running, and pass the minimum of these into asyncio.wait(). Then when something times out, I can cancel only those jobs whose time remaining was equal to the value I passed in for timeout_seconds.
This requires a lot of manual bookkeeping on my part though, and I can't help but wonder about floating point problems which cause me to decide that it's not time to cancel a job even though it really is). So I can't help but think that there's something easier. Would appreciate any ideas.
You can wrap each job into a coroutine that checks its timeout, e.g. using asyncio.wait_for. Limiting the number of parallel invocations could be done in the same coroutine using an asyncio.Semaphore. With those two combined, you only need one call to wait() or even just gather(). For example (untested):
# Run the job, limiting concurrency and time. This code could likely
# be part of _run_job_coroutine, omitted from the question.
async def _run_job_with_limits(job, sem, timeout):
async with sem:
try:
await asyncio.wait_for(_run_job_coroutine(job), timeout)
except asyncio.TimeoutError:
# timed out and canceled, decide what you want to return
pass
async def _run_test_invocations_async(jobs, max_concurrency, timeout):
sem = asyncio.Semaphore(max_concurrency)
return await asyncio.gather(
*(_run_job_with_limits(job, sem, timeout) for job in jobs)
)
poller = zmq.Poller()
poller.register(socket, zmq.POLLIN)
while(1)
try:
sockets = dict(poller.poll(timeout)) #How to detect timeout
# do job
except SomeTimeoutException:
break
As shown, is there a way to detect when timeout occurs while polling? Couldn't find a way on searching
poll returns tuples with the sockets registered as the first item and a flag denoting whether a frame was received or send. Thus you can just check for the flag, as shown in many pyzmq examples:
socket_dict = dict(poller.poll(timeout))
for polled_socket, flag in socket_dict.items():
if not (flag & zmq.POLLIN):
print("timeout")
If you just need to test one socket, this will work too:
if socket.poll(timeout) & zmq.POLLIN:
print("processing data...")
else:
print("timeout")
I would like to use the library threads (or perhaps parallel) for loading/preprocessing data into a queue but I am not entirely sure how it works. In summary;
Load data (tensors), pre-process tensors (this takes time, hence why I am here) and put them in a queue. I would like to have as many threads as possible doing this so that the model is not waiting or not waiting for long.
For the tensor at the top of the queue, extract it and forward it through the model and remove it from the queue.
I don't really understand the example in https://github.com/torch/threads enough. A hint or example as to where I would load data into the queue and train would be great.
EDIT 14/03/2016
In this example "https://github.com/torch/threads/blob/master/test/test-low-level.lua" using a low level thread, does anyone know how I can extract data from these threads into the main thread?
Look at this multi-threaded data provider:
https://github.com/soumith/dcgan.torch/blob/master/data/data.lua
It runs this file in the thread:
https://github.com/soumith/dcgan.torch/blob/master/data/data.lua#L18
by calling it here:
https://github.com/soumith/dcgan.torch/blob/master/data/data.lua#L30-L43
And afterwards, if you want to queue a job into the thread, you provide two functions:
https://github.com/soumith/dcgan.torch/blob/master/data/data.lua#L84
The first one runs inside the thread, and the second one runs in the main thread after the first one completes.
Hopefully that makes it a bit more clear.
If Soumith's examples in the previous answer are not very easy to use, I suggest you build your own pipeline from scratch. I provide here an example of two synchronized threads : one for writing data and one for reading data:
local t = require 'threads'
t.Threads.serialization('threads.sharedserialize')
local tds = require 'tds'
local dict = tds.Hash() -- only local variables work here, and only tables or tds.Hash()
dict[1] = torch.zeros(4)
local m1 = t.Mutex()
local m2 = t.Mutex()
local m1id = m1:id()
local m2id = m2:id()
m1:lock()
local pool = t.Threads(
1,
function(threadIdx)
end
)
pool:addjob(
function()
local t = require 'threads'
local m1 = t.Mutex(m1id)
local m2 = t.Mutex(m2id)
while true do
m2:lock()
dict[1] = torch.randn(4)
m1:unlock()
print ('W ===> ')
print(dict[1])
collectgarbage()
collectgarbage()
end
return __threadid
end,
function(id)
end
)
-- Code executing on master:
local a = 1
while true do
m1:lock()
a = dict[1]
m2:unlock()
print('R --> ')
print(a)
end