Say I have a Process which has two optional View tasks to the finish:
--> Split --> optional_view_1 ---> Join ----> finish
| ^
| |
-----> optional_view_2 -------
Let's say both tasks are assigned, but a human logs in and completes optional_view_1. The Join itself and indeed the End node both end up with status DONE. However, the Process will get stuck and NOT exit cleanly due to optional_view_2.
The questions are:
how to finish the process cleanly (i.e. not cancel it) and without races
from where (e.g. from inside the Join or a Handler after each View to cancel the other one?)
I originally posted this as an issue, along with a possible coded solution. In summary the code:
Overrides the Activation "done" associated with the Join
The override checks if the Join has wait_all=False
And if so, tries to find any other tasks the Join is waiting for
And cancels them
The code for step 3 is the bit I am most worried about. It is a modified version of the Viewflow code handling the wait_all=True case and looks like this in summary:
previous = self.task.previous.exclude(status=STATUS.CANCELED).all()
join_prefixes = set(prev.token.get_common_split_prefix(self.task.token, prev.pk)
for prev in previous)
#
# Don't forget to exclude the previous tasks.
#
join_token_prefix = next(iter(join_prefixes))
active = self.flow_class.task_class._default_manager \
.filter(process=self.process, token__startswith=join_token_prefix) \
.exclude(status__in=[STATUS.DONE, STATUS.CANCELED], pk__in=[prev.pk for prev in previous])
for task in active:
cancel_task(task)
Any comments on whether this is the right approach or not would be welcome.
That BPMN process would not be complete, it's the right behavior. The join outgoing path would be executed twice.
To have an optional branch, you need to implement event-based gateway. That would subscribe to flow task end signal (override ready method) and cancel nonfinished tasks.
https://www.youtube.com/watch?v=pyJM_V8Ji2w
Related
I am currently building a project that requires multiple requests made to various endpoints. I am wrapping these requests in Aiohttp to allow for async.
The problem:
I have three Queues: queue1, queue2, and queue3. Additionally, I have three worker functions (worker1, worker2, worker3) that are associated with their respective Queue. The first queue is populated immediately with a list IDs that is known prior to running. When the request is finished and the data is committed to a database, it passes the ID to queue2. A worker2 will take this ID and request more data. From this data it will begin to generate a list of IDs (different from the IDs in queue1/queue2. worker2 will put the IDs in queue3. Finally worker3 will grab this ID from queue3 and request more data before committing to a database.
The issue arises with the fact queue.join() is a blocking call. Each worker is tied to a separate Queue so the join for queue1 will block until its finished. This is fine, but it also defeats the purpose of using async. Without using join() the program is unable to detect when the Queues are totally empty. The other issue is that there may be silent errors when one of the Queues is empty but there is still data that hasn't been added yet.
The basic code outline is as follows:
queue1 = asyncio.Queue()
queue2 = asyncio.Queue()
queue3 = asyncio.Queue()
async with aiohttp.ClientSession() as session:
for i in range(3):
tasks.append(asyncio.create_task(worker1(queue1)))
for i in range(3):
tasks.append(asyncio.create_task(worker2(queue2)))
for i in range(10):
tasks.append(asyncio.create_task(worker3(queue3)))
for i in IDs:
queue1.put_nowait(i)
await asyncio.gather(*tasks)
The worker functions sit in an infinite loop waiting for items to enter the queue.
When the data has all been processed there will be no exit and the program will hang.
Is there a way to effectively manage the workers and end properly?
As nicely explained in this answer, Queue.join serves to inform the producer when all the work injected into the queue got completed. Since your first queue doesn't know when a particular item is done (it's multiplied and distributed to other queues), join is not the right tool for you.
Judging from your code, it seems that your workers need to run for only as long as it takes to process the queue's initial items. If that is the case, then you can use a shutdown sentinel to signal the workers to exit. For example:
async with aiohttp.ClientSession() as session:
# ... create tasks as above ...
for i in IDs:
queue1.put_nowait(i)
queue1.put_nowait(None) # no more work
await asyncio.gather(*tasks)
This is like your original code, but with an explicit shutdown request. Workers must detect the sentinel and react accordingly: propagate it to the next queue/worker and exit. For example, in worker1:
while True:
item = queue1.get()
if item is None:
# done with processing, propagate sentinel to worker2 and exit
await queue2.put(None)
break
# ... process item as usual ...
Doing the same in other two workers (except for worker3 which won't propagate because there's no next queue) will result in all three tasks completing once the work is done. Since queues are FIFO, the workers can safely exit after encountering the sentinel, knowing that no items have been dropped. The explicit shutdown also distinguishes a shut-down queue from one that happens to be empty, thus preventing workers from exiting prematurely due to a temporarily empty queue.
Up to Python 3.7, this technique was actually demonstrated in the documentation of Queue, but that example somewhat confusingly shows both the use of Queue.join and the use of a shutdown sentinel. The two are separate and can be used independently of one another. (And it might also make sense to use them together, e.g. to use Queue.join to wait for a "milestone", and then put other stuff in the queue, while reserving the sentinel for stopping the workers.)
I have a piece of code which has to get executed every 100ms and update the GUI. When I am updating the GUI - I am pressing a button, which calls a thread and in turn it calls a target function. The target function gives back the message to the GUI thread using pub sub as follows.
wx.CallAfter(pub.sendMessage, "READ EVENT", arg1=data, arg2=status_read) # This command line is in my target function
pub.subscribe(self.ReadEvent, "READ EVENT") # This is in my GUI file - whihc calls the following function
def ReadEvent(self, arg1, arg2):
if arg2 == 0:
self.MessageBox('The program did not properly read data from MCU \n Contact the Program Developer')
return
else:
self.data = arg1
self.firmware_version_text_control.Clear()
#fwversion = '0x' + ''.join('{:02X}'.format(j) for j in reversed(fwversion))
self.firmware_version_text_control.AppendText(str(SortAndDecode(self.data, 'FwVersion')))
# Pump Model
self.pump_model_text_control.Clear()
self.pump_model_text_control.AppendText(str(SortAndDecode(self.data, 'ModelName')))
# Pump Serial Number
self.pump_serial_number_text_control.Clear()
self.pump_serial_number_text_control.AppendText(str(SortAndDecode(self.data, 'SerialNum'))[:10]) # Personal Hack to not to display the AA , AB and A0
# Pressure GAIN
self.gain_text_control.Clear()
self.gain_text_control.AppendText(str(SortAndDecode(self.data, 'PresGain')))
# Pressure OFFSET Offset
self.offset_text_control.Clear()
self.offset_text_control.AppendText(str(SortAndDecode(self.data, 'PresOffset')))
#Wagner Message:
#self.status_text.SetLabel(str(SortAndDecode(self.data, 'WelcomeMsg')))
# PUMP RUNNING OR STOPPED
if PumpState(SortAndDecode(self.data, 'PumpState')) == 1:
self.led6.SetBackgroundColour('GREEN')
elif PumpState(SortAndDecode(self.data, 'PumpState')) == 0:
self.led6.SetBackgroundColour('RED')
else:
self.status_text.SetLabel(PumpState(SortAndDecode(self.data, 'PumpState')))
# PUMP RPM
self.pump_rpm_text_control.Clear()
if not self.new_model_value.GetValue():
self.pump_rpm_text_control.AppendText("000")
else:
self.pump_rpm_text_control.AppendText(str(self.sheet_num.cell_value(self.sel+1,10)*(SortAndDecode(self.data, 'FrqQ5'))/65536))
# PUMP PRESSURE
self.pressure_text_control.Clear()
self.pressure_text_control.AppendText(str(SortAndDecode(self.data, 'PresPsi')))
# ON TIME -- HOURS AND MINUTES --- EDITING IF YOU WANT
self.on_time_text_control.Clear()
self.on_time_text_control.AppendText(str(SortAndDecode(self.data, 'OnTime')))
# JOB ON TIME - HOURS AND MINUTES - EDITING IF YOU WANT
self.job_on_time_text_control.Clear()
self.job_on_time_text_control.AppendText(str(SortAndDecode(self.data, 'JobOnTime')))
# LAST ERROR ----- RECHECK THIS AGAIN
self.last_error_text_control.Clear()
self.last_error_text_control.AppendText(str(SortAndDecode(self.data, 'LastErr')))
# LAST ERROR COUNT --- RECHECK THIS AGAIN
self.error_count_text_control.Clear()
self.error_count_text_control.AppendText("CHECK THIS")
As you can see my READEVENT is very big and it takes a while for the GUI to take enough time to successfully do all these things. My problem here is, when my GUI is updating the values of TEXTCTRL it is going unresponsive - I cannot do anything else. I cant press a button or enter data or anything else. My question is if there is a better way for me to do this, so my GUI wont go unresponsive. I dont know how I can put this in a different thread as all widgets are in the main GUI. But that also requires keep creating threads every 100ms - which is horrible. Any suggestions would be greatly helpful.
Some suggestions:
How long does SortAndDecode take? What about the str() of the result? Those may be good candidates for keeping that processing in the worker thread instead of the UI thread, and passing the values to the UI thread pre-sorted-and-decoded.
You can save a little time in each iteration by calling ChangeValue instead of Clear and AppendText. Why do two function calls for each text widget instead of just one? Function calls are relatively expensive in Python compared to other Python code.
If it's possible that the same value will be sent that was sent on the last iteration then adding checks for the new value matching the old value and skipping the update of the widget could potentially save lots of time. Updating widget values is very expensive compared to leaving them alone.
Unless there is a hard requirement for 100ms updates you may want to try 150 or 200. Fewer updates per second may be fast enough for most people, especially since it's mostly textual. How much text can you read in 100ms?
If you are still having troubles with having more updates than the UI thread can keep up with, then you may want to use a different approach than pubsub and wx.CallAfter. For example you could have the worker thread receive and process the data and then add an object to a Queue.Queue and then call wx.WakeUpIdle(). In the UI thread you can have an EVT_IDLE event handler that checks the queue and pulls the first item out of the queue, if there are any, and then updates the widgets with that data. This will give the benefit of not flooding the pending events list with events from too many wx.CallAfter calls, and you can also do things like remove items from your data queue if there are too many items in it.
Consider the following code
//proces i: //proces j:
flag[i] = true; flag[j] = true;
turn = j; turn = i;
while(flag[j] == true && turn==j); while(flag[i] == true && turn == i);
<critical section> <critical section>
flag[i] = false; flag[j] = false;
<remainder section <remainder section>
I am certain that the above code will satisfies the mutual exclusion property but what I am uncertain about the following
What exactly does progress mean ? and does the above code satisfy it, The above code requires the the critical section being executed in strict alternation. Is that considered as progress ?
From what I see the above code does not maintain any information on the number of times a process has entered the critical section, would that mean that the above code does not satisfy bounded waiting ?
Progress means that the process will eventually do some work - an example of where this may not be the case is when a low-priority thread might be pre-empted and rolled back by high-priority threads. Once your processes reach their critical section they won't be pre-empted, so they'll make progress.
Bounded waiting means that the process will eventually gain control of the processor - an example of where this may not be the case is when another process has a non-terminating loop in a critical section with no possibility of the thread being interrupted. Your code has bounded waiting IF the critical sections terminate AND the remainder section will not re-invoke the process's critical section (otherwise a process might keep running its critical section without the other process ever gaining control of the processor).
Progress of the processes mean that the processes don't enter in a deadlock situation and hence,their execution continues independently! Actually, at any moment of time,only one of the process i or process j will be executing its critical section code and hence,the consistency will be maintained! SO, Progress of both processes are being talked and met successfully in the given code.
Next, this particular code is for processes which are intended to run only for once and hence, they won't be reaching their critical section code again. It is for single execution of process.
Bounded waiting says that a bound must exist on the number of times
that other processes are allowed to enter their critical sections
after a process has made a request to enter its critical section and
before that request is granted.
This particular piece of code has nothing to do with bounded waiting and is for trivial cases where processes execute for once only!
So I have been working with coroutines for a little while and I'm sort of having trouble trying to make what I want. I want a class that I can access objectivley creating objects as tasks or processes. I think showing you code would be irrelevant and its not what I want either. So I'm just going to show you how i want the functionality
local task1 = Tasker.newTask(function()
while true do
print("task 1")
end
end)
local task2 = Tasker.newTask(function()
while true do
print("task 2")
end
end)
task1:start()
task2:start()
This way I can run multiple tasks at once, I want to be able to add new tasks when ever during runtime. Also I would like a way to stop the tasks also for example:
task2:stop()
But I don't want the stop command to entirely delete the task instance, only stop the task itself so I can invoke
task2:start()
Then maybe I could use a command to delete it.
task2:delete()
This would be very helpful and thank you for help if you need more info please ask. Also i posted this on my phone so there may be typos and formatting issues
Lua doesn't natively support operating system threads, i.e. preemptive multitasking.
You can use coroutines to implement your own cooperative "threads", but each thread must relinquish control before another can do anything.
local task1 = Tasker.newTask(function()
while true do
print("task 1")
coroutine.yield()
end
end)
local task2 = Tasker.newTask(function()
while true do
print("task 2")
coroutine.yield()
end
end)
Your Tasker class must take the task function and wrap it in a coroutine, then take care of calling coroutine.resume on them. Operations like stop and start would set flags on the task that tell Tasker whether or not to resume that particular coroutine in the main loop.
You can do this via C. You might be able to use LuaLanes and Linda objects.
I have a web application that requests a report that takes more than 10 minutes to run. Apart from improving that performance, I would for now prefer to set up a thread to run the report and mail it to the user, returning that decision message back to the user immediately.
I have been looking at cherrypy.process.plugins.Monitor, but I'm not clear if it is the correct choice (what to do with the frequency parameter?)
Monitor is not the correct choice; it's for running the same task repeatedly on a schedule. You're probably better off just calling threading.Thread(target=run_report).start(). You can then return 202 Accepted to the user, along with a URL for the client to watch the status and/or retrieve the newly-created report resource when it's ready.
The one caveat to that is that you might want your new thread to shut down gracefully when the cherrypy.engine stops. Have a look at the various plugins for examples of how to hook into the 'stop' channel on the bus. The other option would be to make your thread daemonic, if you don't care if it terminates abnormally.
Besides agreeing with fumanchu's answer, I would like to add that the frequency parameter is actually the period expressed in seconds.cherrypy.process.plugins.Monitor (the name is misleading).
Another possible solution could be having a monitor executed periodically, and a set of working computations which can be checked periodically for completion. The code would be something like
class Scheduler:
def __init__ (self):
self.lock = threading.Lock()
self.mon = Monitor(cherrypy.engine, check_computations, frequency=whatever)
self.mon.start()
self.computations = list() # on which we append stuff
def check_computations (self):
with self.lock:
for i in self.computations:
check(i) # Single check function
Caveats:
The computation time of check matters. You don't want to have workload on this perioic routine
Beware on how you use locks:
It is protecting the computations list;
If you access it (even indirectly) from with check your program gets into deadlock. This could be the case if you want to unsubscribe something from the computations list.