I’m learning concurrent.futures.ThreadPoolExecutor in Py3.6 and a bit confused as to what’s the difference, pros-and-cons between using
1 future.add_done_callback(callback)
2 concurrent.futures.as_completed(futures)
When would you choose one over the other? If I understand correctly the purpose is same for both more or less.. #1 calls the callback(future) fn as soon as the task has finished and corresponding future has settled, and #2 returns the futures object in the order which the tasks finish and futures settle..
In both cases we we can retrieve the returned value using future.results() (or raise future.exception() if exception was raised).
Thanks for any clarification around that.
The definitions of the functions are at https://github.com/python/cpython/blob/f85af035c5cb9a981f5e3164425f27cf73231b5f/Lib/concurrent/futures/_base.py#L200
def as_completed(fs, timeout=None):
"""An iterator over the given futures that yields each as it completes.
add_done_callback is a method within futures class and a lower level function than as_completed. Essentially, as_completed uses add_done_callback internally. as_completed also has timeout argument for callback.
In general, we may use as_completed if working with multiple futures, while add_done_callback is used with single future.
Overall, both add_done_callback and as_completed may achieve the similar objectives for simpler programs.
Just a thought. We can use different callback function for each future in the list of futures with add_done_callback, while as_completed may work only accept a single single callback.
Related
I am rewriting a c++ project in rust as my first non-tiny rust program. I thought I would start with a simple but key gnarly piece of code.
Its a queue of std::packaged_tasks that run at specific times. A client says
running_func_fut_ = bus_->TimerQueue().QueueTask(std::chrono::milliseconds(func_def.delay),
[this, func, &unit]()
{
func(this, &unit);
Done();
}, trace);
func is a std::function, but they key point is that as far as the queue is concerned is queuing up a lambda (closure in rust speak )
It returns a std::future which the client can ignore or can hang onto. If they hang onto it they can see if the task completed yet. (It could return a result but in my current use case the functions are all void, the client just needs to know if the task completed). All the tasks run on a single dedicated thread. The QueueTask method wraps the passed lambda up in a packaged_task and then places it in a multiset of objects that say when and what to run.
I am reading the rust docs and it seems that futures encapsulate both the callable object and the 'get me the result' mechanism.
So I think I need a BTreeSet (I need the queue sorted by launch time so I can pick the next one to run) of futures, but I am not even sure how to declare one of those. SO before I dive into the deep end of futures, is this the right approach? Is there a better , more natural, abstraction for rust?
For the output, you probably do want a Future. However, for the input, you probably want a function object (Box<dyn FnOnce(...)>); see https://doc.rust-lang.org/book/ch19-05-advanced-functions-and-closures.html.
https://www.aeracode.org/2018/02/19/python-async-simplified/
It's not going to ruin your day if you call a non-blocking synchronous
function, like this:
def get_chat_id(name):
return "chat-%s" % name
async def main():
result = get_chat_id("django")
However, if you call a blocking function, like the Django ORM, the
code inside the async function will look identical, but now it's
dangerous code that might block the entire event loop as it's not
awaiting:
def get_chat_id(name):
return Chat.objects.get(name=name).id
async def main():
result = get_chat_id("django")
You can see how it's easy to have a non-blocking function that
"accidentally" becomes blocking if a programmer is not super-aware of
everything that calls it. This is why I recommend you never call
anything synchronous from an async function without doing it safely,
or without knowing beforehand it's a non-blocking standard library
function, like os.path.join.
So I am looking for a way to automatically catch instances of this mistake. Are there any linters for Python which will report sync function calls from within an async function as a violation?
Can I configure Pylint or Flake8 to do this?
I don't necessarily mind if it catches the first case above too (which is harmless).
Update:
On one level I realise this is a stupid question, as pointed out in Mikhail's answer. What we need is a definition of a "dangerous synchronous function" that the linter should detect.
So for purpose of this question I give the following definition:
A "dangerous synchronous function" is one that performs IO operations. These are the same operations which have to be monkey-patched by gevent, for example, or which have to be wrapped in async functions so that the event loop can context switch.
(I would welcome any refinement of this definition)
So I am looking for a way to automatically catch instances of this
mistake.
Let's make few things clear: mistake discussed in article is when you call any long running sync function inside some asyncio coroutine (it can be I/O blocking call or just pure CPU function with a lot of calculations). It's a mistake because it'll block whole event loop what will lead to significant performance downgrade (more about it here including comments below answer).
Is there any way to catch this situation automatically? Before run time - no, no one except you can predict if particular function will take 10 seconds or 0.01 second to execute. On run time it's already built-in asyncio, all you have to do is to enable debug mode.
If you afraid some sync function can vary between being long running (detectable in run time in debug mode) and short running (not detectable) just execute function in background thread using run_in_executor - it'll guarantee event loop will not be blocked.
I'm new to asyncio and I started working with Transports to create a simple server-client program.
on the asyncio page I see the following:
Transport.close() can be called immediately after WriteTransport.write() even if data are not sent yet on the socket: both methods are asynchronous. yield from is not needed because these transport methods are not coroutines.
I searched the web (including stackoverflow) but couldn't find a good answer to the following question: what are the major differences between an asynchronous method and a coroutine?
The only 2 differences I can make are:
in coroutines you have a more fine grained control over the order of the methods the main loop executes using the yield from expression.
coroutines are generators, hence are more memory efficient.
anything else I am missing?
Thank you.
In the context asynchronous means both .write() and .close() are regular methods, not coroutines.
If .write() cannot write data immediately it stores the data in internal buffer.
.close() never closes connection immediately but schedules socket closing after all internal buffer will be sent.
So
transp.write(b'data')
transp.write(b'another data')
transp.close()
is safe and perfectly correct code.
Also .write() and .close() are not coroutines, obviously.
You should call coroutine via yield from expression, e.g. yield from coro().
But these methods are convention functions, so call it without yield from as shown in example above.
In Python, I am using a library called futures, which allows me to do my processing work with a pool of N worker processes, in a succinct and crystal-clear way:
schedulerQ = []
for ... in ...:
workParam = ... # arguments for call to processingFunction(workParam)
schedulerQ.append(workParam)
with futures.ProcessPoolExecutor(max_workers=5) as executor: # 5 CPUs
for retValue in executor.map(processingFunction, schedulerQ):
print "Received result", retValue
(The processingFunction is CPU bound, so there is no point for async machinery here - this is about plain old arithmetic calculations)
I am now looking for the closest possible way to do the same thing in Scala. Notice that in Python, to avoid the GIL issues, I was using processes (hence the use of ProcessPoolExecutor instead of ThreadPoolExecutor) - and the library automagically marshals the workParam argument to each process instance executing processingFunction(workParam) - and it marshals the result back to the main process, for the executor's map loop to consume.
Does this apply to Scala and the JVM? My processingFunction can, in principle, be executed from threads too (there's no global state at all) - but I'd be interested to see solutions for both multiprocessing and multithreading.
The key part of the question is whether there is anything in the world of the JVM with as clear an API as the Python futures you see above... I think this is one of the best SMP APIs I've ever seen - prepare a list with the function arguments of all invocations, and then just two lines: create the poolExecutor, and map the processing function, getting back your results as soon as they are produced by the workers. Results start coming in as soon as the first invocation of processingFunction returns and keep coming until they are all done - at which point the for loop ends.
You have way less boilerplate than that using parallel collections in Scala.
myParameters.par.map(x => f(x))
will do the trick if you want the default number of threads (same as number of cores).
If you insist on setting the number of workers, you can like so:
import scala.collection.parallel._
import scala.concurrent.forkjoin._
val temp = myParameters.par
temp.tasksupport = new ForkJoinTaskSupport(new ForkJoinPool(5))
temp.map(x => f(x))
The exact details of return timing are different, but you can put as much machinery as you want into f(x) (i.e. both compute and do something with the result), so this may satisfy your needs.
In general, simply having the results appear as completed is not enough; you then need to process them, maybe fork them, collect them, etc.. If you want to do this in general, Akka Streams (follow links from here) are nearing 1.0 and will facilitate the production of complex graphs of parallel processing.
There is both a Futures api that allows you to run work-units on a thread pool (docs: http://docs.scala-lang.org/overviews/core/futures.html) and a "parallell collections api" that you can use to perform parallell operations on collections: http://docs.scala-lang.org/overviews/parallel-collections/overview.html
I currently have code in the form of a generator which calls an IO-bound task. The generator actually calls sub-generators as well, so a more general solution would be appreciated.
Something like the following:
def processed_values(list_of_io_tasks):
for task in list_of_io_tasks:
value = slow_io_call(task)
yield postprocess(value) # in real version, would iterate over
# processed_values2(value) here
I have complete control over slow_io_call, and I don't care in which order I get the items from processed_values. Is there something like coroutines I can use to get the yielded results in the fastest order by turning slow_io_call into an asynchronous function and using whichever call returns fastest? I expect list_of_io_tasks to be at least thousands of entries long. I've never done any parallel work other than with explicit threading, and in particular I've never used the various forms of lightweight threading which are available.
I need to use the standard CPython implementation, and I'm running on Linux.
Sounds like you are in search of multiprocessing.Pool(), specifically the Pool.imap_unordered() method.
Here is a port of your function to use imap_unordered() to parallelize calls to slow_io_call().
def processed_values(list_of_io_tasks):
pool = multiprocessing.Pool(4) # num workers
results = pool.imap_unordered(slow_io_call, list_of_io_tasks)
while True:
yield results.next(9999999) # large time-out
Note that you could also iterate over results directly (i.e. for item in results: yield item) without a while True loop, however calling results.next() with a time-out value works around this multiprocessing keyboard interrupt bug and allows you to kill the main process and all subprocesses with Ctrl-C. Also note that the StopIteration exceptions are not caught in this function but one will be raised when results.next() has no more items return. This is legal from generator functions, such as this one, which are expected to either raise StopIteration errors when there are no more values to yield or just stop yielding and a StopIteration exception will be raised on it's behalf.
To use threads in place of processes, replace
import multiprocessing
with
import multiprocessing.dummy as multiprocessing