Python - nested loops and asyncio - python-3.x

Question

I am still quite new to asyncio and struggling a bit with how to deal with loops within loops:
import asyncio
import concurrent.futures
import logging
import sys
import time
sub_dict = {
1: ['one', 'commodore', 'apple', 'linux', 'windows'],
2: ['two', 'commodore', 'apple', 'linux', 'windows'],
3: ['three', 'commodore', 'apple', 'linux', 'windows'],
4: ['four', 'commodore', 'apple', 'linux', 'windows'],
5: ['five', 'commodore', 'apple', 'linux', 'windows'],
6: ['six', 'commodore', 'apple', 'linux', 'windows'],
7: ['seven', 'commodore', 'apple', 'linux', 'windows'],
8: ['eight', 'commodore', 'apple', 'linux', 'windows']
}
def blocks(key, value):
for v in value:
log = logging.getLogger('blocks({} {})'.format(key, v))
log.info('running')
log.info('done')
time.sleep(5)
return key, v
async def run_blocking_tasks(executor, sub_dict2):
log = logging.getLogger('run_blocking_tasks')
log.info('starting')
log.info('creating executor tasks')
loop = asyncio.get_event_loop()
blocking_tasks = [
loop.run_in_executor(executor, blocks, key, value)
for key, value in sub_dict2.items()
]
log.info('waiting for executor tasks')
completed, pending = await asyncio.wait(blocking_tasks)
results = [t.result() for t in completed]
log.info('results: {!r}'.format(results))
log.info('exiting')
def new_func():
logging.basicConfig(
level=logging.INFO,
format='%(threadName)10s %(name)18s: %(message)s',
stream=sys.stderr,
)
executor = concurrent.futures.ThreadPoolExecutor(
max_workers=8,
)
event_loop = asyncio.get_event_loop()
event_loop.run_until_complete(
run_blocking_tasks(executor, sub_dict)
)
event_loop.close()
new_func()
Here you can see all the value items for each element are assigned to the same thread. For example all values of element '1' are on thread zero.
I know enough to understand that this is because my for v in value loop is not plugged into asyncio properly.
My desired output is, if I assigned five workers, each value item for the element '1' would be on it's own thread, numbered 0-4, giving five threads in total. This would then repeat for elements 2 through 8.
Should I assign 40 threads, 8 dictionary elements * 5 value items per element = 1 unique thread for each dictionary item.
Hope that makes sense...

Answer 1

Something about asking a question on SO always seems to trigger extra layers of IQ in me. The answer is as so, should anyone be interested:
def blocks(key, v):
#for v in value:
log = logging.getLogger('blocks({} {})'.format(key,v))
log.info('running')
log.info('done')
time.sleep(30)
return v
async def run_blocking_tasks(executor, sub_dict2):
log = logging.getLogger('run_blocking_tasks')
log.info('starting')
log.info('creating executor tasks')
for key, value in sub_dict2.items():
loop = asyncio.get_event_loop()
blocking_tasks = [
loop.run_in_executor(executor, blocks, key, v)
for v in value
]
log.info('waiting for executor tasks')
completed, pending = await asyncio.wait(blocking_tasks)
results = [t.result() for t in completed]
log.info('results: {!r}'.format(results))
log.info('exiting')
def new_func():
logging.basicConfig(
level=logging.INFO,
format='%(threadName)10s %(name)18s: %(message)s',
stream=sys.stderr,
)
sub_dict2 = dict(list(sub_dict.items())[0:8])
executor = concurrent.futures.ThreadPoolExecutor(
max_workers=5,
)
event_loop = asyncio.get_event_loop()
event_loop.run_until_complete(
run_blocking_tasks(executor, sub_dict2)
)
event_loop.close()
new_func()
EDIT:
Here is a rough layout of my present concurrent.futures code as per comment thread below. Is a layout of the logical order, rather than full code, as this has a few functions of pre steps that are quite lengthy...
#some code here that chunks a bigger dictionary using slicing in a for loop.
#sub_dict a 20 element subset of a bigger dictionary
#slices are parameterised in real code
sub_dict = dict(list(fin_dict.items())[0:20])
#set 20 workers
with concurrent.futures.ThreadPoolExecutor(max_workers=20) as executor:
#submit to the executor an enumerator/counter and my sub dict
future_to_pair = {executor.submit(function_name, v, i): (i, v) for i, v in enumerate(sub_dict.items(), 1)}
#await results
for future in concurrent.futures.as_completed(future_to_pair):
pair = future_to_pair[future]
data = future.result()
#function that is being called by concurrent.futures
#am happy for all the v's in value to be on a single thread
#i want each key to be on an individual thread
#this will process 20 keys simultaneously, but wait for the slowest one before clearing
def function_name(sub_dict, i):
for key, value in sub_dict:
for v in value:
# using subprocess, execute some stuff
# dictionary loops provide parameters for the executables.

Answer 2

I think you have missed a key concept: waiting. An async def function without an await is perfectly legal but rather pointless. The purpose of asyncio programming is to handle situations where your program must wait for something, and the program has something useful it could be doing in the meantime. Otherwise it has very little utility.* It's also not easy to find simple examples that illustrate how useful this is.
Python offers several types of concurrency: processes, which use multiple CPU cores; threads, which use multiple lines of execution within a single process; and asyncio tasks, which use multiple units of execution within a thread. These can be combined in various ways, and have different characteristics.
Threads allow your program to block in one place while waiting for a resource, but continue to execute in another place. But synchronizing between threads is often tricky because scheduling CPU time between threads is pre-emptive. It's not under your direct control.
Tasks also allow your program to stop in one place and continue in another, but switching between tasks is cooperative. It's under your control. When a task encounters an "await" expression, it stops there and allows another task to run. That task continues until it bumps into an await expression, and so on. If this solves a problem you have, great. It's a fantastic tool.
It seems, based on reading a number of SO questions, that programmers sometimes get the impression that asyncio will make their programs run faster by sending them off to some sort of Never-Never Land where they execute without consuming any CPU cycles, and the result will come floating back on the breeze. Sorry, that won't happen. The main use case is what I described: you have to wait for something but you've got something else to do.
*Remark added for completeness: I have used the cross-threading capabilities of asyncio as means of coordinating between threads. For example, create an event loop in Thread B, and cause it to execute a function on demand by Thread A using the "call_soon_threadsafe" method, or the "run_coroutine_threadsafe" method. This is a handy capability even if it doesn't require the use of an await expression.