Best way to run "self monitoring" - asyncio/thread? - multithreading

Question

Python 3.10.6
ManagerTask() - responsible of executing Task() methods .
Note , Task's method are actually are Celery tasks (async ).
I'd like to add the option to track those tasks execution inside ManagerTask class.
I manage to get it work but as it my first asyncio I'm not sure i'm doing it right ,
(I'm aware of Flower).
Second, at current running single Main() which orchestrating executing Task() ,
In real world need to expand it to support executing multiple sessions of Main() in parallel.
class ManagerTask:
def __init__(self ,id:int ) -> None:
self.id = id
self.tasks: List["Task"] = []
self.executed_tasks: List["Task"] = []
self.state = State.SCHEDULED
def load_tasks(self , configs: List[Dict]):
# code for loading tasks
async def run(self):
""" Execute all tasks and start monitor results"""
execute_task = asyncio.create_task(self.task_exec())
monitor_progresss = asyncio.create_task(self.monitor_progress())
await execute_task
await monitor_progresss
async def monitor_progress(self):
""" If one task Failed - Main.state failed if all success mark Main.state -success"""
import time
failure_flag = False
count_success = 0
i = 0
while True:
state = self.executed_tasks[i].task_run.state
if state == 'SUCCESS':
count_success += 1
i += 1
if state == 'FAILURE':
failure_flag = True
i += 1
print(f'Rule_ID:{self.executed_tasks[i].rule_id} \
\nCelery_UUID:{self.executed_tasks[i].task_celery_id} \
\nstatus - {self.executed_tasks[i].task_run.state}')
# all tasks proccessed (either failed/success)
if i == len(self.executed_tasks) -1:
if failure_flag:
self.state = State.FAILED
elif count_success == len(self.executed_tasks) :
self.state = State.FINISHED
break;
# otherwise wait
await asyncio.sleep(3)
async def task_exec(self):
for task in self.starting_tasks:
task.run()
client (execute the app):
...code
cust = MainTask(id=1)
cust.load_tasks(configs=rules ,db_connections=db_connections)
asyncio.run(cust.run())
print("MainTask State:" + cust.state)
Example of output:
Rule_ID:4
Celery_UUID:fecde27c-b58a-43cd-9498-3478404c248b
status - FAILURE
....
Rule_ID:6
Celery_UUID:85df9bba-3d75-4b00-a533-a81cd3f6afb3
status - SUCCESS
MainTask State:Failed
1.Is that is proper way executing asyncio ?
2.for running multiple MainTask , how I should do it? Thread/Asyncio?
As this program executing all task using celery I think i should also run in asyncio but not sure .
second , would be thankful for guidance if this is the right approach
async def exec_id(id):
cust = MainTask(id=id)
cust.load_tasks(configs=rules ,...)
await cust.run()
async def main():
ids = [111,222,333]
for id in ids:
await exec_id(id)
asyncio.run(main())