python threading event.wait() using same object in multiple threads - python-3.x

where is the documentation for the python3 threading library's event.wait() method that explains how 1 event can be used multiple times in different threads?
the example below shows the same event can be used in multiple threads, each with a different wait() duration, probably because each has its own lock under the hood.
But this functionality is not documented in an obvious way, on the threading page.
this works great but it's not clear why it works or if this will continue to work in future python versions.
are there ways this could bonk unexpectedly?
can an inherited event work properly in multiple classes as long as it's used in separate threads?
import logging
import threading
import time
logging.basicConfig(level=logging.DEBUG,
format='[%(levelname)s] (%(threadName)-10s) %(message)s',)
def worker(i,dt,e):
tStart=time.time()
e.wait(dt)
logging.debug('{0} tried to wait {1} seconds but really waited {2}'.format(i,dt, time.time()-tStart ))
e = threading.Event()
maxThreads=10
for i in range(maxThreads):
dt=1+i # (s)
w = threading.Thread(target=worker, args=(i,dt,e))
w.start()
output:
[DEBUG] (Thread-1 ) 0 tried to wait 1 seconds but really waited 1.0003676414489746
[DEBUG] (Thread-2 ) 1 tried to wait 2 seconds but really waited 2.00034761428833
[DEBUG] (Thread-3 ) 2 tried to wait 3 seconds but really waited 3.0001776218414307
[DEBUG] (Thread-4 ) 3 tried to wait 4 seconds but really waited 4.000180244445801
[DEBUG] (Thread-5 ) 4 tried to wait 5 seconds but really waited 5.000337362289429
[DEBUG] (Thread-6 ) 5 tried to wait 6 seconds but really waited 6.000308990478516
[DEBUG] (Thread-7 ) 6 tried to wait 7 seconds but really waited 7.000143051147461
[DEBUG] (Thread-8 ) 7 tried to wait 8 seconds but really waited 8.000152826309204
[DEBUG] (Thread-9 ) 8 tried to wait 9 seconds but really waited 9.00012469291687
[DEBUG] (Thread-10 ) 9 tried to wait 10 seconds but really waited 10.000144481658936

Since e is threading Event,
You are declaring it locally for each thread (all the 10 threads are excecated almost parallelly).
You can check it here:
import logging
import threading
import time
logging.basicConfig(level=logging.DEBUG,
format='[%(levelname)s] (%(threadName)s) %(message)s',)
def worker(i,dt,e):
tStart=time.time()
logging.info('Program will wait for {} time while trying to print the change from {} to {}'.format(dt,i,dt))
e.wait(dt)
logging.debug('{0} tried to wait {1} seconds but really waited {2}'.format(i,dt, time.time()-tStart ))
e = threading.Event()
maxThreads=10
for i in range(maxThreads):
dt=1+i # (s)
w = threading.Thread(target=worker, args=(i,dt,e))
w.start()
It's not the locking, its just about the value passed into the thread target

Related

Data acquisition and parallel analysis

With this example, I am able to start 10 processes and then continue to do "stuff".
import random
import time
import multiprocessing
if __name__ == '__main__':
"""Demonstration of GIL-friendly asynchronous development with Python's multiprocessing module"""
def process(instance):
total_time = random.uniform(0, 2)
time.sleep(total_time)
print('Process %s : completed in %s sec' % (instance, total_time))
return instance
for i in range(10):
multiprocessing.Process(target=process, args=(i,)).start()
for i in range(2):
print("im doing stuff")
output:
>>
im doing stuff
im doing stuff
Process 8 : completed in 0.5390905372395016 sec
Process 6 : completed in 1.2313793332779521 sec
Process 2 : completed in 1.3439237625459899 sec
Process 0 : completed in 2.171809500083049 sec
Process 5 : completed in 2.6980031493633887 sec
Process 1 : completed in 3.3807358192422416 sec
Process 3 : completed in 4.597366303348297 sec
Process 7 : completed in 4.702447947943171 sec
Process 4 : completed in 4.8355495004170965 sec
Process 9 : completed in 4.9917788543156245 sec
I'd like to have a main while True loop which do data acquisition and just start a new process at each iteration (with the new data) and check if any process has finished and look at the output.
How could I verify that a process has ended and what is its return value? Edit: while processes in a list are still executing.
If I had to summarize my problem: how can I know which process is finished in a list of processes - with some still executing or new added?

How to use ApScheduler correctly in FastAPI?

from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
import uvicorn
import time
from loguru import logger
from apscheduler.schedulers.background import BackgroundScheduler
app = FastAPI()
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
test_list = ["1"]*10
def check_list_len():
global test_list
while True:
time.sleep(5)
logger.info(f"check_list_len:{len(test_list)}")
#app.on_event('startup')
def init_data():
scheduler = BackgroundScheduler()
scheduler.add_job(check_list_len, 'cron', second='*/5')
scheduler.start()
#app.get("/pop")
async def list_pop():
global test_list
test_list.pop(1)
logger.info(f"current_list_len:{len(test_list)}")
if __name__ == '__main__':
uvicorn.run(app="main3:app", host="0.0.0.0", port=80, reload=False, debug=False)
Above is my code, I want to take out a list of elements through get request, and set a periodic task constantly check the number of elements in the list, but when I run, always appear the following error:
Execution of job "check_list_len (trigger: cron[second='*/5'], next run at: 2021-11-25 09:48:50 CST)" skipped: maximum number of running instances reached (1)
2021-11-25 09:48:50.016 | INFO | main3:check_list_len:23 - check_list_len:10
Execution of job "check_list_len (trigger: cron[second='*/5'], next run at: 2021-11-25 09:48:55 CST)" skipped: maximum number of running instances reached (1)
2021-11-25 09:48:55.018 | INFO | main3:check_list_len:23 - check_list_len:10
INFO: 127.0.0.1:55961 - "GET /pop HTTP/1.1" 200 OK
2021-11-25 09:48:57.098 | INFO | main3:list_pop:35 - current_list_len:9
Execution of job "check_list_len (trigger: cron[second='*/5'], next run at: 2021-11-25 09:49:00 CST)" skipped: maximum number of running instances reached (1)
2021-11-25 09:49:00.022 | INFO | main3:check_list_len:23 - check_list_len:9
It looks like I started two scheduled tasks and only one succeeded, but I started only one task. How do I avoid this
You're getting the behavior you're asking for. You've configured apscheduler to run check_list_len every five seconds, but you've also made it so that function runs without terminating - just sleeping for five seconds in an endless loop. That function never terminates, so apscheduler doesn't run it again - since it still hasn't finished.
Remove the infinite loop inside your utility function when using apscheduler - it'll call the function every five seconds for you:
def check_list_len():
global test_list # you really don't need this either, since you're not reassigning the variable
logger.info(f"check_list_len:{len(test_list)}")

Laravel-Excel keeps browser busy for 140 seconds after completion of import: how do I correct it?

Using the import to models option, I am importing an XLS file with about 15,000 rows.
With the microtime_float function, the script times and echos out how long it takes. At 29.6 secs, this happens, showing it took less than 30 seconds. At that time, I can see the database has all 15k+ records as expected, no issues there.
Problem is, the browser is kept busy and at 1 min 22 secs, 1 min 55 secs and 2 min 26 secs it prompts me to either wait or kill the process. I keep clicking wait and finally it ends at 2 mins 49 secs.
This is a terrible user experience, how can I cut off this extra wait time?
It's a very basic setup: the route calls importcontroller#import with http get and the code is as follows:
public function import()
{
ini_set('memory_limit', '1024M');
$start = $this->microtime_float();
Excel::import(new myImport, 'myfile.xls' , null, \Maatwebsite\Excel\Excel::XLS);
$end = $this->microtime_float();
$t = $end - $start;
return "Time: $t";
}
The class uses certain concerns as follows:
class myImport implements ToModel, WithBatchInserts, WithChunkReading, WithStartRow

Execute python script every 15 mins starting from 9:30:42 till 15:00:42 every day from Mon-Fri

I need a way to execute my python script every 15 mins starting from 9:30:42 till 15:00:42 every day from Mon-Fri.
I have explored APScheduler with cron syntax but can't figure out how to code the above condition. I tried below but doesn't work (execute is my function name)
sched.add_cron_job(execute, day_of_week='mon-fri', hour='9:30:42-15:00:42', minute='*/15')
Any pointer is deeply appreciated.
As far as I can tell you won't be able to do what you want with a single job.
This is the closest I could get with one:30-59/15 9-14 * * 1-5 which equates to Every 15 minutes, minutes 30 through 59 past the hour, between 09:00 AM and 02:59 PM, Monday through Friday.
Although it isn't exactly what you wanted I hope this helps as a base.
I wrote custom code to solve my problem. Posting here in case it helps someone. Any optimisations suggestions are welcome.
The first infinite loop starts the job when the start time is hit. The 2nd infinite loop wakes up every x minutes to check if next run time has approached. If yes, it executes else goes back to sleep. If the end time for execution has reached, then it breaks out
def execute_schedule_custom():
start_time_of_day = datetime.combine(date.today(), time(9, 30, 42))
next_run_time = start_time_of_day
end_time_of_day = datetime.combine(date.today(), time(15, 0, 42))
interval = 15
sleep_secs = 60 * 5 #sleep for 5 mins
while True:
if datetime.now() >= start_time_of_day:
execute()
next_run_time = start_time_of_day + timedelta(minutes=interval)
break
while True:
if datetime.now() >= end_time_of_day:
break
elif datetime.now() >= next_run_time:
execute()
next_run_time = next_run_time + timedelta(minutes=interval)
t.sleep(sleep_secs)

rampUser method is getting stuck in gatling 3.3

I am having issues using rampUser() method in my gatling script. The request is getting stuck after the following entry which had passed half way through.
Version : 3.3
================================================================================
2019-12-18 09:51:44 45s elapsed
---- Requests ------------------------------------------------------------------
> Global (OK=2 KO=0 )
> graphql / request_0 (OK=1 KO=0 )
> rest / request_0 (OK=1 KO=0 )
---- xxxSimulation ---------------------------------------------------
[##################################### ] 50%
waiting: 1 / active: 0 / done: 1
================================================================================
I am seeing the following in the log which gets repeated for ever and the log size increases
09:35:46.495 [GatlingSystem-akka.actor.default-dispatcher-2] DEBUG io.gatling.core.controller.inject.open.OpenWorkload - Injecting 0 users in scenario xxSimulation, continue=true
09:35:47.494 [GatlingSystem-akka.actor.default-dispatcher-6] DEBUG io.gatling.core.controller.inject.open.OpenWorkload - Injecting 0 users in scenario xxSimulation, continue=true
The above issue is happening only with rampUser and not happening with
atOnceUsers()
rampUsersPerSec()
rampConcurrentUsers()
constantConcurrentUsers()
constantUsersPerSec()
incrementUsersPerSec()
Is there a way to mimic rampUser() in some other way or is there a solution for this.
My code is very minimal
setUp(
scenarioBuilder.inject(
rampUsers(2).during(1 minutes)
)
).protocols(protocolBuilder)
I am stuck with this for some time and my earlier post with more information can be found here
Can any of the gatling experts help me on this?
Thanks for looking into it.
It seems you have slightly incorrect syntax for a rampUsers. You should try remove a . before during.
I have in my own script this code and it works fine:
setUp(userScenario.inject(
// atOnceUsers(4),
rampUsers(24) during (1 seconds))
).protocols(httpProtocol)
Also, in Gatling documentation example is also without a dot Open model:
scn.inject(
nothingFor(4 seconds), // 1
atOnceUsers(10), // 2
rampUsers(10) during (5 seconds), // HERE
constantUsersPerSec(20) during (15 seconds), // 4
constantUsersPerSec(20) during (15 seconds) randomized, // 5
rampUsersPerSec(10) to 20 during (10 minutes), // 6
rampUsersPerSec(10) to 20 during (10 minutes) randomized, // 7
heavisideUsers(1000) during (20 seconds) // 8
).protocols(httpProtocol)
)
My guess is that syntax can't be parsed, so instead 0 is substituted. (Here is example of rounding. Not applicable, but as reference: gatling-user-injection-constantuserspersec)
Also, you mentioned that others method work, could you paste working code as well?

Resources