I'm making unittests for timed processes, in my test there are some cases that depend on a timed action, for instance:
class MovementTest(unittest.TestCase):
def goto_test(self):
self.robot.goto(3, 4, 10) # go to x=3, y=4 in 10 seconds
# wait the 10 seconds
# the objective of the test is that the robot is able to get to (3, 4) in
# 10 seconds, that's why I need the waiting
self.assertEqual(self.robot.position, (3, 4))
The problem is that when I used time.sleep(10), the program just didn't wait, and therefore the test fails. Is there a way I can make unittest wait the desired time for this test case, also it would be nice if some other tests could run while the waiting is being done.
I tried with threads but the result where that the test checking finishes, the test program waits but the assertions where not checked.
Related
I am curious why the threads started in a python script are running even when the last statement of the script is executed (which means, the script has completed (I believe)).
I have shared below the code I am talking about. Any insights on this would be helpful:
======================================================================================
import time
import threading
start=time.perf_counter()
def do_something():
print("Waiting for a sec...")
time.sleep(60)
print("Waiting is over!!!")
mid1=time.perf_counter()
t1=threading.Thread(target=do_something)
t2=threading.Thread(target=do_something)
mid2=time.perf_counter()
t1.start()
mid3=time.perf_counter()
t2.start()
finish=time.perf_counter()
print(start,mid1,mid2,mid3,finish)
What output do you see? This is what I see:
Waiting for a sec...
Waiting for a sec...
95783.4201273 95783.4201278 95783.4201527 95783.4217046 95783.4219945
Then it's quiet for a minute, and displays:
Waiting is over!!!
Waiting is over!!!
and then the script ends.
That's all as expected. As part of shutting down, the interpreter waits for all running threads to complete (unless they were created with daemon=True, which you should probably avoid until you know exactly what you're doing). You told your threads to sleep for 60 seconds before finishing, and that's what they did.
I'm using asyncio to run a subprocess that prints some output which I parse line by line and do various things based on the output I see. I want to put a timeout on this process, but it should not be a global timeout for the life of the entire process. Instead, whenever I see certain specific output from the process, I want to actually reset the timeout so that it starts over. How can I implement this?
For a global timeout I have this working and it's easy, I simply call asyncio.wait_for(_foo(), timeout). But I can't get this to work with resetting the timeout. Here's what I have so far:
# here invocation is my own data structure with some bookkeeping information in it
# (such as the start time from which I want to base my timeout decisions on).
# and process is the value returned by await asyncio.create_subprocess_exec(...)
# _run_one_invocation() is my own function which is just a stdout readline loop
# and some dispatching.
# Make a Task out of the co-routine so that we decide when it gets cancelled, not Python.
run_task = asyncio.Task(_run_one_invocation(invocation, process))
while True:
try:
# Use asyncio.shield so it doesn't get cancelled if the timeout expires
await asyncio.shield(asyncio.wait_for(run_task, 1))
# If the await returns without raising an exception, we got to EOF and we're done.
break
except asyncio.TimeoutError:
# If it's been too long since last reset, this is a "real" timeout.
duration = time.time() - invocation.start_time
if duration > timeout:
run_task.cancel()
raise
When I run this, the if statement which calls run_task.cancel() is not being entered, and nevertheless when I go back to the top of the loop and call asyncio.wait_for() again it immediately raises an asyncio.CancelledError.
What am I doing wrong?
You can fix the issue and simplify the code by completely avoiding wait_for() (and therefore shield()) and just using wait(return_when=FIRST_COMPLETED) to implement the kind of timeout you need:
run_task = asyncio.create_task(_run_one_invocation(invocation, process))
while True:
await asyncio.wait([run_task], timeout=1)
if run_task.done():
break
if time.time() - invocation.start_time > timeout:
run_task.cancel()
raise asyncio.TimeoutErrror()
The downside of this approach is that it introduces 1s wakeups, prohibiting the program (and consequently the computer) from ever going to sleep, even if the task is dormant for hours. Probably not a big deal on a server, but such practices contribute to battery drain on laptops, and it's a good idea to avoid them. Also, the 1s sleep introduces an up to 1s latency to react to a change in timeout.
To resolve that, you can create an event that is fired by the code changing the timeout, and react to that event in addition to the timeout and the task completing:
timeout_changed = asyncio.Event()
# pass timeout_changed where needed, and have the code that changes
# the timeout also call timeout_changed.set()
run_task = asyncio.create_task(_run_one_invocation(invocation, process))
while True:
remaining = timeout - (time.time() - invocation.start_time)
timeout_changed_task = asyncio.ensure_future(timeout_changed.wait())
await asyncio.wait([run_task, timeout_changed_task],
return_when=asyncio.FIRST_COMPLETED, timeout=remaining)
timeout_changed_task.cancel()
# either: 1) the task has completed, 2) the previous timeout has
# expired, or 3) the timeout has changed
if run_task.done():
break # 1
if time.time() - invocation.start_time > timeout:
# 2 or 2+3
run_task.cancel()
raise asyncio.TimeoutErrror()
# 3 - continue waiting with the new timeout
Im trying to schedule a task with the module "schedule" for every hour. My problem is i need the task to first run then run again every hour.
This code works fine but it waits an hour before initial run
import schedule
import time
def job():
print("This happens every hour")
schedule.every().hour.do(job)
while True:
schedule.run_pending()
I would like to avoid doing this:
import schedule
import time
def job():
print("This happens immediately then every hour")
schedule.every().hour.do(job)
while i == 0:
job()
i = i+1
while i == 1:
schedule.run_pending()
Ideally it would be nice to have a option like this:
schedule.run_pending_now()
Probably the easiest solution is to just run it immediately as well as scheduling it, such as with:
import schedule
import time
def job():
print("This happens every hour")
schedule.every().hour.do(job)
job() # Runs now.
while True:
schedule.run_pending() # Runs every hour, starting one hour from now.
To run all jobs regardless if they are scheduled to run or not, use schedule.run_all(). Jobs are re-scheduled after finishing, just like they would if they were executed using run_pending().
def job_1():
print('Foo')
def job_2():
print('Bar')
schedule.every().monday.at("12:40").do(job_1)
schedule.every().tuesday.at("16:40").do(job_2)
schedule.run_all()
# Add the delay_seconds argument to run the jobs with a number
# of seconds delay in between.
schedule.run_all(delay_seconds=10)```
If you have many tasks that takes some time to execute and you want to run them independently during start you can use threading
import schedule
import time
def job():
print("This happens every hour")
def run_threaded(task):
job_thread = threading.Thread(target=task)
job_thread.start()
run_threaded(job) #runs job once during start
schedule.every().hour.do(run_threaded, job)
while True:
schedule.run_pending() # Runs every hour, starting one hour from now.
Actually I don't think that calling the function directly would be so wise, since it will block the thread without the scheduler, right?
I think there is nothing wrong about setting the job to be executed once, and every 30 sec for example like that:
scheduler.add_job(MPOStarter.run, args=ppi_args) # run once, then every 30 sec
scheduler.add_job(MPOStarter.run, "interval", seconds=30, args=ppi_args)
In my program I have this utility function for executing commands in shell, here's a simplified version of it:
def run_command(cmd):
s = time.time()
print('starting subprocess')
proc = subprocess.Popen(cmd.split(),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
universal_newlines=True)
print('subprocess started after ({0}) seconds'.format(time.time() - s))
My program uses this function from different threads to execute commands.
Occasionally the "Popen" line takes around 70 seconds to complete. I mean out of thousands of invocations in a day on different program runs this happens about 4-5 times. As far as I know Popen is non-blocking. What is weird to me is when it does happen it takes the same ~70 seconds to start. It's important to note that while this happen I have 3-4 other threads that are waiting in a loop:
while some_counter > 0:
time.sleep(0.5)
They do so for at most 60 seconds. After they give up and finish their flow I see another ~14 seconds until the "Popen" call finishes. Is there a problem running "Popen" from some threads in parallel to having other threads in a "wait loop"?
Update 1:
I now I see that this problem started after I switched from Fedora27+Python3.6 to Fedora31+python3.7.
I have an array of data to handle and handler that executing long (1-2 minutes) and takes a lot of memory for its calculations.
raw = ['a', 'b', 'c']
def handler():
# do something long
Since handler requires a lot of memory, I want to execute it in separate subprocess and kill it after execution to release memory. Something like the following snippet:
from multiprocessing import Process
for r in raw:
process = Process(target=handler, args=(r))
process.start()
The problem is that such approach leads to immediate running len(raw) processes. And it's not good.
Also, it's not needed to interchange any kind of data between subprocesses. Just run them consequently.
Therefore it would be great to run a few processes at the same time and add a new one once existing finishes.
How could it be implemented (if it's even possible)?
to run your processes sequentially, just join each process within the loop:
from multiprocessing import Process
for r in raw:
process = Process(target=handler, args=(r))
process.start()
process.join()
that way you're sure that only one process is running at the same time (no concurrency)
That's the simplest way. To run more than one process but limit the number of processes running at the same time, you can use a multiprocessing.Pool object and apply_async
I've built a simple example which computes the square of the argument, and simulates an heavy processing:
from multiprocessing import Pool
import time
def target(r):
time.sleep(5)
return(r*r)
raw = [1,2,3,4,5]
if __name__ == '__main__':
with Pool(3) as p: # 3 processes at a time
reslist = [p.apply_async(target, (r,)) for r in raw]
for result in reslist:
print(result.get())
Running this I get:
<5 seconds wait, time to compute the results>
1
4
9
<5 seconds wait, 3 processes max can run at the same time>
16
25