from threading import Timer
def Add():
...Some process...
Timer(86400, Add).start() ## 86400 secs in 24 hours.
if __name__ == '__main__':
Add()
"Consuming Kafka Messages and will run continously once started".
Once the code runs Add function is called and program starts to consume kafka messages and waits for it continously. I want to call the "Add" function every 24 hrs without disturbing kafka process. For this I have tried one threading.timer but I am not sure will it work or not.
I have one more question- is this thread initialize every 24hrs or it one thread just calls every 24 hrs.
Please advice I am doing correct and will it work or not and thanks in advance!!
You can simply use an extra function that calls add() and adds the timer for you, something like this:
def task1():
while True:
add()
time.sleep(86400)
if __name__ == "__main__":
thread = Thread(target=task1)
thread.start()
# add kafka code
Related
Im trying to schedule a task with the module "schedule" for every hour. My problem is i need the task to first run then run again every hour.
This code works fine but it waits an hour before initial run
import schedule
import time
def job():
print("This happens every hour")
schedule.every().hour.do(job)
while True:
schedule.run_pending()
I would like to avoid doing this:
import schedule
import time
def job():
print("This happens immediately then every hour")
schedule.every().hour.do(job)
while i == 0:
job()
i = i+1
while i == 1:
schedule.run_pending()
Ideally it would be nice to have a option like this:
schedule.run_pending_now()
Probably the easiest solution is to just run it immediately as well as scheduling it, such as with:
import schedule
import time
def job():
print("This happens every hour")
schedule.every().hour.do(job)
job() # Runs now.
while True:
schedule.run_pending() # Runs every hour, starting one hour from now.
To run all jobs regardless if they are scheduled to run or not, use schedule.run_all(). Jobs are re-scheduled after finishing, just like they would if they were executed using run_pending().
def job_1():
print('Foo')
def job_2():
print('Bar')
schedule.every().monday.at("12:40").do(job_1)
schedule.every().tuesday.at("16:40").do(job_2)
schedule.run_all()
# Add the delay_seconds argument to run the jobs with a number
# of seconds delay in between.
schedule.run_all(delay_seconds=10)```
If you have many tasks that takes some time to execute and you want to run them independently during start you can use threading
import schedule
import time
def job():
print("This happens every hour")
def run_threaded(task):
job_thread = threading.Thread(target=task)
job_thread.start()
run_threaded(job) #runs job once during start
schedule.every().hour.do(run_threaded, job)
while True:
schedule.run_pending() # Runs every hour, starting one hour from now.
Actually I don't think that calling the function directly would be so wise, since it will block the thread without the scheduler, right?
I think there is nothing wrong about setting the job to be executed once, and every 30 sec for example like that:
scheduler.add_job(MPOStarter.run, args=ppi_args) # run once, then every 30 sec
scheduler.add_job(MPOStarter.run, "interval", seconds=30, args=ppi_args)
I have a script that receives temperature data via using requests. Since I had to make multiple requests (around 13000) I decided to explore the use of multi-threading which I am new at.
The programs work by grabbing longitude/latitude data from a csv file and then makes a request to retrieve the temperature data.
The problem that I am facing is that the script does not finish fully when the last temperature value is retrieved.
Here is the code. I have shortened so it is easy to see what I am doing:
num_threads = 16
q = Queue(maxsize=0)
def get_temp(q):
while not q.empty():
work = q.get()
if work is None:
break
## rest of my code here
q.task_done()
At main:
def main():
for o in range(num_threads):
logging.debug('Starting Thread %s', o)
worker = threading.Thread(target=get_temp, args=(q,))
worker.setDaemon(True)
worker.start()
logging.info("Main Thread Waiting")
q.join()
logging.info("Job complete!")
I do not see any errors on the console and temperature is being successfully being written to another file. I have a tried running a test csv file with only a few longitude/latitude references and the script seems to finish executing fine.
So is there a way of shedding light as to what might be happening in the background? I am using Python 3.7.3 on PyCharm 2019.1 on Linux Mint 19.1.
the .join() function waits for all threads to join before continuing to the next line
I have a simple watchdog in python 3 that reboots my server if something goes wrong:
import time, os
from multiprocessing import Pool
def watchdog(x):
time.sleep(x)
os.system('reboot')
return
def main():
while True:
p = Pool(processes=1)
p.apply_async(watchdog, (60, )) # start watchdog with 60s interval
# here some code thas has a little chance to block permanently...
# reboot is ok because of many other files running independently
# that will get problems too if this one blocks too long and
# this will reset all together and autostart everything back
# block is happening 1-2 time a month, mostly within a http-request
p.terminate()
p.join()
return
if __name__ == '__main__':
main()
p = Pool(processes=1) is declared every time the while loop starts.
Now here the question: Is there any smarter way?
If I p.terminate() to prevent the process from reboot, the Pool becomes closed for any other work. Or is there even nothing wrong with declaring a new Pool every time because of garbage collection.
Use a process. Processes support all of the features you are using, so you don't need to make a pool with size one. While processes do have a warning about using the terminate() method (since it can corrupt pipes, sockets, and locking primitives), you are not using any of those items and don't need to care. (In any event, Pool.terminate() probably has the same issues with pipes etc. even though it lacks a similar warning.)
Let's say I have built a small custom class - I think quickly built a representative prototype below - to help me hold on to and operate on some data.
The ultimate use case, though, is to hold on to data and periodically delete older data, really just holding onto newer data. I was thinking to start I would just try to delete all of the data after 15 seconds. And understand that in trying to do things/operate on the data I have, well, that it's all less than 15 seconds old for sure. So it's all new data.
And then later I could make the class more complicated - do things like - not delete all of the data but just the VERY old stuff, or change the 15 second interval to something dynamic/smarter, etc.
But before I even got near that point, I was stumbled by the way threading/timer worked. I discovered empirically that the below code only prints out 'Fresh!' if I include that second/subsequent threading.Timer call INSIDE the freshen_up method which is the callback. I would have thought I'd only need to start the timer once and it would just call the callback every 15 seconds. Am I crazy, or do I need to set the timer 'every time' (as I do in the code below), for this simple implementation to work? Again, as far as I can tell, this code works, I am just surprised that I needed to start a new timer every time. So I want to make sure I am not doing something inefficient/unnecessary due to my lack of understanding how timer works... thanks
import threading
class Fresh_Data_Container():
def __init__(self):
self.fresh_data_dict = {}
threading.Timer(15.0, self.freshen_up).start()
def freshen_up(self):
self.fresh_data_dict = {} # starting over for now, perhaps more nuanced later
print ('Fresh!')
threading.Timer(15.0, self.freshen_up).start() # does this really have to be here or am i looking at this wrong?
def add_fresh_data(self, some_key, fresh_data):
if some_key in self.fresh_data_dict:
self.fresh_data_dict[some_key].append(fresh_data)
else:
self.fresh_data_dict[some_key] = [fresh_data]
def operate_on_data(self):
pass # do stuff
threading.Timer isn't periodic -- that is, it only calls the given function once after the elapsed 15 seconds, not every 15 seconds.
A better way to do this would be to create a regular Thread instead of a Timer:
def __init__(self):
self.fresh_data_dict = {}
self.running = True
threading.Thread(target=self.freshen_up, daemon=True).start()
def stop(self):
self.running = False
def freshen_up(self):
while self.running:
time.sleep(15)
self.fresh_data_dict.clear()
I'm trying to utilize threading and queueing (based on a recommendation) to pause the main process.
My program basically iterates through images, opening and closing them utilizing a 3-second time-loop for each iteration.
I'm trying to use threading to interject a time.sleep(20) if a certain condition is met (x == True). The condition is being met (evident by the output of the print statement), but time.sleep(20) is not affecting the main process.
I plan to subsitute time.sleep(20) with a more complex process but for simpliclity I've used it here.
import time
import subprocess
import pickle
import keyboard
import threading
from threading import Thread
import multiprocessing
import queue
import time
with open('C:\\Users\Moondra\\Bioteck.pickle', 'rb') as file:
bio = pickle.load(file)
q = queue.LifoQueue(0)
def keyboard_press(): # This is just receiving boolean values based on key presses
while True:
q.put(keyboard.is_pressed('down'))
x = q.get()
print(x)
if x == True:
time.sleep(20)
t = Thread(target = keyboard_press, args= ())
t.start()
if __name__ == "__main__":
for i in bio[:5]:
p = subprocess.Popen(["C:\Program Files\IrfanView\i_view64.exe",'C:\\Users\Moondra\\Bioteck_charts\{}.png'.format(i)])
time.sleep(3)
p.kill()
So why isn't my thread affecting my main process?
Thank you.
Update:
So It seems I have to use flags and use flag as a global variable within my function. I would like to avoid using global but it's not working without globalizing flag within my function.
Second, I don't know how to restart the thread.
Once the thread returns the flag as false, the thread sort of just stalls.
I tried starting the thread again, with t.start, but I received the error:
RuntimeError: threads can only be started once
Here is updated code:
def keyboard_press():
while True:
global flag
q.put(keyboard.is_pressed('down'))
x = q.get()
print(x)
if x == True:
flag = False
#print('keyboard_flag is',flag)
return flag
if __name__ == "__main__":
flag = True
q = queue.LifoQueue(0)
t = Thread(target = keyboard_press, args= ())
t.start()
for i in bio[:5]:
p = subprocess.Popen(["C:\Program Files\IrfanView\i_view64.exe",'C:\\Users\Moondra\\Bioteck_charts\{}.png'.format(i)])
time.sleep(3)
print ('flag is',flag)
if flag == True:
p.kill()
else:
time.sleep(20)
p.kill()
flag = True
#t.start() #doesn't seem to work.
why isn't my thread affecting my main process?
Because you have not written any code to be executed by the keyboard_press() thread that would affect the main process.
It looks like you're trying to create a slide show that shows one image every three seconds, and you want it to pause for an extra twenty seconds when somebody presses a key. Is that right?
So, you've got one thread (the main thread) that runs the slide show, and you've got another that polls the keyboard, but your two threads don't communicate with one another.
You put a time.sleep(20) call in your keyboard thread. But that only pauses the keyboard thread. It doesn't do anything at all to the main thread.
What you need, is for the keyboard thread to set a variable that the main thread looks at after it wakes up from its three second sleep. The main thread can look at the variable, and see if a longer sleep has been requested, and if so, sleep for twenty more seconds.
Of course, after the longer sleep, you will want the main thread to re-set the variable so that it won't always sleep for twenty seconds after the first time the keyboard is touched.
P.S.: I am not a Python expert. I know that in other programming environments (e.g., Java), you also have to worry about "memory visibility." That is, when a variable is changed by one thread, there is no guarantee of when (if ever) some other thread will see the change...
...Unless, the threads use some kind of synchronization when they access the variable.
Based on what I have read (It's on the Internet! It must be true!), Python either does not have that problem now, or it did not have that problem in the recent past. I'm not sure which.
If memory consistency actually is an issue, then you will either have to use a mutex when you access the shared variable, or else you will have to make the threads communicate through some kind of a synchronized object such as a queue.