im using multithreading in python3 with Flask as below.
Would like to know if there is any issue in below code, and if this is efficient way of using threads
import _thread
COUNT = 0
class Myfunction(Resource):
#staticmethod
def post():
global GLOBAL_COUNT
logger = logging.getLogger(__name__)
request_json = request.get_json()
logger.info(request_json)
_thread.start_new_thread(Myfunction._handle_req, (COUNT, request_json))
COUNT += 1
return Response("Request Accepted", status=202, mimetype='application/json')
#staticmethod
def _handle_req(thread_id, request_json):
with lock:
empID = request_json.get("empId", "")
myfunction2(thread_id,empID)
api.add_resource(Myfunction, '/Myfunction')
I think the newer module threading would be better suited for python 3. Its more powerful.
import threading
threading.Thread(target=some_callable_function).start()
or if you wish to pass arguments
threading.Thread(target=some_callable_function,
args=(tuple, of, args),
kwargs={'dict': 'of', 'keyword': 'args'},
).start()
Unless you specifically need _thread for backwards compatibility. Not specifically related to how efficient your code is but good to know anyways.
see What happened to thread.start_new_thread in python 3 and https://www.tutorialspoint.com/python3/python_multithreading.htm
Related
This is Python 3.10. I use a PriorityQueue as a way to track Actors' turn order in my game. It's just a simple roguelike. I don't use the synchronization features of the PriorityQueue. My code:
import pickle
from queue import PriorityQueue
class GameEngine():
def __init__(self):
self.pqueue = PriorityQueue()
def save_to_file(self):
with open('save.pkl', 'wb') as file:
pickle.dump(self, file, pickle.HIGHEST_PROTOCOL)
class Monster():
pass
engine = GameEngine()
orc1 = Monster()
orc2 = Monster()
engine.pqueue.put((20,orc1))
engine.pqueue.put((10,orc2))
engine.save_to_file()
It returns TypeError: cannot pickle '_thread.lock' object. From what I understand PriorityQueue is not pickle-able. I've read here that Queue.Queue has a pickle-able alternative of collections.deque if the synchronization stuff is not necessary. Is there such an alternative to PriorityQueue, or is there a way to pickle it anyway? Other than implementing my own simplified version of PriorityQueue?
As you don't need the synchronisation features of PriorityQueue, just use the light-weight heapq module. It provides functions (not methods) to work on a plain list:
import pickle
from heapq import heappush, heappop
class GameEngine():
def __init__(self):
self.pqueue = []
def save_to_file(self):
with open('save.pkl', 'wb') as file:
pickle.dump(self, file, pickle.HIGHEST_PROTOCOL)
class Monster():
pass
engine = GameEngine()
orc1 = Monster()
orc2 = Monster()
heappush(engine.pqueue, (20,orc1))
heappush(engine.pqueue, (10,orc2))
engine.save_to_file()
I am trying to use multiprocessing for the below code. The code seems to run a bit faster than the for loop inside the function.
How can I confirm I using the library and not the just the for loop?
from multiprocessing import Pool
from multiprocessing import cpu_count
import requests
import pandas as pd
data= pd.read_csv('~/Downloads/50kNAE000.txt.1' ,sep="\t", header=None)
data = data[0].str.strip("0 ")
lst = []
def request(x):
for i,v in x.items():
print(i)
file = requests.get(v)
lst.append(file.text)
#time.sleep(1)
if __name__ == "__main__":
pool = Pool(cpu_count())
results = pool.map(request(data))
pool.close() # 'TERM'
pool.join() # 'KILL'
Multiprocessing has overhead. It has to start the process and transfer function data via interprocess mechanism. Just running a single function in another process vs. running that same function normally is always going to be slower. The advantage is actually doing parallelism with significant work in the functions that makes the overhead minimal.
You can call multiprocessing.current_process().name to see the process name change.
I am trying to update a variable of a class by calling a function of the class from a different function which is being run on multi-process.
To achieve the desired result, process (p1) needs to update the variable "transaction" and which should get then modified by process (p2)
I tried the below code and I know i should use Multiprocess.value or manager to achieve the desired result and I am not sure of how to do it as my variable to be updated is in another class
Below is the code:
from multiprocessing import Process
from helper import Helper
camsource = ['a','b']
Pros = []
def sub(i):
HC.trail_func(i)
def main():
for i in camsource:
print ("Camera Thread {} Started!".format(i))
p = Process(target=sub, args=(i))
Pros.append(p)
p.start()
# block until all the threads finish (i.e. block until all function_x calls finish)
for t in Pros:
t.join()
if __name__ == "__main__":
HC = Helper()
main()
Here is the helper code:
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
class Helper():
def __init__(self):
self.transactions = []
def trail_func(self,preview):
if preview == 'a':
self.transactions.append({"Apple":1})
else:
if self.transactions[0]['Apple'] == 1:
self.transactions[0]['Apple'] = self.transactions[0]['Apple'] + 1
print (self.transactions)
Desired Output:
p1:
transactions = {"Apple":1}
p2:
transactions = {"Apple":2}
I've recently released this module that can help you with your code, as all data frames (data models that can hold any type of data), have locks on them, in order to solve concurrency issues. Anyway, take a look at the README file and the examples.
I've made an example here too, if you'd like to check.
I'm getting very familiar with python's asyncio, the asynchronous programming in python, co-routines etc.
I want to be able to executing several co-routines with my own custom made eventloop.
I'm curious if i can write my own eventloop without importing asyncio at all
I want to be able to executing several co-routines with my own custom made eventloop.
The asyncio event loop is well-tested and can be easily extended to acknowledge non-asyncio events. If you describe the actual use case, it might be easier to help. But if your goal is to learn about async programming and coroutines, read on.
I'm curious if i can write my own eventloop without importing asyncio at all
It's definitely possible - asyncio itself is just a library, after all - but it will take some work for your event loop to be useful. See this excellent talk by David Beazley where he demonstrates writing an event loop in front of a live audience. (Don't be put off by David using the older yield from syntax - await works exactly the same way.)
Ok, so i found an example somewhere (sorry, don't remember where, no link), and changed a little bit.
An eventloop and co-routins without even importing asyncio:
import datetime
import heapq
import types
import time
class Task:
def __init__(self, wait_until, coro):
self.coro = coro
self.waiting_until = wait_until
def __eq__(self, other):
return self.waiting_until == other.waiting_until
def __lt__(self, other):
return self.waiting_until < other.waiting_until
class SleepingLoop:
def __init__(self, *coros):
self._new = coros
self._waiting = []
def run_until_complete(self):
# Start all the coroutines.
for coro in self._new:
wait_for = coro.send(None)
heapq.heappush(self._waiting, Task(wait_for, coro))
# Keep running until there is no more work to do.
while self._waiting:
now = datetime.datetime.now()
# Get the coroutine with the soonest resumption time.
task = heapq.heappop(self._waiting)
if now < task.waiting_until:
# We're ahead of schedule; wait until it's time to resume.
delta = task.waiting_until - now
time.sleep(delta.total_seconds())
now = datetime.datetime.now()
try:
# It's time to resume the coroutine.
wait_until = task.coro.send(now)
heapq.heappush(self._waiting, Task(wait_until, task.coro))
except StopIteration:
# The coroutine is done.
pass
#types.coroutine
def async_sleep(seconds):
now = datetime.datetime.now()
wait_until = now + datetime.timedelta(seconds=seconds)
actual = yield wait_until
return actual - now
async def countdown(label, total_seconds_wait, *, delay=0):
print(label, 'waiting', delay, 'seconds before starting countdown')
delta = await async_sleep(delay)
print(label, 'starting after waiting', delta)
while total_seconds_wait:
print(label, 'T-minus', total_seconds_wait)
waited = await async_sleep(1)
total_seconds_wait -= 1
print(label, 'lift-off!')
def main():
loop = SleepingLoop(countdown('A', 5, delay=0),
countdown('B', 3, delay=2),
countdown('C', 4, delay=1))
start = datetime.datetime.now()
loop.run_until_complete()
print('Total elapsed time is', datetime.datetime.now() - start)
if __name__ == '__main__':
main()
I want to implement GUI using threading and tkinter (python 3.6).
When I run GUIExecution.py, the following error occurs.
"RuntimeError: Calling Tcl from different appartment" on self.root.mainloop() in base_gui_class.py
I am implementing it on a class basis, and the three code files are as follows.
The executable file is GUIExecution.py.
I spent a lot of time trying to fix the error, but I have not been able to fix it yet.
Please give a lot of advice.
Additionally, if I run the following code in a python2 environment, it works fine without error.
GUIExecution.py
from base_gui_class import *
from base_class import *
speed = 1000
height = 500
width = 700
base_model = base_class()
gui = base_gui_class(base_model, speed, height, width)
base_model.visualize()
base_class.py
class base_class():
genes = []
dicLocations = {}
gui = ''
best = ''
time = 0
def __init__(self):
pass
def visualize(self):
if self.gui != '':
self.gui.start()
def registerGUI(self, gui):
self.gui = gui
base_gui_class.py
import threading
import tkinter as tk
import math
import threading
import time
class base_gui_class(threading.Thread):
root = ''
canvas = ''
speed = 0
base_model = ''
def __init__(self, base_model, speed, h, w):
threading.Thread.__init__(self)
self.base_model = base_model
base_model.registerGUI(self)
self.root = tk.Tk()
self.canvas = tk.Canvas(self.root, height=h, width=w)
self.canvas.pack()
self.root.title("Test")
self.speed = 1 / speed
def run(self):
self.root.mainloop()
def update(self):
time.sleep(self.speed)
width = int(self.canvas.cget("width"))
height = int(self.canvas.cget("height"))
self.canvas.create_rectangle(0, 0, width, height, fill='white')
def stop(self):
self.root.quit()
To a very good first and second approximation, the core of Tk is single threaded. It can be used from multiple threads, but only by initialising it separately in each of those threads. Internally, it uses thread-specific variables extensively to avoid the need for major locking (that is, it has nothing like a big Global Interpreter Lock) but that means you must not cheat. Whatever thread initialises a Tk context must be the only thread that interacts with that Tk context. This includes loading the Tkinter module so you are effectively restricted to using Tkinter from your main thread only; working around this is serious expert's-only stuff.
I recommend that you make your worker threads make changes to your GUI by posting events to it using a queue (or otherwise interlock with critical sections and condition variables, though I find queues easier in practice).
pip install tkthread
#call the function which shows error like this:
tkthread.call_nosync(yourfunction)
this tkthread library handles all the threading internally by itself.
I recommend you reading the documentation of this library:https://pypi.org/project/tkthread/ .