I have two relay boards, each works on a separate serial port. I am trying to build a script so that the individual boards accept commands in their own thread. I want to somehow store the command queue and communication thread based on this device's ID (in a keyed dict) to target it for actions. The original script I'm using for a guide is 650 lines so I haven't posted it here. I'm in over my head a bit, can anyone make some suggestions?
import queue
class easydaq(object):
def __init__(self):
self.commQueue = {"/dev/ttyUSB0": None, "/dev/ttyUSB1": None}
q = queue.Queue
def _getDevCommQueue(self, dev_id):
return (self.commQueue.get(dev_id, 0))
def _sendToDevCommQueue(self, dev_id, action):
q = self._getDevCommQueue(dev_id)
q.put(action)
dev_id = "/dev/ttyUSB0"
action = b'C\x01'
my_new_instance = easydaq()
my_new_instance._sendToDevCommQueue(dev_id, action)
Related
First, I'd like to thank the StackOverflow community for the tremendous help it provided me over the years, without me having to ask a single question.
I could not find anything that I can relate to my problem, though it is probably due to my lack of understanding of the subject, rather than the absence of a response on the website. My apologies in advance if this is a duplicate.
I am relatively new to multiprocess; some time ago I succeeded in using multiprocessing.pools in a very simple way, where I didn't need any feedback between the child processes.
Now I am facing a much more complicated problem, and I am just lost in the documentation about multiprocessing. I hence ask for you help, your kindness and your patience.
I am trying to build a parallel tempering monte-carlo algorithm, from a class.
The basic class very roughly goes as follows:
import numpy as np
class monte_carlo:
def __init__(self):
self.x=np.ones((1000,3))
self.E=np.mean(self.x)
self.Elist=[]
def simulation(self,temperature):
self.T=temperature
for i in range(3000):
self.MC_step()
if i%10==0:
self.Elist.append(self.E)
return
def MC_step(self):
x=self.x.copy()
k = np.random.randint(1000)
x[k] = (x[k] + np.random.uniform(-1,1,3))
temp_E=np.mean(self.x)
if np.random.random()<np.exp((self.E-temp_E)/self.T):
self.E=temp_E
self.x=x
return
Obviously, I simplified a great deal (actual class is 500 lines long!), and built fake functions for simplicity: __init__ takes a bunch of parameters as arguments, there are many more lists of measurement else than self.Elist, and also many arrays derived from self.X that I use to compute them. The key point is that each instance of the class contains a lot of informations that I want to keep in memory, and that I don't want to copy over and over again, to avoid dramatic slowing down. Else I would just use the multiprocessing.pool module.
Now, the parallelization I want to do, in pseudo-code:
def proba(dE,pT):
return np.exp(-dE/pT)
Tlist=[1.1,1.2,1.3]
N=len(Tlist)
G=[]
for _ in range(N):
G.append(monte_carlo())
for _ in range(5):
for i in range(N): # this loop should be ran in multiprocess
G[i].simulation(Tlist[i])
for i in range(N//2):
dE=G[i].E-G[i+1].E
pT=G[i].T + G[i+1].T
p=proba(dE,pT) # (proba is a function, giving a probability depending on dE)
if np.random.random() < p:
T_temp = G[i].T
G[i].T = G[i+1].T
G[i+1].T = T_temp
Synthesis: I want to run several instances of my monte-carlo class in parallel child processes, with different values for a parameter T, then periodically pause everything to change the different T's, and run again the child processes/class instances, from where they paused.
Doing this, I want each class-instance/child-process to stay independent from one another, save its current state with all internal variables while it is paused, and do as few copies as possible. This last point is critical, as the arrays inside the class are quite big (some are 1000x1000), and a copy will therefore very quickly become quite time-costly.
Thanks in advance, and sorry if I am not clear...
Edit:
I am using a distant machine with many (64) CPUs, running on Debian GNU/Linux 10 (buster).
Edit2:
I made a mistake in my original post: in the end, the temperatures must be exchanged between the class-instances, and not inside the global Tlist.
Edit3: Charchit answer works perfectly for the test code, on both my personal machine and the distant machine I am usually using for running my codes. I hence check this as the accepted answer.
However, I want to report here that, inserting the actual, more complicated code, instead of the oversimplified monte_carlo class, the distant machine gives me some strange errors:
Unable to init server: Could not connect: Connection refused
(CMC_temper_all.py:55509): Gtk-WARNING **: ##:##:##:###: Locale not supported by C library.
Using the fallback 'C' locale.
Unable to init server: Could not connect: Connection refused
(CMC_temper_all.py:55509): Gdk-CRITICAL **: ##:##:##:###:
gdk_cursor_new_for_display: assertion 'GDK_IS_DISPLAY (display)' failed
(CMC_temper_all.py:55509): Gdk-CRITICAL **: ##:##:##:###: gdk_cursor_new_for_display: assertion 'GDK_IS_DISPLAY (display)' failed
The "##:##:##:###" are (or seems like) IP adresses.
Without the call to set_start_method('spawn') this error shows only once, in the very beginning, while when I use this method, it seems to show at every occurrence of result.get()...
The strangest thing is that the code seems otherwise to work fine, does not crash, produces the datafiles I then ask it to, etc...
I think this would deserve to publish a new question, but I put it here nonetheless in case someone has a quick answer.
If not, I will resort to add one by one the variables, methods, etc... that are present in my actual code but not in the test example, to try and find the origin of the bug. My best guess for now is that the memory space required by each child-process with the actual code, is too large for the distant machine to accept it, due to some restrictions implemented by the admin.
What you are looking for is sharing state between processes. As per the documentation, you can either create shared memory, which is restrictive about the data it can store and is not thread-safe, but offers better speed and performance; or you can use server processes through managers. The latter is what we are going to use since you want to share whole objects of user-defined datatypes. Keep in mind that using managers will impact speed of your code depending on the complexity of the arguments that you pass and receive, to and from the managed objects.
Managers, proxies and pickling
As mentioned, managers create server processes to store objects, and allow access to them through proxies. I have answered a question with better details on how they work, and how to create a suitable proxy here. We are going to use the same proxy defined in the linked answer, with some variations. Namely, I have replaced the factory functions inside the __getattr__ to something that can be pickled using pickle. This means that you can run instance methods of managed objects created with this proxy without resorting to using multiprocess. The result is this modified proxy:
from multiprocessing.managers import NamespaceProxy, BaseManager
import types
import numpy as np
class A:
def __init__(self, name, method):
self.name = name
self.method = method
def get(self, *args, **kwargs):
return self.method(self.name, args, kwargs)
class ObjProxy(NamespaceProxy):
"""Returns a proxy instance for any user defined data-type. The proxy instance will have the namespace and
functions of the data-type (except private/protected callables/attributes). Furthermore, the proxy will be
pickable and can its state can be shared among different processes. """
def __getattr__(self, name):
result = super().__getattr__(name)
if isinstance(result, types.MethodType):
return A(name, self._callmethod).get
return result
Solution
Now we only need to make sure that when we are creating objects of monte_carlo, we do so using managers and the above proxy. For that, we create a class constructor called create. All objects for monte_carlo should be created with this function. With that, the final code looks like this:
from multiprocessing import Pool
from multiprocessing.managers import NamespaceProxy, BaseManager
import types
import numpy as np
class A:
def __init__(self, name, method):
self.name = name
self.method = method
def get(self, *args, **kwargs):
return self.method(self.name, args, kwargs)
class ObjProxy(NamespaceProxy):
"""Returns a proxy instance for any user defined data-type. The proxy instance will have the namespace and
functions of the data-type (except private/protected callables/attributes). Furthermore, the proxy will be
pickable and can its state can be shared among different processes. """
def __getattr__(self, name):
result = super().__getattr__(name)
if isinstance(result, types.MethodType):
return A(name, self._callmethod).get
return result
class monte_carlo:
def __init__(self, ):
self.x = np.ones((1000, 3))
self.E = np.mean(self.x)
self.Elist = []
self.T = None
def simulation(self, temperature):
self.T = temperature
for i in range(3000):
self.MC_step()
if i % 10 == 0:
self.Elist.append(self.E)
return
def MC_step(self):
x = self.x.copy()
k = np.random.randint(1000)
x[k] = (x[k] + np.random.uniform(-1, 1, 3))
temp_E = np.mean(self.x)
if np.random.random() < np.exp((self.E - temp_E) / self.T):
self.E = temp_E
self.x = x
return
#classmethod
def create(cls, *args, **kwargs):
# Register class
class_str = cls.__name__
BaseManager.register(class_str, cls, ObjProxy, exposed=tuple(dir(cls)))
# Start a manager process
manager = BaseManager()
manager.start()
# Create and return this proxy instance. Using this proxy allows sharing of state between processes.
inst = eval("manager.{}(*args, **kwargs)".format(class_str))
return inst
def proba(dE,pT):
return np.exp(-dE/pT)
if __name__ == "__main__":
Tlist = [1.1, 1.2, 1.3]
N = len(Tlist)
G = []
# Create our managed instances
for _ in range(N):
G.append(monte_carlo.create())
for _ in range(5):
# Run simulations in the manager server
results = []
with Pool(8) as pool:
for i in range(N): # this loop should be ran in multiprocess
results.append(pool.apply_async(G[i].simulation, (Tlist[i], )))
# Wait for the simulations to complete
for result in results:
result.get()
for i in range(N // 2):
dE = G[i].E - G[i + 1].E
pT = G[i].T + G[i + 1].T
p = proba(dE, pT) # (proba is a function, giving a probability depending on dE)
if np.random.random() < p:
T_temp = Tlist[i]
Tlist[i] = Tlist[i + 1]
Tlist[i + 1] = T_temp
print(Tlist)
This meets the criteria you wanted. It does not create any copies at all, rather, all arguments to the simulation method call are serialized inside the pool and sent to the manager server where the object is actually stored. It gets executed there, and the results (if any) are serialized and returned in the main process. All of this, with only using the builtins!
Output
[1.2, 1.1, 1.3]
Edit
Since you are using Linux, I encourage you to use multiprocessing.set_start_method inside the if __name__ ... clause to set the start method to "spawn". Doing this will ensure that the child processes do not have access to variables defined inside the clause.
is it possible in Dart to instantiate a class in an isolate, and then send message to this isolate to receive a return value from its methods (instead of spawning a new isolate and re instantiate the same class every time)? I have a class with a long initialization, and heavy methods. I want to initialize it once and then access its methods without compromising the performance of the main isolate.
Edit: I mistakenly answered this question thinking python rather than dart. snakes on the brain / snakes on a plane
I am not familiar with dart programming, but it would seem the concurrency model has a lot of similarities (isolated memory, message passing, etc..). I was able to find an example of 2 way message passing with a dart isolate. There's a little difference in how it gets set-up, and the streams are a bit simpler than python Queue's, but in general the idea is the same.
Basically:
Create a port to receive data from the isolate
Create the isolate passing it the port it will send data back on
Within the isolate, create the port it will listen on, and send the other end of it back to main (so main can send messages)
Determine and implement a simple messaging protocol for remote method call on an object contained within the isolate.
This is basically duplicating what a multiprocessing.Manager class does, however it can be helpful to have a simplified example of how it can work:
from multiprocessing import Process, Lock, Queue
from time import sleep
class HeavyObject:
def __init__(self, x):
self._x = x
sleep(5) #heavy init
def heavy_method(self, y):
sleep(.2) #medium weight method
return self._x + y
def HO_server(in_q, out_q):
ho = HeavyObject(5)
#msg format for remote method call: ("method_name", (arg1, arg2, ...), {"kwarg1": 1, "kwarg2": 2, ...})
#pass None to exit worker cleanly
for msg in iter(in_q.get, None): #get a remote call message from the queue
out_q.put(getattr(ho, msg[0])(*msg[1], **msg[2])) #call the method with the args, and put the result back on the queue
class RMC_helper: #remote method caller for convienience
def __init__(self, in_queue, out_queue, lock):
self.in_q = in_queue
self.out_q = out_queue
self.l = lock
self.method = None
def __call__(self, *args, **kwargs):
if self.method is None:
raise Exception("no method to call")
with self.l: #isolate access to queue so results don't pile up and get popped off in possibly wrong order
print("put to queue: ", (self.method, args, kwargs))
self.in_q.put((self.method, args, kwargs))
return self.out_q.get()
def __getattr__(self, name):
if not name.startswith("__"):
self.method = name
return self
else:
super().__getattr__(name)
def child_worker(remote):
print("child", remote.heavy_method(5)) #prints 10
sleep(3) #child works on something else
print("child", remote.heavy_method(2)) #prints 7
if __name__ == "__main__":
in_queue = Queue()
out_queue = Queue()
lock = Lock() #lock is used as to not confuse which reply goes to which request
remote = RMC_helper(in_queue, out_queue, lock)
Server = Process(target=HO_server, args=(in_queue, out_queue))
Server.start()
Worker = Process(target=child_worker, args=(remote, ))
Worker.start()
print("main", remote.heavy_method(3)) #this will *probably* start first due to startup time of child
Worker.join()
with lock:
in_queue.put(None)
Server.join()
print("done")
I have been searching for days for a solution to this issue, if anyone can point me in the right direction, I will be so grateful.
I have been writing a program that allows a user to create multiple trading bots at the same time, each bot is created as an individual thread that makes an api call to Binance for the latest market data and evaluates this data with conditional statements and place trades accordingly. The code I am using for this has been trimmed to only the essential parts for simplicity.. I have a Bot class...
class Bot(threading.Thread):
def __init__(self, symbol, time, exposure, strategy):
threading.Thread.__init__(self)
self.symbol = symbol
self.time = time
self.exposure = exposure
self.strategy = strategy
self.stop_bot = False
def scanner(self):
while self.stop_bot == False:
client = Client(self.api_key, self.secret_key, {"verify": True, "timeout": 20})
price_data = client.get_klines(symbol=self.symbol, interval=self.time_interval)
self.df = pd.DataFrame(price_data)
self.modified_df = pd.DataFrame(price_data)
time.sleep(10)
def kill(self):
self.stop_bot = True
And the Bot class is called from the bot manager terminal class...
class Bot_Manager:
def __init__(self):
self.bot_list = []
bot = object
def terminal(self):
while True:
user_input = input('(1) Create new Bot (2) Stop Bot')
if user_input == '1':
symbol = 'OMGUSDT'
time = '1D'
exposure = 1000
strategy = 'SAR'
bot_name = input('name your bot: ')
bot = Bot(symbol=symbol, time=time, exposure=exposure, strategy=strategy, bot_name=bot_name)
scanner_thread = threading.Thread(target=bot.scanner, daemon=True)
scanner_thread.name = bot_name
scanner_thread.start()
self.bot_list.append(scanner_thread)
elif user_input == '2':
for thread in threading.enumerate():
print(thread.name)
print(self.bot_list)
user_input = input('Bot to stop: ')
i = int(user_input)
print(self.bot_list[i])
Now I am able to create multiple threads / bots by repeatedly selecting option 1. However the issue I am facing is stopping these threads when a user selects option 2. If for example I create 3 bots and name them Bot A, Bot B and Bot C.. when I enumerate this in a for loop, i get the following:
MainThread
Bot A
Bot B
Bot C
and when I store each thread into a list and print the list I see the following:
[<Thread(Bot A, started 8932)>, <Thread(Bot B, started 12268)>, <Thread(Bot C, started 13436)>]
I would like the user to be able to select the thread / bot they want to stop from the list, so in this example if the user types 1, it should return the thread <Thread(Bot B, started 12268)>and stop this thread by passing the variable stop_bot = True. However I haven't had much luck with this.
When I call the function bot.kill() it only stops the last thread that was created, so for this example, Bot C. When it runs again, it doesn't remove any other thread. Is there anyway to pass in the variable stop_bot = True on an already created object / thread? Or is there another method to this that I have overlooked... Any help would be greatly appreciated....
I managed to find a solution to this by adding multiple bots to a data frame and passing this into the bot class. 1 thread was created to iterate through the data frame and by removing rows of the data frame, the bots would be removed.
I've been reading up on the Observer Design Pattern.
Take the below code from the following article:
https://sourcemaking.com/design_patterns/observer/python/1
I understand how the code works and how the two classes, subject and observer, interrelate. What I can't quite get my head round is how this would all work in reality.
In the real world examples of the observer pattern I have read about, the interaction between the subject and observer tends to be open ended.
For example, if you subscribe to a question on Quora and get notified of answers via email, theoretically, you could receive updates indefinitely.
How then if the code below was applied to a real world scenario (like Quora) do the classes persist? Are the classes and their states stored somewhere on the server?
import abc
class Subject:
"""
Know its observers. Any number of Observer objects may observe a
subject.
Send a notification to its observers when its state changes.
"""
def __init__(self):
self._observers = set()
self._subject_state = None
def attach(self, observer):
observer._subject = self
self._observers.add(observer)
def detach(self, observer):
observer._subject = None
self._observers.discard(observer)
def _notify(self):
for observer in self._observers:
observer.update(self._subject_state)
#property
def subject_state(self):
return self._subject_state
#subject_state.setter
def subject_state(self, arg):
self._subject_state = arg
self._notify()
class Observer(metaclass=abc.ABCMeta):
"""
Define an updating interface for objects that should be notified of
changes in a subject.
"""
def __init__(self):
self._subject = None
self._observer_state = None
#abc.abstractmethod
def update(self, arg):
pass
class ConcreteObserver(Observer):
"""
Implement the Observer updating interface to keep its state
consistent with the subject's.
Store state that should stay consistent with the subject's.
"""
def update(self, arg):
self._observer_state = arg
# ...
def main():
subject = Subject()
concrete_observer = ConcreteObserver()
subject.attach(concrete_observer)
subject.subject_state = 123
if __name__ == "__main__":
main()
You do not need to persist the classes. All you need to do is persist the data. In your real-world example, Quora just stores the list of subscribers for each questions. If there is any activity, it just needs to call some Notification service, passing along the list, which eventually will send the mail to subscribed users.
Need some help please to explain why the following does not work.
Environment: Python 3.4, Gtk3.0, limited experience of Python
File selectcontact.py contains code to select one of a number of records and pass its key back to its parent process for use in one of at least three other actions.
Code snippet from the parent class:
….
self.cindex = 0
….
def editcontact_clicked (self, menuitem):
import selectcontact
selectcontact.SelectContactGUI(self)
print ('From Manage ', self.cindex)
if self.cindex > 0:
import editcontact
editcontact.EditContactGUI(self.db, self.cindex)
….
Code snippet from selectcontact:
class SelectContactGUI:
def init(self, parent_class):
self.builder = Gtk.Builder()
self.builder.add_from_file(UI_FILE)
self.builder.connect_signals(self)
self.parent_class = parent_class
self.db = parent_class.db
self.cursor = self.db.cursor(cursor_factory = psycopg2.extras.NamedTupleCursor)
self.contact_store = self.builder.get_object('contact_store')
self.window = self.builder.get_object('window1')
self.window.show_all()
def select_contact_path(self, path):
self.builder.get_object('treeview_selection1').select_path(path)
def contact_treerow_changed (self, treeview):
selection = self.builder.get_object('treeview_selection1')
model, path = selection.get_selected()
if path != None:
self.parent_class.cindex = model[path][0]
print ('From select ', self.parent_class.cindex)
self.window.destroy()
….
window1 is declared as “modal”, so I was expecting the call to selectcontact to act as a subroutine, so that editcontact wouldn’t be called until control was passed back to the parent. The parent_class bit works because the contact_store is correctly populated. However the transfer back to the parent appears not to work, and the two print statements occur in the wrong order:
From Manage 0
From select 2
Comments gratefully received.
Graeme
"Modal" refers to windows only. That is, a modal window prevents accessing the parent window.
It has little to do with what code is running. I am not familiar with this particular windowing framework, but any I have worked with has had a separate thread for GUI and at least one for processing, to keep the GUI responsive, and message loops running in all active windows, not just the one currently with the focus. The modal dialog has no control over what code in other threads are executed when.
You should be able to break into the debugger and see what threads are running and what is running in each thread at any given time.