First off, I'm relatively new to Python, so pardon the rough code I have. Here goes: I have a class with a constructor. It inherits a few properties from a superclass. This same class can also be a child object of itself. So an Epic can hold an Epic, which I'm appending to the Children property as a list.
class Epic(V1Superclass):
Type = 'Epic'
Children = []
ParentAssetID = []
#classmethod
def __init__(self, **kwargs):
for key, value in kwargs.items():
if key == 'Name':
self.Name = value
elif key == 'Description':
self.Description = value
elif key == 'Super':
self.Super = value
elif key == 'Scope':
self.Scope = value
if not bool(self.Super):
self.Super = None
if not bool(self.Scope):
self.Scope = 445082
def populateChildren(self,env,key):
children = V1.utl.getChildren(env,key,'Epic',self.AssetID)
for child in children:
if child['_oid'].split(':')[0] == 'Story':
print('Got another backlog child')
elif child['_oid'].split(':')[0] == 'Defect':
print('Got another Defect child')
elif child['_oid'].split(':')[0] == 'Epic':
childEpic = V1.Epic()
self.Children.append(childEpic)
If I create two instances of this class in the python console, everything's fine. I do "a = V1.Epic()" and "b = V1.Epic()", and the world is good. They all initialize to the proper default values (or empty). However, when I run my code in the populateChildren function, when I instantiate a new Epic object, rather than create a default version of the Epic, it is creating a new instance, but with all the properties of the parent (self) object. In essence, it's an exact copy, but if I do a "self is childEpic" command, it returns false, which (if I understand things correctly) means that childEpic is not a copy of the parent object. I can manipulate the child object and set properties with no problem, but obviously that's not how this should work. This is kind of maddening, as I'm not even sure what to google to see what I'm doing wrong syntax wise.
TIA!
I've tried adding an addChild function which tries to instantiate an instance outside the scope of the parent object, but even within that function, the object created is a duplicate of the parent
Your __init__ function is defined as a class method, which means that its first parameter is not a reference to the instance of the new object, but instead a reference to the current class. These properties are present on each of the class's instances, so it appears that they are being copied over.
Remove the #classmethod decorator from the __init__ function and the self parameter will refer to the object being instantiated.
First, I'd like to thank the StackOverflow community for the tremendous help it provided me over the years, without me having to ask a single question.
I could not find anything that I can relate to my problem, though it is probably due to my lack of understanding of the subject, rather than the absence of a response on the website. My apologies in advance if this is a duplicate.
I am relatively new to multiprocess; some time ago I succeeded in using multiprocessing.pools in a very simple way, where I didn't need any feedback between the child processes.
Now I am facing a much more complicated problem, and I am just lost in the documentation about multiprocessing. I hence ask for you help, your kindness and your patience.
I am trying to build a parallel tempering monte-carlo algorithm, from a class.
The basic class very roughly goes as follows:
import numpy as np
class monte_carlo:
def __init__(self):
self.x=np.ones((1000,3))
self.E=np.mean(self.x)
self.Elist=[]
def simulation(self,temperature):
self.T=temperature
for i in range(3000):
self.MC_step()
if i%10==0:
self.Elist.append(self.E)
return
def MC_step(self):
x=self.x.copy()
k = np.random.randint(1000)
x[k] = (x[k] + np.random.uniform(-1,1,3))
temp_E=np.mean(self.x)
if np.random.random()<np.exp((self.E-temp_E)/self.T):
self.E=temp_E
self.x=x
return
Obviously, I simplified a great deal (actual class is 500 lines long!), and built fake functions for simplicity: __init__ takes a bunch of parameters as arguments, there are many more lists of measurement else than self.Elist, and also many arrays derived from self.X that I use to compute them. The key point is that each instance of the class contains a lot of informations that I want to keep in memory, and that I don't want to copy over and over again, to avoid dramatic slowing down. Else I would just use the multiprocessing.pool module.
Now, the parallelization I want to do, in pseudo-code:
def proba(dE,pT):
return np.exp(-dE/pT)
Tlist=[1.1,1.2,1.3]
N=len(Tlist)
G=[]
for _ in range(N):
G.append(monte_carlo())
for _ in range(5):
for i in range(N): # this loop should be ran in multiprocess
G[i].simulation(Tlist[i])
for i in range(N//2):
dE=G[i].E-G[i+1].E
pT=G[i].T + G[i+1].T
p=proba(dE,pT) # (proba is a function, giving a probability depending on dE)
if np.random.random() < p:
T_temp = G[i].T
G[i].T = G[i+1].T
G[i+1].T = T_temp
Synthesis: I want to run several instances of my monte-carlo class in parallel child processes, with different values for a parameter T, then periodically pause everything to change the different T's, and run again the child processes/class instances, from where they paused.
Doing this, I want each class-instance/child-process to stay independent from one another, save its current state with all internal variables while it is paused, and do as few copies as possible. This last point is critical, as the arrays inside the class are quite big (some are 1000x1000), and a copy will therefore very quickly become quite time-costly.
Thanks in advance, and sorry if I am not clear...
Edit:
I am using a distant machine with many (64) CPUs, running on Debian GNU/Linux 10 (buster).
Edit2:
I made a mistake in my original post: in the end, the temperatures must be exchanged between the class-instances, and not inside the global Tlist.
Edit3: Charchit answer works perfectly for the test code, on both my personal machine and the distant machine I am usually using for running my codes. I hence check this as the accepted answer.
However, I want to report here that, inserting the actual, more complicated code, instead of the oversimplified monte_carlo class, the distant machine gives me some strange errors:
Unable to init server: Could not connect: Connection refused
(CMC_temper_all.py:55509): Gtk-WARNING **: ##:##:##:###: Locale not supported by C library.
Using the fallback 'C' locale.
Unable to init server: Could not connect: Connection refused
(CMC_temper_all.py:55509): Gdk-CRITICAL **: ##:##:##:###:
gdk_cursor_new_for_display: assertion 'GDK_IS_DISPLAY (display)' failed
(CMC_temper_all.py:55509): Gdk-CRITICAL **: ##:##:##:###: gdk_cursor_new_for_display: assertion 'GDK_IS_DISPLAY (display)' failed
The "##:##:##:###" are (or seems like) IP adresses.
Without the call to set_start_method('spawn') this error shows only once, in the very beginning, while when I use this method, it seems to show at every occurrence of result.get()...
The strangest thing is that the code seems otherwise to work fine, does not crash, produces the datafiles I then ask it to, etc...
I think this would deserve to publish a new question, but I put it here nonetheless in case someone has a quick answer.
If not, I will resort to add one by one the variables, methods, etc... that are present in my actual code but not in the test example, to try and find the origin of the bug. My best guess for now is that the memory space required by each child-process with the actual code, is too large for the distant machine to accept it, due to some restrictions implemented by the admin.
What you are looking for is sharing state between processes. As per the documentation, you can either create shared memory, which is restrictive about the data it can store and is not thread-safe, but offers better speed and performance; or you can use server processes through managers. The latter is what we are going to use since you want to share whole objects of user-defined datatypes. Keep in mind that using managers will impact speed of your code depending on the complexity of the arguments that you pass and receive, to and from the managed objects.
Managers, proxies and pickling
As mentioned, managers create server processes to store objects, and allow access to them through proxies. I have answered a question with better details on how they work, and how to create a suitable proxy here. We are going to use the same proxy defined in the linked answer, with some variations. Namely, I have replaced the factory functions inside the __getattr__ to something that can be pickled using pickle. This means that you can run instance methods of managed objects created with this proxy without resorting to using multiprocess. The result is this modified proxy:
from multiprocessing.managers import NamespaceProxy, BaseManager
import types
import numpy as np
class A:
def __init__(self, name, method):
self.name = name
self.method = method
def get(self, *args, **kwargs):
return self.method(self.name, args, kwargs)
class ObjProxy(NamespaceProxy):
"""Returns a proxy instance for any user defined data-type. The proxy instance will have the namespace and
functions of the data-type (except private/protected callables/attributes). Furthermore, the proxy will be
pickable and can its state can be shared among different processes. """
def __getattr__(self, name):
result = super().__getattr__(name)
if isinstance(result, types.MethodType):
return A(name, self._callmethod).get
return result
Solution
Now we only need to make sure that when we are creating objects of monte_carlo, we do so using managers and the above proxy. For that, we create a class constructor called create. All objects for monte_carlo should be created with this function. With that, the final code looks like this:
from multiprocessing import Pool
from multiprocessing.managers import NamespaceProxy, BaseManager
import types
import numpy as np
class A:
def __init__(self, name, method):
self.name = name
self.method = method
def get(self, *args, **kwargs):
return self.method(self.name, args, kwargs)
class ObjProxy(NamespaceProxy):
"""Returns a proxy instance for any user defined data-type. The proxy instance will have the namespace and
functions of the data-type (except private/protected callables/attributes). Furthermore, the proxy will be
pickable and can its state can be shared among different processes. """
def __getattr__(self, name):
result = super().__getattr__(name)
if isinstance(result, types.MethodType):
return A(name, self._callmethod).get
return result
class monte_carlo:
def __init__(self, ):
self.x = np.ones((1000, 3))
self.E = np.mean(self.x)
self.Elist = []
self.T = None
def simulation(self, temperature):
self.T = temperature
for i in range(3000):
self.MC_step()
if i % 10 == 0:
self.Elist.append(self.E)
return
def MC_step(self):
x = self.x.copy()
k = np.random.randint(1000)
x[k] = (x[k] + np.random.uniform(-1, 1, 3))
temp_E = np.mean(self.x)
if np.random.random() < np.exp((self.E - temp_E) / self.T):
self.E = temp_E
self.x = x
return
#classmethod
def create(cls, *args, **kwargs):
# Register class
class_str = cls.__name__
BaseManager.register(class_str, cls, ObjProxy, exposed=tuple(dir(cls)))
# Start a manager process
manager = BaseManager()
manager.start()
# Create and return this proxy instance. Using this proxy allows sharing of state between processes.
inst = eval("manager.{}(*args, **kwargs)".format(class_str))
return inst
def proba(dE,pT):
return np.exp(-dE/pT)
if __name__ == "__main__":
Tlist = [1.1, 1.2, 1.3]
N = len(Tlist)
G = []
# Create our managed instances
for _ in range(N):
G.append(monte_carlo.create())
for _ in range(5):
# Run simulations in the manager server
results = []
with Pool(8) as pool:
for i in range(N): # this loop should be ran in multiprocess
results.append(pool.apply_async(G[i].simulation, (Tlist[i], )))
# Wait for the simulations to complete
for result in results:
result.get()
for i in range(N // 2):
dE = G[i].E - G[i + 1].E
pT = G[i].T + G[i + 1].T
p = proba(dE, pT) # (proba is a function, giving a probability depending on dE)
if np.random.random() < p:
T_temp = Tlist[i]
Tlist[i] = Tlist[i + 1]
Tlist[i + 1] = T_temp
print(Tlist)
This meets the criteria you wanted. It does not create any copies at all, rather, all arguments to the simulation method call are serialized inside the pool and sent to the manager server where the object is actually stored. It gets executed there, and the results (if any) are serialized and returned in the main process. All of this, with only using the builtins!
Output
[1.2, 1.1, 1.3]
Edit
Since you are using Linux, I encourage you to use multiprocessing.set_start_method inside the if __name__ ... clause to set the start method to "spawn". Doing this will ensure that the child processes do not have access to variables defined inside the clause.
So I am getting used to working with OOP in python, it has been a bumpy road but so far things seem to be working. I have, however hit a snag and i cannot seem to figure this out. here is the premise.
I call a class and pass 2 variables to it, a report and location. From there, I need to take the location variable, pass it to a database and get a list of filters it is supposed to run through, and this is done through a dictionary call. Finally, once that dictionary call happens, i need to take that report and run it through the filters. here is the code i have.
class Filters(object):
def __init__ (self, report, location):
self.report = report
self.location = location
def get_location(self):
return self.location
def run(self):
cursor = con.cursor()
filters = cursor.execute(filterqry).fetchall()
for i in filters:
f = ReportFilters.fd.get(i[0])
f.run()
cursor.close()
class Filter1(Filters):
def __init__(self):
self.f1 = None
''' here is where i tried super() and Filters.__init__.() etc.... but couldn't make it work'''
def run(self):
'''Here is where i want to run the filters but as of now i am trying to print out the
location and the report to see if it gets the variables.'''
print(Filters.get_location())
class ReportFilters(Filters):
fd = {
'filter_1': Filter1(),
'filter_2': Filter2(),
'filter_3': Filter3()
}
My errors come from the dictionary call, as when i tried to call it as it is asking for the report and location variables.
Hope this is clear enough for you to help out with, as always it is duly appreciated.
DamnGroundHog
The call to its parent class should be defined inside the init function and you should pass the arguments 'self', 'report' and 'location' into init() and Filters.init() call to parent class so that it can find those variables.
If the error is in the Filters1 class object, when you try to use run method and you do not see a location or a report variable passed in from parent class, that is because you haven't defined them when you instantiated those object in ReportFilters.fd
It should be:
class ReportFilters(Filters):
fd = {
'filter_1': Filter1(report1, location1),
'filter_2': Filter2(report2, location2),
'filter_3': Filter3(report3, location3)
}
class Filter1(Filters):
def __init__(self, report, location):
Filters.__init__(self, report, location)
self.f1 = None
def run(self):
print(self.get_location())
I don't very understand the classes logic in python but cannot answer on web.
I have create a class to generate person info:
class person:
def fristnameGen(gender):
...
def emailGen(firstname,surname):
...
i create a bot to call it like this:
from person import *
class bots:
def __init__(self):
self.person = person()
def createDB(self):
print(self.person.name)
#do something...
finally i call it by a button with thread
from bots import *
import threading
class Panel:
def __init__(self):
self.top = tk.Tk()
self.bot = bots()
self.buildUI()
def foo(self):
self.bot.createDB(self.stringPhone.get())
def threadTheAction(func, *args):
t = threading.Thread(target=func, args=args)
t.setDaemon(True)
t.start()
def buildUI(self):
Button = tk.Button(self.top, text ="Start", command = lambda :self.threadTheAction(self.foo))
I get this error:
TypeError: 'Panel' object is not callable
However, i call it directly, its work
Button = tk.Button(self.top, text ="Start", command = lambda :self.foo())
How to fix the error?
...
2. Moreover, i tried create p1 = person() and p2= person() and print it. Found p1 and p2 is a same person, i prefer each new a class have a new one. How to generate "new person" using classes?
Thank you
You seem to have a lot of confusion about Object Oriented Programming in Python. Some of your methods have self parameters and some do not, seemingly at random. That's the source of your current error.
The threadTheAction method in your Panel class is getting the Panel instance passed in as its first argument, but that's named func in the method (since you omitted self). The real function you're passing as an argument gets caught in the variable argument *args. When the thread tries unsuccessfully to call it, you get an exception. Adding self before func would fix the immediate problem:
def threadTheAction(self, func, *args):
I suspect if your code got further along, you'd run into other errors with other methods without self in their parameter lists. For instance, none of the methods you've shown in person are likely to work correctly.
As for your second question, you haven't shown enough of person to know what's happening, but you're probably doing instance variables wrong somehow. With no self parameter in the methods, that's almost inevitable (since you assign to self.whatever to set a whatever attribute on the current instance). If you need help squaring that away, I suggest you ask a separate question (Stack Overflow is best when each question is self-contained) and provide the full code for your person class.
Will class variable ever reset on its own when instance for that particular class is still present?
I have a class and during instantiating an object, I update class variable within init for future use where I would not have access to the instantiated object. I know for a fact that the object is no out of scope when I try to access this class variable. Sample snippet is given below.
Class A:
var = ""
def __init__(self,name):
self.name = name
A.var = name
A_obj = A("John")
I want to use var (which is "John") at a later part. when I get to that part, value of "A.var" is "" and not "John" as I expected The complete code is complicated to be posted here. So I have just provided basic scenario of what is happening
No.
Rather than a working example which would let us reproduce the symptom you see, you chose to provide code which works as documented and never shows the symptom, leaving us to guess about your situation. I enclose my guess, a slightly longer version of your code:
def empty():
print('empty')
return ''
class A:
var = empty()
def __init__(self, name):
self.name = name
A.var = name
obj_john = A('John')
print(A.var)
obj_mary = A('Mary')
print(A.var)
The big difference is logging of the empty string assignment, which seems to be your chief concern. Unsurprisingly, the output produced is:
empty
John
Mary
That is, the empty string was assigned exactly once, and then the ctor repeatedly overwrote the singleton.
If you abuse repeated imports of your module then you might manage to invoke the empty assignment twice, but you haven't described any of those interactions. Setting a debugger watchpoint might prove useful. Dig a bit further, and share with us what you found.