All
I try to inherit networkx.Graph with my own, adding two node and an edge when the graph is created. But it fail with
networkx.exception.NetworkXError: ('The node sets of G and H are not disjoint.', 'Use appropriate rename=(Gprefix,Hprefix)or use disjoint_union(G,H).')
when I am trying to union my graphs, here is my code. Anything do I miss?
#!/usr/bin/python3
import networkx as nx
class die(nx.Graph):
nLatency = 2
def __init__(self):
super().__init__()
self.addNet()
def addNet(self):
self.add_node('N0')
self.add_node('N1')
self.add_edge('N0', 'N1', name='nLink', latency=self.nLatency)
S0D0 = die()
S1D0 = die()
Top = nx.union(S0D0, S1D0, rename=('S0D0', 'S1D0'))
So what is happening here is that networkx tries to create two temporary graphs whose nodes are 'S0D0-N0', 'S0D0-N1' for one and 'S1D0-N1', 'S1D0-N2' for the other. Then it tries to join them.
However as you dig through the code when it does that, the two new graphs created have the same class as the originals. So, let's call the new graphs created H1 and H2. Because H1 and H2 also both have class die, they are initialized with the nodes 'N0' and 'N1' and then 'S0D0-N0', 'S0D0-N1' or 'S1D0-N1', 'S1D0-N2' are added. So both are initialized with 'N0' and 'N1'.
So then at the next stage in the union process it tests whether or not H1 and H2 have any common nodes, and they do. So you get the error.
So that's the cause of the error. How to fix it probably depends on why you are initializing the graphs with these nodes, and what class you want Top to have.
If Top has class die, it's going to have to have 'N0' and 'N1' (because of the initialization), which I suspect you don't actually want. If you just want Top to be a Graph, you can first turn S0D0 and S1D0 into Graphs:
Top = nx.union(nx.Graph(S0D0), nx.Graph(S1D0), rename=('S0D0', 'S1D0'))
Related
First, I'd like to thank the StackOverflow community for the tremendous help it provided me over the years, without me having to ask a single question.
I could not find anything that I can relate to my problem, though it is probably due to my lack of understanding of the subject, rather than the absence of a response on the website. My apologies in advance if this is a duplicate.
I am relatively new to multiprocess; some time ago I succeeded in using multiprocessing.pools in a very simple way, where I didn't need any feedback between the child processes.
Now I am facing a much more complicated problem, and I am just lost in the documentation about multiprocessing. I hence ask for you help, your kindness and your patience.
I am trying to build a parallel tempering monte-carlo algorithm, from a class.
The basic class very roughly goes as follows:
import numpy as np
class monte_carlo:
def __init__(self):
self.x=np.ones((1000,3))
self.E=np.mean(self.x)
self.Elist=[]
def simulation(self,temperature):
self.T=temperature
for i in range(3000):
self.MC_step()
if i%10==0:
self.Elist.append(self.E)
return
def MC_step(self):
x=self.x.copy()
k = np.random.randint(1000)
x[k] = (x[k] + np.random.uniform(-1,1,3))
temp_E=np.mean(self.x)
if np.random.random()<np.exp((self.E-temp_E)/self.T):
self.E=temp_E
self.x=x
return
Obviously, I simplified a great deal (actual class is 500 lines long!), and built fake functions for simplicity: __init__ takes a bunch of parameters as arguments, there are many more lists of measurement else than self.Elist, and also many arrays derived from self.X that I use to compute them. The key point is that each instance of the class contains a lot of informations that I want to keep in memory, and that I don't want to copy over and over again, to avoid dramatic slowing down. Else I would just use the multiprocessing.pool module.
Now, the parallelization I want to do, in pseudo-code:
def proba(dE,pT):
return np.exp(-dE/pT)
Tlist=[1.1,1.2,1.3]
N=len(Tlist)
G=[]
for _ in range(N):
G.append(monte_carlo())
for _ in range(5):
for i in range(N): # this loop should be ran in multiprocess
G[i].simulation(Tlist[i])
for i in range(N//2):
dE=G[i].E-G[i+1].E
pT=G[i].T + G[i+1].T
p=proba(dE,pT) # (proba is a function, giving a probability depending on dE)
if np.random.random() < p:
T_temp = G[i].T
G[i].T = G[i+1].T
G[i+1].T = T_temp
Synthesis: I want to run several instances of my monte-carlo class in parallel child processes, with different values for a parameter T, then periodically pause everything to change the different T's, and run again the child processes/class instances, from where they paused.
Doing this, I want each class-instance/child-process to stay independent from one another, save its current state with all internal variables while it is paused, and do as few copies as possible. This last point is critical, as the arrays inside the class are quite big (some are 1000x1000), and a copy will therefore very quickly become quite time-costly.
Thanks in advance, and sorry if I am not clear...
Edit:
I am using a distant machine with many (64) CPUs, running on Debian GNU/Linux 10 (buster).
Edit2:
I made a mistake in my original post: in the end, the temperatures must be exchanged between the class-instances, and not inside the global Tlist.
Edit3: Charchit answer works perfectly for the test code, on both my personal machine and the distant machine I am usually using for running my codes. I hence check this as the accepted answer.
However, I want to report here that, inserting the actual, more complicated code, instead of the oversimplified monte_carlo class, the distant machine gives me some strange errors:
Unable to init server: Could not connect: Connection refused
(CMC_temper_all.py:55509): Gtk-WARNING **: ##:##:##:###: Locale not supported by C library.
Using the fallback 'C' locale.
Unable to init server: Could not connect: Connection refused
(CMC_temper_all.py:55509): Gdk-CRITICAL **: ##:##:##:###:
gdk_cursor_new_for_display: assertion 'GDK_IS_DISPLAY (display)' failed
(CMC_temper_all.py:55509): Gdk-CRITICAL **: ##:##:##:###: gdk_cursor_new_for_display: assertion 'GDK_IS_DISPLAY (display)' failed
The "##:##:##:###" are (or seems like) IP adresses.
Without the call to set_start_method('spawn') this error shows only once, in the very beginning, while when I use this method, it seems to show at every occurrence of result.get()...
The strangest thing is that the code seems otherwise to work fine, does not crash, produces the datafiles I then ask it to, etc...
I think this would deserve to publish a new question, but I put it here nonetheless in case someone has a quick answer.
If not, I will resort to add one by one the variables, methods, etc... that are present in my actual code but not in the test example, to try and find the origin of the bug. My best guess for now is that the memory space required by each child-process with the actual code, is too large for the distant machine to accept it, due to some restrictions implemented by the admin.
What you are looking for is sharing state between processes. As per the documentation, you can either create shared memory, which is restrictive about the data it can store and is not thread-safe, but offers better speed and performance; or you can use server processes through managers. The latter is what we are going to use since you want to share whole objects of user-defined datatypes. Keep in mind that using managers will impact speed of your code depending on the complexity of the arguments that you pass and receive, to and from the managed objects.
Managers, proxies and pickling
As mentioned, managers create server processes to store objects, and allow access to them through proxies. I have answered a question with better details on how they work, and how to create a suitable proxy here. We are going to use the same proxy defined in the linked answer, with some variations. Namely, I have replaced the factory functions inside the __getattr__ to something that can be pickled using pickle. This means that you can run instance methods of managed objects created with this proxy without resorting to using multiprocess. The result is this modified proxy:
from multiprocessing.managers import NamespaceProxy, BaseManager
import types
import numpy as np
class A:
def __init__(self, name, method):
self.name = name
self.method = method
def get(self, *args, **kwargs):
return self.method(self.name, args, kwargs)
class ObjProxy(NamespaceProxy):
"""Returns a proxy instance for any user defined data-type. The proxy instance will have the namespace and
functions of the data-type (except private/protected callables/attributes). Furthermore, the proxy will be
pickable and can its state can be shared among different processes. """
def __getattr__(self, name):
result = super().__getattr__(name)
if isinstance(result, types.MethodType):
return A(name, self._callmethod).get
return result
Solution
Now we only need to make sure that when we are creating objects of monte_carlo, we do so using managers and the above proxy. For that, we create a class constructor called create. All objects for monte_carlo should be created with this function. With that, the final code looks like this:
from multiprocessing import Pool
from multiprocessing.managers import NamespaceProxy, BaseManager
import types
import numpy as np
class A:
def __init__(self, name, method):
self.name = name
self.method = method
def get(self, *args, **kwargs):
return self.method(self.name, args, kwargs)
class ObjProxy(NamespaceProxy):
"""Returns a proxy instance for any user defined data-type. The proxy instance will have the namespace and
functions of the data-type (except private/protected callables/attributes). Furthermore, the proxy will be
pickable and can its state can be shared among different processes. """
def __getattr__(self, name):
result = super().__getattr__(name)
if isinstance(result, types.MethodType):
return A(name, self._callmethod).get
return result
class monte_carlo:
def __init__(self, ):
self.x = np.ones((1000, 3))
self.E = np.mean(self.x)
self.Elist = []
self.T = None
def simulation(self, temperature):
self.T = temperature
for i in range(3000):
self.MC_step()
if i % 10 == 0:
self.Elist.append(self.E)
return
def MC_step(self):
x = self.x.copy()
k = np.random.randint(1000)
x[k] = (x[k] + np.random.uniform(-1, 1, 3))
temp_E = np.mean(self.x)
if np.random.random() < np.exp((self.E - temp_E) / self.T):
self.E = temp_E
self.x = x
return
#classmethod
def create(cls, *args, **kwargs):
# Register class
class_str = cls.__name__
BaseManager.register(class_str, cls, ObjProxy, exposed=tuple(dir(cls)))
# Start a manager process
manager = BaseManager()
manager.start()
# Create and return this proxy instance. Using this proxy allows sharing of state between processes.
inst = eval("manager.{}(*args, **kwargs)".format(class_str))
return inst
def proba(dE,pT):
return np.exp(-dE/pT)
if __name__ == "__main__":
Tlist = [1.1, 1.2, 1.3]
N = len(Tlist)
G = []
# Create our managed instances
for _ in range(N):
G.append(monte_carlo.create())
for _ in range(5):
# Run simulations in the manager server
results = []
with Pool(8) as pool:
for i in range(N): # this loop should be ran in multiprocess
results.append(pool.apply_async(G[i].simulation, (Tlist[i], )))
# Wait for the simulations to complete
for result in results:
result.get()
for i in range(N // 2):
dE = G[i].E - G[i + 1].E
pT = G[i].T + G[i + 1].T
p = proba(dE, pT) # (proba is a function, giving a probability depending on dE)
if np.random.random() < p:
T_temp = Tlist[i]
Tlist[i] = Tlist[i + 1]
Tlist[i + 1] = T_temp
print(Tlist)
This meets the criteria you wanted. It does not create any copies at all, rather, all arguments to the simulation method call are serialized inside the pool and sent to the manager server where the object is actually stored. It gets executed there, and the results (if any) are serialized and returned in the main process. All of this, with only using the builtins!
Output
[1.2, 1.1, 1.3]
Edit
Since you are using Linux, I encourage you to use multiprocessing.set_start_method inside the if __name__ ... clause to set the start method to "spawn". Doing this will ensure that the child processes do not have access to variables defined inside the clause.
We use n external binning rules engine vs using the Native binning API in origen_testers and need to pass it a list of the test names in the proper order. The issue arises when we use on_fail or on_pass flow arguments that contain procs. These child tests can contain several layers of similar proc calls. The child tests are processed first before the parent tests so the binning order gets reversed. Is there a way to extract a list of test names that preserves the parent to child relationship? I know flow.atp.raw shows the hierarchy.
How can I select out the TestSuite names in the proper order?
thx
EDIT
Here is an example of the ATP:
s(:test,
s(:object, <TestSuite: jtag_ccra_all_vmin>),
s(:name, "jtag_ccra_all_vmin"),
s(:number, 0),
s(:id, "jtag_ccra_all_vmin"),
s(:on_fail,
s(:if_flag, "$Alarm",
s(:render, "multi_bin;")),
s(:test,
s(:object, <TestSuite: jtag_ccra_top_vmin>),
s(:name, "jtag_ccra_top_vmin"),
s(:number, 0),
s(:id, "jtag_ccra_top_vmin"),
s(:on_fail,
s(:if_flag, "$Alarm",
s(:render, "multi_bin;")))),
s(:test,
s(:object, <TestSuite: jtag_ccra_gasket_vmin>),
s(:name, "jtag_ccra_gasket_vmin"),
s(:number, 0),
s(:id, "jtag_ccra_gasket_vmin"),
s(:on_fail,
I am wondering if there is way to extract the names in order:
['jtag_ccra_all_vmin', 'jtag_ccra_top_vmin', 'jtag_ccra_gasket_vmin']
flow.atp.raw.children returns them in order as an Array but I would like to just query out the name attribute versus the ATP node objects.
thx
The way to extract information from the AST is to make a processor, something like this: https://github.com/Origen-SDK/origen_testers/blob/master/lib/origen_testers/atp/processors/extract_set_flags.rb
In your processor you would make a method called on_test and then from there you can extract the name.
This is untested, but something like this:
class ExtractTestNames < OrigenTesters::ATP::Processor
def run(node)
#results = []
process(node)
#results
end
# All 'test' nodes in the AST will be handed to this method, all nodes which don't have
# a specific handler defined will pick up a default handler which simply processes
# (looks for a handler for) all of the node's children
def on_test(node)
#results << node.find(:name).value
# Keep processing the children of this node, so that any tests embedded
# in on_fail nodes, etc. are picked up
process_all(node.children)
end
end
To call it:
ExtractTestNames.new.run(flow.atp.raw) # => ['jtag_ccra_all_vmin', 'jtag_ccra_top_vmin', ...]
Ultimately I want pyqtgraph to display a single GraphicsObject simultaneously in several ViewBoxes, sharing a single scene.
At the same time I want to have some other GraphicsObjects in a single ViewBox only.
Something like this:
vb0 = ViewBox()
vb1 = ViewBox()
# shown only in first ViewBox vb0
local_item = GraphicsObject()
# shown in all ViewBoxes
global_item = GraphicsObject()
vb0.addItem(local_item)
assert vb0.scene() is vb1.scene()
# the magic function i am looking for
vb0.scene().addItemGlobally(global_item)
So very naively I looked into the ViewBox sourcecode and reproduced the steps for addItem() like here:
import pyqtgraph as pg
from pyqtgraph.Qt import QtCore as qc, QtGui as qg, QtWidgets as qw
class Polygon(pg.GraphicsObject):
"""Just a Triangle..."""
app = qw.QApplication([])
viewer = pg.GraphicsWindow()
vb0 = viewer.addViewBox(0, 0)
vb1 = viewer.addViewBox(0, 1)
viewer.show()
poly_yellow = Polygon((125, 125, 0, 255))
scene = vb0.scene()
added_items = vb1.addedItems = vb0.addedItems
child_group = vb1.childGroup = vb0.childGroup
child_group.setParent(scene)
child_group.itemsChangedListeners.append(vb1)
# here reproducing steps found in addItem()
if scene is not poly_yellow.scene():
scene.addItem(poly_yellow)
poly_yellow.setParentItem(child_group)
added_items.append(poly_yellow)
vb0.updateAutoRange()
# vb1.updateAutoRange()
# checking if ViewBoxes share relevant attributes
assert vb0.scene() is vb1.scene()
assert vb0.scene() is poly_yellow.scene()
assert vb0.addedItems is vb1.addedItems
assert vb0.childGroup is vb1.childGroup
app.exec_()
Running this gives me two ViewBoxes, but only vb0 showing the triangle. Also this approach would give me global only Items. Is there any way to get something like local/global items without re-implementing ViewBoxes completely?
EDIT: I think it is impossible to achieve what I want with pyqtgraph ViewBoxes. A transform of the global items must happen just before the painting.
I found it is not easily doable using ViewBox. However, it is possible to us pyqtgraphs GraphicsView, implementing a lot of functionality found in the ViewBox class already.
My approach now is to generate as many GraphicsView as I need, and set them all to one scene via GraphicsView.setScene()
The scene contains the 'global' items, displayed in every View. The Viewspecific, local Items are drawn using the GraphicsView.drawBackground() function.
I haven't tested it to much, but it seems quite good working, for the case, were several thousend items are added to the scene, but only few items are drawn to the background.
I am developing an Intelligent agent for board games using MCTS algorithm.
Monte carlo tree search (MCTS) is a popular method in AI which is mostly used for games (like GO, Chess, ...). In this method, An agent builds a tree based on states which would be a result of choosing moves allowed in current state. Agent is allowed to search through the tree for limited time. in this period, Agent expands the tree to the nodes which are most promising (for winning a game).
The picture below shows the process:
For more information you can check this link:
1 - http://www.cameronius.com/research/mcts/about/index.html
In root node of the tree, there would be a variable rootstate which shows the current state of game. A deepcopy of rootstate is used to simulate the tree states (future states) as we go deep in the tree.
I used this code for deepcopy of gamestate class because deepcopy doesn't work fine with cython objects due to their problem with pickle protocol:
cdef class gamestate:
# ... other functions
def __deepcopy__(self,memo_dictionary):
res = gamestate(self.size)
res.PLAYERS = self.PLAYERS
res.size = int(self.size)
res.board = np.array(self.board, dtype=np.int32)
res.white_groups = deepcopy(self.white_groups) # a module which checks if white player has won the game
res.black_groups = deepcopy(self.black_groups) # a module which checks if black player has won the game
# the black_groups and white_groups are also cython objects which the same deepcopy function is implemented for them
# .... etc
return res
Whenever an MCTS iteration starts, a deepcopy of the state is stored in memory.
The problem which occurs is that in the begining of the game,
the iterations per 1 second is between 2000 and 3000 which is expected, but as the game tree expands,the iterations per 1 second decreases to 1. It get even worse when each iteration takes more time to
be completed. When I checked the memory usage, I noticed that it increases from 0.6 percent to 90 percent for each time I call the agent to search. I had implemented the same algorithm in pure python and it has no issues of this type. So I guess the __deepcopy__ function causes the problem. I was once suggested to make my own pickle protocol for cython objects in here, but I am not very much familiar with pickle module.
Can anyone suggest me some protocol to use for my cython objects to get rid of this obstacle.
Edit 2:
I add some parts of the code which might help more.
The code below belongs to deepcopy of class unionfind which is used for white_groups and black_groups in gamestate:
cdef class unionfind:
cdef public:
dict parent
dict rank
dict groups
list ignored
cdef __init__(self):
# initialize variables ...
def __deepcopy__(self, memo_dictionary):
res = unionfind()
res.parent = self.parent
res.rank = self.rank
res.groups = self.groups
res.ignored = self.ignored
return res
this one is the search function which is run during allowed time:
cdef class mctsagent:
def search(time_budget):
cdef int num_rollouts = 0
while (num_rollouts < time_budget):
state_copy = deepcopy(self.rootstate)
node, state = self.select_node(state_copy) # expansion runs inside the select_node function
turn = state.turn()
outcome = self.roll_out(state)
self.backup(node, turn, outcome)
num_rollouts += 1
This issue is probably the lines
res.white_groups = deepcopy(self.white_groups) # a module which checks if white player has won the game
res.black_groups = deepcopy(self.black_groups) # a module which checks if black player has won the game
What you should be doing is calling deepcopy with the second argument memo_dictionary. This is deepcopys record of if it's already copied an object. Without it deepcopy ends up copying the same object multiple times (hence the huge memory use)
res.white_groups = deepcopy(self.white_groups, memo_dictionary) # a module which checks if white player has won the game
res.black_groups = deepcopy(self.black_groups, memo_dictionary) # a module which checks if black player has won the game
If the __deepcopy__() implementation needs to make a deep copy of a component, it should call the deepcopy() function with the component as first argument and the memo dictionary as second argument.
(edit: just seen that #Blckknght already pointed this out in the comments)
(edit2: unionfind looks to mainly contain Python objects. There probably isn't a huge value in it being a cdef class and not just a normal class. Also, your current __deepcopy__ for it doesn't actually make a copy of those dictionaries - you should be doing res.parent = deepcopy(self.parent, memo_dictionary) etc.. If you just made it a normal class this would be implemented automatatically)
I'm making a basic DFA (deterministic finite automata) generator. If you don't know what it is, it's more or less a basic state machine. Nodes are generated and systematically linked to another node with some data specified in JSON.
To do this, I have a dictionary of nodes. Each node is guaranteed have a unique ID, so I use the ID as the key, and the node as the value.
I populate my dictionary, and from there, go about linking each node to following nodes (states) it can transfer to. The transition state has an ID to the node it starts at, and an ID to the node it ends at. Each node has a map of references to nodes with the transition as the hash value (which is also guaranteed to be unique by the nature of being a DFA).
Here's where the issue is, when linking one node in my map with a following node, every other node simultaneously gets the link too!
So if I have nodes
[q0] [q1] [q2]
And I link q0 with q1 with a transition of 1, this happens:
[q0] -> [q1] <- [q2]
↺
I know I do not have collision errors though. If I perform any other action on a node here, the action is not acted on my every other node.
I create the dictionary of nodes like so:
dict_nodes = {}
alphabet = json_values["alphabet"]
for state in json_values["possible_states"]:
dict_nodes[state["state_key"]] = node()
Then link them like this:
for transition in json_values["transitions"]:
lexicon = transition["transition"]
if lexicon not in alphabet:
raise ValueError("Nodes with transitions outside of alphabet are not accepted")
dict_nodes[transition["start_key"]].add_following_state(dict_nodes[transition["end_key"]], lexicon)
The node class looks like this:
class node(object):
""" Following node paths that can be walked and their states """
following_states = {}
""" Determine if node is an accept state or not """
is_accept_state = False
def __init__(self):
pass
def add_following_state(self, new_node, state):
self.following_states[state] = new_node
""" Extra methods omitted """
Any idea why everything is getting linked at once?