First of all, I'm pretty new in multiprocessing and I'm here for learning of all of you. I have several files doing something similar to this:
SharedClass.py:
class simpleClass():
a = 0
b = ""
.....
MyProcess.py:
import multiprocessing
import SharedClass
class FirstProcess(multiprocessing.Process):
def __init__(self):
multiprocessing.Process.__init__(self)
def modifySharedClass():
# Here I want to modify the object shared with main.py defined in SharedClass.py
Main.py:
from MyProcess import FirstProcess
import sharedClass
if __name__ == '__main__':
pr = FirstProcess()
pr.start()
# Here I want to print the initial value of the shared class
pr.modifySharedClass()
# Here I want to print the modified value of the shared class
I want to define a shared class (in SharedClass.py), in a kind of shared memory that can be readed and writted for both files Main.py and MyProcess.py.
I have try to use the Manager of multiprocessing and multiprocessing.array but Im not having good results, the changes made in one file are not beeing reflected in the other file (maybe Im doing this in the wrong way).
Any ideas? Thank you.
Related
I have more of a design question, but I am not sure how to handle that. I have a script preprocessing.py where I read a .csv file of text column that I would like to preprocess by removing punctuations, characters, ...etc.
What I have done now is that I have written a class with several functions as follows:
class Preprocessing(object):
def __init__(self, file):
self.my_data = pd.read_csv(file)
def remove_punctuation(self):
self.my_data['text'] = self.my_data['text'].str.replace('#','')
def remove_hyphen(self):
self.my_data['text'] = self.my_data['text'].str.replace('-','')
def remove_words(self):
self.my_data['text'] = self.my_data['text'].str.replace('reference','')
def save_data(self):
self.my_data.to_csv('my_data.csv')
def preprocessing(file_my):
f = Preprocessing(file_my)
f.remove_punctuation()
f.remove_hyphen()
f.remove_words()
f.save_data()
return f
if __name__ == '__main__':
preprocessing('/path/to/file.csv')
although it works fine, i would like to be able to expand the code easily and have smaller classes instead of having one large class. So i decided to use abstract class:
import pandas as pd
from abc import ABC, abstractmethod
my_data = pd.read_csv('/Users/kgz/Desktop/german_web_scraping/file.csv')
class Preprocessing(ABC):
#abstractmethod
def processor(self):
pass
class RemovePunctuation(Preprocessing):
def processor(self):
return my_data['text'].str.replace('#', '')
class RemoveHyphen(Preprocessing):
def processor(self):
return my_data['text'].str.replace('-', '')
class Removewords(Preprocessing):
def processor(self):
return my_data['text'].str.replace('reference', '')
final_result = [cls().processor() for cls in Preprocessing.__subclasses__()]
print(final_result)
So now each class is responsible for one task but there are a few issues I do not know how to handle since I am new to abstract classes. first, I am reading the file outside the classes, and I am not sure if that is good practice? if not, should i pass it as an argument to the processor function or have another class who is responsible to read the data.
Second, having one class with several functions allowed for a flow, so every transformation happened in order (i.e, first punctuation is removes, then hyphen is removed,...etc) but I do not know how to handle this order and dependency in abstract classes.
Currently I am extracting files to the temp directory of the operating system. One of the files is a Python file containing a class which I need to get a handle of. The Python's file is known, but the name of the class inside the file is unknown. But it is safe to assume, that the there is only one single class, and that the class is a subclass of another.
I tried to work with importlib, but I am not able to get a handle of the class.
So far I tried:
# Assume
# module_name contains the name of the class and -> "MyClass"
# path_module contains the path to the python file -> "../Module.py"
spec = spec_from_file_location(module_name, path_module)
module = module_from_spec(spec)
for pair in inspect.getmembers(module):
print(f"{pair[1]} is class: {inspect.isclass(pair[1])}")
When I iterate over the members of the module, none of them get printed as a class.
My class in this case is called BasicModel and the Output on the console looks like this:
BasicModel is class: False
What is the correct approach to this?
Edit:
As the content of the file was requested, here you go:
class BasicModel(Sequential):
def __init__(self, class_count: int, input_shape: tuple):
Sequential.__init__(self)
self.add(Input(shape=input_shape))
self.add(Flatten())
self.add(Dense(128, activation=nn.relu))
self.add(Dense(128, activation=nn.relu))
self.add(Dense(class_count, activation=nn.softmax))
Use dir() to get the attributes of the file and inspect to check if the attribute is a class. If so, you can create an object.
Assuming that your file's path is /tmp/mysterious you can do this:
import importlib
import inspect
from pathlib import Path
import sys
path_pyfile = Path('/tmp/mysterious.py')
sys.path.append(str(path_pyfile.parent))
mysterious = importlib.import_module(path_pyfile.stem)
for name_local in dir(mysterious):
if inspect.isclass(getattr(mysterious, name_local)):
print(f'{name_local} is a class')
MysteriousClass = getattr(mysterious, name_local)
mysterious_object = MysteriousClass()
I have a class to represent a resource (for the sake of the example, let's say a folder), and a factory class to create those resources.
The issue is that, for clarity purposes, I wish to provide each folder with it's own factory, allowing it to generate it's sub-folders.
This currently looks like this :
folder.py
import FolderFactory
class Folder
def __init__(self):
self.file_factory = FolderFactory(self)
self.sub_folders = []
folder_factory.py
import File
class FolderFactory
def __init__(self, folder):
assert isinstance(folder, Folder)
self.folder = Folder(folder)
def create_sub_folder(name):
// Create Folder
self.folder.sub_folders.append(new_folder)
But obviously this generate a circular reference, which cause an error.
Is there an issue with my design, or is there a way to implement this cleanly ?
The end goal would be to be able to write something along the lines of
folder.folder_factory.create_folder("new")
Please note that the actual factory class I'm working with have a large amount of functions to generate different resources in the parent resource, which explains the need to separate the object definition and the management functions.
Thanks a lot for taking the time to read !
You could pass a self from the FolderFactory to the constructor of the folder, to be used as a reference to its 'parent'.
import FolderFactory
class Folder
def __init__(self, factory):
self.file_factory = FolderFactory(self)
self.folder_factory = factory
self.sub_folders = []
import File
class FolderFactory
def __init__(self, folder):
assert isinstance(folder, Folder)
self.folder = Folder(folder, self)
def create_sub_folder(name):
// Create Folder
self.folder.sub_folders.append(new_folder)
I want to access the calling environment from an imported module.
import child
…
def test(xxx):
print("This is test " + str(xxx))
child.main()
…
now on child:
import inspect
def main():
caller = inspect.currentframe().f_back
caller.f_globals['test']("This is my test")
This works, but it's not fancy. Is there a simplification like 'self' when use in a class? the idea is to do: caller.test('abc') instead.
One option to pass the caller as a parameter like: child.main(self), however self is not available in this context.
Python only load one version of a module so, tempted with this idea:
import sys
myself=sys.modules[__name__]
a then sending myself to the child:
…
child.main(myself)
…
Creates a reference to (a new) module, but not the running one, this is like creating a new class: one code buy a different environment.
If you already have a way of accessing the correct functions and data that works, why not just store f_globals on an instance of a wrapper class and then call things from the instance as if they were unbound properties? You could use the class itself, but using an instance ensures that the data you get from the imported file are valid when you create the object. Then you can access using the dot operator the way you want. This is your child file:
import inspect
class ImportFile:
def __init__(self, members):
self.__dict__.update(members)
def main():
caller = inspect.currentframe().f_back
imported_file = ImportFile(caller.f_globals)
imported_file.test("This is my test")
Outputs:
This is test This is my test
Admittedly, I don't have your setup, importantly the module you're trying to pull from, so it's hard to confirm whether or not this will work for you even though it has for me, but I think you could also use your method of calling main with globals() or even inspect.getmembers() since while inside the module you're importing you're still on the frame you're accessing with f_back from inside child.
The imported module:
import child
def test(xxx):
print("This is test " + str(xxx))
child.main(globals())
Child:
import inspect
class ImportFile:
def __init__(self, members):
self.__dict__.update(members)
def main(caller):
imported_file = ImportFile(caller)
imported_file.test("This is my test")
Outputs:
This is test This is my test
I run the main module, which should work correctly. But an error gets returned. 'spaceship' is not defined when I define 's=spaceship(parameters)' why is this I don't get it. I'm using zelle graphics for python. thank you
Functions from main module:
spaceshipGame file
from graphics import *
from spaceshipClass import *
def main():
window=createGraphicsWindow()
runGame(window)
def createGraphicsWindow():
win=GraphWin("Spaceship game",800,800)
return win
def createSpaceship(window,p1,p2,p3,speed,colour):
s=spaceship(window,p1,p2,p3,speed,colour)
return s
def runGame(window):
player=createSpaceship(window,Point(500,500),Point(500,470),Point(520,485),0.5,"red")
player.draw(window)
main()
spaceshipClass file
from spaceshipGame import *
from graphics import *
class spaceship:
def __init__(self,window,p1,p2,p3,speed,colour):
self.p1=p1
self.p2=p2
self.p3=p3
self.speed=speed
self.colour=colour
self.window=window
Never mind, I see the problem. Consult this example for more information:
Simple cross import in python
But the problem is the way you are cross importing, so delete from spaceshipGame import * from spaceshipClass or vise-versa (i.e. delete from spaceshipClass import * from spaceshipGame). You can import individually if you need to like in the example I provided.
There are also many other ways around it if you read the example. One of the easiest would be just merging them in the same file if they need to share a lot of methods.