I have 2 different python scripts, the first called main.py and the second called running.py
the main.py script is the following:
#setting things up and stuff
while True:
#stuff
Here running.py should be started
#stuff
while running.py contains the following
#various imports
while True:
#stuff
My question is, how can i run running.py from main.py as a new thread knowing only the name of the running script?
I looked a bit into it and, since i need to communicate and share data between main.py and running.py, I don't think creating a subprocess is the best course of action but I haven't found a way to run a whole script in a thread only knowing the script's name.
For various (stupid) company reasons I can't change the content of running.py and i can't import it into main.py so creating a threading class inside it is not a possibility but i have free rein on main.py
Is what i'm trying to do even possible?
Related
Suppose I have a python script program.py
that looks as follows:
# program.py
import random
class Program
def __init__(self):
self.data = []
def run():
while True:
self.data.append(random.random()]
if __name__ == "__main__":
program = Program()
program.run()
Now suppose that I have another script script.py that calls program.py as a separate process.
# script.py
import subprocess
import time
process = subprocess.Popen(["py", "program.py"])
while True:
data = get_program_data(process)# how?
time.sleep(10)
The purpose of script.py is to illustrate the fact that I don't have access to the class Program. In my case this is due to the fact that I will be triggering program.py from a .NET application. I thought I'll try to understand how to deal with this problem from a python script first then apply it to the .NET application. So here is my question (keep in mind that I can alter the code in program.py and script.py, but script.py can not access program.py):
How should I go about accessing self.data from the process? I have been searching all over and I'm not quite sure in what direction I should be going. In my case, I will need to trigger difference commands for different ind of data generated in Program, i.e. get_program_data1(), get_program_data2(),....
The way I have been "solving" this issue is to have a file controller.json that script.py modifies and program.py reads and acts accordingly. I does not feel quite right doing and so I want your opinion about this. Remember that ultimately, script.py is a .NET application. Thanks
I am pretty confused about the best way to do what I am trying to do.
What do I want?
API call to the flask application
Flask route starts 4-5 multiprocess using Process module and combine results(on a sliced pandas dataframe) using a shared Managers().list()
Return computed results back to the client.
My implementation:
pos_iter_list = get_chunking_iter_list(len(position_records), 10000)
manager = Manager()
data_dict = manager.list()
processes = []
for i in range(len(pos_iter_list) - 1):
temp_list = data_dict[pos_iter_list[i]:pos_iter_list[i + 1]]
p = Process(
target=transpose_dataset,
args=(temp_list, name_space, align_namespace, measure_master_id, df_searchable, products,
channels, all_cols, potential_col, adoption_col, final_segment, col_map, product_segments,
data_dict)
)
p.start()
processes.append(p)
for p in processes:
p.join()
My directory structure:
- main.py(flask entry point)
- helper.py(contains function where above code is executed & calls transpose_dataset function)
Error that i am getting while running the same?
RuntimeError: No root path can be found for the provided module "mp_main". This can happen because the module came from an import hook that does not provide file name information or because it's a namespace package. In this case the root path needs to be explicitly provided.
Not sure what went wong here, manager list works fine when called from a sample.py file using if __name__ == '__main__':
Update: The same piece of code is working fine on my MacBook and not on windows os.
A sample flask API call:
#app.route(PREFIX + "ping", methods=['GET'])
def ping():
man = mp.Manager()
data = man.list()
processes = []
for i in range(0,5):
pr = mp.Process(target=test_func, args=(data, i))
pr.start()
processes.append(pr)
for pr in processes:
pr.join()
return json.dumps(list(data))
Stack has an ongoing bug preventing me from commenting, so I'll just write up an answer..
Python has 2 (main) ways to start a new process: "spawn", and "fork". Fork is a system command only available in *nix (read: linux or macos), and therefore spawn is the only option in windows. After 3.8 spawn will be the default on MacOS, but fork is still available. The big difference is that fork basically makes a copy of the existing process while spawn starts a whole new process (like just opening a new cmd window). There's a lot of nuance to why and how, but in order to be able to run the function you want the child process to run using spawn, the child has to import the main file. Importing a file is tantamount to just executing that file and then typically binding its namespace to a variable: import flask will run the flask/__ini__.py file, and bind its global namespace to the variable flask. There's often code however that is only used by the main process, and doesn't need to be imported / executed in the child process. In some cases running that code again actually breaks things, so instead you need to prevent it from running outside of the main process. This is taken into account in that the "magic" variable __name__ is only equal to "__main__" in the main file (and not in child processes or when importing modules).
In your specific case, you're creating a new app = Flask(__name__), which does some amount of validation and checks before you ever run the server. It's one of these setup/validation steps that it's tripping over when run from the child process. Fixing it by not letting it run at all is imao the cleaner solution, but you can also fix it by giving it a value that it won't trip over, then just never start that secondary server (again by protecting it with if __name__ == "__main__":)
I'm developing a small app using kivy and python3.6 (I'm still a beginner). I'm planning to separate the code in different files for clarity, however I have encountered a problem in a specific situation. I have made minimal working example to illustrate.
I have the following files:
main.py
main.kv
module.py
module.kv
Here a minimal code:
main.py:
from kivy.app import App
from kivy.uix.button import Button
from kivy.lang import Builder
import module
Builder.load_file('module.kv')
class MainApp(App):
pass
def function():
print('parent function')
if __name__ == '__main__':
MainApp().run()
main.kv:
CallFunction
module.py:
from kivy.uix.button import Button
class CallFunction(Button):
def call_function(self):
from main import function
function()
module.kv:
<CallFunction>:
id : parent_button
text: 'Call parent button'
on_press: self.call_function()
So the problem is that when I run this code, I receive a warning
The file /home/kivy/python_exp/test/module.kv is loaded multiples times, you might have unwanted behaviors.
What works:
If the function I want to call is part of the main app class, there is no problem
If the function is part of the module.py there is no problem
If the function is part of another module, there is no problem
What doesn't work
I cannot call a function which is in the main.py. If I use the import the function as the beginning of module.py, kivy has a weird behavior and call everything twice. Calling within this call_function allows to have a proper interface, but I get the warning that the file has been loaded multiple time.
There are easy workarounds, I'm well aware of that, so it's more about curiosity and understanding better how the imports in kivy works. Is there a way to make it work?
I wanted to use the main.py to initialize different things at the startup of the app. In particular I wanted to create an instance of another class (not a kivy class) in the main.py and when clicking on the button on the interface, calling a method on this instance.
Thanks :)
When you import something from another python module the python virtual machine execute this module. In the call_function you import function from the main file so everytime you press the CallFunction the module.kv is loaded.
To solve this it is recommended to include the other kv files in your main kv file.
You can also move the import statement from the method to the top of the module file.
Your kv file is loaded twice because the code is executed twice. This is due to how pythons module system works and kivy just realized that loading the kv twice is probably not what you want.
Generally python objects live in a namespace. So when a function in the module foo looks up a variable the variable is searched in the namespace of the module. That way if you define two variables foo.var and bar.var (in the modules foo and bar resp.) they don't clash and get confused for each other.
The tricky thing is that the python file you execute is special: It does not create a module namespace but the __main__ namespace. Thus if you import the file you are executing as __main__ it will create a whole new namespace with new objects and execute the module code. If you import a module that was already imported in the current session the module code is not executed again, but the namespace already created is made available. You don't even need two files for that, put the following in test.py:
print("hello!")
print(__name__)
import test
If you now execute python test.py you will see two hello! and once __main__ and once test.
You can find more information on namespaces and how variable lookups works in python in the documentation.
Also if your function actually does some work and mutates an object that lives in main.py you might want to rethink the information flow. Often it is a good idea to bind the state and functions working on them together in classes and passing the objects then where they are called i.e. to CallFunction in your example.
I am confused about some behavior of Python. I always thought importing a module basically meant executing it. (Like they say here: Does python execute imports on importation) So I created three simple scripts to test something:
main.py
import config
print(config.a)
config.a += 1
print(config.a)
import test
print(config.a)
config.py
def get_a():
print("get_a is called")
return 1
a = get_a()
test.py
import config
print(config.a)
config.a += 1
The output when running main.py is:
get_a is called
1
2
2
3
Now I am confused because I expected get_a() to be called twice, once from main.py and once from test.py. Can someone please explain why it is not? What if I really wanted to import config a second time, like it was in the beginning with a=1?
(Fortunately, for my project this behavior is exactly what I wanted, because get_a() corresponds to a function, which reads lots of data from a database and of course I only want to read it once, but it should be accessible from multiple modules.)
Because the config module is already loaded so there's no need to 'run' it anymore, just return the loaded instance.
Some standard library modules make use of this, from example random. It creates an object of class Random on first import and reuses it when it gets imported again. A comment on the module reads:
# Create one instance, seeded from current time, and export its methods
# as module-level functions. The functions share state across all uses
#(both in the user's code and in the Python libraries), but that's fine
# for most programs and is easier for the casual user than making them
# instantiate their own Random() instance.
I have a number of python scripts that I would like to automate using Python's Datetime and Schedule module.
They are too numerous to consider breaking apart and adding to one large file.
What is the easiest way to write a python script that will open and run these other python scripts?
I have browsed similar questions but none offered a concrete answer that I could find. Thanks for your help.
A minimally demonstrative example
In a file called "child.py", write a file to the current directory:
with open('test', 'w') as f:
f.write('hello world')
Then, in a file called "parent.py", execute the "child.py" script:
import subprocess
subprocess.call(['python', 'child.py'])
Now, from your command line, you can type (assuming both "parent.py" and "child.py" are in the current directory):
python parent.py
In the next instant, you should see a file called "test" in your current directory. Open it up. What do you see?
Well, hello world of course!
The above example makes a child of the current process (meaning it inherits the environment variables in the parent), and waits until the child process completes before returning control to the parent. If you want the child script to run in the background, then you need to use Popen:
subprocess.Popen(['python', 'child.py'])