Will Multithreading and Multiprocessing work with or without this,
if __name__=="__main__" ??
__name__ is a global variable that is automatically initialized to refer to a string—either the name of the module in which __name__ appears or, the special value, "__main__", if it appears in a script or top-level program.
This pattern is used in many Python source files:
if __name__ == "__main__":
# test or demo the module
...
The test/demo code is run if you run the source file as a script, or it doesn't get run if the source file is imported as a module.
Importing multiprocessing or threading doesn't change what if __name__ == "__main__" does, and if __name__ == "__main__" doesn't change what multiprocessing or threading do. Those modules are no different in that respect from any other Python module (e.g., math, or time, or sys)
Related
My case is a little bit specific. I'm trying to run a Python program using Python for testing purposes. The case is as follows:
# file1.py
print("Hello world")
# file1.test.py
import io
import sys
import os
import unittest
EXPECTED_OUTPUT = "Hello world"
class TestHello(unittest.TestCase):
def test_hello(self):
sio = io.StringIO()
sys.stdout = sio
os.system("python3 path/to/file1.py")
sys.stdout = sys.__stdout__
print("captured value:", sio.getvalue())
self.assertEqual(sio.getvalue(), EXPECTED_STDOUT)
if __name__ == "__main__":
unittest.main()
But nothing ends up in the sio variable. This way and similar ways are introduced online but they don't seem to work for me. My Python version is 3.8.10 but it doesn't really matter if this works better in some other version, I can switch to that.
Note: I know that if I was using an importable object this might be easier, but right now I need to know how to catch the output of another file.
Thanks!
stdout redirection does not work like this - this will change the stdout variable inside your Python process. But by using os.system, you are running another process, that will re-use the same terminal pseudo-files your parent process is using.
If you want to log a subprocess, the way to do it is to use the subprocess modules calls, which allow you to redirect the subprocess output. https://docs.python.org/3/library/subprocess.html
Also, the subprocess won't be able to use a StringIO object from the parent process (it is not an O.S. level object, just an in-process Python object with a write method). The docs above include instructions about using the special object subprocess.PIPE which allows for in-memory communication, or, you can just pass an ordinary filesystem file, which you can read afterwards.
I have 2 different python scripts, the first called main.py and the second called running.py
the main.py script is the following:
#setting things up and stuff
while True:
#stuff
Here running.py should be started
#stuff
while running.py contains the following
#various imports
while True:
#stuff
My question is, how can i run running.py from main.py as a new thread knowing only the name of the running script?
I looked a bit into it and, since i need to communicate and share data between main.py and running.py, I don't think creating a subprocess is the best course of action but I haven't found a way to run a whole script in a thread only knowing the script's name.
For various (stupid) company reasons I can't change the content of running.py and i can't import it into main.py so creating a threading class inside it is not a possibility but i have free rein on main.py
Is what i'm trying to do even possible?
I am currently running a unittest script which successfully passes the various specified test with a nagging ImportWarning message in the console:
...../lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
return f(*args, **kwds)
....
----------------------------------------------------------------------
Ran 7 tests in 1.950s
OK
The script is run with this main function:
if __name__ == '__main__':
unittest.main()
I have read that warnings can be surpressed when the script is called like this:
python -W ignore:ImportWarning -m unittest testscript.py
However, is there a way of specifying this ignore warning in the script itself so that I don't have to call -W ignore:ImportWarning every time that the testscript is run?
Thanks in advance.
To programmatically prevent such warnings from showing up, adjust your code so that:
import warnings
if __name__ == '__main__':
with warnings.catch_warnings():
warnings.simplefilter('ignore', category=ImportWarning)
unittest.main()
Source: https://stackoverflow.com/a/40994600/328469
Update:
#billjoie is certainly correct. If the OP chooses to make answer 52463661 the accepted answer, I am OK with that. I can confirm that the following is effective at suppressing such warning messages at run-time using python versions 2.7.11, 3.4.3, 3.5.4, 3.6.5, and 3.7.1:
#! /usr/bin/env python
# -*- coding: utf-8 -*-
import unittest
import warnings
class TestPandasImport(unittest.TestCase):
def setUp(self):
warnings.simplefilter('ignore', category=ImportWarning)
def test_01(self):
import pandas # noqa: E402
self.assertTrue(True)
def test_02(self):
import pandas # noqa: E402
self.assertFalse(False)
if __name__ == '__main__':
unittest.main()
However, I think that the OP should consider doing some deeper investigation into the application code targets of the unit tests, and try to identify the specific package import or operation which is causing the actual warning, and then suppress the warning as closely as possible to the location in code where the violation takes place. This will obviate the suppression of warnings throughout the entirety of one's unit test class, which may be inadvertently obscuring warnings from other parts of the program.
Outside the unit test, somewhere in the application code:
with warnings.catch_warnings():
warnings.simplefilter('ignore', category=ImportWarning)
# import pandas
# or_ideally_the_application_code_unit_that_imports_pandas()
It could take a bit of work to isolate the specific spot in the code that is either causing the warning or leveraging third-party software which causes the warning, but the developer will obtain a clearer understanding of the reason for the warning, and this will only improve the overall maintainability of the program.
I had the same problem, and starting my unittest script with a warnings.simplefilter() statement, as described by Nels, dit not work for me. According to this source, this is because:
[...] as of Python 3.2, the unittest module was updated to use the warnings module default filter when running tests, and [...] resets to the default filter before each test, meaning that any change you may think you are making scriptwide by using warnings.simplefilter(“ignore”) at the beginning of your script gets overridden in between every test.
This same source recommends to renew the filter inside of each test function, either directly or with an elegant decorator. A simpler solution is to define the warnings filter inside unittest's setUp() method, which is run right before each test.
import unittest
class TestSomething(unittest.TestCase):
def setUp(self):
warnings.simplefilter('ignore', category=ImportWarning)
# Other initialization stuff here
def test_a(self):
# Test assertion here.
if __name__ == '__main__':
unittest.main()
I had the same warning in Pycharm for one test when using unittest. This warning disappeared when I stopped trying to import a library during the test (I moved the import to the top where it's supposed to be). I know the request was for suppression, but this would also make it disappear if it's only happening in a select number of tests.
Solutions with def setUp suppress warnings for all methods within class. If you don't want to suppress it for all of them, you can use decorator.
From Neural Dump:
def ignore_warnings(test_func):
def do_test(self, *args, **kwargs):
with warnings.catch_warnings():
warnings.simplefilter("ignore")
test_func(self, *args, **kwargs)
return do_test
Then you can use it to decorate single test method in your test class:
class TestClass(unittest.TestCase):
#ignore_warnings
def test_do_something_without_warning()
self.assertEqual(whatever)
def test_something_else_with_warning()
self.assertEqual(whatever)
I'm developing a small app using kivy and python3.6 (I'm still a beginner). I'm planning to separate the code in different files for clarity, however I have encountered a problem in a specific situation. I have made minimal working example to illustrate.
I have the following files:
main.py
main.kv
module.py
module.kv
Here a minimal code:
main.py:
from kivy.app import App
from kivy.uix.button import Button
from kivy.lang import Builder
import module
Builder.load_file('module.kv')
class MainApp(App):
pass
def function():
print('parent function')
if __name__ == '__main__':
MainApp().run()
main.kv:
CallFunction
module.py:
from kivy.uix.button import Button
class CallFunction(Button):
def call_function(self):
from main import function
function()
module.kv:
<CallFunction>:
id : parent_button
text: 'Call parent button'
on_press: self.call_function()
So the problem is that when I run this code, I receive a warning
The file /home/kivy/python_exp/test/module.kv is loaded multiples times, you might have unwanted behaviors.
What works:
If the function I want to call is part of the main app class, there is no problem
If the function is part of the module.py there is no problem
If the function is part of another module, there is no problem
What doesn't work
I cannot call a function which is in the main.py. If I use the import the function as the beginning of module.py, kivy has a weird behavior and call everything twice. Calling within this call_function allows to have a proper interface, but I get the warning that the file has been loaded multiple time.
There are easy workarounds, I'm well aware of that, so it's more about curiosity and understanding better how the imports in kivy works. Is there a way to make it work?
I wanted to use the main.py to initialize different things at the startup of the app. In particular I wanted to create an instance of another class (not a kivy class) in the main.py and when clicking on the button on the interface, calling a method on this instance.
Thanks :)
When you import something from another python module the python virtual machine execute this module. In the call_function you import function from the main file so everytime you press the CallFunction the module.kv is loaded.
To solve this it is recommended to include the other kv files in your main kv file.
You can also move the import statement from the method to the top of the module file.
Your kv file is loaded twice because the code is executed twice. This is due to how pythons module system works and kivy just realized that loading the kv twice is probably not what you want.
Generally python objects live in a namespace. So when a function in the module foo looks up a variable the variable is searched in the namespace of the module. That way if you define two variables foo.var and bar.var (in the modules foo and bar resp.) they don't clash and get confused for each other.
The tricky thing is that the python file you execute is special: It does not create a module namespace but the __main__ namespace. Thus if you import the file you are executing as __main__ it will create a whole new namespace with new objects and execute the module code. If you import a module that was already imported in the current session the module code is not executed again, but the namespace already created is made available. You don't even need two files for that, put the following in test.py:
print("hello!")
print(__name__)
import test
If you now execute python test.py you will see two hello! and once __main__ and once test.
You can find more information on namespaces and how variable lookups works in python in the documentation.
Also if your function actually does some work and mutates an object that lives in main.py you might want to rethink the information flow. Often it is a good idea to bind the state and functions working on them together in classes and passing the objects then where they are called i.e. to CallFunction in your example.
I have a (python3) package that has completely different behaviour depending on how it's init()ed (perhaps not the best design, but rewriting is not an option). The module can only be init()ed once, a second time gives an error. I want to test this package (both behaviours) using py.test.
Note: the nature of the package makes the two behaviours mutually exclusive, there is no possible reason to ever want both in a singular program.
I have serveral test_xxx.py modules in my test directory. Each module will init the package in the way in needs (using fixtures). Since py.test starts the python interpreter once, running all test-modules in one py.test run fails.
Monkey-patching the package to allow a second init() is not something I want to do, since there is internal caching etc that might result in unexplained behaviour.
Is it possible to tell py.test to run each test module in a separate python process (thereby not being influenced by inits in another test-module)
Is there a way to reliably reload a package (including all sub-dependencies, etc)?
Is there another solution (I'm thinking of importing and then unimporting the package in a fixture, but this seems excessive)?
To reload a module, try using the reload() from library importlib
Example:
from importlib import reload
import some_lib
#do something
reload(some_lib)
Also, launching each test in a new process is viable, but multiprocessed code is kind of painful to debug.
Example
import some_test
from multiprocessing import Manager, Process
#create new return value holder, in this case a list
manager = Manager()
return_value = manager.list()
#create new process
process = Process(target=some_test.some_function, args=(arg, return_value))
#execute process
process.start()
#finish and return process
process.join()
#you can now use your return value as if it were a normal list,
#as long as it was assigned in your subprocess
Delete all your module imports and also your tests import that also import your modules:
import sys
for key in list(sys.modules.keys()):
if key.startswith("your_package_name") or key.startswith("test"):
del sys.modules[key]
you can use this as a fixture by configuring on your conftest.py file a fixture using the #pytest.fixture decorator.
Once I had similar problem, quite bad design though..
#pytest.fixture()
def module_type1():
mod = importlib.import_module('example')
mod._init(10)
yield mod
del sys.modules['example']
#pytest.fixture()
def module_type2():
mod = importlib.import_module('example')
mod._init(20)
yield mod
del sys.modules['example']
def test1(module_type1)
pass
def test2(module_type2)
pass
The example/init.py had something like this
def _init(val):
if 'sample' in globals():
logger.info(f'example already imported, val{sample}' )
else:
globals()['sample'] = val
logger.info(f'importing example with val : {val}')
output:
importing example with val : 10
importing example with val : 20
No clue as to how complex your package is, but if its just global variables, then this probably helps.
I have the same problem, and found three solutions:
reload(some_lib)
patch SUT, as the imported method is a key and value in SUT, you can patch the
SUT. Example, if you use f2 of m2 in m1, you can patch m1.f2 instead of m2.f2
import module, and use module.function.