How to install python plugin before running pytest tests? - python-3.x

I need to install a python plugin that is a simple python file before the starting tests using pytest. I have used entry_points in setup.py. My problem is a bit complex so let's get into the problem with an example and we will come back to the problem later.
There are two packages- one is core and another one is mypackage.
Core provided functionality to add a plugin with group name 'my.plugin'.
core package logic
from importlib_metadata import entry_points
def plugin_method(some_data):
plugins = entry_points()['my.plugin']
loaded_plugins = []
for p in plugins:
loaded_plugins.apend(p.load())
#Does some processing on the data and decides which method to call from plugin
#call plugin method
return result
mypackage logic
setup.py
setup(
...
entry_points={'my.plugin': 'plugin1= plugin1.logic'}
...
)
logic.py
def method1(method_data):
print('method1 called')
return 1
def method2(method_data):
print('method1 called')
return 2
main.py
def method_uses_plugin()
# create data
plugin_method(data)
The plugin works fine. :)
Problem
I have written a test case for the method_uses_plugin method. It works fine if I have installed pypackage on my machine but it fails if installation is not done (in jenkins pipeline 🙁 )
We don't usually install the package to run test cases because test cases should use source code directly.
We might need to do something with pytest to register the plugin in entry_points. I have tried many links but nothing worked.
My use case a bit complex but a similar question can be found here

There is two usecase to run the test on the actual source code.
In your Local machine
If you want to test the source code while working, you can simply install your package in editable mode with the command:
pip install -e .
Documentation of -e from the man page:
-e,--editable <path/url>
Install a project in editable mode (i.e. setuptools "develop mode") from a local project path or a VCS url.
This will link the package to the . location of the code, meaning that any change made to the source code will be reflected on the package.
In Continuous Integration (CI)
As your CI is running on a docker container, you can simply copy the source code inside it, install it with pip install . and finally run pytest.

If all else fails you can try to convert your code into an executable and use batch commands to run pip install for as many packages as you need and then run your program. I believe in Jenkins you can run batch files as an administrator.
Invoke pip install from batch file
Run Batch file as an administrator in Jenkins

Related

Dynamically update virtual environment

I have created a virtual environment for my Python project and have installed my requirement.txt in it and every time my Python code ex executes it first activate my virtual environment and then run the .py code which is what I want.
I have an issue here - I want to do everything programmatically.
Below are the high level steps i'm looking for :
1)pipreqs requirements.txt using py code
2)check if .txt file has changed, create a flag(1,0)
3)check if virtualenv is created
2.1 - if not created, create and install 1 and run py code
2.2 - if created - check if flag=1 then re-install 1 and run py code
if flag=0 , just run py code
I'm struggling hard to achieve this. Could somebody help.
maybe
pip install -r requirements.txt
is a good starting point (if your paint point is to manually install all the required libraries). It installs all the libraries listed in requirements.txt.
If this is not enough putting this command in a bash / cmd file might be an option.

How to simply compile a big python program with huge number of files, for linux, mac and windows [duplicate]

I'm trying to convert a fairly simple Python program to an executable and couldn't find what I was looking for, so I have a few questions (I'm running Python 3.6):
The methods of doing this that I have found so far are as follows
downloading an old version of Python and using pyinstaller/py2exe
setting up a virtual environment in Python 3.6 that will allow me to do 1.
downloading a Python to C++ converter and using that.
Here is what I've tried/what problems I've run into.
I installed pyinstaller before the required download before it (pypi-something) so it did not work. After downloading the prerequisite file, pyinstaller still does not recognize it.
If I'm setting up a virtualenv in Python 2.7, do I actually need to have Python 2.7 installed?
similarly, the only python to C++ converters I see work only up until Python 3.5 - do I need to download and use this version if attempting this?
Steps to convert .py to .exe in Python 3.6
Install Python 3.6.
Install cx_Freeze, (open your command prompt and type pip install cx_Freeze.
Install idna, (open your command prompt and type pip install idna.
Write a .py program named myfirstprog.py.
Create a new python file named setup.py on the current directory of your script.
In the setup.py file, copy the code below and save it.
With shift pressed right click on the same directory, so you are able to open a command prompt window.
In the prompt, type python setup.py build
If your script is error free, then there will be no problem on creating application.
Check the newly created folder build. It has another folder in it. Within that folder you can find your application. Run it. Make yourself happy.
See the original script in my blog.
setup.py:
from cx_Freeze import setup, Executable
base = None
executables = [Executable("myfirstprog.py", base=base)]
packages = ["idna"]
options = {
'build_exe': {
'packages':packages,
},
}
setup(
name = "<any name>",
options = options,
version = "<any number>",
description = '<any description>',
executables = executables
)
EDIT:
be sure that instead of myfirstprog.py you should put your .pyextension file name as created in step 4;
you should include each imported package in your .py into packages list (ex: packages = ["idna", "os","sys"])
any name, any number, any description in setup.py file should not remain the same, you should change it accordingly (ex:name = "<first_ever>", version = "0.11", description = '' )
the imported packages must be installed before you start step 8.
Python 3.6 is supported by PyInstaller.
Open a cmd window in your Python folder (open a command window and use cd or while holding shift, right click it on Windows Explorer and choose 'Open command window here'). Then just enter
pip install pyinstaller
And that's it.
The simplest way to use it is by entering on your command prompt
pyinstaller file_name.py
For more details on how to use it, take a look at this question.
There is an open source project called auto-py-to-exe on GitHub. Actually it also just uses PyInstaller internally but since it is has a simple GUI that controls PyInstaller it may be a comfortable alternative. It can also output a standalone file in contrast to other solutions. They also provide a video showing how to set it up.
GUI:
Output:
Alternatively use pyinstaller directly:
pip install pyinstaller
pyinstaller filename
I can't tell you what's best, but a tool I have used with success in the past was cx_Freeze. They recently updated (on Jan. 7, '17) to version 5.0.1 and it supports Python 3.6.
Here's the pypi
https://pypi.python.org/pypi/cx_Freeze
The documentation shows that there is more than one way to do it, depending on your needs.
http://cx-freeze.readthedocs.io/en/latest/overview.html
I have not tried it out yet, so I'm going to point to a post where the simple way of doing it was discussed. Some things may or may not have changed though.
How do I use cx_freeze?
Now you can convert it by using PyInstaller. It works with even Python 3.
Steps:
Fire up your PC
Open command prompt
Enter command pip install pyinstaller
When it is installed, use the command 'cd' to go to the working directory.
Run command pyinstaller <filename>
I've been using Nuitka and PyInstaller with my package, PySimpleGUI.
Nuitka
There were issues getting tkinter to compile with Nuikta. One of the project contributors developed a script that fixed the problem.
If you're not using tkinter it may "just work" for you. If you are using tkinter say so and I'll try to get the script and instructions published.
PyInstaller
I'm running 3.6 and PyInstaller is working great!
The command I use to create my exe file is:
pyinstaller -wF myfile.py
The -wF will create a single EXE file. Because all of my programs have a GUI and I do not want to command window to show, the -w option will hide the command window.
This is as close to getting what looks like a Winforms program to run that was written in Python.
[Update 20-Jul-2019]
There is PySimpleGUI GUI based solution that uses PyInstaller. It uses PySimpleGUI. It's called pysimplegui-exemaker and can be pip installed.
pip install PySimpleGUI-exemaker
To run it after installing:
python -m pysimplegui-exemaker.pysimplegui-exemaker
The best and easiest way is auto-py-to-exe for sure, and I have given all the steps and red flags below which will take you just 5 mins to get a final .exe file as you don't have to learn anything to use it.
1.) It may not work for python 3.9 on some devices I guess.
2.) While installing python, if you had selected 'add python 3.x to path', open command prompt from start menu and you will have to type pip install auto-py-to-exe to install it. You will have to press enter on command prompt to get the result of the line that you are typing.
3.) Once it is installed, on command prompt itself, you can simply type just auto-py-to-exe to open it. It will open a new window. It may take up to a minute the first time. Also, closing command prompt will close auto-py-to-exe also so don't close it till you have your .exe file ready.
4.) There will be buttons for everything you need to make a .exe file and the screenshot of it is shared below. Also, for the icon, you need a .ico file instead of an image so to convert it, you can use https://convertio.co/
5.) If your script uses external files, you can add them through auto-py-to-exe and in the script, you will have to do some changes to their path. First, you have to write import sys if not written already, second, you have to make a variable for eg, location=getattr(sys,"_MEIPASS",".")+"/", third, the location of example.png would be location+"/example.png" if it is not in any folder.
6.) If it is showing any error, it may probably be because of a module called setuptools not being at the latest version. To upgrade it to the latest version, on command prompt, you will have to write pip install --upgrade setuptools. Also, in the script, writing import setuptools may help. If the version of setuptools is more than 50.0.0 then everything should be fine.
7.) After all these steps, in auto-py-to-exe, when the conversion is complete, the .exe file will be in the folder that you would have chosen (by default, it is 'c:/users/name/output') or it would have been removed by your antivirus if you have one. Every antivirus has different methods to restore a file so just experiment if you don't know.
Here is how the simple GUI of auto-py-to-exe can be used to make a .exe file.
PyOxidizer can be an option here. It's pretty popular with 3.3k stars on Github. Its documentation says
PyOxidizer is capable of producing a single file executable - with a copy of Python and all its dependencies statically linked and all resources (like .pyc files) embedded in the executable. You can copy a single executable file to another machine and run a Python application contained within. It just works.
While I'm not sure if it is capable of producing .exe file PyOxidizer definitely helps with packaging and distribution.

Yocto Bitbake Recipe for Custom Python Script and PyTest

I have a custom Python Flask App and I build it for my embedded target using Yocto and Bitbake.
I also have a set of test cases that I run on my Build Machine against the app using PyTest. I'd like the build to fail if the tests fail.
I'm looking for a Yocto way of running these tests as a custom task as part of my recipe. So far by Google search has ( unusually ) come up empty.
Here is what I have implemented so far - it works but it requires the system Python3 and various pip installs. Ideally the requirements would be built in Yocto but only for the Host machine and not the target. I haven't figured out how to do that yet.
# Run the pytest test cases against the app code after it is installed
python do_run_pytest_testsuite(){
# Change to the testing directory
import os
testDir = "%s"%(d.getVar('WORKDIR', True))
os.chdir(testDir)
# Run pytest execute the test suite
from subprocess import Popen, PIPE
with open("%s/TestOutput.txt"%(testDir), "w") as testReportFile:
with Popen(['/usr/bin/python3','-m','pytest','-v','tests/test_app.py'], stdout=PIPE, bufsize=1, universal_newlines=True) as proc:
for line in proc.stdout:
testReportFile.write(line)
# Get the return code
if not proc.returncode == 0:
# Force Failure
bb.fatal("Test Cases Failed! ( %s )"%(testDir))
}
addtask run_pytest_testsuite before do_install after do_configure
How can I accomplish this self-contained within the Yocto environment and not install any PyTest dependencies for the target board.
First I would have a look at poky/meta/recipes-devtools/python to get an idea what python recipes are available (mind which release you are working on).
Then you can add a dependency on the native version of the recipes e.g. DEPENDS = "python3-native python3-yaml-native" (or whatever packages you need to run your python application)
Then add a task that runs your application
do_run_testsuite(){
python3 ${WORKDIR}/test/test_app.py
}
addtask do_run_testsuite before do_install after do_configure
Maybe also check the openembedded python layer
If you don't find all the dependencies you need, it is relatively easy to add a pip package to your own layer (just have a look at the recipes in the mentioned layers).

Getting Error: 'No module named flask' in VSCode even when I have installed flask

I want to debug an application using Python and Flask in VSCode. I have installed Flask and the app runs perfectly fine through cmd. But, when I try to debug it through VSCode, it gives the following error:
cd 'c:\Users\Aditi\CleanHandymanApp';
${env:FLASK_APP}='NewApp'; ${env:PYTHONIOENCODING}='UTF-8';
${env:PYTHONUNBUFFERED}='1'; & 'C:\Users\Aditi\envs\CleanHandymanApp\Scripts\python.exe'
'c:\Users\Aditi\.vscode\extensions\ms-python.python-2018.10.1\pythonFiles\experimental\ptvsd_launcher.py' '--client' '--host'
'localhost' '--port' '63143' '-m' 'flask' 'run' '--no-debugger' '--no-reload'
No module named flask
Can you please help me.
This error message can occur if you installed the python3 version of flask but Visual Studio Code tries to run your project with python2.
Make sure to select the correct version of python in the editor. This can be done by running the command Python: Select Interpreter from the Command Palette (Ctrl+Shift+P).
Activate your virtualenv and run
pip3 install -r requirements.txt
to reinstall all packages inside the venv.
For some reason VS Code thought I was missing all my packages the first time I debugged even though the app was running fine locally.
Sometimes you can get this error if you loaded Flask into a folder which has sub-files. For instance, if you loaded flask into the parent folder with the virtual shell instance but you're running your code in the child file (let's say parent is called crypto_files and inside that is a python source code file called blockchain.py ), then in order to get flask to run properly you'd have to run the file like this:
python crypto_files/blockchain.py
This allows your machine to see Flask running inside crypto_files but also run blockchain.py .
OR, it's possibly you could just reload Flask into the sub(child)file... blockchain.py and then you'd run it from within the subfile.
This complication is mainly due to modern "virtual instances" and shells which are basically like creating a virtual computer-machine inside your ACTUAL hard machine. Flask does this to avoid running everywhere, and since Flask is modular it allows each of your projects to run different modular configurations of Flask to suit each project precisely. The alternative would be awful: you'd have to load the fattest version of Flask with dozens of add-ons for each project, and so all your git and all your projects would have tons of extra code. Flask is built to be very small at the core to avoid this problem (too verbose!).
If you have installed flask in virtual environment, you should have activated it first.
source /path to env dir/bin/activate #in linux
workon 'name of env' #windows
Another option is add sys.path.append('d:/programas/anaconda3/lib/site-packages') in c:\Users\Aditi.vscode\extensions\ms-python.python-2018.10.1\pythonFiles\experimental\ptvsd_launcher.py
Being that "d:/programas/anaconda3/lib/site-packages" should be modified by your local python packages.
Use this command in the terminal instead of selecting run code:
python3 "insert your file name here without the quotes"
e.g.: python3 example.py
I had a variant of the issue mentioned by #confusius. I had installed both Python 3.9 and Python 3.10. I had added Flask to Python 3.10. I had been using one vscode workspace which had selected Python 3.10. I started another project in a different vscode workspace and it had selected Python 3.9 by default, which I didn't notice because I thought it would select the same Python I had already selected in the other workspace.

Run a python script with Popen with PyInstaller

I need to run a script inside a Popen inside a packaged project (with pyinstaller). The command NEEDS to run in python3 and I (obviously) need it to be portable, which means that I can't rely on the config (python/python3 notation) used on the machine I've used to generate the package neither the one where I am using it (I can't also guarantee that there will be any python in the deployment machine...).
I have computers where python is already the correct version, and others where the correct one is python3. I tried to insert #!/usr/bin python3 in the beginning of the file and then run as python but it didn't work.
subprocess.Popen(['python', 'internal.py', arg1, arg2], universal_newlines=True)
In the non-frozen environment I am able to run using sys.executable instead of python. I tried to "pyinstall" the internal.py then, "pyinstall" the rest copying the internal folder to the new folder and then call:
subprocess.Popen(['./internal/internal', arg1, arg2], universal_newlines=True)
But this doesn't work... In the console (moving to the bigger project folder) it does, but when running the package, it doesn't...
Any idea?
Ps. I can't just import the script as a class or whatever. I can't modify system environment vars (except the ones inside the pyinstaller thing...)
I saw something about the PyInstaller.compat that has a exec_python and a __wrap_python, but I couldn't use it I get errors when trying to run the package due to pkg_resources.DistributionNotFound: The 'PyInstaller' distribution was not found and is required by the application, I tried to create a hook, to add as hidden import... None worked (but the hook part maybe I got it wrong...)

Resources