For my personal project, I'm trying to analyze the technical debt of various python libraries. For this, I made python script which downloads the library, checks out to each merged commit and runs the analysis. One part of the analysis is to get the test coverage.
The easiest way I found to get it is to run
coverage run --source={library}/ setup.py test
However, I want to have this automated so I'm trying to run this command in a shell script from the python script mentioned above.
The script shall change the dir to the library, switch to virtualenv (automatically created in the previous steps of the analysis) and run the coverage. However, it fails on Requirement error which indicates that it does not actually switch the virtualenv and stays in the virtualenv of the analysis project.
The shell script looks like this:
#!/usr/bin/env bash
# Args
# $1 proj_path
# $2 proj_name
# $3 venv_name
cd $1
source `which virtualenvwrapper.sh`
workon $3
coverage run --source=$2/ setup.py test
coverage report
And it's called from the python script like this:
subprocess.call(["analyzer/run_coverage.sh", self.repo_path, self.repo_name, self.venv_name])
Could you please help me how to switch the venv in the shell script? Thanks!
Script 'activate' activates virtual env, and 'deactivate' deactivates virtual env:
https://virtualenv.pypa.io/en/stable/userguide/
Example of the loop over two virtual environments stored in bash array:
venvs=(~/venvs/py3.6.4 ~/venvs/py3.7.0)
$ for i in "${venvs[#]}"
do
source "$i"/bin/activate
which python
python --version
deactivate
done
/home/gbajson/venvs/py3.6.4/bin/python
Python 3.6.4
/home/gbajson/venvs/py3.7.0/bin/python
Python 3.7.0
Related
I am trying to setup my project environment from a shell script on ubuntu so that I can skip basic setup every time.
I use my virtual environmet cv using the command workon cv.
But I am having trouble doing so using a shell script. I tried the the script
#!/bin/bash
workon cv
But I get the error
workon: command not found
I try to list all the venv I have
pran#pran-HP-65-Notebook-PC:~$ lsvirtualenv
cv
==
virtual-py2
===========
Also, I thought of locating it
(cv) pran#pran-HP-65-Notebook-PC:~/.virtualenvs$ l
cv/ postdeactivate preactivate* prermvirtualenv*
get_env_details* postmkproject* predeactivate virtual-py2/
initialize postmkvirtualenv premkproject*
postactivate postrmvirtualenv* premkvirtualenv*
How can I do it?
You cannot execute workon command outside of Python virtual environment.
If you are interested, check out this article that will walk you through all of the steps required to set up your virutal env.
If it is not what you need then please clarify your requirements for a project environment.
I found the solution:
My venv was located in .virtualenvs. So, I put the command in the bash file start.sh (meant to be executed before working on my project).
#start.sh
source ~/.virtualenvs/cv/bin/activate
And run it using
$ source start.sh
It works perfectly 👍
In order to get some software running I need to 1. Run a script that will execute a remote license manage, 2. Execute a shell script to start the software. I can do this by opening a command window in the directory with the rlm , and then type ./rlm to run the Linux executable. Then I can go into the directory that contains the shell script, open a terminal in that location and run ./myshell.sh. This opens the GUI for my software.
I would like to run these steps using a single Python script. I have tried:
#change the working directory...
os.chdir(r'/opt/mysoftwarelocation')
#confirm location change...
print(os.getcwd() )
#run ./rlm...
os.system('./rlm')
At this point I can see from a python terminal that the rlm is running.
I would then like to run the code below to run the shell script...
os.chdir(r'/opt/mysoftwarelocation/sumsubdirectory')
print(os.getcwd() )
os.system('./some.sh')
Unfortunately, after os.system('./rlm') finishes the script stalls and will not execute further and without an errors.
How to I get the second part of my script to run within a single Python script?
Have you tried to run the rlm command in the background?
subprocess module gives a nice interface for that
I'm trying to set up a recurring Python task through windows task scheduler.
I have had success when I input the path to 'python.exe' and provide the script's path as a parameter to windows task scheduler (see screenshot below)
However, I want to be able to choose a particular virtual environment in which to run the script. I don't have much knowledge of venv, and I typically use it by opening cmd and running Scripts\activate.bat in the desired virtual environment directory.
How can I accomplish 'run task x in venvxxx every 24 hours' using windows task scheduler?
Create batch file with these commands:
c:\__full_path_to_virtualenv__\Scripts\activate.bat && python __full_path_to_python_script__.py
&& means run command2 if command1 completed successfully.
Then set that batch file as script to run. You don't need to set any additional arguments in task scheduler (or you can set them in batch file anyway) and can set Start in if script has to read/write from specific directory and uses relative paths.
Though the answer by mx0 above seems to work, I have set up Task Scheduler to run a flask web app on bootup. In this case, manual starting works fine, but manual ending does not. Ending the task kills the cmd.exe task that sets up the virtual environment, but the python.exe continues to run.
The solution that I found worked was from this reddit post which skips the virtual environment activation to call the python executable directly:
path\to\venv\Scripts\python.exe path\to\script.py
I'm not sure how robust this will be, but at least this way ending the task will end the python.exe
This is more verbose but very easy to understand, and - I found the most important - much easier than using Windows Task Scheduler settings when you have lots of scripts. To create another you just copy the .bat file and change one line.
Save this as a .bat file and point to it under Actions > Start a Program > Program/Script:, with no arguments or "Start in" necessary.
set original_dir=%CD%
set venv_root_dir="C:\Python-Venvs\env-name"
cd %venv_root_dir%
call %venv_root_dir%\Scripts\activate.bat
python your_script.py <arg1> <arg2>
call %venv_root_dir%\Scripts\deactivate.bat
cd %original_dir%
exit /B 1
For an installed command-line program, you can replace python your_script.py <arg1> <arg2> ... with <program name> <arg1> <arg2> ....
In addition it's simple to add another script on the following line, rather than attempting to parse sequential scripts into a one-liner for Task Scheduler.
I tried with mx0's answer and it works fine as long as your script does not take too long to finish.
I use a different approach in the task scheduler instead using batch files:
In "Program/script" textbox you set the path to Python executable (in my case is inside the virtualenv folder).
"Add arguments" => Just the name of your Python script (name.ppy).
"Start in" => The full path of your Python script (without the name.py).
This way the script runs and wait until the end.
My solution is almost identical to mx0, but I've added an extra step to ensure environment parameters each time. Edit the path/to/app for the app_path variable.
It may be a little redundant to check the environment setup every time, but I like ensuring my environment is set.
Simply schedule the execute_app.bat file or run in the cmd prompt. Deactivate command is not needed unless running from an Anaconda prompt. If you use a full path for path/to/app this file can be executed from any directory. I also have a Linux solution using execute_app.sh file below from a terminal.
This answer has been edited to simplify, and to use variables to make this easier to adapt to new projects.
App structure:
app/bin/app.py
app/bin/execute_app.bat
app/env/requirements.txt
# execute_app.bat file
# windows solution
SETLOCAL
set app_path=path/to/app
set env_path=%app_path%/env
set activ=%env_path%/Scripts/activate.bat
set req=%env_path%/requirements.txt
set app=%app_path%/bin/app.py
py -m venv %env_path% && %activ% && python -m pip install --upgrade pip && pip install -r %req% && python %app%
ENDLOCAL
#!/bin/bash
# execute_app.sh file
# Linux solution
app_path='path/to/app'
env_path=$app_path'/env'
activ=$env_path'/bin/activate'
req=$env_path'/requirements.txt'
app=$app_path'/bin/app.py'
python3 -m venv $env_path &&
. $activ &&
python3 -m pip install --upgrade pip &&
pip install -r $req &&
python $app &&
deactivate
The selected answer for this question is not correct. If you review the comments, you'll see the problem.
My answer builds off of #Nick P's answer (the #2 answer currently). His batch file will work, but you'll want to change the exit code from 1 to 0 if you want Windows Task Scheduler to report the task completed successfully. Also, simply calling the .bat file on the "Program/Script" line will not work. Instead, you need to put the name of your shell as the "Program/Script" to run (for instance, cmd), then put "/c name-of-batch-file.bat" goes in the "Add arguments (optional):" field. And finally, put the path to the batch file (minus the file name) in the "Start in (optional):" field.
It should look something like this:
REM Windows batch script to run 1+ Python program/scripts, sequentially, within
REM their virtual environment. This can be called from Windows Task Scheduler.
set original_dir=%CD%
set venv_root_dir="C:\Users\myUsername\myProjects\nameOfProject"
cd %venv_root_dir%
call %venv_root_dir%\Scripts\activate.bat
python nameOfPythonProgram.py
call %venv_root_dir%\Scripts\deactivate.bat
cd %original_dir%
exit /B 1
Copied this from nmpowell on github and it works fine. Run from task scheduler
Solution
Patching the path is all necessary.
The quick way would be a bat script that source environment activation script at the beginning
#call PATH_TO_MY_VENV/bin/activate.bat
python app.py
Proceeding Problem
Later you will realize that the python job starts fine, but won't stop when Windows Scheduler stop it.
When taskengine.exe decides to stop the job, the intermediate cmd.exe (bat script) process is killed and the Python.exe will be left straw. since the cmd.exe(bat script) won't signal python.exe to stop on exit.
Final Solution
Let Windows Task Scheduler or taskengine.exe launch python.exe directly without a middle-man script.
Previous answers launch python.exe directly with py script, this works for simple modules, but not for some binary module in conda environment.
For binary modules in conda to work, you can create a utility module named e.g. patch_conda_path to patch PATH variable in os.environ based on sys.base_exec_prefix. Copy the patching work that activate.bat does, just in python.
below example has been tested for conda virtual environment:
import is, sys
conda_base = sys.base_exec_prefix
subps = [";", "library\\mingw-w64\\bin;", "library\\usr\\bin;", "library\\bin;", "scripts;", "bin;", "condabin;"]
conda_paths = ""
for p in subps :
_p = os.path.join(conda_base, p)
if _p in os.environ["PATH"]:
continue
else:
conda_paths += _p
os.environ["PATH'"] = conda_paths + os.environ["PATH"]
Import this module at the beginning of your main script.
import patch_conda_path
... original main script ...
In task scheduler set
program to ... conda environment path...\python.exe,
arguments to your py script file name and
start in to your py script folder.
I am trying to make python3 executable scripts and run them from shell.I have python 3.4.0 installed on my system.
So, I added '/home/spandan/python_codes' directory to PYTHONPATH, as I am planning to keep my scripts and modules here.
However, while trying to execute these, the above error is thrown by the system, and the scripts wont execute unless I go into the python_codes directory and then execute them.
Executing python program : Here I found out that PYTHONPATH is irrelevant while making scripts, and also how to set the python shebang. So I set mine as #!/usr/bin/env python3.4.0
Is it correct?
You don't have to put your python codes in a global path. Just make your python 3.4 interpreter interpreter available globally. For that, edit .bash_profile or .bashrc file in your home directory and add the following line:
export PATH=${PATH}:/usr/bin/python3
That will make python3 executable irrespective of your current working directory.
In order to execute code from your codes directory, you just have to write:
$ python3 ./your_code.py
Another way is to add the shebang at the top of your code as
#/usr/bin/python3
and change the permission to executable by the current user (by default it will not have execute permission).
$ chmod 744 your_code.py
and then executing the script directly as
$ your_code.py
I hope I could address your problem.
Another way to do it is to use python-is-python2 or python-is-python3 debian packages with /usr/bin/env python sheabang.
This option will let the end user select which interpreter he want to use while maintaining your code versionless and avoiding some people to install unwanted interpreter versions.
As an example, run this commands to make a sample file:
cat << EOF > version.py && chmod +x version.py
#!/usr/bin/env python
import sys
print(sys.version)
EOF
And run the following command if you want to set python2 as default:
sudo apt install python-is-python2
Or run this command if you want to set python3 as default:
sudo apt install python-is-python3
Finally you could see the python interpreter version that you have been selected by running:
./version.py
The only drawback of this solution is that you could make your python scripts compatible across python interpreter versions, but thats your decision!
You can install which you want and change from python2 to python3 and vice- versa as you want, but the idea is to fix a python interpreter version in a system and not change it again unless definitive upgrade.
References:
https://askubuntu.com/questions/1296790/python-is-python3-package-in-ubuntu-20-04-what-is-it-and-what-does-it-actually
I have an account to a computing cluster that uses Scientific Linux. Of course I only have user access. I'm working with python and I need to run python scripts, so I need to import some python modules. Since I don't have root access, I installed a local python copy on my $HOME with all the required modules. When I run the scripts on my account (hosting node), they run correctly. But in order to submit jobs to the computing queues (to process on much faster machines), I need to submit a bash script that has a line that executes the scripts. The computing cluster uses SunGrid Engine. However when I submit the bash script, I get an error that the modules I installed can't be found! I can't figure out what is wrong. I hope if you can help.
You could simply call your python program from the bash script with something like: PYTHONPATH=$HOME/lib/python /path/to/my/python my_python_script
I don't know how SunGrid works, but if it uses a different user than yours, you'll need global read access to your $HOME. Or at least to the python libraries.
First, whether or not this solution works for you depends heavily on how the cluster is set up. That said, the general solution to your problem is below. If the compute cluster has access to the same files as you do in your home directory, I see no reason why this would not work.
You need to be using a virtualenv. Install your software inside that virtualenv along with any additional python packages you need. Then in your batch bash script, provide the full path to the python interpreter within that virtualenv.
Note: to install python packages inside your virtualenv, you need to use the pip instance that is in your virtualenv, not the system pip.
Example:
$ virtualenv foo
$ cd foo
$ ./bin/pip install numpy
Then in your bash script:
/path/to/foo/bin/python /path/to/your/script.py
Have you tried to add these in your python code:
import sys
sys.path.append("..")
from myOtherPackage import myPythonFile
This works very well for my code when I run it on Cluster and I wanted to call my "myPythonFile" from other package "myOtherPackage"