Im working on my Mac and I just fixed the script.
Then I wanted to make an edit and no nothing seem to work anymore.
I get the error message;
Traceback (most recent call last):
File "lu.py", line 1, in
import cplex
ModuleNotFoundError: No module named 'cplex'
I did not made the script and my knowledge about python is not much
When I want to start the script I use
terminal -- cd (file location) pyzton3 lu.py
Anyone who can help me with this problem?
This happens when the Python interpreter cannot find the CPLEX installation.
Usually this can be fixed by setting the PYTHONPATH variable correctly. Make sure that this variable contains the CPLEX Python libraries.
Assuming you are on Linux, you can do the following:
Find the CPLEX (or COS) installation folder.
Assuming this is /path/to/COS then you should have folder /path/to/COS/cplex/python/ and in that folder different sub-directories that correspond to different Python versions.
Assuming you are running Python 3.6, execute the following in your shell: export PYTHONPATH="$PYTHONPATH:/path/to/COS/cplex/python/3.6/x86-64_linux" and then execute your script as before in the same terminal.
Related
I am encoutering import errors, but only when running my python scripts from cmd or windows task scheduler (effectively the same issue I assume). I have researched answers already and attempted various solutions (detailed below), but nothing has worked yet. I need to understand the problem in any case so that I can manage anything like it in the future.
Here is the issue:
Windows 10. Anaconda Python 3.9.7. Virtual enviromnent.
I have a script that works fine if I open an anaconda prompt, activate the virtual environment and run it.
However, this is where the fun starts. If I try to run the script from the non-anaconda cmd prompt deploying the commands: "C:\Users\user\anaconda3\envs\venv\python.exe" "C:\Users\user\scripts\script.py" if get the following error:
ImportError: DLL load failed while importing etree: The specified module could not be found.
Traceback includes:
"C:\Users\user\anaconda3\envs\venv\lib\site-packages\lxml\html\__init__.py", line 53, in <module>
from ..import etree
This is not as simple as one specific module not being installed, because of course running the script from within the anaconda prompt and the virtual environment works. Similar also happens when I run other scripts. Other errors I have seen include, for example:
ImportError: DLL load failed while importing _imaging: The specified module could not be found.
Traceback includes:
"C:\Users\user\anaconda3\envs\venv\lib\site-packages\PIL\Image.py", line 114, in <module>
from . import _imaging as core
Also, I think this may be somehow related. Importing numpy (1.22.3) from within the python interpreter in the virtual environment works fine, but when I try to run a test script that imports numpy it fails both from anaconda and the cmd with the following error:
ImportError: cannot import name SystemRandom
The oveall issue was noted originally when trying to run various scripts from Windows Task Scheduler with the path to python "C:\Users\user\anaconda3\envs\venv\python.exe" entered as the Program/script and the script "script.py" entered as an argument. The above errors were produced, then reproduced by running the scripts from a non-anaconda cmd.
I am looking to understand what is happening here and for a solution that can get the scripts running from the virtual enviroment from Windows Task Scheduler effectively.
Update:
I have uninstalled and reinstalled numpy (and pandas) using conda. This has left the venv with numpy==1.20.3 (and pandas=1.4.2). On attempting to re-run one of the scripts, it runs fine from within the venv in anaconda, but produces the following error when attempting to run from cmd or from within Windows Task Scheduler as above:
ImportError: Unable to import required dependencies:
numpy:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions faled. This error can happen for many reasons, often due to issues with your setup or how NumPy was installed.
We have complied some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
* The Python version is: Python3.9 from "C:\Users\user\anaconda3\envs\venv\python.exe"
* The NumPy version is "1.20.3"
and make sure that they are the versions you expect.
Please carefull study the documentation linked above for further help.
Original error was: DLL load failed while importing _multiarray_umath: The specified module could not be found.
I have looked into the solutions suggested, but am still completely at a loss, especially as to why the script runs from the venv in one place, but NOT the other.
I am trying to make a Nextflow script that utilizes a python script. My python script imports a number of modules but within Nextflow python3 does not find two (cv2 and matplotlib) of 7 modules and crashes. If I call the script directly from bash it works fine. I would like to avoid creating a docker image to run this script.
Error executing process > 'grab_images (1)'
Caused by:
Process `grab_images (1)` terminated with an error exit status (1)
Command executed:
python3 --version
echo 'processing image-1.npy'
python3 /home/hq/cv_proj/k_means2.py image-1.npy
Command exit status:
1
Command output:
Python 3.7.3
processing image-1.npy
Command error:
Traceback (most recent call last):
File "/home/hq/cv_proj/k_means2.py", line 5, in <module>
import matplotlib.pyplot as plt
ModuleNotFoundError: No module named 'matplotlib'
Work dir:
/home/hq/cv_proj/work/7f/b787c62ec420b2b5eb490603ef913f
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
I think there is a path issue as modules like numpy, sys, re, time are successfully loaded. How can I fix?
Thanks in advance
UPDATE
To assist other who may have problems using python in nextflow scripts... Make sure your shebang is correct. I was using
#!/usr/bin/python
instead of
#!/usr/bin/python3
Since all of my packages were installed with pip3 and I exclusively use python3 you need to have the right shebang.
Best to avoid absolute paths to your script(s) in your process declarations. This section of the docs is worth taking some time to read: https://www.nextflow.io/docs/latest/sharing.html#manage-dependencies, particularly the subsection on how to manage third party scripts:
Any third party script that does not need to be compiled (Bash,
Python, Perl, etc) can be included in the pipeline project repository,
so that they are distributed with it.
Grant the execute permission to these files and copy them into a
folder named bin/ in the root directory of your project repository.
Nextflow will automatically add this folder to the PATH environment
variable, and the scripts will automatically be accessible in your
pipeline without the need to specify an absolute path to invoke them.
Then the problem is how to manage your Python dependencies. You mentioned Docker is not an option. Is Conda also not an option? The config for Conda might look something like:
name: myenv
channels:
- conda-forge
- bioconda
- defaults
dependencies:
- conda-forge::matplotlib-base=3.4.3
- conda-forge::numpy=1.21.2
- conda-forge::opencv=4.5.2
Then if the above is in a file called environment.yml, create the environment with:
conda env create
See also the best practices for using Conda.
I installed the library ase. Since then I can't execute python3 on terminal neither run its python files. I found out with the command which python3 command that with python3 is located at '/usr/local/bin/python3'. The libraries, including ase, are located at '/usr/local/lib/python3.9/site-packages' apparently.
See:
pip3 install --upgrade git+https://gitlab.com/ase/ase.git#master
WARNING: The scripts ase, ase-build, ase-db, ase-gui, ase-info and ase-run are installed in '/usr/local/Cellar/python#3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
This is the error I got when I try to run python3 on terminal:
Fatal Python error: init_sys_streams: can't initialize sys standard streams
Python runtime state: core initialized
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/ase/io/__init__.py", line 1, in <module>
ModuleNotFoundError: No module named 'ase'
I think that something has to be wrong at my .bash_profile file however, I'm not getting the error. Here is my edited .bash_profile:
export PYTHONPATH=/usr/local/lib/python3.9/site-packages/ase:$PYTHONPATH
export PATH=/usr/local/Cellar/python#3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/bin:$PATH
Hope you can help me, thank you!
Update:
I changed my PYTHONPATH from:
export PYTHONPATH=/usr/local/lib/python3.9/site-packages/ase:$PYTHONPATH
to:
export PYTHONPATH=/usr/local/lib/python3.9/site-packages:$PYTHONPATH
Now I can run python3 on terminal again, however I try to execute ase directly on the terminal it seems that the command is not found.
-bash: ase: command not found
I have already checked out the requires (matplotlib, scipy, numpy) and they are all installed. Seems that everything is wrong with the PATH. I don't know what could be.
The binaries files related to ase package are not anymore into this directory "/usr/local/Cellar/python#3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/bin". I don't know where they are since the installation was successfully done.
According to https://unix.stackexchange.com/questions/240037/why-did-pip-install-a-package-into-local-bin I redefined my paths to:
export PYTHONPATH=/usr/local/lib/python3.9/site-packages:$PYTHONPATH
export PATH=/usr/local/bin:$PATH
and seems to be working, I already ran a program containing that package, however when I enter the folder '/usr/local/bin' I don't find those binary files.
If you can help me again I will be grateful.
I am installing Cplex python API into my MacBook. So I followed the tutorial given in YouTube (https://www.youtube.com/watch?v=oINxanpv_2M) and have installed the script & library for Cplex using the Terminal instructions:
cd /Users/my_name/IBM_CPLEX/CPLEX_Studio1210/python
python3 setup.py install
this part works alright with everything installed, just a note from the Terminal at the beginning saying 'python wrapper is not found'.
Then I tested an existing Cplex problem using
cd IBM_CPLEX/CPLEX_Studio1210/python/examples/mp/modeling
python3 production.py
, which returned:
Model: production
- number of variables: 6
- binary=0, integer=0, continuous=6
- number of constraints: 5
- linear=5
- parameters: defaults
Traceback (most recent call last):
File "production.py", line 101, in <module>
if model.solve():
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/docplex/mp/model.py", line 4152, in solve
return self.fatal("CPLEX runtime not found: please install CPLEX or solve this model on DOcplexcloud")
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/docplex/mp/model.py", line 884, in fatal
self._error_handler.fatal(msg, args)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/docplex/mp/error_handler.py", line 210, in fatal
raise DOcplexException(resolved_message)
docplex.mp.utils.DOcplexException: CPLEX runtime not found: please install CPLEX or solve this model on DOcplexcloud
So my initial guess is that my python 3 has been installed in another directory under the Library folder, and a path will be needed to link the Cplex engine to the python solver (if correct). Inside the YouTube tutorial, a PATH is actually provided, so in my case it's written as:
PATH="/Library/Frameworks/Python.framework/Versions/3.8/bin:${PATH}"
export PATH
export PYTHONPATH=/Users/my_name/IBM_CPLEX/CPLEX_Studio1210/cplex/python/3.7/x86-64_osx
I have actually installed Cplex python API into a Windows PC previously. The path-adding step was done by visiting:
Control Panel -> System and Security -> System -> Advanced system settings -> Environment Variables. Visit (https://www.youtube.com/watch?v=hqGZzRh00y0) for specific further steps.
However, I tried quite a lot of ways adding this path to my Mac, and so far I've used those to write the above path but none worked:
nano .bash_profile.pysave
nano .bash_profile
sudo nano /etc/paths
That's where I got stuck. Any help and comment would be highly appreciated. I am new to Mac Terminal. Anaconda is also installed in my Mac if that's helpful.
Sorry for the massive details - was trying to put down everything in case there was something unnoticed that I have done wrong. Thanks.
Added:
A small issue in the process above was that my Python was actually 3.8. So after adding the path export PYTHONPATH=/Users/my_name/IBM_CPLEX/CPLEX_Studio1210/cplex/python/3.7/x86-64_osx and running a cplex example, a message popped up inside the Terminal:
Exception: CPLEX 12.10.0.0 is not compatible with this version of Python.
Need to download an older 3.7 version then did the $ python3.7 setup.py install process again after changing my directory to IBM_CPLEX/CPLEX_Studio1210/python.
Some similar issues about Python version have been discussed in the post: CPLEX and Python 3.7.
I have packages installed under /usr/local/lib and I added that in my PATH as well, but then I try to import it in any of my python scripts I get an error saying module not found.
-bash-4.2$ pip2 list | grep pytest
pytest-mock 2.0.0
My PATH:
echo $PATH
/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/bin
ERROR:
-bash-4.2$ python2
>>> import pytest
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named pytest
Only if the packages is installed under my /users/user-name/.local/bin folder, it is reflected else it is not.
My usecase is to use this machine as a slave for my Jenkins setup. I tried injecting this PATH directly to the job during build process as well. Didn't work for me.
I have been stuck on this for quite some while, any help on this is greatly appreciated.
First thing, it's generally a good idea to use virtualenv to create Python environments - installing Python packages system-wide is asking for trouble.
Second, your path may not work because you are setting PATH in the way that Jenkins ignores. The simplest solution is to provide full path to file: /usr/local/bin/pytest.
The safest way is to combine two above - create virtualenv, install pytest in it and rovide full path when using (note: you don't need to activate virtualenv to use it).