files used
data.csv - dummy data set i am using
main.py - my core flask program
problem i am facing
even though pandas is installed in my virtual environment(python3.8 - virtualenv) i am getting ModuleNotFoundError for pandas when used in main.py
additional information
There is no error in interactive environment when importing pandas
within the virtual environment.
Same error is thrown by webUI by
flask application.
Your main.py script has the shebang
#!/usr/bin/python
pointing to the system interpreter. If you execute ./main.py, your script is launched with the system interpreter, not the virtual env.
Change the shebang or execute the script with your venv python:
$ python main.py
Related
I am encoutering import errors, but only when running my python scripts from cmd or windows task scheduler (effectively the same issue I assume). I have researched answers already and attempted various solutions (detailed below), but nothing has worked yet. I need to understand the problem in any case so that I can manage anything like it in the future.
Here is the issue:
Windows 10. Anaconda Python 3.9.7. Virtual enviromnent.
I have a script that works fine if I open an anaconda prompt, activate the virtual environment and run it.
However, this is where the fun starts. If I try to run the script from the non-anaconda cmd prompt deploying the commands: "C:\Users\user\anaconda3\envs\venv\python.exe" "C:\Users\user\scripts\script.py" if get the following error:
ImportError: DLL load failed while importing etree: The specified module could not be found.
Traceback includes:
"C:\Users\user\anaconda3\envs\venv\lib\site-packages\lxml\html\__init__.py", line 53, in <module>
from ..import etree
This is not as simple as one specific module not being installed, because of course running the script from within the anaconda prompt and the virtual environment works. Similar also happens when I run other scripts. Other errors I have seen include, for example:
ImportError: DLL load failed while importing _imaging: The specified module could not be found.
Traceback includes:
"C:\Users\user\anaconda3\envs\venv\lib\site-packages\PIL\Image.py", line 114, in <module>
from . import _imaging as core
Also, I think this may be somehow related. Importing numpy (1.22.3) from within the python interpreter in the virtual environment works fine, but when I try to run a test script that imports numpy it fails both from anaconda and the cmd with the following error:
ImportError: cannot import name SystemRandom
The oveall issue was noted originally when trying to run various scripts from Windows Task Scheduler with the path to python "C:\Users\user\anaconda3\envs\venv\python.exe" entered as the Program/script and the script "script.py" entered as an argument. The above errors were produced, then reproduced by running the scripts from a non-anaconda cmd.
I am looking to understand what is happening here and for a solution that can get the scripts running from the virtual enviroment from Windows Task Scheduler effectively.
Update:
I have uninstalled and reinstalled numpy (and pandas) using conda. This has left the venv with numpy==1.20.3 (and pandas=1.4.2). On attempting to re-run one of the scripts, it runs fine from within the venv in anaconda, but produces the following error when attempting to run from cmd or from within Windows Task Scheduler as above:
ImportError: Unable to import required dependencies:
numpy:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions faled. This error can happen for many reasons, often due to issues with your setup or how NumPy was installed.
We have complied some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
* The Python version is: Python3.9 from "C:\Users\user\anaconda3\envs\venv\python.exe"
* The NumPy version is "1.20.3"
and make sure that they are the versions you expect.
Please carefull study the documentation linked above for further help.
Original error was: DLL load failed while importing _multiarray_umath: The specified module could not be found.
I have looked into the solutions suggested, but am still completely at a loss, especially as to why the script runs from the venv in one place, but NOT the other.
I am trying to make a Nextflow script that utilizes a python script. My python script imports a number of modules but within Nextflow python3 does not find two (cv2 and matplotlib) of 7 modules and crashes. If I call the script directly from bash it works fine. I would like to avoid creating a docker image to run this script.
Error executing process > 'grab_images (1)'
Caused by:
Process `grab_images (1)` terminated with an error exit status (1)
Command executed:
python3 --version
echo 'processing image-1.npy'
python3 /home/hq/cv_proj/k_means2.py image-1.npy
Command exit status:
1
Command output:
Python 3.7.3
processing image-1.npy
Command error:
Traceback (most recent call last):
File "/home/hq/cv_proj/k_means2.py", line 5, in <module>
import matplotlib.pyplot as plt
ModuleNotFoundError: No module named 'matplotlib'
Work dir:
/home/hq/cv_proj/work/7f/b787c62ec420b2b5eb490603ef913f
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
I think there is a path issue as modules like numpy, sys, re, time are successfully loaded. How can I fix?
Thanks in advance
UPDATE
To assist other who may have problems using python in nextflow scripts... Make sure your shebang is correct. I was using
#!/usr/bin/python
instead of
#!/usr/bin/python3
Since all of my packages were installed with pip3 and I exclusively use python3 you need to have the right shebang.
Best to avoid absolute paths to your script(s) in your process declarations. This section of the docs is worth taking some time to read: https://www.nextflow.io/docs/latest/sharing.html#manage-dependencies, particularly the subsection on how to manage third party scripts:
Any third party script that does not need to be compiled (Bash,
Python, Perl, etc) can be included in the pipeline project repository,
so that they are distributed with it.
Grant the execute permission to these files and copy them into a
folder named bin/ in the root directory of your project repository.
Nextflow will automatically add this folder to the PATH environment
variable, and the scripts will automatically be accessible in your
pipeline without the need to specify an absolute path to invoke them.
Then the problem is how to manage your Python dependencies. You mentioned Docker is not an option. Is Conda also not an option? The config for Conda might look something like:
name: myenv
channels:
- conda-forge
- bioconda
- defaults
dependencies:
- conda-forge::matplotlib-base=3.4.3
- conda-forge::numpy=1.21.2
- conda-forge::opencv=4.5.2
Then if the above is in a file called environment.yml, create the environment with:
conda env create
See also the best practices for using Conda.
Soo, I created a small Flask Website which I wanna deploy on my Debian 10 server with Apache 2.4
At first I followed a tutorial and created the apache2 conf file and the .wsgi file. Since that didn't work well, I got the error "ModuleNotFoundError: No module named 'flask'" so I set up an virtual enviroment with
python3 -m venv venv
and changed my .wsgi to
#!/usr/bin/python3
import sys
import logging
activate_this = '/home/bot/Club-Manager/Webinterface/venv/bin/activate_this.py'
with open(activate_this) as file_:
exec(file_.read(), dict(__file__=activate_this))
logging.basicConfig(stream=sys.stderr)
sys.path.insert(0,"/home/bot/Club-Manager/Webinterface/")
import app as application
Now I get
Failed to exec Python script file '/var/www/html/Webinterface/Webinterface.wsgi'
Exception occurred processing WSGI script '/var/www/html/Webinterface/Webinterface.wsgi'., file "/var/www/html/Webinterface/Webinterface.wsgi", line 8, in <module>,
exec(file_.read(), dict(__file__=activate_this))
File "<string>", line 28, in <module>
AttributeError: 'str' object has no attribute 'decode'
At first the "activate_this.py" didn't exist for me, some research told me to just copy it from the lib/virtualenv folder which I tried (The one in the venv and the one in ~/.local/).
After some more research I found out that you should create the venv with the same python version you're using to run it (Duh), I check that and created the venv with python3 instead of just python because python3 and python2 are installed (even if "python" gets me to python 3.7, but /usr/bin/python is python 2).
I also tried the "activator.py" file from the venv/lib/virtualenv but this also didn't work (well it did but threw the also the flask not found error)
I checked and reinstalled the dependencies in the venv, I checked if they are install for python3 and they are.
So now I'm pretty lost, I guess the problem is the "activate_this.py" file but I dont know how I get a functional one or what else could lead this to break. The "Apache2 Part" to forward my request from the domain to the python script seems to work (well I guess because why else could I see the errors) so I didn't append the apache conf, but if needed I can post it ofc. I also checked the permissions for the folders, /var/www/html/Webinterface is a symlink to /home/bot/Club-Manager/Webinterface and all of them got bot:www-data which is the user and group apache2 should use
Thanks for any tips in advance
I know this is a year old, but in case anyone has the same problem...
the easiest/fastest/cleanest way I've found is to simply reference the python interpreter from within the venv from your wsgi file, like this:
#!/path/to/your/venv/bin/python
From there, your wsgi app will be using the corresponding python, no need for activating, searching for packages, etc.
I have a python file that I need to run. I have made sure to make it executable. I have added a shebang to environment python3 and I have been having no such file or directory error when trying to run this code.
I have tried to convert the line endings to unix format using dos2unix and the error still occurs. The works fine when i run it directly without shebang using python3 file_name.py command.
#!/usr/bin/env python3
import tensorflow as tf
import numpy as np
import roslib
roslib.load_manifest('joint_states_listener')
roslib.load_manifest('spider_control')
import pylab as plt
This python file a custom plugin that has to run with ros. I have made sure to source the catkin environment. Do you think that shebang expression error is caused because I have sourced the terminal to another catkin directory where my ros packages are located ?
I have launched my robot in a terminal using
roslaunch spider_gazebo spider_world.launch
this command says ros to launch the spider_world launch file from spider_gazebo package. This has started all my controllers. Later i opened terminal and I exported my ros environment paths and sourced that terminal to catkin workspace and then used this command
rosrun spider_control control.py
where control.py is my python node and spider_control is the package. the error is
/usr/bin/env: python3.6.7: No such file or directory
I am using linux. I am trying to run daemon from function in django views. I want to run shell command from a view in Djangp app. I am using python 2.7. Command needs python2.7 path.
My app will be like plug n play. So on system on which it is going to install may have python installed on different location. So I want to make python path dynamic.
Command will be
usr/bin/python2.7 filename.py --start
On my system path is usr/bin/python2.7.
I found follwing using os.
On python shell I tried following code & I get what I want
import os
getPyPath = os.popen('which python2.7', 'r')
pyPath = getPyPath.read()
pyPath.rstrip()
I got o/p as which is expected as below
usr/bin/python2.7
So now how to get this code is django app view function & run it so that I can get python path in a variable.
I found pythons subprocess module call using which we can run command through shell using shell=True.
So can I get above code running in django view function using subprocess call??
If not what is the other ways to get python path in variable in function django views.
Thanks in advance.
To view the full path to the current Python interpreter, use sys.executable
import sys
print(sys.executable)