I am a beginner and I am starting using a local cluster that works with Slurm.
I am able to execute some python codes with the usual modules (numpy, scipy, etc..) but as I try to run a script that includes my own library: myownlib.py, the following message is displayed:
No module named myownlib
I sought a lot for the solution, probably looking in the wrong direction. Hereby what I tried to fix this:
I created an environment file, with conda;
I wrote the following test.sh
(That led to the error mentioned before)
#!/bin/bash
module purge
source myownlib-devel #This is the name I gave into the environment file)
/usr/bin/python ~/filexample.py
Any suggestions?
(Thank you in advance...)
One of the most probable cause could be a difference in Python version between the login node where you created the environment and the compute nodes. If you loaded a specific Python module with module load for creating the virtual env, you should load the same module in the submission script. The default Python version for the login node could be Python 3 while the default version for the compute nodes could be Python 2, depending on the Linux distribution and list of modules loaded.
Related
How do I setup a Python virtual environment with the FreeCAD library embedded as to enable import as a module into scripts?
I would like to avoid using the FreeCAD GUI as well as being dependent on having FreeCAD installed on the system, when working with Python scripts that use FreeCAD libraries to create and modify 3D geometry. I hope a Python virtual environment can make that possible.
I am working in PyCharm 2021.1.1 with Python 3.8 in virtualenv on Debian 10.
I started out with FreeCAD documentation for embedding in scripts as a basis:
https://wiki.freecadweb.org/Embedding_FreeCAD
In one attempt, I downloaded and unpacked .deb packages from Debian, taking care to get the correct versions required by each dependency. In another attempt, I copied the contents of a FreeCAD flatpak install, as it should contain all the libraries that FreeCAD depends on.
After placing the libraries to be imported in the virtual maching folder, I have pointed to them with sys.path.append() as well as PyCharm's Project Structure tool in various attempts. In both cases the virtual environment detects where FreeCAD.so is located, but fails to find any of its dependencies, even when located in the same folder. When importing these dependencies explicitly, each of them have the same issue. This leads to a dead end when an import fails because it does not define a module export function according to Python:
ImportError: dynamic module does not define module export function (PyInit_libnghttp2)
I seem to be looking at a very long chain of broken dependencies, even though I make the required libraries available and inform Python where they are located.
I would appreciate either straight up instructions for how to do this or pointers to documentation that describes importing FreeCAD libraries in Python virtual environments, as I have not come across anything that specific yet.
I came across a few prior questions which seemed to have similar intent, but no answers:
Embedding FreeCAD in python script
Is it possible to embed Blender/Freecad in a python program?
Similar questions for Conda focus on importing libraries from the host system rather than embedding them in the virtual environment:
Incude FreeCAD in system path for just one conda virtual environment
Other people's questions on FreeCAD forums went unanswered:
https://forum.freecadweb.org/viewtopic.php?t=27929
EDIT:
Figuring this out was a great learning experience. The problem with piecing dependencies together is that for that approach to work out, everything from the FreeCAD and its dependencies to the Python interpreter and its dependencies seems to need to be built on the same versions of the libraries that they depend on to avoid causing segmentation faults that brings everything to a crashing halt. This means that the idea of grabbing FreeCAD modules and libraries it depends on from a Flatpak installation is in theory not horrible, as all parts are built together using the same library versions. I just couldn't make it work out, presumably due to how the included libraries are located and difficulty identifying an executable for the included Python interpreter. In the end, I looked into the contents of the FreeCAD AppImage, and that turned out to have everything needed in a folder structure that appears to be very friendly to what PyCharm and Python expects from modules and libraries.
This is what I did to get FreeCAD to work with PyCharm and virtualenv:
Download FreeCAD AppImage
https://www.freecadweb.org/downloads.php
Make AppImage executable
chmod -v +x ~/Downloads/FreeCAD_*.AppImage
Create folder for extracting AppImage
mkdir -v ~/Documents/freecad_appimage
Extract AppImage from folder (note: this expands to close to 30000 files requiring in excess of 2 GB disk space)
cd ~/Documents/freecad_appimage
~/Downloads/./FreeCAD_*.AppImage --appimage-extract
Create folder for PyCharm project
mkdir -v ~/Documents/pycharm_freecad_project
Create pycharm project using Python interpreter from extracted AppImage
Location: ~/Documents/pycharm_freecad_project
New environment using: Virtualenv
Location: ~/Documents/pycharm_freecad_project/venv
Base interpreter: ~/Documents/freecad_appimage/squashfs-root/usr/bin/python
Inherit global site-packages: False
Make available to all projects: False
Add folder containing FreeCAD.so library as Content Root to PyCharm Project Structure and mark as Sources (by doing so, you shouldn't have to set PYTHONPATH or sys.path values, as PyCharm provides module location information to the interpreter)
File: Settings: Project: Project Structure: Add Content Root
~/Documents/freecad_appimage/squashfs-root/usr/lib
After this PyCharm is busy indexing files for a while.
Open Python Console in PyCharm and run command to check basic functioning
import FreeCAD
Create python script with example functionality
import FreeCAD
vec = FreeCAD.Base.Vector(0, 0, 0)
print(vec)
Run script
Debug script
All FreeCAD functionality I have used in my scripts so far has worked. However, one kink seems to be that the FreeCAD module needs to be imported before the Path module. Otherwise the Python interpreter exits with code 139 (interrupted by signal 11: SIGSEGV).
There are a couple of issues with PyCharm: It is showing a red squiggly line under the import name and claiming that an error has happened because "No module named 'FreeCAD'", even though the script is running perfectly. Also, PyCharm fails to provide code completion in the code view, even though it manages to do so in it's Python Console. I am creating new questions to address those issues and will update info here if I find a solution.
I faced this silly problem since the last 2 days, and it turns me mad.
Since I installed python 3.9 on my new Windows 10 machine, when I run the command python3 -m venv .\venv, I get the error "Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases."
If I switch off the app execution aliases, as mentioned in the message, I got a message:
'python3' is not recognized as an internal or external command,
operable program or batch file.
So, I looked at similar issues on Stackoverflow and fixed the path in the environment variables (in the user variables and in the system variables).
I added to the variable 'paths' the 2 following entries:
C:\Users{myuser}\AppData\Local\Programs\Python\Python39
C:\Users{myuser}\AppData\Local\Programs\Python\Python39\Scripts\
In all cases seen on Stack, that should have been enough. Not in my case.
I rebooted my machine twice, to make sure the new variables were picked up, but it's still the same.
I must add that I am using daily my Jupyter Lab environment, and everything works fine.
It's just this venv that refuses to work.
Does anyone has a hint on other places I should look for ?
Solution found thanks to Michael Vine.
To summarize:
referring to python instead of python3 in the command line
using [virtualenv venv] instead of [python -m venv .\venv] as I did
I want to debug an application using Python and Flask in VSCode. I have installed Flask and the app runs perfectly fine through cmd. But, when I try to debug it through VSCode, it gives the following error:
cd 'c:\Users\Aditi\CleanHandymanApp';
${env:FLASK_APP}='NewApp'; ${env:PYTHONIOENCODING}='UTF-8';
${env:PYTHONUNBUFFERED}='1'; & 'C:\Users\Aditi\envs\CleanHandymanApp\Scripts\python.exe'
'c:\Users\Aditi\.vscode\extensions\ms-python.python-2018.10.1\pythonFiles\experimental\ptvsd_launcher.py' '--client' '--host'
'localhost' '--port' '63143' '-m' 'flask' 'run' '--no-debugger' '--no-reload'
No module named flask
Can you please help me.
This error message can occur if you installed the python3 version of flask but Visual Studio Code tries to run your project with python2.
Make sure to select the correct version of python in the editor. This can be done by running the command Python: Select Interpreter from the Command Palette (Ctrl+Shift+P).
Activate your virtualenv and run
pip3 install -r requirements.txt
to reinstall all packages inside the venv.
For some reason VS Code thought I was missing all my packages the first time I debugged even though the app was running fine locally.
Sometimes you can get this error if you loaded Flask into a folder which has sub-files. For instance, if you loaded flask into the parent folder with the virtual shell instance but you're running your code in the child file (let's say parent is called crypto_files and inside that is a python source code file called blockchain.py ), then in order to get flask to run properly you'd have to run the file like this:
python crypto_files/blockchain.py
This allows your machine to see Flask running inside crypto_files but also run blockchain.py .
OR, it's possibly you could just reload Flask into the sub(child)file... blockchain.py and then you'd run it from within the subfile.
This complication is mainly due to modern "virtual instances" and shells which are basically like creating a virtual computer-machine inside your ACTUAL hard machine. Flask does this to avoid running everywhere, and since Flask is modular it allows each of your projects to run different modular configurations of Flask to suit each project precisely. The alternative would be awful: you'd have to load the fattest version of Flask with dozens of add-ons for each project, and so all your git and all your projects would have tons of extra code. Flask is built to be very small at the core to avoid this problem (too verbose!).
If you have installed flask in virtual environment, you should have activated it first.
source /path to env dir/bin/activate #in linux
workon 'name of env' #windows
Another option is add sys.path.append('d:/programas/anaconda3/lib/site-packages') in c:\Users\Aditi.vscode\extensions\ms-python.python-2018.10.1\pythonFiles\experimental\ptvsd_launcher.py
Being that "d:/programas/anaconda3/lib/site-packages" should be modified by your local python packages.
Use this command in the terminal instead of selecting run code:
python3 "insert your file name here without the quotes"
e.g.: python3 example.py
I had a variant of the issue mentioned by #confusius. I had installed both Python 3.9 and Python 3.10. I had added Flask to Python 3.10. I had been using one vscode workspace which had selected Python 3.10. I started another project in a different vscode workspace and it had selected Python 3.9 by default, which I didn't notice because I thought it would select the same Python I had already selected in the other workspace.
I have a code in python that I use
import requests
It´s works if I run the code on command line.
But when I put in systemd It didn't work.
In log I see error:
Aug 2 15:08:19 beaglebone python3[2140]: import requests
Aug 2 15:08:19 beaglebone python3[2140]: ImportError: No module named 'requests'
I´m using debian in BeagleBone
You've asked a variation of the FAQ Why do things behave differently under systemd?.
This may have to do with the current working directory or environment variables that provide Python library paths.
At top of your python code before your import line, immediately dump out the current working directory and all environment variables. Review the differences under systemd vs manual use.
I think you'll find the issue there. For example, if requests was found relative to your current working directory and systemd is using a different directory.
See the answer to above linked FAQ for more possibilities.
I have an account to a computing cluster that uses Scientific Linux. Of course I only have user access. I'm working with python and I need to run python scripts, so I need to import some python modules. Since I don't have root access, I installed a local python copy on my $HOME with all the required modules. When I run the scripts on my account (hosting node), they run correctly. But in order to submit jobs to the computing queues (to process on much faster machines), I need to submit a bash script that has a line that executes the scripts. The computing cluster uses SunGrid Engine. However when I submit the bash script, I get an error that the modules I installed can't be found! I can't figure out what is wrong. I hope if you can help.
You could simply call your python program from the bash script with something like: PYTHONPATH=$HOME/lib/python /path/to/my/python my_python_script
I don't know how SunGrid works, but if it uses a different user than yours, you'll need global read access to your $HOME. Or at least to the python libraries.
First, whether or not this solution works for you depends heavily on how the cluster is set up. That said, the general solution to your problem is below. If the compute cluster has access to the same files as you do in your home directory, I see no reason why this would not work.
You need to be using a virtualenv. Install your software inside that virtualenv along with any additional python packages you need. Then in your batch bash script, provide the full path to the python interpreter within that virtualenv.
Note: to install python packages inside your virtualenv, you need to use the pip instance that is in your virtualenv, not the system pip.
Example:
$ virtualenv foo
$ cd foo
$ ./bin/pip install numpy
Then in your bash script:
/path/to/foo/bin/python /path/to/your/script.py
Have you tried to add these in your python code:
import sys
sys.path.append("..")
from myOtherPackage import myPythonFile
This works very well for my code when I run it on Cluster and I wanted to call my "myPythonFile" from other package "myOtherPackage"