Specs:
Macbook Air M1 Ventura
Visual Studio Code Version: 1.75.0
which brew
/opt/homebrew/bin/brew
which pip3
/opt/homebrew/bin/pip3
(which is a link to:
/opt/homebrew/Cellar/python#3.10/3.10.10/bin/pip3 )
tl;dr:
I don't have python installed through brew anymore, but only as pyenv (and only version 3.11.1) - my Mac native version is 3.9.6)
YET whenever I install something through the activated virtualenv it will re-create a folder in /opt/homebrew/lib/python3.10/site-packages.
I don't understand why it keeps fussing about python3.10 so much and how I can get packages installed to any other environment. It seems my pc is only the folder mentioned above.
What I tried:
Since a module (twarc-csv as part of the twarc2 package) didn't work anymore (when trying to run, it says Error: No such command 'csv'.; which twarc-csv in terminal also didn't find it), I started looking into my python environments. the developer of the package suggested it was related to versionings and recommend using pyenv.
I have to say that before installing pyenv I upgraded twarc (to be sure) in terminal with brew install twarc --upgrade (as I already figured my pip3 path and my python path didn't align see python mess) which automatically fetched python3.11.1 as a dependency for twarc for some reason. As a result besides the Mac native python version I now had python 3.11 and 3.10 installed.
Even when I explicitly ran 3.11 though (e.g. python3.11 -m pip install twarc --upgrade twarc-csv), packages would still be installed to this folder /opt/homebrew/lib/python3.10/site-packages though. So something was off.
I decided to go with the suggestion from said developer and try pyenv. I strictly stuck to this guide: clean pyenv environment and VSCode seemed to find my pyenv virtualenv installed python version. However even though which pip3 led to the virtual shims path, the packages were still installed to /opt/homebrew/lib/python3.10/site-packages
Ok, if I wouldn't really need this package to run, I would already have given up, but since things are the way they are... I decided to delete pyenv and reinstall python3 through brew.
As you might have figured by now, the cycle started from anew. I wanted to check into why this path is so dear to my setup /opt/homebrew/lib/python3.10/site-packages Visual Studio would also still indicate that python 3.10.10 is global, so I checked
import sys
import pprint
pprint.pprint(sys.path)
I checked this for 3.11 and 3.10 environment in VSCode
the one for 3.10 being
'/opt/homebrew/Cellar/python#3.10/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python310.zip',
'/opt/homebrew/Cellar/python#3.10/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10',
'/opt/homebrew/Cellar/python#3.10/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload',
'/opt/homebrew/lib/python3.10/site-packages']
the one for 3.11 being
'/opt/homebrew/Cellar/python#3.11/3.11.1/Frameworks/Python.framework/Versions/3.11/lib/python311.zip',
'/opt/homebrew/Cellar/python#3.11/3.11.1/Frameworks/Python.framework/Versions/3.11/lib/python3.11',
'/opt/homebrew/Cellar/python#3.11/3.11.1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/lib-dynload',
'/opt/homebrew/lib/python3.11/site-packages']
plus my desktop, so this seems to be fine. Nevertheless even if I use interpreter 3.11 in VSCode it will still look for packages in python3.10 and not install them to 3.11
Now I checked $PATH and see what it says, and it still refers to pyenv it seems:
$PATH
zsh: no such file or directory: /Users/myusername/.pyenv/shims:/Users/myusername/.pyenv/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/go/bin:/Library/Apple/usr/bin
So how do I clean that up?
Also, how do I get things to run correctly? I do not understand that affection to the sitepackages 3.10 at all.
Update:
I am wondering whether my zprofile, zshrc or bashrc or bash_profile files have to do with it.
That's how they look:
(bashrc is empty)
bash_profile
export PATH="/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
zprofile
# Set PATH, MANPATH, etc., for Homebrew.
eval "$(/opt/homebrew/bin/brew shellenv)"
zshrc
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
if command -v pyenv 1>/dev/null 2>&1; then
eval "$(pyenv init -)"
fi
Related
EDIT:
In addition to the behaviour outlined below, the Python3.10 based environment seems to be ignoring packages installed with the pip -e option (development mode).
Specifically, a package installed in development mode is not listed in pip freeze (which, it does in the Python3.9 based virtual environment) and a simple import <package_name> fails. If the package is installed normally (i.e. not in development mode) everything works as expected.
I am using virtualenv to create a virtual environment based around Python 3.10. While virtualenv finishes without errors and does seem to activate, it still fails to pick up its own Python unless the PYTHONPATH environment variable has been set manually.
I am not sure if the situation I am faced with is due to Ubuntu's way of incorporating Python or the way virtualenv is setting up the environment so that it picks up a local interpreter. Here is what I have gathered this far:
My base system is an Ubuntu 21.04. It has its own Python 3 (Python3.9.5) installation which I have not touched at all except installing the python3-virtualenv package using apt.
With Python 3.10, I installed the python3.10-dev package and proceeded to create a virtual environment in the usual way:
> virtualenv -p python3.10 the_env/
> source the_env/bin/activate
Although this looks OK so far, this environment does not have any information about its own site-packages directory. Not even the one that virtualenv is supposed to be creating which includes pip. In this installation, if you try to > pip --version you simply get an error that
the pip package does not exist (the pip "executable" location is picked up correctly, but because the interpreter does not know anything about its site-packages it fails to start pip properly).
Long story short, I created two environments, one based on Python3.9 (which works perfectly) and one based on Python3.10 (which does not work) and did a very simple test in each environment:
> python -m site
On the Python3.9 environment, sys.path includes a path that leads all the way to this particular environment's site-packages
On the Python3.10 environment, sys.path does not include that particular path but still includes the typical paths you expect to find (e.g. those pointing to the interpreter itself and the top environment directory but not the specific path that points to the site-packages location.
Following this, I defined a PYTHONPATH manually, before activating the environment which points exactly to the site-packages for that particular environment and everything worked as expected.
I suspect that this might be something to do with the fact that my system's Python is 3.9 which means that the USER_SITE variable is valid while in the case of Python3.10, it is not (because, I do not have a use for it, this is just a virtual environment I am creating). So, I suspect that this might be throwing off the way the site module determines where things are.
As I am not sure, I would like to ask the following:
Could this be something to do with the way Ubuntu handles the Python installation that might just be creating this small problem with virtual environments?
Could the problem be with virtualenv that does not explicitly specify a PYTHONPATH?
Could this behaviour be something of a corner case of the site module?
What worked for me is an installation from source. After unpacking the source code, as a summary:
$ ./configure --enable-optimizations --with-ensurepip=install --prefix=/path/to/install/to/
$ make -j
$ make test
$ make install
$ /path/to/install/to/bin/python3.10 -m venv /path/to/test
$ source /path/to/test/bin/activate
$ pip list
Package Version
---------- -------
pip 21.2.4
setuptools 58.1.0
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/path/to/test/bin/python3.10 -m pip install --upgrade pip' command.
$ python -m pip install --upgrade pip
[...]
$ pip list
Package Version
---------- -------
pip 21.3.1
setuptools 58.1.0
Edit
As mentioned below, I believe the reason why the procedure above just works is simply that when building and installing Python from source, the installation simply will be correct, which I suspect the 3.10 installation in Ubuntu 21.04 is not.
I made my Python installation from source somewhere under my home directory, in order to not risk messing things up, and I also did not permanently modify the PATH.
Setting the path to $PATH:/path/to/install/to/bin should be fine however, either permanently, or just when running mkvirtualenv.
Doing that, my new installation even seems to integrate seamlessly with my system's virtualenvwrapper.
I don't really need to be on the Python leading edge myself, but if I did, I would definitely make myself independent of Ubuntu's latest development by using the procedure described above.
By the way, if I update pip directly in the Python installation built from source, I will no longer get the message about the older pip (see above) when I install new virtual environments.
Edit 2
Bug reported to Ubuntu:
https://bugs.launchpad.net/ubuntu/+source/python3.10/+bug/1955742
Edit 3
Also, since Ubuntu 21.04 has end of life in a month or so, upgrading to 21.10 could really make sense. I have just tried that, and it seems that Python 3.10 virtual environments work just fine in that release.
With the advent of Ubuntu 22.04 (which caused a few minor issues with some specific python virtualenv setups I had) and having already spent some time figureing this out, I ended up switching to using pyenv.
Pyenv made it very easy to install any python version and any number of virtual environments within it which can be further customised in complete isolation.
The only thing to be careful of is installing all necessary python prerequisites before installing a specific version to avoid missing functionality from some packages (e.g. not including the lzma library will generate an ominous warning from pandas ("your python installation is incomplete..."). This can be ignored if you are not using that functionality or otherwise, easily fixed.
I think that this is a better option overall if you have to manage different versions and specific configurations for python, even across distros.
This solution is very close to the one suggested before ("install from source") so I will be accepting that one and leave my contribution as additional information about this problem.
It's extremely useful for the development workflow to be able to build and install the latest version of a package into a local environment. You can then interactively validate and debug by importing this latest version into a Python shell or a Jupyter notebook. The problem is I've recently adopted Poetry and cannot figure how to do this now. So...
How do I install the latest version of my package from the current working directory into my local environment using Poetry?
Moving on from setuptools
Back in the day, I used to always use setuptools and it worked great. I'd put a setup.py file in the root of my repository, create a virtual environment (let's say using conda) for the project, and do...
pip install -e .
From here, I could fire up a python shell, or even configure a jupyter kernel to use this virtual environment, and I'd always have the latest version of my package to interact with.
Now setuptools has its limitations, and we've since moved on to Poetry to more tightly control dependencies and handle more sophisticated build needs and such.
The problem with poetry
If you look up what's the pip install -e . equivalent in poetry you will find this issue. Looks like the creator of poetry thinks installing directly from source like this is a hack and has no interest in supporting it. (BTW: I've tried poetry build and then pulling out the setup.py file like he suggested and it does not work)
Linking directly to source is not necessary, I'm willing to run an install command to get the latest version of the package. And when I do this with poetry, it appears to work.
cd root/of/my/project
poetry install
Installing dependencies from lock file
No dependencies to install or update
Installing the current project: my-project (0.4.8) <-- this is the latest version according to the source code in the working directory
The problem is that if I open a Python shell and try and import my package for instance, it is linked to the last version of my package that installed from a remote artifact repository (via pip install my-package) – not what's in my working directory.
python
...
>>> import my_package
>>> my_package.__version__
'0.4.7`
Now, even though I'm using poetry I'm using a conda environment to specify the Python version for my project and then installing, and using, poetry from inside that.
source activate my-package
(my-package) ... $ poetry update
I also know that poetry (not very transparently) can create and manage its own virtual environment on your behalf. I thought maybe the reason this is not working is because I need to be inside this environment (whereas I was only inside my conda environment, while poetry is installing the 0.4.8 version of my package within the virtual environment it manages).
I tried both shell and run to test this out. I get the same result.
poetry shell
Virtual environment already activated: /Users/U6020643/.conda/envs/my-package
Python 3.8.5
...
>>> import my_package
>>> my_package.__version__
'0.4.7`
What gives?
The way I fixed this: I stopped using conda to manage the Python version for projects that involve poetry and instead use pyenv.
Below is how I made that work. This was very helpful!
1. Removing conda as default environment manager.
This involved removing conda base activate from ~/.bash_profile.
Now open a new shell and verify that there's no conda environment prefix, e.g. (base) ... $.
2. Installing pyenv using Homebrew
Had been awhile, and an OS upgrade or two, since I interacted with Homebrew. Needed to do some housekeeping.
brew cleanup # This made it so that brew update didn't take forever
brew update
brew upgrade
brew cleanup
Then...
brew install pyenv
3. Install Python version(s)
Let's say you need/want Python 3.
pyenv install 3.8.5
4. Set the local Python version for your project
cd your/project/root/
pyenv local 3.8.5
5. Install poetry
See here.
6. Use it
Now install and use shell. Check the version -> hey it works!
cd your/project/root
poetry install
poetry shell
my_package --version # Package has a CLI.
I am having trouble with my current python, so I wanted to uninstall my python and install the latest version. I installed with homebrew, so I uninstalled it with homebrew and reinstalled python 3.8.1 with the installer from the official site. Python3.8 was installed, but my python3 was not upgraded.
~ which python3
/usr/bin/python3
~ python3 --version
Python 3.7.3
I know I'm not supposed to(and I can't) manually delete things inside /usr/bin. What am I supposed to do?
When you installed Python with homebrew it told you this:
Unversioned symlinks python, python-config, pip etc. pointing to
python3, python3-config, pip3 etc., respectively, have been
installed into /usr/local/opt/python/libexec/bin
If you need a reminder, post install, you will get the same message if you run:
brew info python
It says "unversioned links are in /usr/local/opt/python/libexec/bin". That means, if you want to run Python without specifying the version, i.e. if you want to type this:
python
and this:
pip
to start Python 3 and its corresponding pip, you need to make sure your PATH has /usr/local/opt/python/libexec/bin at the start, i.e.
export PATH=/usr/local/opt/python/libexec/bin:$PATH
I could not uninstall the python3 in /usr/bin but found a workaround to give the python3 in /usr/loca/bin precedence by setting the PATH env variable as PATH=/usr/local/bin:$PATH. This gives binaries in /usr/local/bin precedence. Not a full fledged solution, but got me moving.
Question
I am not sure why when virtualenv active, pip freeze stills showing all modules when it suppose NOT to. What am I doing wrong?
On the terminal
I tried two ways to create a virtual env:
virtualenv my-virtualenv
virtualenv --no-site-packages my-virtualenv
then activate it
source my-virtualenv/bin/activate
Results
Both show all packages (when they suppose not to). Running command pip freeze I get:
(my-virtualenv)$ pip freeze
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
actionlib==1.11.9
angles==1.9.11
bondpy==1.7.19
camera-calibration==1.12.20
camera-calibration-parsers==1.11.12
catkin==0.7.6
cv-bridge==1.12.4
diagnostic-analysis==1.9.2
diagnostic-common-diagnostics==1.9.2
diagnostic-updater==1.9.2
dynamic-reconfigure==1.5.48
gazebo-plugins==2.5.13
gazebo-ros==2.5.13
...
rosnode==1.12.7
rosparam==1.12.7
rospy==1.12.7
rosservice==1.12.7
rostest==1.12.7
rostopic==1.12.7
rosunit==1.13.5
roswtf==1.12.7
tf2-ros==0.5.16
topic-tools==1.12.7
xacro==1.11.2
Running
(my-virtualenv)$ which python
/home/user/.../my-virtualenv/bin/python
(my-virtualenv)$ python -V
Python 2.7.12
and
(my-virtualenv)$ which python3
/usr/bin/python3
(my-virtualenv)$ python3 -V
Python 3.5.2
Hence, I even can use python3 when virtualenv is activated but it uses the pc installed module. Moreover, new installed packages are install in the pc and not the virtualenv and I can't install python3 in the vitualenv because it "exists" already (but in the pc).
I was having the same issue and it was because, somehow, Python configuration of ROS was generating it.
I solved it removing the source commands of ROS from the ~/.bashrc. E.g.:
source /opt/ros/melodic/setup.bash
source $HOME/ROS/aslam_ws/devel/setup.bash
By the way, I realised that, before changing anything, creating the environment from PyCharm works properly too (when in the terminal I still had the issue).
There must be a better solution but I haven't found it yet.
I debated which Stackoverflow site this best fit but couldn't decide.
I'd like to contribute to an open-source project on Github, but I can't figure out how to prevent the stable version already installed on my machine and the development version I'd like to make a patch for from clashing on import.
The repository only suggests pip installing with the editable.
What I've done so far is: clone the repository locally, and then trying to import it in a Jupyter Notebook from the directory above. However, the Jupyter Notebook is referencing the stable version installed earlier with pip. I tried to append to sys.path the child directory holding the package but still the same issue. I can't seem to get relative imports working either. Do I need to uninstall the stable version?
Any tips are appreciated!
You'd use virtualenv for this. It will allow you to create an environment that is isolated from your system python and you can install the dev version of the library on it.
Basic usage (for Unix-like systems) is:
$ pip install virtualenv
$ virtualenv MY_ENV
$ cd MY_ENV
$ source bin/activate # activates the local python for this shell only
(MY_ENV)$ pip install <some-module> # installs to a local and isolated python
(MY_ENV)$ python ... # runs python in the local environment
(MY_ENV)$ deactivate # disable the isolated python
$