EDIT:
In addition to the behaviour outlined below, the Python3.10 based environment seems to be ignoring packages installed with the pip -e option (development mode).
Specifically, a package installed in development mode is not listed in pip freeze (which, it does in the Python3.9 based virtual environment) and a simple import <package_name> fails. If the package is installed normally (i.e. not in development mode) everything works as expected.
I am using virtualenv to create a virtual environment based around Python 3.10. While virtualenv finishes without errors and does seem to activate, it still fails to pick up its own Python unless the PYTHONPATH environment variable has been set manually.
I am not sure if the situation I am faced with is due to Ubuntu's way of incorporating Python or the way virtualenv is setting up the environment so that it picks up a local interpreter. Here is what I have gathered this far:
My base system is an Ubuntu 21.04. It has its own Python 3 (Python3.9.5) installation which I have not touched at all except installing the python3-virtualenv package using apt.
With Python 3.10, I installed the python3.10-dev package and proceeded to create a virtual environment in the usual way:
> virtualenv -p python3.10 the_env/
> source the_env/bin/activate
Although this looks OK so far, this environment does not have any information about its own site-packages directory. Not even the one that virtualenv is supposed to be creating which includes pip. In this installation, if you try to > pip --version you simply get an error that
the pip package does not exist (the pip "executable" location is picked up correctly, but because the interpreter does not know anything about its site-packages it fails to start pip properly).
Long story short, I created two environments, one based on Python3.9 (which works perfectly) and one based on Python3.10 (which does not work) and did a very simple test in each environment:
> python -m site
On the Python3.9 environment, sys.path includes a path that leads all the way to this particular environment's site-packages
On the Python3.10 environment, sys.path does not include that particular path but still includes the typical paths you expect to find (e.g. those pointing to the interpreter itself and the top environment directory but not the specific path that points to the site-packages location.
Following this, I defined a PYTHONPATH manually, before activating the environment which points exactly to the site-packages for that particular environment and everything worked as expected.
I suspect that this might be something to do with the fact that my system's Python is 3.9 which means that the USER_SITE variable is valid while in the case of Python3.10, it is not (because, I do not have a use for it, this is just a virtual environment I am creating). So, I suspect that this might be throwing off the way the site module determines where things are.
As I am not sure, I would like to ask the following:
Could this be something to do with the way Ubuntu handles the Python installation that might just be creating this small problem with virtual environments?
Could the problem be with virtualenv that does not explicitly specify a PYTHONPATH?
Could this behaviour be something of a corner case of the site module?
What worked for me is an installation from source. After unpacking the source code, as a summary:
$ ./configure --enable-optimizations --with-ensurepip=install --prefix=/path/to/install/to/
$ make -j
$ make test
$ make install
$ /path/to/install/to/bin/python3.10 -m venv /path/to/test
$ source /path/to/test/bin/activate
$ pip list
Package Version
---------- -------
pip 21.2.4
setuptools 58.1.0
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/path/to/test/bin/python3.10 -m pip install --upgrade pip' command.
$ python -m pip install --upgrade pip
[...]
$ pip list
Package Version
---------- -------
pip 21.3.1
setuptools 58.1.0
Edit
As mentioned below, I believe the reason why the procedure above just works is simply that when building and installing Python from source, the installation simply will be correct, which I suspect the 3.10 installation in Ubuntu 21.04 is not.
I made my Python installation from source somewhere under my home directory, in order to not risk messing things up, and I also did not permanently modify the PATH.
Setting the path to $PATH:/path/to/install/to/bin should be fine however, either permanently, or just when running mkvirtualenv.
Doing that, my new installation even seems to integrate seamlessly with my system's virtualenvwrapper.
I don't really need to be on the Python leading edge myself, but if I did, I would definitely make myself independent of Ubuntu's latest development by using the procedure described above.
By the way, if I update pip directly in the Python installation built from source, I will no longer get the message about the older pip (see above) when I install new virtual environments.
Edit 2
Bug reported to Ubuntu:
https://bugs.launchpad.net/ubuntu/+source/python3.10/+bug/1955742
Edit 3
Also, since Ubuntu 21.04 has end of life in a month or so, upgrading to 21.10 could really make sense. I have just tried that, and it seems that Python 3.10 virtual environments work just fine in that release.
With the advent of Ubuntu 22.04 (which caused a few minor issues with some specific python virtualenv setups I had) and having already spent some time figureing this out, I ended up switching to using pyenv.
Pyenv made it very easy to install any python version and any number of virtual environments within it which can be further customised in complete isolation.
The only thing to be careful of is installing all necessary python prerequisites before installing a specific version to avoid missing functionality from some packages (e.g. not including the lzma library will generate an ominous warning from pandas ("your python installation is incomplete..."). This can be ignored if you are not using that functionality or otherwise, easily fixed.
I think that this is a better option overall if you have to manage different versions and specific configurations for python, even across distros.
This solution is very close to the one suggested before ("install from source") so I will be accepting that one and leave my contribution as additional information about this problem.
Long story short, I messed with my Python environment too much (moving files around, creating new folders, trying to reinstall packages, deleting files etc.) My google package doesn't work anymore. Everytime I try to import the package, it says it can't find the module, even though I did a pip install.
I was wondering how I could do a hard reset/delete python off my computer and reinstall it.
Thanks.
is your pip working try extracting the required modules.
pip freeze > requirements.txt
create new Venv
pip install -r requirements.txt
I figured it out. My pip was installing to a site packages folder inside a local folder, while my jupyter notebook was trying to pull from the anaconda site packages folder.
I debated which Stackoverflow site this best fit but couldn't decide.
I'd like to contribute to an open-source project on Github, but I can't figure out how to prevent the stable version already installed on my machine and the development version I'd like to make a patch for from clashing on import.
The repository only suggests pip installing with the editable.
What I've done so far is: clone the repository locally, and then trying to import it in a Jupyter Notebook from the directory above. However, the Jupyter Notebook is referencing the stable version installed earlier with pip. I tried to append to sys.path the child directory holding the package but still the same issue. I can't seem to get relative imports working either. Do I need to uninstall the stable version?
Any tips are appreciated!
You'd use virtualenv for this. It will allow you to create an environment that is isolated from your system python and you can install the dev version of the library on it.
Basic usage (for Unix-like systems) is:
$ pip install virtualenv
$ virtualenv MY_ENV
$ cd MY_ENV
$ source bin/activate # activates the local python for this shell only
(MY_ENV)$ pip install <some-module> # installs to a local and isolated python
(MY_ENV)$ python ... # runs python in the local environment
(MY_ENV)$ deactivate # disable the isolated python
$
I am having an issue when it comes to installing python3.3 modules. The current modules are installing to /usr/lib64/python3.3/site-packages but they need to be installed to /usr/lib/python3.3/site-packages. Is there a way to point the installation to put the modules in the appropriate location?
I am using CentOS 6.6
While you could specify a prefix directory to tell pip to install into a specific place:
pip install --install-option="--prefix=/usr/lib/python3.3/site-packages/" my_module
I would instead heavily recommend learning about and using virtual environments to give you greater control of your Python programming environment.
The creation and activation of a virtual environment looks like this:
pyvenv-3.4 my_new_environment
source my_new_environment/bin/activate
pip install my_module
which will make my_module install reliably inside of my_new_environment.
Whenever you want to "activate" this environment, you simply source the activate script to tell the Python interpreter where to find its libraries!
I am having trouble installing and installing again the same package. I get an 'AssertionError: Multiple .dist-info directories' error. I think that it has something to do with the pip uninstall not completely getting rid of all the files of a package, but am not sure how to remedy the situation. I don't know what information would be helpful here, except maybe that this is a Django project, running on virtualenv on IIS.
EDIT: I just re-installed python and with it, a new version of pip.
Remove the related pip build directory for that particular package and try installing it again. That'd work.
If you're working in a virtual environment say venv, pip build directory will be <path-to-venv>/venv/build/<package-name>.