Given the pending retirement of python 2 support in January 2020, we should be thinking about migrating from Miniconda2, which is python 2 based, to Miniconda3, which is python 3 based.
As of the date of asking this question, the Miniconda web pages don't address this topic.
Is there a good procedure for converting from Miniconda2 to Miniconda3 that will preserve the existing environments created under Miniconda2?
You could try to upgrade your python version as suggested in this answer:
conda install python=3.7
But I'm not sure how safe that would be. (unsafe)
A safe approach is to simply install miniconda3 into a new path on your machine reproduce identically your current environments on the new miniconda installation.
To do that, you'll have to create a spec list for each of your environments in miniconda2 by:
conda activate myenv
conda list --explicit > myenv-spec-file.txt
Then under your miniconda3 installation, do:
conda create --name myenv --file myenv-spec-file.txt
The conda docs have detailed instructions on this process.
Keep in mind that when you install miniconda3, it will add an entry into your .*rc file (e.g. .bashrc, if using bash) and the new conda based on python 3 will be used when running any conda command. So you should create your spec files prior to installing miniconda3.
Edit: As pointed out by merv and nekomatic, upgrading conda in-place is not safe.
Related
This may be a noob question, I am new to using Conda environments. I am looking for some advice on how to tackle the following workflow in the best way.
I have both a work desktop and a desktop at home. I want to be able to, at the end of the day, take my work environment home.
Note: I work in Ubuntu on subsystems for windows
Say I start a project from scratch. I currently use the following workflow:
I create a conda environment.
conda create --name my_new_project python=3.10
activate my workspace.
conda activate my_new_project
I install python packages I need:
conda install -c conda-forge opencv
etc...
At the end of the day I want to copy that environment and take it to another PC, so I do the following:
conda env export --f my_new_project.yml
Finally on my home PC I do
conda env create --file my_new_project.yml
This works but requires me to make a new environment every time I switch PC. Is there a way to load the differences between the two conda environments and only add the new packages? Or is there another better way to tackle this?
There's no need to create a new environment every time. You only do that once and then update the existing environment, i.e. use the following as step 5:
conda env update -f dependencies.yml
I also suggest you to put your code, including the dependencies file, into version control if you aren't doing that already. Then, getting up to speed with your project on a different computer will only require two steps.
I installed latest version of Anaconda3-2020-02.
I was trying to follow this instruction in order to create environment for running with python==3.6 instead of python==3.7., because I have python 3.6. installed.
So, running the
conda create --name snakes python=3.6
and then activating my environment with conda activate snakes, it enters environment (snakes). However there are no anaconda packages inside like jupyter notebook or others, no anaconda-navigator... So whats the purpose of it and how can I run these programs from environment?
Also, for some reason (when I am not in the environment, just regular bash shell) $PATH is not set up to the /bin directory in anaconda, just to /condabin. Can you explain this also, because I am not able to run nothing except conda command from shell after recommended installation.
If by default after installation environment is not activated, you should activate it by sourcing a file anaconda3/bin/activate. You will see indicator (base) on the left of your bash prompt.
Good thing about anaconda3-2020-02 is that it is not messing with system python, as newest python is being launched inside environment and proper$PATH is set up just inside environment. If only, for some reason, specific version of python is needed, then it makes sense to set it up with this instruction. I would be just using default one with python 3.7 probably from (base) environment.
Keep in mind that by default anaconda components are not being set up inside new environment being created. In order to bring them, for example jupyter, you should run the command like this:
conda create --name snakes python=3.6 jupyter
I'm trying to get wxPython to work in a pyenv-based virtualenv with the virtualenv and virtualenvwrapper plugins on MacOS. I've read several questions about making this work, but most of the answers seem to assume that I'm using the system python version, and/or only address one aspect of a broken setup. I haven't yet seen anything that explains what wxPython is checking for when it starts.
I have python 3.7 compiled by pyenv with --enable-framework.
env PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install 3.7.4
pyenv itself is installed in my homedir from a reasonably recent (within the last couple of weeks) pull from git.
To be clear, none of the tools or libraries in my Python toolchain are installed by Homebrew.
My virtualenv has access to the framework by virtue of --system-site-packages. Having access to the framework and to the display are supposedly all that's required for wxPython to work, yet I'm still getting the same error on start of any test app:
This program needs access to the screen. Please run with a
Framework build of python, and only when you are logged in
on the main display of your Mac.
It looks to me like everything I should need is available.
% pyenv which python3.7
/Users/matt/.pyenv/versions/3.7.4/bin/python3.7
% mkvirtualenv --system-site-packages --python python3.7 wxtest
Running virtualenv with interpreter /Users/matt/.pyenv/shims/python3.7
Already using interpreter /Users/matt/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/bin/python3.7
Using base prefix '/Users/matt/.pyenv/versions/3.7.4/Python.framework/Versions/3.7'
New python executable in /Users/matt/.ve/wxtest/bin/python3.7
Also creating executable in /Users/matt/.ve/wxtest/bin/python
Installing setuptools, pip, wheel...
done.
virtualenvwrapper.user_scripts creating /Users/matt/.ve/wxtest/bin/predeactivate
virtualenvwrapper.user_scripts creating /Users/matt/.ve/wxtest/bin/postdeactivate
virtualenvwrapper.user_scripts creating /Users/matt/.ve/wxtest/bin/preactivate
virtualenvwrapper.user_scripts creating /Users/matt/.ve/wxtest/bin/postactivate
virtualenvwrapper.user_scripts creating /Users/matt/.ve/wxtest/bin/get_env_details
% python -m site
sys.path = [
'/Users/matt/devel/wxtest',
'/Users/matt/.ve/wxtest/lib/python37.zip',
'/Users/matt/.ve/wxtest/lib/python3.7',
'/Users/matt/.ve/wxtest/lib/python3.7/lib-dynload',
'/Users/matt/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7',
'/Users/matt/.ve/wxtest/lib/python3.7/site-packages',
'/Users/matt/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages',
]
USER_BASE: '/Users/matt/.local' (exists)
USER_SITE: '/Users/matt/.local/lib/python3.7/site-packages' (doesn't exist)
ENABLE_USER_SITE: True
% pip install wxPython
Collecting wxPython
[...]
Successfully installed numpy-1.18.0 pillow-6.2.1 six-1.13.0 wxPython-4.0.7.post2
The code for IsDisplayAvailable() is buried in the _core shared object library, so not particularly easy to examine. I'm not a C++ coder, and digging around in the code repository all I've been able to find so far is the docstring in src/_app.i, not the actual code.
On Mac OS X a False return value will mean that wx is not
able to access the window manager, which can happen if logged in
remotely or if running from the normal version of python instead
of the framework version, (i.e., pythonw.)
That list of requirements seem to be satisfied by what I have here. I don't have a pythonw binary, but as the pythonw(1) man page says:
Actually, since Python 2.5, the normal python also allows GUI access, so
python and pythonw are now interchangeable.
Does anyone have an exhaustive list of what wxPython actually expects to find before it runs?
The error you're encountering is likely to be a "brew-hole" (a.k.a homebrew installation/compatibility issue for wxpython v3 and sometimes v4).
To test some issues that might be at hand perform the following in your pyenv:
python --version
python3 --version
How are they installed? Did you use brew?
If brew or any non-anaconda: uninstall all.
Then : reinstall python via the anaconda 2019. (xx >04) packaged version and it gives you 3.7.4 or 3.7.5 depending on what you choose.
reinstall via conda/pip cmd-line the required packages that are not with default install.
Anaconda "base" is your default environment.
Then conda create --name myenv where myenv is any name you give your environment. For example "myPythonwx408" environment.
cmdline: conda activate myenv
... and tada.. up you go...
If the error persist in anaconda env you can post the environment.yml file so I can recreate your environment for testing.
You mention "virtualenv". Is that the actual virtualenv package or just what you are calling the pyenv instance? If the former then try python -m venv instead. The virtualenv package still hides the framework-ness of the parent Python, even after all these years. The venv package is included with Python3 and is a much better/simpler implementation of the concept.
I'm a bit new to Python and all its "deployment" related tools: pip3,setuptools, virtualenv wheel etc, so I hope my question will make sense...anyways it is like so:
I have a Python3 project which is "managed" with virtual environment using virtualenv where all the projects' dependencies are "listed" within the project's setup.py file. The contents of the setup.py files are as follows:
setup(name="MyProjectName",
version="0.1",
description="Some description",
url="someURL",
author="My Name",
author_email="someemail",
license="MIT",
packages=find_packages(),
include_package_data=True,
install_requires=["robotframework", "paramiko"])
As you can see, the only 3rd party packages the project uses (explicitly) are robotframework & paramiko.
Now when I'm deploying the project, I do the following actions (in that order):
Create a virtual environment with the command:
virtualenv -p python3 virtualEnvFolderName
Switching "into" the virtual environment like so (I'm deploying it on a Linux machine):
source virtualEnvFolderName/bin/activate
Running the setup.py script with the install argument to "automatically" install all the project's dependencies with the following command:
python3 setup.py install
--> Up until couple of days ago, all the 3rd party packages (and their "dependencies sub-packages") listed in the setup.py file where downloaded (and then installed) using their whl file, i.e. - for example:The output for the paramiko package installation would have been:
Reading https://pypi.org/simple/paramiko/
Downloading https://files.pythonhosted.org/packages/4b/80/74dace9e48b0ef923633dfb5e48798f58a168e4734bca8ecfaf839ba051a/paramiko-2.6.0-py2.py3-none-any.whl#sha256=99f0179bdc176281d21961a003ffdb2ec369daac1a1007241f53374e376576cf
Best match: paramiko 2.6.0
Processing paramiko-2.6.0-py2.py3-none-any.whl
Installing paramiko-2.6.0-py2.py3-none-any.whl to
--> This way, the installation was very quick (~1-3 seconds per package).
Today, when I performed the same procedure, on the same machine (I'm quiet sure I did not change any settings on my Ubuntu 16.04 machine), for each package the setup.py tried to install, it installed "via" the tar.gz file (i.e. sources ?) and NOT using the whl file --> which takes MUCH longer since for some of the packages it actually builds (complies) all the "underlying C libraries". This "change" makes my "installation procedure" execution time increase from ~20 seconds to ~4 minutes.
My questions are:
a) How can I resolve this situation - preferably without changing the deployment procedure, i.e. - still perform the 3 steps mentioned above, taking into account that perhaps one or more of the commands will be slightly modified (the creation of the virtual environment and/or some additional argument in required to the setup.py ? ).
b) If I have no other option, then using a pip3 install -r requirement.txt ... "procedure" will also be good, if it also will use whl file(s) whenever applicable.
c) If I will need to "switch" my virtual environment "generator" to venv it is OK (and actually preferred, in case it will deploy the project in the "same" duration).
NOTES:
I tested it both on Ubuntu 16.04 and Ubuntu 18.04 machines with Python 3.5 and Python 3.6 respectively.
Thanks !!
It seems like there is no wheel compatible with your environment (operating system, Python interpreter version, CPU bitness) for the current version of the project PyNaCl. If you have a recent version of pip, the command path/to/venv/bin/python -m pip debug --verbose should list the tags that are compatible with your environment, so that you can compare with the list of wheels available on PyPI.
I am using conda version 4.5.11, python 3.6.6, and Windows 10.
I create a virtual environment using conda
conda create --name venv
When I check for installed packages
conda list
it is (as expected), empty.
But
pip list
is quite long.
Question #1: Why? - when I create a virtual environment using
python -m venv venv
the pip list is empty.
When I am not in an activated virtual environment, then
conda list
is also quite long, but it isn't the same as the pip list (* see follow up below)
In general, the pip list is a subset of the conda list. There is at least one exception ('tables' in the pip list, not in conda list) but I haven't analysed too closely. The conda list changes/displays some (all?) hyphens to underscores (or pip does the reverse). There also a few instances of versions being different.
Question #2: Why? (and follow up questions - can they be? and should I care?)
I was hoping to have a baseline conda 'environment' (that may not be the right word) -ie, the packages I have installed/updated into Ananconda/conda and then all virtual environments would be pulled from that. If I needed to install something new, it would be first installed into the baseline. Only when I need to create an application using different versions of packages from the baseline (which I don't envision in the foreseeable future) would I need to update the virtual environments differently.
Question #3: Am I overthinking this? I am looking for consistency and hoping for understanding.
-- Thanks.
Craig
Follow Up #1: After installing some packages to my empty conda venv, the results of conda list and pip list are still different, but the pip list is much shorter than it was, but is a subset of the conda list (it does not include two packages I don't use, so I don't care)
Follow Up #2: In the empty environment, I ran some code
python my-app.py
and was only mildly surprised that it ran without errors. As expected, when I installed a package (pytest), it failed to run due to the missing dependencies. So ... empty is not empty.
1. conda list vs pip list
If all you did was create the environment (conda create -n venv), then nothing is installed in there, including pip. Nevertheless, the shell is still going to try to resolve pip on using the PATH environment variable, and is possibly finding the pip in the Anaconda/Miniconda base environment.
2. pip list is subset of conda list outside env
This could simply be a matter of conda installing things other than Python packages, which pip doesn't have the option to install. Conda is a more generic package manager and brings in all the dependencies (e.g., shared libraries) necessary to run each package - by very definition this is a broader range than what is available from the PyPI.
3. Overthinking
I think this is more of a workflow style question, and generally outside the scope of StackOverflow because it's going to get opinionated answers. Try searching around for best practice recommendations and pick a style suited to your goals.
Personally, I would never try to install everything into my base/root Conda environment simply because the more one installs, the more one has dependency requirements pulling in different directions. In the end, Conda will centralize all packages anyway (anaconda/pkgs or miniconda3/pkgs), so I focus on making modular environments that serve specific purposes.