We have Anaconda installed on my cloudera cluster via parcels. We have python 2.7.13 available with the version of Anaconda. We wanted to have another version of python (3.6) across all nodes.
My challenge here is, when I followed the Conda documentation to create a new environment and install python 3.6 on that using "conda create -n py36 python=3.6 anaconda". For few nodes, I am getting python3.6.6 installed for few nodes, and python3.6.7 for few nodes and 3.6.1 for few.
I would like to know if there is a way to choose the version of python while installing 3.6 on a separate environment. Or am I doing something wrong? Please help me.
Thanks
Kancharlapalli
You can specify the patch as well
conda create -n py36 python=3.6.7 anaconda
Related
Spyder-terminal does not install with Anaconda Navigator running Python v3.8 or v3.9 or v3.10.
My question is "Is there a way to install the Spyder-Terminal Plugin using Anaconda Navigator running Python 3.8 or 3.9 or 3.10, and use Spyder v5.x " ?
Is there a hack for this to run inside Spyder 5.0.x on Python 3.8, 3.9 or 3.10, WITHOUT creating a new virtual environment?
My Software Versions
Spyder-Terminal Version: spyder-terminal-0.5.0 (spyder-ide/win-64)
Python Version: 3.8 or 3.9
Anaconda v2.0.1
Operating system: Windows 10
What I tried:
I tried installing 'conda install -c spyder-ide spyder-terminal' from Anaconda Command Prompt. I did several trials with MANY errors to learn that Spyder-terminal only works with Spyder v4.2 and NOT Spyder v5.0.3.
Spyder-terminal is (nominally) supposed to work in Python 3.9. So I tried upgrading Anaconda base environment to Python 3.9. However, that BROKE the Anaconda base environment, which required uninstalling and reinstalling the entire Anaconda distribution. (What a hassle. Good thing this was on someone else's computer! 8^))
I finally got it to work in a separate conda virtual environment running Python 3.6. Just to be conservative, I used Python 3.6 instead of Python 3.7. (I later installed Spyder-Terminal in a Python 3.7-based virtual environment and it worked fine.)
The steps that finally worked were to Create a new conda environment with Python 3.6 and install spyder-terminal and spyder-notebook there.
I used the following commands...
conda create -n python36 -c conda-forge python=3.6
conda install spyder-terminal -c conda-forge spyder-ide
conda install spyder-notebook -c conda-forge spyder-ide
The correct version of Spyder (for Python 3.6) was installed along with all of its correct dependencies.
What steps will reproduce the problem?
try installing into an Anaconda distribution with Python 3.8 or Python 3.9 kernels
Wait a long time and get many errors.
Rip out a few locks of hair (preferably someone else's)
**What is the expected output?
Spyder-Terminal plugin installs and is available for use.
What do you see instead?
A bunch of errors. The exact errors were lost when I closed the installation console.
Suggestions
The Spyder-terminal plugin failed repeatedly in Anaconda "base" running v3.8.x or a new environment running v3.9.x. I used a new conda virtual environment, running Python 3.6 runtime kernel and install spyder-terminal there.
The Spyder-terminal installer should do version checks for Python and Spyder FIRST, warn the user, and stop if they are not using Python 3.6 or Python 3.7.
Clearer documentation for Spyder-Terminal plugin installation on the Github.com home page (or built into Spyder Menu "Help" will help the "HELP" system. These docs should state WHICH version(s) of Spyder and Python that Spyder-Terminal v0.5.x requires. A dependency table would also be useful.
In one existing conda environment, the python is 3.8. Is that possible to downgrade the python version for this specific environment from 3.8 to 3.6 or 3.7?
Check this,
Open your terminal and search for available versions using the following command.
conda search python
If the python version you are searching is available then use the command
conda install python=3.8 (0r 3.6 or 3.7 depending to your requirement)
This will change the python version in a specific environment.
Note: This command will overwrite the default python version.
I suggest you open a new conda environment using the following command.
conda create --name py38 python=3.8
//This lines will create a new environment named py38
Now you can work into this environment without interfering with the libraries of the other environment.
Hope this will help you.
I learned that there are some issues in the latest Anaconda 3 python 3.7 version for object detection and face recognition deep learning problems from various posts. The official Anaconda site is only providing the latest python 3.7 version here. I want to work on a similar deep learning project in Windows 10(64 bit) which requires Anaconda 3 with python 3.6 version.
I found several posts providing solutions on StackOverflow for a Linux environment but I could not find any solution for the Windows 10 operating system. Can anyone share with me how to download the Anaconda 3 python 3.6 version?.
Any help will be highly appreciated.
Recommendation: Learn how to use conda by reading get started with conda. The problem you encounter is a very common case could be solved by conda, as an environment manager.
Solution 1
Use an environment with Python 3.6 installed, and activate this environment each time to work with your project.
# create an environment with python 3.6
conda create -n py36 python=3.6
# activate this environment
conda activate py36
This is also the preferred way to work with different projects. One environment for one project.
Solution 2
The default environment you're using with conda is base. You can override the Python 3.7 within base with Python 3.6
conda install -n base python=3.6
I create a dataproc cluster using the following command
gcloud dataproc clusters create datascience \
--initialization-actions \
gs://dataproc-initialization-actions/jupyter/jupyter.sh \
However when I submit my PySpark Job I got the following error
Exception: Python in worker has different version 3.4 than that in driver 3.7, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.
Any Thoughts?
This is due to a difference in the python versions between the master and the worker. By default, the jupyter image installs the latest version of miniconda, which uses the python3.7. However, the worker is still using the default python3.6.
Solution:
- specify the miniconda version when creating the master node i.e to install python3.6 in the master node
gcloud dataproc clusters create example-cluster --metadata=MINICONDA_VERSION=4.3.30
Note:
may need updating to have a more sustainable solution to managing the environment
UPDATE THE SPARK ENVIRONMENT TO USE PYTHON 3.7:
Open a new terminal and type the following command: export PYSPARK_PYTHON=python3.7 This will ensure that the worker nodes use Python 3.7 (same as the Driver) and not the default Python 3.4
DEPENDING ON VERSIONS OF PYTHON YOU HAVE, YOU MAY HAVE TO DO SOME INSTALL/UPDATE ANACONDA:
(To install see: https://www.digitalocean.com/community/tutorials/how-to-install-anaconda-on-ubuntu-18-04-quickstart)
Make sure you have anaconda 4.1.0 or higher. Open a new terminal and check your conda version by typing into a new terminal:
conda --version
checking conda version
if you are below anaconda 4.1.0, type conda update conda
Next we check to see if we have the library nb_conda_kernels by typing
conda list
Checking if we have nb_conda_kernels
If you don’t see nb_conda_kernels type
conda install nb_conda_kernels
Installing nb_conda_kernels
If you are using Python 2 and want a separate Python 3 environment please type the following
conda create -n py36 python=3.6 ipykernel
py35 is the name of the environment. You could literally name it anything you want.
Alternatively, If you are using Python 3 and want a separate Python 2 environment, you could type the following.
conda create -n py27 python=2.7 ipykernel
py27 is the name of the environment. It uses python 2.7.
Ensure the versions of python are installed successfully and close the terminal. Open a new terminal and type pyspark. You should see the new environments appearing.
We fixed it now -- thanks for the intermediate workaround #brotich. Check out the discussion in #300.
PR #306 keeps python at the same version as was already installed (3.6), and installs packages on all nodes to ensure that the master and worker python environments stay identical.
As a side effect, you can choose your python version by passing an argument to the conda init action to change the python version. E.g. --metadata 'CONDA_PACKAGES="python==3.5"'.
PR #311 pins miniconda to a particular version (currently 4.5.4), so we avoid issues like this again. You can use --metadata 'MINICONDA_VERSION=latest' to use the old behavior of always downloading the latest miniconda.
I have the Anaconda distribution installed for Mac. I have Mac OSX 10.8 (Mountain lion). The problem I don't often use Anaconda is because the default Python which it uses is 2.7 while I work on 3.3 or atleast prefer to work on that.
I really like the Spyder IDE of Anaconda. Is there a way I can get the default environment on Anaconda changed to 3.3 instead of 2.7 so that Spyder and iPython all use 3.3 default?
I see the following help from Anaconda site:
$ conda create -n py3k python=3 anaconda
Here python=3 and anaconda are package specifications, and it is the job of the SAT solver inside conda to find a consistent set of packages which satisfies these requirements. As the root environment uses Python 2, we had to specify the major version explicitly.
After adding the binary directory of the newly created environment to the PATH environment variable, which may be done using
$ source activate py3k
My question is I have separately installed Python 3.3 and associated Scientific Python packages like Pandas, numpy, scipy, scikit-learn etc using Homebrew so that it doesn't conflict with my Mac OS default Python 2.7. So now if I run the above Conda commands in Mac Terminal will it interfere with other Python packages I have installed using Homebrew? or will it automatically install/upgrade the python and other packages in the Anaconda library without interfering with either the Homebrew installed Python or Mac OS default Python?
Please advise.
No, the Homebrew and Anaconda Pythons will stay completely independent of one another. Just make sure you don't have PYTHONPATH set, which causes this to not be true.
Also, you should know that Spyder is not available for Python 3 in Anaconda yet, because PySide has not yet been built for Python 3.
I had installed Anaconda with python 2.7, but even after adding python3:
conda create --name Py3 python=3
spyder continued to call python2.7
creating an environment with BOTH spyder and python=3 worked for me:
conda create --name SpyPy3 python=3 spyder