I have a dependency for my project that requires python v3.6+. It therefore throws an error during installation via pip in a python 3 kernel, because AI Platform Notebooks ship with v3.5 by default. How can I run a GCP AI Platform Notebook with the latest version of python?
The answer is simpler than I thought. Since the AI notebook is a GCE instance, I simply ssh'ed into the machine, and followed the instructions here to install Python 3.7.
Click on AI Platform notebook name and you will reach VM Instance details page and get remote access SSH option(option will be enabled only if AI Platform notebook is running and not stopped)
Once you SSH into notebook VM , you can install using following commands: How do I install Python 3.7 in google cloud shell
# Install requirements
sudo apt-get install -y build-essential checkinstall libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev zlib1g-dev openssl libffi-dev python3-dev python3-setuptools wget
# Prepare to build
mkdir /tmp/Python37
cd /tmp/Python37
# Pull down Python 3.7, build, and install
wget https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tar.xz
tar xvf Python-3.7.0.tar.xz
cd /tmp/Python37/Python-3.7.0
./configure
sudo make altinstall
Now you can create kernel in notebook using below commands
you can do this inside the virtual environment:
Open up your terminal and enter the following line by line
virtualenv -p python3.6 py_36_env
. py_36_env/bin/activate # if . does not work then use source py_36_env/bin/activate
pip install ipykernel
python -m ipykernel install --user --name=py_36_env
jupyter notebook
Then in jupyter notebook you can select the 3.6 environment (py_36_env) from the 'New' drop down menu shown above or from the 'Kernel' drop down menu within a given jupyter notebook.
Related
So i am aiming to install Ludwig for experimentation but i didn't found any solution on the internet to this issue enter image description here
screenshot of the error message
i am using windows subsytem for linux (debian)
Your python version is probably unsupported by tensorflow 1.15.3. I ran into the same issue trying to install with python 3.8.
https://github.com/tensorflow/tensorflow/issues/34302
When I compiled Ludwig from GitHub source, there were a lot of dependencies to patch. I experienced the same error message and I gave up solving it.
I retried with clean installation from the very beginning, I managed to install Ludwig successfully on Google Cloud - Debian 9 VM.
Now I redo the steps on Oracle Cloud - Ubuntu 20.04 VM.
Steps:
Ensure the following dependencies are ready, which I consolidate from various sources.
$ sudo apt update
$ sudo apt install build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev curl libbz2-dev lzma liblzma-dev python3-tk
Clean install Python 3.6. I choose to use older Python 3.6 to execute Ludwig, because Ludwig is using the old Tensorflow 1.15.3. I reserve Python 3.7 and 3.8 environment for other newer Python projects, for example Tensorflow 2.
# Simply use a temporary working folder.
$ cd /tmp
# Download the newest Python3.6 source.
$ curl -O https://www.python.org/ftp/python/3.6.12/Python-3.6.12.tgz
$ tar -xzvf Python-3.6.12.tgz
$ cd Python-3.6.12
# --prefix=/usr/local ensures the newly installed Python3.6 does not mess up with the default Python executables in the OS. This is specially warned in Google Cloud.
$ ./configure --prefix=/usr/local --enable-optimizations
$ sudo make altinstall
$ python3.6 --version
Python 3.6.12
# Upgrade pip and virtualenv
$ sudo python3.6 -m pip install --upgrade pip
$ sudo python3.6 -m pip install --upgrade virtualenv
Prepare the virtual environment for Ludwig.
Reference https://ludwig-ai.github.io/ludwig-docs/getting_started/#installation.
# Go back to home.
$ cd
# Create a Working directory.
$ mkdir Works
$ cd Works
# Initialize a virtual environment with Python3.6
$ virtualenv -p python3.6 ludwig
$ source ludwig/bin/activate
# Install Ludwig
$ pip install ludwig
You can see every dependencies are taken care of and Ludwig is ready to use.
$ pip list
Package Version
-------------------- -------
... ...
ludwig 0.2.2.8
...
tensorflow 1.15.3
...
# Execute Ludwig
$ ludwig
I'm trying a simple thing like creating a Resource group in Azure via an Ansible Playbook, but I need the Azure Python SDK for it. I can't install the SDK with pip install azure because Microsoft doesn't support it anymore, which is completely illogical because Ansible keeps showing ModuleNotFoundError errors every time I execute the playbook, and I have to install them all manually...
Any more efficient way to do this?
I believe you're looking to install Ansible Azure module. So please try the following,
If you already have python and pip installed, use the command pip install ansible[azure], to install Azure modules.
Otherwise Try the following, and see if that helps,
On Windows,
Install Python
Install PIP
Add python and pip installation path to Windows PATH variable
Install Ansible Azure Module: pip install ansible[azure]
On CentOS 7.4,
Install python and pip:
sudo yum check-update; sudo yum install -y gcc libffi-devel python-devel openssl-devel epel-release
sudo yum install -y python-pip python-wheel
Install Ansible Azure Module:
sudo pip install ansible[azure]
On Ubuntu 16.04 LTS,
Install python and pip:
sudo apt-get update && sudo apt-get install -y libssl-dev libffi-dev python-dev python-pip
Install Ansible Azure Module:
sudo pip install ansible[azure]
I'm able to install it with root user but I wanted to install it in a clean environment. My use case is to test the installation of another application with pip for the customer who is using python3.7.0
sudo apt-get update
sudo apt-get install build-essential libpq-dev libssl-dev openssl libffi-dev zlib1g-dev
sudo apt-get install python3-pip python3-dev
sudo apt-get install python3.7
Thanks.
(assuming python3.7 is installed)
Install virtualenv package:
pip3.7 install virtualenv
Create new environment:
python3.7 -m virtualenv MyEnv
Activate environment:
source MyEnv/bin/activate
To help anyone else who runs into the chicken & egg situation trying to use the above chosen answer, here's what solved it for me:
sudo apt install python3.7-venv
python3.7 -m venv env37
source env37/bin/activate
deactivate (when done using the environment)
I had installed python 3.7 using deadsnakes vs source:
sudo apt update
sudo apt install software-properties-common
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt install python3.7
In doing so I could run python3.7 --version but since I had no pip3.7 I could not install virtualenv as directed in the solution above. Luck would have it that deadsnakes has venv! Once I installed venv I could create my environment & be on my merry way
Handy official python page with venv info
So why didn't I use?:
python3.7 -m ensurepip
That was giving me:
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/lib/python3.7/dist-packages/easy_install.py'
Consider using the --user option or check the permissions.
Which left me with 3 choices:
use sudo (which is simple but I keep being told is frowned upon)
install with --user option which wasn't ideal in that I may not always be logged in as the same user
or install it in an environment which I'm told is the recommended route.
But see chicken egg above.. How do I install pip in environment when I can't create venv or virtualenv? Thus my workaround solution of installing venv from deadsnakes which allowed me to create the virtual environment to then install pip3.7:
(env37) user#ubuntu:~$ python3.7 -m ensurepip
(env37) user#ubuntu:~$ pip3.7 --version
pip 19.2.3 from /home/user/env37/lib/python3.7/site-packages/pip (python 3.7)
Some added information, if you are trying for some version like python 3.7.10, which might give following error upon executing pip3.7.10 install virtualenv
.pyenv/versions/3.7.10/bin/python: No module named virtualenv
So, in a general sense you can do the following steps:
[commands are specific to MacOs, I am currently using with the new M1 chip]
After installing 3.7.10 using pyenv, make it global.
brew update
brew install pyenv
set environment variables
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile
echo 'eval "$(pyenv init -)"' >> ~/.bash_profile
source ~/.bash_profile
look at the pyenv list to see if the version you install is there or not and install and make it global
pyenv install --list
pyenv install 3.7.10
pyenv global 3.7.10
create your virtual environment now with this version
python -m venv MyEnv
activate it
source MyEnv/bin/activate
Using pip on windows, you can do the following:
1.virtualenv --python "C:\\Python37\\python.exe" venv# use your own path
You will see something like this:
Running virtualenv with interpreter C:\Python37\python.exe
Using base prefix 'C:\Python37'
New python executable in C:\Users\XXXX\Documents\GitHub\MyProject\venv\Scripts\python.exe
Installing setuptools, pip, wheel...
done.
2.C:\Users\XXXXX\Documents\GitHub\MyProject>cd venv
C:\Users\XXXXX\Documents\GitHub\MyProject\venv>cd Scripts
C:\Users\XXXXX\Documents\GitHub\MyProject\venv\Scripts>activate.
At the beginning of the command path, when you see (environment variable name) in this case (venv), this is a sign that your virtual environment is activated.
(venv) C:\Users\tuscar2001\Documents\GitHub\MyProject\venv\Scripts>
Please check the following link for more details:http://www.datasciencetopics.com/2020/03/how-to-set-up-virtual-environment-in.html
Figure out python3.7 path on your system. For mac with python3.7 in brew you can use the following
virtualenv env -p /usr/local/opt/python#3.7/bin/python3
source ./env/bin/activate
I want to run a pyspark job through Google Cloud Platform dataproc, but I can't figure out how to setup pyspark to run python3 instead of 2.7 by default.
The best I've been able to find is adding these initialization commands
However, when I ssh into the cluster then
(a) python command is still python2,
(b) my job fails due to a python 2 incompatibility.
I've tried uninstalling python2 and also aliasing alias python='python3' in my init.sh script, but alas, no success. The alias doesn't seem to stick.
I create the cluster like this
cluster_config = {
"projectId": self.project_id,
"clusterName": cluster_name,
"config": {
"gceClusterConfig": gce_cluster_config,
"masterConfig": master_config,
"workerConfig": worker_config,
"initializationActions": [
[{
"executableFile": executable_file_uri,
"executionTimeout": execution_timeout,
}]
],
}
}
credentials = GoogleCredentials.get_application_default()
api = build('dataproc', 'v1', credentials=credentials)
response = api.projects().regions().clusters().create(
projectId=self.project_id,
region=self.region, body=cluster_config
).execute()
My executable_file_uri is sits on google storage; init.sh:
apt-get -y update
apt-get install -y python-dev
wget -O /root/get-pip.py https://bootstrap.pypa.io/get-pip.py
python /root/get-pip.py
apt-get install -y python-pip
pip install --upgrade pip
pip install --upgrade six
pip install --upgrade gcloud
pip install --upgrade requests
pip install numpy
I found an answer to this here such that my initialization script now looks like this:
#!/bin/bash
# Install tools
apt-get -y install python3 python-dev build-essential python3-pip
easy_install3 -U pip
# Install requirements
pip3 install --upgrade google-cloud==0.27.0
pip3 install --upgrade google-api-python-client==1.6.2
pip3 install --upgrade pytz==2013.7
# Setup python3 for Dataproc
echo "export PYSPARK_PYTHON=python3" | tee -a /etc/profile.d/spark_config.sh /etc/*bashrc /usr/lib/spark/conf/spark-env.sh
echo "export PYTHONHASHSEED=0" | tee -a /etc/profile.d/spark_config.sh /etc/*bashrc /usr/lib/spark/conf/spark-env.sh
echo "spark.executorEnv.PYTHONHASHSEED=0" >> /etc/spark/conf/spark-defaults.conf
Configure the Dataproc cluster's Python environment explained it in detail. Basically, you need init actions before 1.4, and the default is Python3 from Miniconda3 in 1.4+.
You can also use the Conda init action to setup Python 3 and optionally install pip/conda packages: https://github.com/GoogleCloudPlatform/dataproc-initialization-actions/tree/master/conda.
Something like:
gcloud dataproc clusters create foo --initialization-actions \
gs://dataproc-initialization-actions/conda/bootstrap-conda.sh,gs://dataproc-initialization-actions/conda/install-conda-env.sh
There are couple of ways to select python interpreter for pyspark.
1.If you want to set python3 as default, set export PYSPARK_PYTHON=python3 while creating dataproc cluster. I added couple of code in the init scripts.
sudo echo "export PYSPARK_PYTHON=python3" | sudo tee -a /etc/profile.d/effective-python.sh
source /etc/profile.d/effective-python.sh
2.Otherwise it's also possible to specify the python version through --properties to use while submitting pyspark job to dataproc cluster. The python version can be passed in the following way:
--properties spark.pyspark.python=python3.7,spark.pyspark.driver=python3.7
I'm running Ubuntu 12.04 and do not have the option of upgrading / using something else. It has Python3.2 on it. I have found out that pip doesn't come automatically with < 3.4. How can I get pip to work with Python3 on this machine?
I tried downloading a copy of get-pip.py and running it with python3.2, but I kept getting an error message about how < 3.4 support was dropped.
I spent a little bit getting this all to work so I wanted to make a detailed post.
First I found out I could get Python3.4 onto my Ubuntu 12.04 machine. To do this run:
sudo add-apt-repository ppa:fkrull/deadsnakes
sudo apt-get update
sudo apt-get install python3.4
Source: Is There An Easy Way To Install Python 34 On
Next, I removed Python3.2 (I only wanted to have a single Python3.x version on my machine) by running:
sudo apt-get remove python3.2
sudo apt-get remove python3.2-minimal
Then, to get pip I ran:
sudo curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
sudo python3.4 get-pip.py
Source: How To Install Pip On Ubuntu 12 04 LTS
At this point Python3.4 and pip can both be ran. Example:
python3.4 main.py
pip3 install requests
At this point I wanted to be able to just use python3 to run python3.4 - trying just python3 kept telling me I had to install python-minimal. Doing this reinstalled python3.2 for me which I did not want. Instead, I created an alias by doing the following:
vim ~/.bashrc # Open the file for editing
Add: alias python3=python3.4 to the file.
. ~/.bashrc # Make the changes apply to the current session
Source: How Do I Create A Permanent Bash Alias