I'm working with a Jupyter Notebook written in Python 3, and I would like to run Python 2 scripts from within that Notebook. I was wondering if it's possible to run Shell commands from within the Notebook, and have these Shell commands running under a different environment.
For example, if env2 is a Conda environment that runs Python 2, and env3 runs Python 3, and my Jupyter Notebook runs in env3, maybe I could write within my Notebook:
! source activate env2
! script_that_uses_python2.py
and then continue with python 3 code that is in the Notebook (and uses the output of script_that_uses_python2.py).
I tried it and it didn't work (! conda info --envs showed that env3 is still running). Any advice on how to change the environment in the middle of the Notebook and then go back to the original environment?
As far as I know, you cannot activate another environment and have it work like that. What you can do is run that Python explicitly, something like
!/path/to/anaconda/envs/python2env/bin/python script_that_uses_python2.py
If I run
!/path/to/anaconda/envs/python2env/bin/python -c "import sys; print sys.path"
on my system, it only shows Python 2 directories, so it would probably find the correct imports. However, the variables from that script won't be available in your notebook. You could have the Python 2 file write out a Pickle file and try to read that, maybe...
This worked for me:
! source /home/ubuntu/miniconda/etc/profile.d/conda.sh && conda activate envname && python run.py
Note: It only works if you run all commands in one line connecting them with &&
Related
can i list the python packages that are actually required for running exiting python program running in linux.
I tried running following commands.
pip3 freeze
pip3 list
To find the modules used by a single python script, you can try to use ModuleFinder:
Create a new python script to analyze the modules your script is using:
New Script:
from modulefinder import ModuleFinder
finder = ModuleFinder()
finder.run_script('MultiProcess.py')
print('Loaded modules:')
for name, mod in finder.modules.items():
print(('%s: ' % name))
print((','.join(list(mod.globalnames.keys())[:3])))
print(('-'*50))
print('Modules not imported:')
print(('\n'.join(iter(finder.badmodules.keys()))))
The output is very verbose and detailed
reference:
https://docs.python.org/2/library/modulefinder.html
I have installed the opensource version of pymol on windows (https://github.com/schrodinger/pymol-open-source).
To run pymol, I can open a python3.8 command prompt on windows and type the following:
import pymol
pymol.finish_launching()
This launches the pymol gui.
I'd now like to create a simple python script so that I can click the script (or convert to a windows executible using for example pyinstaller), however if I put the above commands in a script pymol.py and then run with:
python pymol.py
where python points to python3.8 I get the following error:
AttributeError: partially initialize module 'pymol' has no attribute 'finish_launching' most likely due to a circular import.
Why does this work on the command line python but not in a script and how can I fix this so the script runs and can be converted to a windows executible?
I have created an virtual env gcloudenv on my nividia nano running ubuntu. I was able to successfully install flask and required libraries and able to deploy my appengine into GCP from this virtual env. All my work is in python and I was using nano as my editor to get my code up and running. No issues so far.
my virtual env gcloudenv already has all the required packages for flask, jinga etc and I can see them when I run pip freeze.
Then I tried to work on Jupyter notebook as my code was getting little complicated and I didn't want to write full code and then run.
I already had jupyter notebook installed before creating the virtual env. I also installed jupyter within in virtual env as well.
So I followed the instruction to create a new kernel by running the following command:-
(gcloudenv) sunny#my-nano:~gcloudenv/MyApp/mainfolder$ pip install ipykernel
(gcloudenv) sunny#my-nano:~gcloudenv/MyApp/mainfolder$ ipython kernel install --user --
name=gcloudenv
Now, I ran the notebook as:
(gcloudenv) sunny#my-nano:~gcloudenv/MyApp/mainfolder$
/home/gcloudenv/bin/jupyter notebook
When trying to import the flask I get the following error:
ModuleNotFoundError: No module named 'flask'
Note sure what is going on as I getting blanked out.
Add
!pip install flask
in the beginning of your Jupyter notebook.
Finally I managed to solve my problem. Thanks to a wonderful post https://jakevdp.github.io/blog/2017/12/05/installing-python-packages-from-jupyter/.
In essence, I had 2 problems:-
1. I did not have a jupyter-notebook within my virtual env. Originally I thought I had it installed but that was incorrect. so whenever I tried to launch one, it was picking the first jupyter notebook it could find in the path.
The good way to find out which one is it pointing to , is to run which command
(gcloudenv) sunny#my-nano:~/gcloudenv$ which jupyter-notebook
For me, that was at:
/home/sunny/archiconda3/bin/jupyter-notebook
I had in fact 3 copies of jupyter-notebook on my system. One was probably installed using sudo pip and therefore went into the root folder. Probably not a good thing to do.
So I installed a fresh jupyter-notebook with the following command:-
(gcloudenv) $ pip install jupyter notebook
2.Next is to check list of Jupyter kernels available by running the following from the jupyter notebook ( or from command line):
!jupyter kernelspec list (OR (gcloudenv) $jupyter kernelspec list
My jupyter notebook was not able to import flask libraries because it was pointing to a wrong kernel config outside of my virtualenv gcloudenv.
Available kernels:
gcloudenv /home/sunny/.local/share/jupyter/kernels/gcloudenv ( correct one)
python3 /home/sunny/gcloudenv/share/jupyter/kernels/python3
You can determine which python version it is picking by doing a 'more' on the file:-
(gcloudenv) $
/more/home/sunny/.local/share/jupyter/kernels/gcloudenv/kernel.json
Once I changed my kernel to point to python3 from within the notebook, it picked the correct path and all the relevant libraries I needed.
In summary when you hit the problem as mentioned above, do the following:-
check the path of the python ( whereis python or which python)
check if you are running the 'right' notebook. This is determined by the path and if you have sourced your virtualenv.
Install jupyter notebook using pip from within your virtualenv( do not use sudo)
Check the Jupyter kernel. This may be particularly relevant if you have a common jupyter notebook and you want to work with multiple virtualenv.
I'm using Anaconda to manage both Python and Jupyter. That is:
>> which python
>> /home/.../software/anaconda3/bin/python
and
>> which jupyter
>> /home/.../software/anaconda3/bin/jupyter
But Jupyter's python kernel seems to be pointing to a system version of Python rather than my local version through Anaconda, since the sys.path is different in a Jupyter Python 3 notebook. Also, jupyter kernelspec list gives the following:
Available kernels:
ir /usr/local/share/jupyter/kernels/ir
matlab /usr/local/share/jupyter/kernels/matlab
python3 /usr/local/share/jupyter/kernels/python3
This doesn't seem altogether surprising since the docs say in section 1.5.5:
By default, kernel specs will go in a system-wide location (e.g. /usr/local/share/jupyter/kernels). If doing a --user install, the kernel specs will go in the JUPYTER_DATA_DIR location.
For personal sanity and organization, I want the version of Python that I use in the command line to be the same that is accessed in Jupyter. As a result, I think that what I should do is change my jupyter kernelspec list for python3 so that it points to my desired Anaconda python version, i.e. /home/.../software/anaconda3/bin/python. My questions are: 1) is that indeed the best solution for my stated preferences, and 2) how do I actually change my jupyter kernelspec entry for python3? Not sure if this will come up, but I don't want to be using virtual environments--I want the default to be same version of Python across both the command line and Jupyter.
I ended up reposting this to the Jupyter Github issues page, and was recommended to delete /usr/local/share/jupyter/kernels/python3. This allows Jupyter to find a default Python kernel using the same Python running Jupyter itself (i.e. Anaconda), and this worked for me.
You can find my post on Jupyter's Github page as well as an explanation for why the above solution works here.
1) Jupyter kernels in /usr/local/ are indeed a global install. But I do not see why it couldn't be linked to your anaconda python3 interpreter.
2) To explicitly link your anaconda interpreter to your jupyter install you can run :
pip install ipykernel
python -m ipykernel install --prefix=/usr/local/ --name "anaconda_kernel"
for a global install, or change /usr/local/ if you want a per user install. A doc is specially set for anaconda here
If you combine it with jupyter kernelspec remove python3 beforehand, you can then reset your anaconda kernel as default to be sure.
Apologies in advance, I think the issue is quite perplexing!
I would like to use TensorFlow through Jupyter, with a Python3 kernel.
However the command import tensorflow as tf returns the error: ImportError: No module named tensorflow when either Python2 or Python3 is specified as the Jupyter kernel.
I have Python 2 and Python 3 installed on my Mac and can access both
versions through Terminal.
I installed TensorFlow for Python 3, however I can only access it via Python 2 on the Terminal.
As such, this question is really two-fold:
I want to get TensorFlow working with Python3
...which should lead to TensorFlow working with Jupyter on the Python3 terminal.
I had the same problem and solved it using the tutorial Using a virtualenv in an IPython notebook. I'll walk you through the steps I took.
I am using Anaconda, and I installed a new environment tensorflow using these instructions at tensorflow.org. After that, here is how I got tensorflow to work in a Jupyter notebook:
Open Terminal
Run source activate tensorflow. You should now see (tensorflow) at the beginning of the prompt.
Now that we are in the tensorflow environment, we want to install ipython and jupyter in this environment: Run
conda install ipython
and
conda install jupyter
Now follow the instructions in the tutorial linked above. I'll repeat them here with a bit more information added. First run
ipython kernelspec install-self --user
The result for me was Installed kernelspec python3 in /Users/charliebrummitt/Library/Jupyter/kernels/python3
Run the following:
mkdir -p ~/.ipython/kernels
Then run the following with <kernel_name> replaced by a name of your choice (I chose tfkernel) and replace the first path (i.e., ~/.local/share/jupyter/kernels/pythonX) by the path generated in step 4:
mv ~/.local/share/jupyter/kernels/pythonX ~/.ipython/kernels/<kernel_name>
Now you'll see a new kernel if you open a Jupyter notebook and select Kernel -> Change kernel from the menu. But the new kernel will have the same name as your previous kernel (for me it was called Python 3). To give your new kernel a unique name, run in Terminal
cd ~/.ipython/kernels/tfkernel/
and then run vim kernel.json to edit the file kernel.json so that you replace the value of "display_name" from the default (Python 3) to a new name (I chose to call it "tfkernel"). Save and exit vim by typing :wq while in command mode.
Open a new Jupyter notebook and type import tensorflow as tf. If you didn't get ImportError then you are ready to go!