For my IPython QtConsole, I have a startup script in my profile_default/startup folder, which contains the following lines:
ipy = get_ipython()
try:
plot
except NameError: # not loaded yet
ipy.run_line_magic("pylab", "inline")
This works fine in the QtConsole and Notebook, but if I now run ipython with no subcommand, i.e. in the Windows command line, it prints an error that the 'inline' GUI is invalid.
Is there any way I can check for the subcommand (qtconsole, notebook or "None") inside the startup script, so I can avoid that message?
(IPython QtConsole 3.2.0, WinPython-64bit-3.4.3.4)
I'm going to assume that ipy is the result of get_ipython().
Warning: You should not try to have code that behave differently on various frontend. It will break at some point, and lead to hard to debug issues that don't make sens. Nasal demons will be on a the lookout for the smallest misstep to haunt you.
That being say, in the pure, classical terminal IPython you can can verify that:
In [1]: type(get_ipython())
Out[1]: IPython.terminal.interactiveshell.TerminalInteractiveShell
Which is not true for Notebook and QtConsole (which are ZMQInteractiveShell). In both case IPython.terminal.interactiveshell.TerminalInteractiveShell should be importable, and you can check with issubclass in which case you are.
Now, you can also create your own aliases on windows (not sure how), that pass extra command line arguments to IPython, in order for notebook and qtconsole to not have the same startup sequence.
Related
I'm a new beginner with python und have recently installed Julia. And as I wrote "from julia import Main" in Jupyter note through Anaconda, an error would occur which is "no module named 'julia'". I don't know which part is wrong or is this about the path of the installation of Julia? enter image description here
I would appreciate it so much if you could help me.
"from x import y" is the syntax for importing Python packages in a Python script or notebook, but as you probably know, Julia is a separate language of its own, not a Python package.
If you want to use Julia in a Jupyter notebook, you will instead generally want to change the whole Jupyter kernel to Julia from Python, rather than just adding a package (though see comment from Przemyslaw above if you want to call Julia from Python for any reason).
The easiest way to install a Julia kernel for Jupyter and start a notebook using that kernel is to just start the Jupyter notebook from Julia. In other words, first start Julia, so that you have a Julia REPL that looks something like the image below, and then type
julia> using Pkg; Pkg.add("IJulia") # If IJulia is not already installed
julia> using IJulia
julia> notebook()
which will open up a Jupyter instance with a Julia kernel installed, from which you can open a new notebook using that Julia kernel
which should look like the following (note the Julia symbol in the upper right corner)
Note that at no point in this process do you use Python or Anaconda for anything, and note also that in Julia there is no from keyword and also generally no such thing as Main
Recently I'm considering using mainly python 3 although I have used python 2.7 so far.
But I encountered variable problem on python 3 as follows.
For example, the below code properly works in python 2.
#print a
a=1
I run the code named test.py many times on ipython console (python 2.7.16) in spyder 3.3.6.
After the first run, I remove # in the first line.
Then, ipython console outputs 1 which is a's value.
However, when I run the above code (the first line is replaced by #print(a)) similar to the above on ipython console (python 3.7.6) in spyder 4.0.1, ipython console outputs an error message,
NameError: name 'a' is not defined.
When I input a in the ipython console, the console outputs 1.
Can I do the same thing in my python 3 environment as I do in python 2?
Thank you in advance.
There is a setting when running the script called "Run in console's namespace instead of an empty one".
If you tick that box it keeps the variables in the namespace.
When I run pip freeze in the command line, I see: numpy==1.16.3. I am trying to run numpy.isin(...) (documented here), but I get the error
AttributeError: 'module' object has no attribute 'isin'.
So I check the version of python being run in the python script as:
import numpy as np
print(np.__version__)
This outputs:
1.8.0rc1
And since the function was introduced in version 1.13.0, I need to figure out why the older version of numpy is being used instead of the up-to-date version. I usually upgrade with pip in the command line. How can I fix this problem?
Just in case somebody else has a similar problem, I followed the steps in this tutorial to familiarize myself with pipvirtualenv. I installed the updated version of numpy in the pipvirtualenvwhile in the same directory as my python script(s). While there, I entered the virtual env shell and then ran the command - it worked. I then used exit (as opposed to deactivate) to deactivate the shell, and tried the script while outside the shell - it worked again.
I've spent a few days now trying to make Spark work with my Jupyter Notebook and Anaconda. Here's what my .bash_profile looks like:
PATH="/my/path/to/anaconda3/bin:$PATH"
export JAVA_HOME="/my/path/to/jdk"
export PYTHON_PATH="/my/path/to/anaconda3/bin/python"
export PYSPARK_PYTHON="/my/path/to/anaconda3/bin/python"
export PATH=$PATH:/my/path/to/spark-2.1.0-bin-hadoop2.7/bin
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS="notebook" pyspark
export SPARK_HOME=/my/path/to/spark-2.1.0-bin-hadoop2.7
alias pyspark="pyspark --conf spark.local.dir=/home/puifais --num-executors 30 --driver-memory 128g --executor-memory 6g --packages com.databricks:spark-csv_2.11:1.5.0"
When I type /my/path/to/spark-2.1.0-bin-hadoop2.7/bin/spark-shell, I can launch Spark just fine in my command line shell. And the output sc is not empty. It seems to work fine.
When I type pyspark, it launches my Jupyter Notebook fine. When I create a new Python3 notebook, this error appears:
[IPKernelApp] WARNING | Unknown error in handling PYTHONSTARTUP file /my/path/to/spark-2.1.0-bin-hadoop2.7/python/pyspark/shell.py:
And sc in my Jupyter Notebook is empty.
Can anyone help solve this situation?
Just want to clarify: There is nothing after the colon at the end of the error. I also tried to create my own start-up file using this post and I quote here so you don't have to go look there:
I created a short initialization script init_spark.py as follows:
from pyspark import SparkConf, SparkContext
conf = SparkConf().setMaster("yarn-client")
sc = SparkContext(conf = conf)
and placed it in the ~/.ipython/profile_default/startup/ directory
When I did this, the error then became:
[IPKernelApp] WARNING | Unknown error in handling PYTHONSTARTUP file /my/path/to/spark-2.1.0-bin-hadoop2.7/python/pyspark/shell.py:
[IPKernelApp] WARNING | Unknown error in handling startup files:
Well, it really gives me pain to see how crappy hacks, like setting PYSPARK_DRIVER_PYTHON=jupyter, have been promoted to "solutions" and tend now to become standard practices, despite the fact that they evidently lead to ugly outcomes, like typing pyspark and ending up with a Jupyter notebook instead of a PySpark shell, plus yet-unseen problems lurking downstream, such as when you try to use spark-submit with the above settings... :(
(Don't get me wrong, it is not your fault and I am not blaming you; I have seen dozens of posts here at SO where this "solution" has been proposed, accepted, and upvoted...).
At the time of writing (Dec 2017), there is one and only one proper way to customize a Jupyter notebook in order to work with other languages (PySpark here), and this is the use of Jupyter kernels.
The first thing to do is run a jupyter kernelspec list command, to get the list of any already available kernels in your machine; here is the result in my case (Ubuntu):
$ jupyter kernelspec list
Available kernels:
python2 /usr/lib/python2.7/site-packages/ipykernel/resources
caffe /usr/local/share/jupyter/kernels/caffe
ir /usr/local/share/jupyter/kernels/ir
pyspark /usr/local/share/jupyter/kernels/pyspark
pyspark2 /usr/local/share/jupyter/kernels/pyspark2
tensorflow /usr/local/share/jupyter/kernels/tensorflow
The first kernel, python2, is the "default" one coming with IPython (there is a great chance of this being the only one present in your system); as for the rest, I have 2 more Python kernels (caffe & tensorflow), an R one (ir), and two PySpark kernels for use with Spark 1.6 and Spark 2.0 respectively.
The entries of the list above are directories, and each one contains one single file, named kernel.json. Let's see the contents of this file for my pyspark2 kernel:
{
"display_name": "PySpark (Spark 2.0)",
"language": "python",
"argv": [
"/opt/intel/intelpython27/bin/python2",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"env": {
"SPARK_HOME": "/home/ctsats/spark-2.0.0-bin-hadoop2.6",
"PYTHONPATH": "/home/ctsats/spark-2.0.0-bin-hadoop2.6/python:/home/ctsats/spark-2.0.0-bin-hadoop2.6/python/lib/py4j-0.10.1-src.zip",
"PYTHONSTARTUP": "/home/ctsats/spark-2.0.0-bin-hadoop2.6/python/pyspark/shell.py",
"PYSPARK_PYTHON": "/opt/intel/intelpython27/bin/python2"
}
}
I have not bothered to change my details to /my/path/to etc., and you can already see that there are some differences between our cases (I use Intel Python 2.7, and not Anaconda Python 3), but hopefully you get the idea (BTW, don't worry about the connection_file - I don't use one either).
Now, the easiest way for you would be to manually do the necessary changes (paths only) to my above shown kernel and save it in a new subfolder of the .../jupyter/kernels directory (that way, it should be visible if you run again a jupyter kernelspec list command). And if you think this approach is also a hack, well, I would agree with you, but it is the one recommended in the Jupyter documentation (page 12):
However, there isn’t a great way to modify the kernelspecs. One approach uses jupyter kernelspec list to find the kernel.json file and then modifies it, e.g. kernels/python3/kernel.json, by hand.
If you don't have already a .../jupyter/kernels folder, you can still install a new kernel using jupyter kernelspec install - haven't tried it, but have a look at this SO answer.
Finally, don't forget to remove all the PySpark-related environment variables from your bash profile (leaving only SPARK_HOME should be OK). And confirm that, when you type pyspark, you find yourself with a PySpark shell, as it should be, and not with a Jupyter notebook...
UPDATE (after comment): If you want to pass command-line arguments to PySpark, you should add the PYSPARK_SUBMIT_ARGS setting under env; for example, here is the last line of my respective kernel file for Spark 1.6.0, where we still had to use the external spark-csv package for reading CSV files:
"PYSPARK_SUBMIT_ARGS": "--master local --packages com.databricks:spark-csv_2.10:1.4.0 pyspark-shell"
Conda can help correctly manage a lot of dependencies...
Install spark. Assuming spark is installed in /opt/spark, include this in your ~/.bashrc:
export SPARK_HOME=/opt/spark
export PATH=$SPARK_HOME/bin:$PATH
Create a conda environment with all needed dependencies apart from spark:
conda create -n findspark-jupyter-openjdk8-py3 -c conda-forge python=3.5 jupyter=1.0 notebook=5.0 openjdk=8.0.144 findspark=1.1.0
Activate the environment
$ source activate findspark-jupyter-openjdk8-py3
Launch a Jupyter Notebook server:
$ jupyter notebook
In your browser, create a new Python3 notebook
Try calculating PI with the following script (borrowed from this)
import findspark
findspark.init()
import pyspark
import random
sc = pyspark.SparkContext(appName="Pi")
num_samples = 100000000
def inside(p):
x, y = random.random(), random.random()
return x*x + y*y < 1
count = sc.parallelize(range(0, num_samples)).filter(inside).count()
pi = 4 * count / num_samples
print(pi)
sc.stop()
I just conda installed sparkmagic (after re-installing a newer version of Spark).
I think that alone simply works, and it is much simpler than fiddling configuration files by hand.
UPDATE: Actually, now I have checked, and PyInstaller is saying Invalid Syntax for EVERY script I have, even ones that I have previously packaged with PyInstaller without any issues. I uninstalled and reinstalled PyInstaller, but it's still having the same problem. Is PyInstaller not compatible with Python 3.5.1? That's the only thing I can think of that I might have updated between now and when everything was working fine
Original Question: I'm sure there is a really simple and stupid answer for what I'm doing wrong, because I can't seem to find any other cases of people having this problem.
I have a script I want to package into a standalone executable. In the past, I have used PyInstaller with minimal hassle. Py2exe and cx_freeze have never worked for me. I'm using Python version 3.5.1 and PyInstaller version 3.2, which I believe is the current version since I just uninstalled and reinstalled.
The command I am trying to use is so simple I feel like an idiot for having trouble.
pyinstaller --onefile myscript.py
File "<stdin>", line 1
pyinstaller --onefile myscript.py
SyntaxError: invalid syntax
It's giving a generic SyntaxError: invalid syntax even though that is the exact command straight from the PyInstaller docs.
To be sure, I also tried to include the entire path to my script in the command, added and took out quotation marks, and tried every variation I could think of but it gives me the same syntax error every time.
I'm pretty much a beginner, so any really advanced fixes will go over my head. But like I said, I assume it's something silly I've missed. Thanks in advance.
The syntax error is caused by your command itself, not by the code it calls.
This part is very indicative:
File "<stdin>", line 1
pyinstaller --onefile myscript.py
You actually tried to run that command in a Python shell.
But it is not Python code. You should run it in a usual shell (cli.exe, bash, …)
Run It In CMD
Why are you running it in python shell? It's a problem with python syntax because it is not even defined.
>>> pyinstaller --onefile myscript.py
And, by the way. You are not even importing the PyInstaller module.
Run this line on your CMD:
pyinstaller --onefile filename.py
Ensure your script doesn't has any syntax errors. If it's so, then pyinstaller will rethrow the exception and this could be one of the reason.