We have a shared ai-platform jupyter notebook instance, is there anyway for each user to have their own directory that is not shared with the whole team?
Related
I am running Spark jobs in my Jupyter notebook deployed in an EKS Cluster. Jupyterlab provides a Spark UI Monitoring extension where I can view my Spark jobs by clicking on the "SparkMonitor" tab. I am also trying to access the History Server that is deployed on a different pod. What is the best way for me to access the History Server? Is there any way I can route to the History Server within the Jupyter Notebook?
If a shared cluster is being used by development team in Databricks, how is isolation done to ensure that one developer does not impact another developer's work by installing a particular package. Does CONDA help in isolation?
When you're working on Databricks you have multiple levels of libraries (doc):
Cluster level libraries that are installed using cluster UI or REST API - these libraries are shared by all users of the cluster
For Python & R there is support for notebook-level libraries. For Python, libraries installed with %pip install are installed to a virtual environment specific to a given notebook. So different users can install different libraries or different versions of the same library on the same cluster without breaking other's work.
I have an Ubuntu virtual machine on Azure. I am running a Jupyter notebook on this VM to train a Keras sequential model. The model takes ~24 hours to fully train. Is there a way to run the notebook without using my browser. So leaving the Jupyter server and notebook running in the background BUT for all results/outputs/plots to be displayed within the Jupyter notebook?
When I try to do this by starting the Jupyter notebook interactively through my browser, then closing my browser but leaving the Jupyter kernel running, the cells within the notebook will execute but no output is displayed.
I have looked at this similar question: Running an IPython/Jupyter notebook non-interactively
But this is dealing with a different issue where the kernel is not running when trying to execute
This question is for the same issue: Jupyter notebook output cell freezes (Keras related?)
But the solution is to save some outputs in csv files which is not ideal
For anyone that finds this question looking for the same functionality in Jupyter;
Unfortunately as of May 2020 an issue relating to this is still open: https://github.com/jupyter/notebook/issues/1647
Therefore it looks like it is currently not possible to run Jupyter notebooks with the browser closed.
We have installed and setup jupyter notebook on two of our linux hadoop servers with pyspark kernels. Both the servers have same kernel.jason configuration which has same spark and pyhton versions.
on one server jupyter notebook ui --> pyspark kernel is working fine but on other server when launching pyspark kernel, a file(with name - ??????????) is getting generated in the users home directory, able to execute queries in opened pyspark kernel session, but when new jupyternotebook ui is launched and in corresponding pyspark kernel, unable to execute the queries. We are able to execute only after removing the ??????? file which was generated and relauching the jupyter notebook again.
-We see this behaviuor for all users, and is happening only on one server. Can someone please help out with in resolving this issue.
versions:
Python 2.7.12
Spark 2.1.1
Steps performed:
-Verified pyspark kernel configs with jupyter running on other server which has no issues.
-restarted spark client on the server
-Tried rebooting the server which did not resolve the issue.
looks like might be an issue with the server hardware
I am trying to figure out if it is possible to work locally in python with a spark context of a remote EMR cluster(AWS). I've set up the cluster but a locally defined SparkContext with remote master doesn't seem to work. Does anybody have experience with that? Working on a remote notebook is limited because you cannot create python modules and files. Working locally is limited due to computing resources. There is the option to SSH to the master node but then I cannot use a graphical IDE such as pyCharm