Not able to serve jupyter notebooks in binder - python-3.x

Binder project looks promising.
It helps in executing notebooks in a github repository by building an executable container.
I am trying to build an executable container in binder with the following Dockerfile that has Perl 6 and Python 3 kernels:
FROM sumdoc/perl-6
ENV NB_USER jovyan
ENV NB_UID 1000
ENV HOME /home/${NB_USER}
RUN adduser --disabled-password \
--gecos "Default user" \
--uid ${NB_UID} \
${NB_USER}
RUN apt-get update \
&& apt-get install -y build-essential \
git wget libzmq3-dev ca-certificates python3-pip \
&& rm -rf /var/lib/apt/lists/* && pip3 install jupyter notebook --no-cache-dir \
&& zef -v install https://github.com/bduggan/p6-jupyter-kernel.git --force-test \
&& jupyter-kernel.p6 --generate-config
ENV TINI_VERSION v0.16.1
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /usr/bin/tini
RUN chmod +x /usr/bin/tini
ENTRYPOINT ["/usr/bin/tini", "--"]
COPY . ${HOME}
USER root
RUN chown -R ${NB_UID} ${HOME}
USER ${NB_USER}
EXPOSE 8888
CMD ["jupyter", "notebook", "--port=8888", "--no-browser", "--ip=0.0.0.0", "--allow-root"]
Binder launches this window after building a container:
While trying to run Perl 6 or Python 3 notebook I get this error:
I read this documentation on binder but could not succeed.
What things I am missing? Any help with explanations would be appreciated.

After going through this Dockerfile, I solved the issue.
I even wrote a blog on using Perl 6 notebook in Binder here.
What I was missing was to add WORKDIR $HOME after USER ${NB_USER}in my Dockerfile as follows:
FROM sumankhanal/perl-6
ENV NB_USER jovyan
ENV NB_UID 1000
ENV HOME /home/${NB_USER}
RUN adduser --disabled-password \
--gecos "Default user" \
--uid ${NB_UID} \
${NB_USER}
RUN apt-get update \
&& apt-get install -y build-essential \
git wget libzmq3-dev ca-certificates python3-pip \
&& rm -rf /var/lib/apt/lists/* && pip3 install jupyter notebook --no-cache-dir \
&& zef -v install https://github.com/bduggan/p6-jupyter-kernel.git --force-test \
&& jupyter-kernel.p6 --generate-config
ENV TINI_VERSION v0.16.1
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /usr/bin/tini
RUN chmod +x /usr/bin/tini
ENTRYPOINT ["/usr/bin/tini", "--"]
COPY . ${HOME}
USER root
RUN chown -R ${NB_UID} ${HOME}
USER ${NB_USER}
WORKDIR ${HOME}
EXPOSE 8888
CMD ["jupyter", "notebook", "--port=8888", "--no-browser", "--ip=0.0.0.0", "--allow-root"]

Related

Why would Python not be available to a Docker Entrypoint Script?

This Python 3.9 project has a Dockerfile, that builds successfully. The file makes use of an ENTRYPOINT script to create some directories and handle some clean-up at run time. It is a bash script. The ENTRYPOINT script has no problem running until the very end, where it is expected to execute the CMD that is passed next. Well, I should say this behavior only happens when Kaniko builds the image. When the image is built locally, no such problem occurs. However, I am willing to chalk that up to the fact that locally is on a Windows machine. However, that shouldn't matter here because the error thrown is:
/opt/project/conf/entrypoint.sh: /usr/bin/supervisord: /usr/bin/python3: bad interpreter: No such file or directory
/opt/project/conf/entrypoint.sh: line 8: /usr/bin/supervisord: Success
Now I have looked at many "bad interpreter" questions. They all seem to revolve around the interpreter being in a custom place. I am reliant upon the default spot for the Python 3.9 interpreter. On Debian Bullseye (The OS behind the base image) that should be /usr/local/bin/python or /usr/local/bin/python3. So I am completely stumped as to why it is unable to find or use it.
Here are the implementation details:
Dockerfile:
FROM python:3.9-slim-bullseye
# Minimum Required Environment Variables
ENV SHELL=/bin/bash
ENV CC /usr/bin/gcc
ENV CXX /usr/bin/g++
ENV LANG=C.UTF-8
ENV DEBIAN_FRONTEND=noninteractive
ENV PYMSSQL_BUILD_WITH_BUNDLED_FREETDS=1
ENV PIP_CONFIG_FILE=/etc/pip.conf
ENV TZ=America/Los_Angeles
# Project Specific Environment Variables
ENV PROJECT_LOGFILE=/var/log/project/project.log
ENV PROJECT_CONFIG_DIRECTORY=/opt/project/conf
ENV PROJECT_SETTINGS_MODULE="project.settings"
# Files Needed for Dependency Installation
COPY dev/.pip.conf /etc/pip.conf
COPY dev/dev-requirements.txt /usr/local/requirements.txt
# Dependency Installation
WORKDIR /tmp
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get install musl-dev g++ bash curl gnupg -y \
&& curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \
&& curl https://packages.microsoft.com/config/debian/11/prod.list > /etc/apt/sources.list.d/mssql-release.list \
&& apt-get update \
&& apt-get install --no-install-recommends libfreetype-dev freetds-dev python-dev git libpng-dev libxml2-dev \
libxslt-dev libssl-dev libopenblas-dev rsyslog supervisor tini tzdata libghc-zlib-dev libjpeg-dev cron \
libgssapi-krb5-2 unixodbc-dev -y \
&& ACCEPT_EULA=Y apt-get install -y msodbcsql18 \
&& ln -s /usr/include/locale.h /usr/include/xlocale.h \
&& pip install --no-cache-dir --upgrade pip setuptools wheel \
&& pip install matplotlib --no-cache-dir \
&& pip install --no-cache-dir -r /usr/local/requirements.txt
# Setting Up For Install
COPY conf/ /opt/project/conf/
RUN mkdir -p /var/log/project /conf \
&& cp /opt/project/conf/supervisord.conf /conf/supervisord.conf \
&& cp /opt/project/conf/rsyslog.conf /conf/rsyslog.conf
WORKDIR /opt
# Copy Over Packages
COPY project-db-migrations /opt/project/project-db-migrations
COPY infrastructure /opt/infrastructure
COPY project /opt/project/src
COPY README.md /opt/project/README.md
# Install Infrastructure
RUN cd /opt/infrastructure && python3 setup.py install
# Install Project Service
RUN cd /opt/project/src && python3 setup.py install
RUN ["chmod", "+x", "/opt/project/conf/entrypoint.sh"]
WORKDIR /
EXPOSE 80
ENTRYPOINT ["tini", "--", "/opt/project/conf/entrypoint.sh"]
CMD ["supervisord", "-c", "/conf/supervisord.conf"]
entrypoint.sh
#!/bin/bash
set -eu
echo "Setting Up Project Service"
# Adding Temp Directory
mkdir -p /opt/project/tmp
echo "Service has been setup"
exec $#
supervisord.conf
[supervisord]
nodaemon=true
logfile=/var/log/project/supervisord.log
childlogdir=/var/log/project
[program:rsyslogd]
command=/usr/sbin/rsyslogd -n -f /conf/rsyslog.conf
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:crond]
command=/usr/sbin/cron -f -l 15
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:project]
command=python -m project.run --server
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
The image is ran and deployed without changes to the user, so it should be running as root.
In this case, after some digging I found there is an issue with the Kaniko version DevOps had running. That was causing the issue. Because the image wasn't being flattened correctly, Python could not start properly.

Fixing security vulnerabilities in docker image

I have the following docker file
FROM debian:stable
# Avoid warnings by switching to noninteractive
ENV DEBIAN_FRONTEND=noninteractive
#Versions
ENV HELM_VERSION=v3.10.0
ENV KUBECTL_VERSION=v1.20.9
ENV MAVEN_OPTS="-Djavax.net.ssl.trustStore=/cicd/assets/truststore.jks"
ENV TERRAFORM_VERSION=1.2.0
ENV GOLANG_VERSION=1.19.1
ENV TERRAGRUNT_VERSION=v0.38.7
RUN set -xe \
&& apt-get update -y \
&& apt-get install -y python3-pip
RUN apt-get install zip unzip
#Copy python requirements file
COPY requirements.txt /tmp/pip-tmp/
# Makes the Ansible directories
RUN mkdir /etc/ansible /ansible
RUN mkdir ~/.ssh
# Configure apt and install python packages
RUN apt-get update -y -q \
&& apt-get upgrade -y -q \
&& apt-get install -y wget \
&& apt-get -y install --no-install-recommends apt-utils dialog 2>&1 \
&& apt-get install -y --no-install-recommends apt-utils \
&& apt-get -y install ca-certificates software-properties-common build-essential curl git gettext-base maven sshpass krb5-user \
&& pip --disable-pip-version-check --no-cache-dir install -r /tmp/pip-tmp/requirements.txt \
&& apt-get -y install jq \
&& rm -rf /tmp/pip-tmp
#Install helm
RUN wget https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz \
&& tar -zxvf helm-${HELM_VERSION}-linux-amd64.tar.gz \
&& mv linux-amd64/helm /usr/local/bin/helm
#Install kubectl
RUN curl --silent https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl --output /usr/local/bin/kubectl \
&& chmod +x /usr/local/bin/kubectl
#Install Docker CLI
RUN curl -sSL https://get.docker.com/ | sh \
&& curl -L "https://github.com/docker/compose/releases/download/2.10.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose \
&& chmod +x /usr/local/bin/docker-compose
#Install AWS CLI
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
&& unzip awscliv2.zip \
&& ./aws/install
#Copy Assets
#RUN mkdir -p /cicd
#COPY assets /cicd
#Install helm plugins
#RUN helm plugin add https://github.com/databus23/helm-diff
#RUN helm plugin install /cicd/helm-nexus-push
# Downloading gcloud package
RUN curl https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz > /tmp/google-cloud-sdk.tar.gz
# Installing the package
RUN mkdir -p /usr/local/gcloud \
&& tar -C /usr/local/gcloud -xvf /tmp/google-cloud-sdk.tar.gz \
&& /usr/local/gcloud/google-cloud-sdk/install.sh
# Adding the package path to local
ENV PATH $PATH:/usr/local/gcloud/google-cloud-sdk/bin
RUN cd /tmp && \
wget https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip && \
unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip -d /usr/local/bin && \
rm -rf /tmp/*
RUN cd /tmp && \
wget https://dl.google.com/go/go${GOLANG_VERSION}.linux-amd64.tar.gz && \
tar -xzf go${GOLANG_VERSION}.linux-amd64.tar.gz -C /usr/local && \
rm -rf /tmp/*
RUN cd /tmp && \
wget https://github.com/gruntwork-io/terragrunt/releases/download/${TERRAGRUNT_VERSION}/terragrunt_linux_amd64 && \
mv terragrunt_linux_amd64 /usr/local/bin/terragrunt && \
chmod +x /usr/local/bin/terragrunt && \
rm -rf /tmp/*
RUN git config --global http.sslCAinfo /etc/ssl/certs/ca-certificates.crt
ENV GOPATH=/usr/local/go
ENV PATH=/usr/local/go/bin:$PATH
ENV CGO_ENABLED=0
RUN go version
RUN terraform --version
RUN terragrunt --version
RUN ansible --version
CMD bash
I build the docker image and upload it to google artifact registry, but I always come across security vulnerabilities I have tried to fix it but unfortunately I'm unable to fix the security vulnerabilities. Please look at the critical errors and let me know how I can fix this, Any recommendation is appreciated. Thank You.
It looks like the DockerFile is trying to a specific version of golang by hand into "/usr/local" rather than using the Debian package manager. According to the info at https://security-tracker.debian.org/tracker/CVE-2021-38297, that bug is fixed in 1.17.3-3 and the Dockerfile are using 1.19.1. So perhaps there is an old golang installation in the base image ... and that is what the scanner is picking up. Check that, and if necessary apt install a newer version.
Likewise, https://security-tracker.debian.org/tracker/CVE-2022-23806 should be fixed by a newer version of golang. See the CVE link for versions.
https://security-tracker.debian.org/tracker/CVE-2015-20107 could be fixed by upgrading to Python 3.10.6-1 or later.
https://security-tracker.debian.org/tracker/CVE-2019-19814 doesn't appear to have a fix from upstream, so there is nothing you can do about it except not use f2fs.
https://security-tracker.debian.org/tracker/CVE-2022-29599 can be fixed by updating the maven-shared-utils package; see the CVE link for versions.
https://security-tracker.debian.org/tracker/CVE-2022-1996 has a fix upstream but it is awaiting triage by the Debian team.
In summary, some of the vulnerabilities can be fixed, but for a couple of them no fix is readily available. So:
Apply the fixes that are available.
Then read the CVEs and accompanying explanations and 1) make a judgement whether they represent a risk that you can take, and 2) figure out if you can mitigate the risk; e.g. by locking down access to the running Docker container.

Setting Up Stand-alone LanguageTool in Docker container

Im trying to setup LanguageTool as a standalone server as a Docker Container. So what I did is download the standalone system provided at -> https://languagetool.org/download/LanguageTool-stable.zip and put it in my project. I setup the docker-compose.yml file like so
version: '3'
services:
grammar:
build: ./services/grammar
image: dev/grammar:1
restart: always
container_name: dev.grammar
ports:
- "8130:8130"
And I created the dockerfile inside the LanguageTool folder like so
FROM ubuntu:18.04
WORKDIR /tmp
RUN apt-get update
RUN apt-get install unzip
ADD https://languagetool.org/download/LanguageTool-stable.zip /tmp/LanguageTool-stable.zip
#RUN apt-get install -y unzip
RUN unzip /tmp/LanguageTool-stable.zip
RUN mv /tmp/LanguageTool-5.7 /usr/languagetool
CMD ["java", "-jar", "languagetool-server.jar", "--port", "8130", "--public", "--allow-origin", "'*'" ]
EXPOSE 8130
I have actually tried many iterations of the dockerfile like another example here
FROM debian:stretch
RUN set -ex \
&& mkdir -p /uploads /etc/apt/sources.list.d /var/cache/apt/archives/ \
&& export DEBIAN_FRONTEND=noninteractive \
&& apt-get clean \
&& apt-get update -y \
&& apt-get install -y \
bash \
curl \
openjdk-8-jre-headless \
unzip \
libhunspell-1.4-0 \
hunspell-de-at
ENV VERSION 5.7
COPY LanguageTool-$VERSION.zip /LanguageTool-$VERSION.zip
RUN unzip LanguageTool-$VERSION.zip \
&& rm LanguageTool-$VERSION.zip
WORKDIR /LanguageTool-$VERSION
CMD ["java", "-cp", "languagetool-server.jar", "org.languagetool.server.HTTPServer", "--port", "8130", "--public", "--allow-origin", "'*'" ]
EXPOSE 8130
But none of them seems to work. Please let me know what I am doing wrong here. Thanks in advance !!
Edit: Here is what my file/folder structure looks like here
I found the solution. Had to tinker around some configurations but I finally got it working. here is the dockerconfig file that worked for me.
FROM debian:stretch
RUN set -ex \
&& mkdir -p /uploads /etc/apt/sources.list.d /var/cache/apt/archives/ \
&& export DEBIAN_FRONTEND=noninteractive \
&& apt-get clean \
&& apt-get update -y \
&& apt-get install -y \
bash \
curl \
openjdk-8-jre-headless \
unzip \
libhunspell-1.4-0 \
hunspell-de-at
ENV VERSION 5.1
COPY LanguageTool-$VERSION.zip /LanguageTool-$VERSION.zip
RUN unzip LanguageTool-$VERSION.zip \
&& rm LanguageTool-$VERSION.zip
WORKDIR /LanguageTool-$VERSION
CMD ["java", "-cp", "languagetool-server.jar", "org.languagetool.server.HTTPServer", "--port", "8130", "--public", "--allow-origin", "'*'" ]
EXPOSE 8130

Docker build for Mattermost, /bin/sh dnf not found

I've got a CentOS 8 install, and I'm trying to use a docker container to run Mattermost to set up a local node for my family to use. I've been searching a lot online, but my google-fu appears to be weak as I can't get answers that address my issue.
I've downloaded docker, and docker compose using the following guide, again tailoring it to Centos - https://docs.mattermost.com/install/prod-docker.htm I've successfully run the "Hello World" container.
I'm using this guide and trying to tailor the Mattermost container install - https://wiki.archlinux.org/index.php/Ma ... ith_Docker
I've edited the ~/mattermost-docker/db/Dockerfile to remove references to apk, and put in yum and then dnf, and tried to execute with SUDO in the script and using SU account to run the script. Latest Dockerfile:
FROM postgres:9.4-alpine
ENV DEFAULT_TIMEZONE UTC
# Install some packages to use WAL
RUN echo "azure<5.0.0" > pip-constraints.txt
RUN dnf install -y \
build-base \
curl \
libc6-compat \
libffi-dev \
linux-headers \
python-dev \
py-pip \
py-cryptography \
pv \
libressl-dev \
&& pip install --upgrade pip \
&& pip --no-cache-dir install -c pip-constraints.txt 'wal-e<1.0.0' envdir \
&& rm -rf /tmp/* /var/tmp/* \
&& dnf clean all
# Add wale script
COPY setup-wale.sh /docker-entrypoint-initdb.d/
#Healthcheck to make sure container is ready
HEALTHCHECK CMD pg_isready -U $POSTGRES_USER -d $POSTGRES_DB || exit 1
# Add and configure entrypoint and command
COPY entrypoint.sh /
ENTRYPOINT ["/entrypoint.sh"]
CMD ["postgres"]
VOLUME ["/var/run/postgresql", "/usr/share/postgresql/", "/var/lib/postgresql/data", "/tmp", "/etc/wal-e.d/env"]
However it still fails on: docker-compose build
Error -
Building db
Step 1/10 : FROM postgres:9.4-alpine
---> 4e66908aa630
Step 2/10 : ENV DEFAULT_TIMEZONE UTC
---> Using cache
---> 03d176f9f783
Step 3/10 : RUN echo "azure<5.0.0" > pip-constraints.txt
---> Using cache
---> 35dbc995f705
Step 4/10 : RUN sudo dnf install -y build-base curl libc6-compat libffi-dev linux-headers python-dev py-pip py-cryptography pv libressl-dev && pip install --upgrade pip && pip --no-cache-dir install -c pip-constraints.txt 'wal-e<1.0.0' envdir && rm -rf /tmp/* /var/tmp/* && dnf clean all
---> Running in 4b89205fdca3
/bin/sh: dnf: not found
ERROR:Service 'db' failed to build : The command '/bin/sh -c sudo dnf install -y build-base curl libc6-compat libffi-dev linux-headers python-dev py-pip py-cryptography pv libressl-dev && pip install --upgrade pip && pip --no-cache-dir install -c pip-constraints.txt 'wal-e<1.0.0' envdir && rm -rf /tmp/* /var/tmp/* && dnf clean all' returned a non-zero code: 127````
Confirmed dnf, and yum are present in /bin and /usr/bin, confirmed /bin/sh -> /bin/bash. I'm not even sure what question I should be asking, so I'd appreciate some assistance in figuring out how I can get this container stood up.
Thanks.

Run npm test inside a docker image and exit

I have basically a docker image of a node js application.
REPOSITORY TAG IMAGE ID CREATED SIZE
abc-test 0.1 1ba85e0ca455 7 hours ago 1.37GB
I want to run npm test from folder /data/node/src but that doesn't seems to be working.
Here is the command what I am trying:
docker run -p 80:80 --entrypoint="cd /data/node/src && npm run test" abc-test:0.1
But that doesn't seems to be working.
Here is my dockerfile:
FROM python:2.7.13-slim
RUN apt-get update && apt-get install -y apt-utils curl
RUN echo 'deb http://nginx.org/packages/debian/ jessie nginx' > /etc/apt/sources.list.d/nginx.list
RUN apt-get update && apt-get install -y \
build-essential \
gcc \
git \
libcurl4-openssl-dev \
libldap-2.4-2 \
libldap2-dev \
libmysqlclient-dev \
libpq-dev \
libsasl2-dev \
nano \
nginx=1.8.* \
nodejs \
python-dev \
supervisor
ENV SERVER_DIR /data/applicationui/current/server
ADD src/application/server $SERVER_DIR
EXPOSE 14000 80
# version A: only start tornado, without nginx.
WORKDIR $SERVER_DIR/src
CMD ["npm","run","start:staging"]
Can anyone please help me here.
Pretty sure you can only run one command with ENTRYPOINT and with CMD.
From their docs:
There can only be one CMD instruction in a Dockerfile. If you list more than one CMD then only the last CMD will take effect.
Same thing with Entrypoint:
ENTRYPOINT has two forms:
ENTRYPOINT ["executable", "param1", "param2"] (exec form, preferred)
ENTRYPOINT command param1 param2 (shell form)
https://docs.docker.com/engine/reference/builder/#cmd
https://docs.docker.com/engine/reference/builder/#entrypoint
A work around that I do is the following
FROM ubuntu:16.04
WORKDIR /home/coins
RUN apt-get update
...
OTHER DOCKERFILE STUFF HERE
...
COPY ./entrypoint.sh /home/coins/
RUN chmod +x ./entrypoint.sh
ENTRYPOINT ./entrypoint.sh
entrypoint.sh:
#!/bin/bash
Can write whatever sh commands you need here..
exec sh ./some_script
EDIT:
One idea is you can add a test sh script and just trigger those 2 commands in it, and you'd be able to launch it with --entrypoint="test.sh"

Resources