Optimize Dockerfile - node.js

Our frontends have such Dockerfile:
FROM node:13.12.0
RUN apt-get update && apt-get install -y --no-install-recommends \
apt-utils \
git \
xvfb \
libgtk-3-0 \
libxtst6 \
libgconf-2-4 \
libgtk2.0-0 \
libnotify-dev \
libnss3 \
libxss1 \
libasound2 \
tzdata && \
rm -rf /var/lib/apt/lists/* && \
ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && \
echo $TZ > /etc/timezone
COPY ./docker-entrypoint.sh /docker-entrypoint.sh
ENTRYPOINT ["/docker-entrypoint.sh"]
WORKDIR /code
COPY ./ /code
RUN npm set registry <registry-url> && \
npm cache clean --force && npm install && npm run bootstrap
As far as I can see it is not optimized because code copying happens before dependencies installation, right? And a better way would be to copy package.json and install dependencies first and then code copying? Something like this:
FROM node:13.12.0
RUN apt-get update && apt-get install -y --no-install-recommends \
apt-utils \
git \
xvfb \
libgtk-3-0 \
libxtst6 \
libgconf-2-4 \
libgtk2.0-0 \
libnotify-dev \
libnss3 \
libxss1 \
libasound2 \
tzdata && \
rm -rf /var/lib/apt/lists/* && \
ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && \
echo $TZ > /etc/timezone
COPY ./docker-entrypoint.sh /docker-entrypoint.sh
WORKDIR /code
COPY package*.json ./
RUN npm set registry <registry-url> && \
npm cache clean --force && npm install && npm run bootstrap
COPY ./ /code
ENTRYPOINT ["/docker-entrypoint.sh"]

I think one of the most important thing related to Dockerfile optimization is to put the elements that might be changing in future versions of your container, the latest cause in case of any change on the code part, being the latest will not force recreation on other layers.
I think that's the reason for the Dockerfile to look like it does in your first example
There are other considerations regarding Dockerfile optimization that you can read for example here:
https://linuxhint.com/optimizing-docker-images/

The hadolint/hadolint Dockerfile linter is a good starting point. Linting your Dockerfile using the Haskell Dockerfile Linter i.e. docker run --rm -i hadolint/hadolint < Dockerfile:
/dev/stdin:5 SC2086 info: Double quote to prevent globbing and word splitting.
/dev/stdin:5 DL3008 warning: Pin versions in apt get install. Instead of `apt-get install <package>` use `apt-get install <package>=<version>`
... after fixing the issues and sorting the packages alphanumerically following the best practices with a couple of minor modifications your Dockerfile might look like:
FROM node:13.12.0
ARG NPM_REGISTRY
RUN apt-get update && \
apt-get install -y --no-install-recommends \
apt-utils=1.4.11 \
git=1:2.11.0-3+deb9u7 \
libasound2=1.1.3-5 \
libgconf-2-4=3.2.6-4+b1 \
libgtk2.0-0=2.24.31-2 \
libgtk-3-0=3.22.11-1 \
libnotify-dev=0.7.7-2 \
libnss3=2:3.26.2-1.1+deb9u2 \
libxss1=1:1.2.2-1 \
libxtst6=2:1.2.3-1 \
tzdata=2021a-0+deb9u1 \
xvfb=2:1.19.2-1+deb9u7 && \
rm -rf /var/lib/apt/lists/* && \
ln -snf "/usr/share/zoneinfo/$TZ" /etc/localtime && \
echo "$TZ" > /etc/timezone
COPY docker-entrypoint.sh /docker-entrypoint.sh
WORKDIR /code
COPY package*.json ./
RUN npm set registry "${NPM_REGISTRY}" && \
npm cache clean --force && \
npm install && \
npm run bootstrap
COPY . .
ENTRYPOINT ["/docker-entrypoint.sh"]
Note: the minor changes are by preference, i.e. COPY . . to COPY from the context into the /code directory which is set by the WORKDIR instruction.
Build the image passing the NPM_REGISTRY as a build arg i.e.: docker build --rm --build-arg NPM_REGISTRY=https://yarn.npmjs.org -t so:66493910 .

Related

Fixing security vulnerabilities in docker image

I have the following docker file
FROM debian:stable
# Avoid warnings by switching to noninteractive
ENV DEBIAN_FRONTEND=noninteractive
#Versions
ENV HELM_VERSION=v3.10.0
ENV KUBECTL_VERSION=v1.20.9
ENV MAVEN_OPTS="-Djavax.net.ssl.trustStore=/cicd/assets/truststore.jks"
ENV TERRAFORM_VERSION=1.2.0
ENV GOLANG_VERSION=1.19.1
ENV TERRAGRUNT_VERSION=v0.38.7
RUN set -xe \
&& apt-get update -y \
&& apt-get install -y python3-pip
RUN apt-get install zip unzip
#Copy python requirements file
COPY requirements.txt /tmp/pip-tmp/
# Makes the Ansible directories
RUN mkdir /etc/ansible /ansible
RUN mkdir ~/.ssh
# Configure apt and install python packages
RUN apt-get update -y -q \
&& apt-get upgrade -y -q \
&& apt-get install -y wget \
&& apt-get -y install --no-install-recommends apt-utils dialog 2>&1 \
&& apt-get install -y --no-install-recommends apt-utils \
&& apt-get -y install ca-certificates software-properties-common build-essential curl git gettext-base maven sshpass krb5-user \
&& pip --disable-pip-version-check --no-cache-dir install -r /tmp/pip-tmp/requirements.txt \
&& apt-get -y install jq \
&& rm -rf /tmp/pip-tmp
#Install helm
RUN wget https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz \
&& tar -zxvf helm-${HELM_VERSION}-linux-amd64.tar.gz \
&& mv linux-amd64/helm /usr/local/bin/helm
#Install kubectl
RUN curl --silent https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl --output /usr/local/bin/kubectl \
&& chmod +x /usr/local/bin/kubectl
#Install Docker CLI
RUN curl -sSL https://get.docker.com/ | sh \
&& curl -L "https://github.com/docker/compose/releases/download/2.10.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose \
&& chmod +x /usr/local/bin/docker-compose
#Install AWS CLI
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
&& unzip awscliv2.zip \
&& ./aws/install
#Copy Assets
#RUN mkdir -p /cicd
#COPY assets /cicd
#Install helm plugins
#RUN helm plugin add https://github.com/databus23/helm-diff
#RUN helm plugin install /cicd/helm-nexus-push
# Downloading gcloud package
RUN curl https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz > /tmp/google-cloud-sdk.tar.gz
# Installing the package
RUN mkdir -p /usr/local/gcloud \
&& tar -C /usr/local/gcloud -xvf /tmp/google-cloud-sdk.tar.gz \
&& /usr/local/gcloud/google-cloud-sdk/install.sh
# Adding the package path to local
ENV PATH $PATH:/usr/local/gcloud/google-cloud-sdk/bin
RUN cd /tmp && \
wget https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip && \
unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip -d /usr/local/bin && \
rm -rf /tmp/*
RUN cd /tmp && \
wget https://dl.google.com/go/go${GOLANG_VERSION}.linux-amd64.tar.gz && \
tar -xzf go${GOLANG_VERSION}.linux-amd64.tar.gz -C /usr/local && \
rm -rf /tmp/*
RUN cd /tmp && \
wget https://github.com/gruntwork-io/terragrunt/releases/download/${TERRAGRUNT_VERSION}/terragrunt_linux_amd64 && \
mv terragrunt_linux_amd64 /usr/local/bin/terragrunt && \
chmod +x /usr/local/bin/terragrunt && \
rm -rf /tmp/*
RUN git config --global http.sslCAinfo /etc/ssl/certs/ca-certificates.crt
ENV GOPATH=/usr/local/go
ENV PATH=/usr/local/go/bin:$PATH
ENV CGO_ENABLED=0
RUN go version
RUN terraform --version
RUN terragrunt --version
RUN ansible --version
CMD bash
I build the docker image and upload it to google artifact registry, but I always come across security vulnerabilities I have tried to fix it but unfortunately I'm unable to fix the security vulnerabilities. Please look at the critical errors and let me know how I can fix this, Any recommendation is appreciated. Thank You.
It looks like the DockerFile is trying to a specific version of golang by hand into "/usr/local" rather than using the Debian package manager. According to the info at https://security-tracker.debian.org/tracker/CVE-2021-38297, that bug is fixed in 1.17.3-3 and the Dockerfile are using 1.19.1. So perhaps there is an old golang installation in the base image ... and that is what the scanner is picking up. Check that, and if necessary apt install a newer version.
Likewise, https://security-tracker.debian.org/tracker/CVE-2022-23806 should be fixed by a newer version of golang. See the CVE link for versions.
https://security-tracker.debian.org/tracker/CVE-2015-20107 could be fixed by upgrading to Python 3.10.6-1 or later.
https://security-tracker.debian.org/tracker/CVE-2019-19814 doesn't appear to have a fix from upstream, so there is nothing you can do about it except not use f2fs.
https://security-tracker.debian.org/tracker/CVE-2022-29599 can be fixed by updating the maven-shared-utils package; see the CVE link for versions.
https://security-tracker.debian.org/tracker/CVE-2022-1996 has a fix upstream but it is awaiting triage by the Debian team.
In summary, some of the vulnerabilities can be fixed, but for a couple of them no fix is readily available. So:
Apply the fixes that are available.
Then read the CVEs and accompanying explanations and 1) make a judgement whether they represent a risk that you can take, and 2) figure out if you can mitigate the risk; e.g. by locking down access to the running Docker container.

Setting Up Stand-alone LanguageTool in Docker container

Im trying to setup LanguageTool as a standalone server as a Docker Container. So what I did is download the standalone system provided at -> https://languagetool.org/download/LanguageTool-stable.zip and put it in my project. I setup the docker-compose.yml file like so
version: '3'
services:
grammar:
build: ./services/grammar
image: dev/grammar:1
restart: always
container_name: dev.grammar
ports:
- "8130:8130"
And I created the dockerfile inside the LanguageTool folder like so
FROM ubuntu:18.04
WORKDIR /tmp
RUN apt-get update
RUN apt-get install unzip
ADD https://languagetool.org/download/LanguageTool-stable.zip /tmp/LanguageTool-stable.zip
#RUN apt-get install -y unzip
RUN unzip /tmp/LanguageTool-stable.zip
RUN mv /tmp/LanguageTool-5.7 /usr/languagetool
CMD ["java", "-jar", "languagetool-server.jar", "--port", "8130", "--public", "--allow-origin", "'*'" ]
EXPOSE 8130
I have actually tried many iterations of the dockerfile like another example here
FROM debian:stretch
RUN set -ex \
&& mkdir -p /uploads /etc/apt/sources.list.d /var/cache/apt/archives/ \
&& export DEBIAN_FRONTEND=noninteractive \
&& apt-get clean \
&& apt-get update -y \
&& apt-get install -y \
bash \
curl \
openjdk-8-jre-headless \
unzip \
libhunspell-1.4-0 \
hunspell-de-at
ENV VERSION 5.7
COPY LanguageTool-$VERSION.zip /LanguageTool-$VERSION.zip
RUN unzip LanguageTool-$VERSION.zip \
&& rm LanguageTool-$VERSION.zip
WORKDIR /LanguageTool-$VERSION
CMD ["java", "-cp", "languagetool-server.jar", "org.languagetool.server.HTTPServer", "--port", "8130", "--public", "--allow-origin", "'*'" ]
EXPOSE 8130
But none of them seems to work. Please let me know what I am doing wrong here. Thanks in advance !!
Edit: Here is what my file/folder structure looks like here
I found the solution. Had to tinker around some configurations but I finally got it working. here is the dockerconfig file that worked for me.
FROM debian:stretch
RUN set -ex \
&& mkdir -p /uploads /etc/apt/sources.list.d /var/cache/apt/archives/ \
&& export DEBIAN_FRONTEND=noninteractive \
&& apt-get clean \
&& apt-get update -y \
&& apt-get install -y \
bash \
curl \
openjdk-8-jre-headless \
unzip \
libhunspell-1.4-0 \
hunspell-de-at
ENV VERSION 5.1
COPY LanguageTool-$VERSION.zip /LanguageTool-$VERSION.zip
RUN unzip LanguageTool-$VERSION.zip \
&& rm LanguageTool-$VERSION.zip
WORKDIR /LanguageTool-$VERSION
CMD ["java", "-cp", "languagetool-server.jar", "org.languagetool.server.HTTPServer", "--port", "8130", "--public", "--allow-origin", "'*'" ]
EXPOSE 8130

OpenCV in AWS Lambda container image

I am trying to build a docker image which will be deployed as a function on AWS Lambda. Able to build and test the image successfully but facing an issue when I try to import OpenCV in the function.
I do not face this issue when I remove the import statement from app.py
The error I am facing -
{"errorMessage": "Unable to import module 'app': libGL.so.1: cannot open shared object file: No such file or directory", "errorType": "Runtime.ImportModuleError"}
My Dockerfile -
# Define custom function directory
ARG FUNCTION_DIR="/function"
FROM python:3.9 as build-image
# Include global arg in this stage of the build
ARG FUNCTION_DIR
# Install aws-lambda-cpp build dependencies
RUN apt-get update && \
apt-get install -y \
g++ \
make \
cmake \
unzip \
libcurl4-openssl-dev
RUN apt-get install -y --fix-missing \
build-essential \
cmake \
gfortran \
git \
wget \
curl \
graphicsmagick \
libgraphicsmagick1-dev \
libatlas-base-dev \
libavcodec-dev \
libavformat-dev \
libgtk2.0-dev \
libjpeg-dev \
liblapack-dev \
libswscale-dev \
pkg-config \
python3-dev \
python3-numpy \
software-properties-common \
zip \
&& apt-get clean && rm -rf /tmp/* /var/tmp/*
# Copy function code
RUN mkdir -p ${FUNCTION_DIR}
COPY app/* ${FUNCTION_DIR}
WORKDIR ${FUNCTION_DIR}
RUN pip install -r requirements.txt --target ${FUNCTION_DIR}
# Install the function's dependencies
RUN pip install \
--target ${FUNCTION_DIR} \
awslambdaric
FROM python:3.9
# Include global arg in this stage of the build
ARG FUNCTION_DIR
# Set working directory to function root directory
WORKDIR ${FUNCTION_DIR}
# Copy in the built dependencies
COPY --from=build-image ${FUNCTION_DIR} ${FUNCTION_DIR}
ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ]
CMD [ "app.handler" ]
My requirements.txt -
mediapipe<=0.8.3.1
numpy<=1.19.4
opencv-python<=4.4.0.46
boto3<=1.17.64
My app.py
import cv2
def handler(event, context):
return cv2.__version__
I came across the same issue getting the same error when trying to use a mediapipe container. pip installing python-opencv-headless solved the issue for me, I did not need to install any additional dependencies.
FROM public.ecr.aws/lambda/python:3.8
# Copy function code
COPY app.py ${LAMBDA_TASK_ROOT}
# Install the function's dependencies using file requirements.txt
# from your project folder.
COPY requirements.txt .
RUN pip3 install mediapipe opencv-python-headless --target "${LAMBDA_TASK_ROOT}"
# Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)
CMD [ "app.handler" ]
Managed to solve it by adding these to the Dockerfile -
RUN apt-get install ffmpeg libsm6 libxext6 -y
New Dockerfile looks like this -
# Define custom function directory
ARG FUNCTION_DIR="/function"
FROM python:3.9 as build-image
# Include global arg in this stage of the build
ARG FUNCTION_DIR
# Install aws-lambda-cpp build dependencies
RUN apt-get update && \
apt-get install -y \
g++ \
make \
cmake \
unzip \
libcurl4-openssl-dev
RUN apt-get install -y --fix-missing \
build-essential \
cmake \
gfortran \
git \
wget \
curl \
ffmpeg \
libsm6 \
libxext6 \
graphicsmagick \
libgraphicsmagick1-dev \
libatlas-base-dev \
libavcodec-dev \
libavformat-dev \
libgtk2.0-dev \
libjpeg-dev \
liblapack-dev \
libswscale-dev \
pkg-config \
python3-dev \
python3-numpy \
software-properties-common \
zip \
&& apt-get clean && rm -rf /tmp/* /var/tmp/*
# Copy function code
RUN mkdir -p ${FUNCTION_DIR}
COPY app/* ${FUNCTION_DIR}
WORKDIR ${FUNCTION_DIR}
RUN pip install -r requirements.txt --target ${FUNCTION_DIR}
# Install the function's dependencies
RUN pip install \
--target ${FUNCTION_DIR} \
awslambdaric
FROM python:3.9
# Include global arg in this stage of the build
ARG FUNCTION_DIR
# Set working directory to function root directory
WORKDIR ${FUNCTION_DIR}
# Copy in the built dependencies
COPY --from=build-image ${FUNCTION_DIR} ${FUNCTION_DIR}
ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ]
CMD [ "app.handler" ]

Heroku Exec on Docker container not installing

Im trying to install Heroku Exec on a Docker container so that I can run scripts through a shell. I cannot access a shell on the container. Ive followed the below linked instructions on using heroku ps:exec with Docker, but get this error when running heroku:ps exec . You can see Ive installed the packages in my dockerfile, and I created a /.profile.d folder in my base /app directory, and added the heroku-exec.sh file and pasted the line of code they wanted. seems like Heroku Exec is not installed or working though.
establishing credentials... error
Heroku Exec is not running!
▸ Could not connect to dyno!
▸ Check if the dyno is running with heroku ps
Heroku Exec with Docker:
https://devcenter.heroku.com/articles/exec#using-with-docker
Dockerfile
FROM node:latest
# update and add all the steps for running with xvfb
RUN apt-get update &&\
apt-get install -yq gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 \
libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 \
libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 \
libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 \
ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget \
xvfb x11vnc x11-xkb-utils xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic x11-apps \
&& apt-get install -y wget gnupg \
&& wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
&& apt-get install -y google-chrome-stable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf libxss1 \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
RUN apt-get install bash
RUN apt-get install python
RUN apt-get install curl
RUN apt-get install openssh-client
#remove zombie images
ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.2/dumb-init_1.2.2_x86_64 /usr/local/bin/dumb-init
RUN chmod +x /usr/local/bin/dumb-init
ENTRYPOINT ["dumb-init", "--"]
# WORKDIR /app
# using /tmp in order to read/write to container fs as non-root user
WORKDIR /tmp
# add the required dependencies - this breaks the build process for some reason, manually npm install below works
# COPY node_modules /app/node_modules
RUN npm install puppeteer \
# Add user so we don't need --no-sandbox.
# same layer as npm install to keep re-chowned files from using up several hundred MBs more space
&& groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
&& mkdir -p /home/pptruser/Downloads \
&& chown -R pptruser:pptruser /home/pptruser \
&& npm install express \
&& npm install basic-ftp
# && chown -R pptruser:pptruser /node_modules
# For Heroku exec
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
ADD ./.profile.d /app/.profile.d
EXPOSE 3000
# Run everything after as non-privileged user.
USER pptruser
# USER root
# Finally copy the build application
COPY . .
# make sure we can run without a UI
ENV DISPLAY :99
CMD Xvfb :99 -screen 0 1024x768x16 -nolisten unix & node server.js

Not able to serve jupyter notebooks in binder

Binder project looks promising.
It helps in executing notebooks in a github repository by building an executable container.
I am trying to build an executable container in binder with the following Dockerfile that has Perl 6 and Python 3 kernels:
FROM sumdoc/perl-6
ENV NB_USER jovyan
ENV NB_UID 1000
ENV HOME /home/${NB_USER}
RUN adduser --disabled-password \
--gecos "Default user" \
--uid ${NB_UID} \
${NB_USER}
RUN apt-get update \
&& apt-get install -y build-essential \
git wget libzmq3-dev ca-certificates python3-pip \
&& rm -rf /var/lib/apt/lists/* && pip3 install jupyter notebook --no-cache-dir \
&& zef -v install https://github.com/bduggan/p6-jupyter-kernel.git --force-test \
&& jupyter-kernel.p6 --generate-config
ENV TINI_VERSION v0.16.1
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /usr/bin/tini
RUN chmod +x /usr/bin/tini
ENTRYPOINT ["/usr/bin/tini", "--"]
COPY . ${HOME}
USER root
RUN chown -R ${NB_UID} ${HOME}
USER ${NB_USER}
EXPOSE 8888
CMD ["jupyter", "notebook", "--port=8888", "--no-browser", "--ip=0.0.0.0", "--allow-root"]
Binder launches this window after building a container:
While trying to run Perl 6 or Python 3 notebook I get this error:
I read this documentation on binder but could not succeed.
What things I am missing? Any help with explanations would be appreciated.
After going through this Dockerfile, I solved the issue.
I even wrote a blog on using Perl 6 notebook in Binder here.
What I was missing was to add WORKDIR $HOME after USER ${NB_USER}in my Dockerfile as follows:
FROM sumankhanal/perl-6
ENV NB_USER jovyan
ENV NB_UID 1000
ENV HOME /home/${NB_USER}
RUN adduser --disabled-password \
--gecos "Default user" \
--uid ${NB_UID} \
${NB_USER}
RUN apt-get update \
&& apt-get install -y build-essential \
git wget libzmq3-dev ca-certificates python3-pip \
&& rm -rf /var/lib/apt/lists/* && pip3 install jupyter notebook --no-cache-dir \
&& zef -v install https://github.com/bduggan/p6-jupyter-kernel.git --force-test \
&& jupyter-kernel.p6 --generate-config
ENV TINI_VERSION v0.16.1
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /usr/bin/tini
RUN chmod +x /usr/bin/tini
ENTRYPOINT ["/usr/bin/tini", "--"]
COPY . ${HOME}
USER root
RUN chown -R ${NB_UID} ${HOME}
USER ${NB_USER}
WORKDIR ${HOME}
EXPOSE 8888
CMD ["jupyter", "notebook", "--port=8888", "--no-browser", "--ip=0.0.0.0", "--allow-root"]

Resources