I am fairly new to Docker and containerisation. I am wanting to decrease the size of my_proj docker container in production.
I prefer installing packages and managing dependencies via Poetry.
How can I specify using CPU-only PyTorch in a Dockerfile?
To do this via. bash terminal, it would be:
poetry add pytorch-cpu torchvision-cpu -c pytorch
(or conda install...)
My existing Dockerfile:
FROM python:3.7-slim as base
RUN apt-get update -y \
&& apt-get -y --no-install-recommends install curl wget\
&& rm -rf /var/lib/apt/lists/*
ENV ROOT /home/worker/python/my_proj
WORKDIR $ROOT
ARG ATLASSIAN_TOKEN
ARG POETRY_HTTP_BASIC_AZURE_PASSWORD
ARG ACCESS_KEY
ENV AWS_ACCESS_KEY_ID=$ACCESS_KEY
ARG SECRET_KEY
ENV AWS_SECRET_ACCESS_KEY=$SECRET_KEY
ARG REPO
ENV REPO_URL=$REPO
ENV PYPIRC_PATH=$ROOT/.pypirc
ENV \
PYTHONFAULTHANDLER=1 \
POETRY_VERSION=1.1.4 \
POETRY_HOME=/etc/poetry \
XDG_CACHE_HOME=/home/worker/.cache \
POETRY_VIRTUALENVS_IN_PROJECT=true \
MPLCONFIGDIR=/home/worker/matplotlib \
PATH=/home/worker/python/my_proj/.venv/bin:/usr/local/bin:/etc/poetry/bin:$PATH
ADD https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py ./
RUN python get-poetry.py && chmod +x /etc/poetry/bin/poetry
RUN --mount=type=cache,target=/root/.cache pip install twine keyring artifacts-keyring
RUN --mount=type=cache,target=/root/.cache apt update && apt install gcc -y
FROM base as ws
ARG WS_APIKEY
ARG WS_PROJECTVERSION=
ARG WS_PROJECTNAME=workers-python-my_proj
ARG WS_PRODUCTNAME=HALO
COPY --chown=worker:worker . .
RUN --mount=type=cache,uid=1000,target=/home/worker/.cache poetry install --no-dev
COPY --from=openjdk:15-slim-buster /usr/local/openjdk-15 /usr/local/openjdk-15
ENV JAVA_HOME /usr/local/openjdk-15
ENV PATH $JAVA_HOME/bin:$PATH
RUN --mount=type=cache,uid=1000,target=/home/worker/.cache ./wss_agent.sh
FROM base as test
COPY . .
RUN poetry config experimental.new-installer false
RUN poetry install
RUN cd my_proj && poetry run invoke deployconfluence_server_pass=$ATLASSIAN_TOKEN
FROM base as package
COPY . .
RUN poetry build
RUN python -m pip install --upgrade pip && \
pip install twine keyring artifacts-keyring && \
twine upload -r $REPO_URL --config-file $PYPIRC_PATH dist/* --skip-existing
FROM base as build
COPY . .
RUN poetry config experimental.new-installer false
RUN poetry install --no-dev
RUN pip3 --no-cache-dir install --upgrade awscli
RUN aws s3 cp s3://....tar.gz $ROOT/my_proj # censored url
RUN mkdir $ROOT/my_proj/bert-base-cased && cd $ROOT/my_proj/bert-base-cased && \
wget https://huggingface.co/bert-base-cased/resolve/main/config.json && \
wget https://huggingface.co/bert-base-cased/resolve/main/tokenizer.json && \
wget https://huggingface.co/bert-base-cased/resolve/main/tokenizer_config.json
FROM python:3.7-slim as production
ENV ROOT=/home/worker/python/my_proj \
VIRTUAL_ENV=/home/worker/python/my_proj/.venv\
PATH=/home/worker/python/my_proj/.venv/bin:/home/worker/python/my_proj:$PATH
COPY --from=build /home/worker/python/my_proj/pyproject.toml /home/worker/python/
COPY --from=build /home/worker/python/my_proj/.venv /home/worker/python/my_proj/.venv
COPY --from=build /home/worker/python/my_proj/my_proj /home/worker/python/my_proj
WORKDIR $ROOT
ENV PYTHONPATH=$ROOT:/home/worker/python/
ENTRYPOINT [ "primary_worker", "--mongo" ]
Installing it via pip should work:
RUN pip3 install torch==1.9.0+cpu torchvision==0.10.0+cpu torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
Related
this is my dockerfile:
FROM public.ecr.aws/lambda/python:3.8-arm64
COPY requirements.txt ./
RUN yum update -y && \
yum install -y gifsicle && \
pip install -r requirements.txt
COPY . .
CMD ["app.handler"]
I'm getting the following error:
#8 200.6 No package gifsicle available.
#8 200.7 Error: Nothing to do
I finally managed to do it by building the package from source. My Dockerfile:
FROM public.ecr.aws/lambda/python:3.8-arm64
RUN yum -y install install make gcc wget gzip
RUN wget https://www.lcdf.org/gifsicle/gifsicle-1.93.tar.gz
RUN tar -xzf gifsicle-1.93.tar.gz
RUN cd gifsicle-1.93 && \
./configure && \
make && \
make install
COPY requirements.txt ./
RUN yum update -y && \
pip install -r requirements.txt
COPY . .
CMD ["app.handler"]
This Python 3.9 project has a Dockerfile, that builds successfully. The file makes use of an ENTRYPOINT script to create some directories and handle some clean-up at run time. It is a bash script. The ENTRYPOINT script has no problem running until the very end, where it is expected to execute the CMD that is passed next. Well, I should say this behavior only happens when Kaniko builds the image. When the image is built locally, no such problem occurs. However, I am willing to chalk that up to the fact that locally is on a Windows machine. However, that shouldn't matter here because the error thrown is:
/opt/project/conf/entrypoint.sh: /usr/bin/supervisord: /usr/bin/python3: bad interpreter: No such file or directory
/opt/project/conf/entrypoint.sh: line 8: /usr/bin/supervisord: Success
Now I have looked at many "bad interpreter" questions. They all seem to revolve around the interpreter being in a custom place. I am reliant upon the default spot for the Python 3.9 interpreter. On Debian Bullseye (The OS behind the base image) that should be /usr/local/bin/python or /usr/local/bin/python3. So I am completely stumped as to why it is unable to find or use it.
Here are the implementation details:
Dockerfile:
FROM python:3.9-slim-bullseye
# Minimum Required Environment Variables
ENV SHELL=/bin/bash
ENV CC /usr/bin/gcc
ENV CXX /usr/bin/g++
ENV LANG=C.UTF-8
ENV DEBIAN_FRONTEND=noninteractive
ENV PYMSSQL_BUILD_WITH_BUNDLED_FREETDS=1
ENV PIP_CONFIG_FILE=/etc/pip.conf
ENV TZ=America/Los_Angeles
# Project Specific Environment Variables
ENV PROJECT_LOGFILE=/var/log/project/project.log
ENV PROJECT_CONFIG_DIRECTORY=/opt/project/conf
ENV PROJECT_SETTINGS_MODULE="project.settings"
# Files Needed for Dependency Installation
COPY dev/.pip.conf /etc/pip.conf
COPY dev/dev-requirements.txt /usr/local/requirements.txt
# Dependency Installation
WORKDIR /tmp
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get install musl-dev g++ bash curl gnupg -y \
&& curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \
&& curl https://packages.microsoft.com/config/debian/11/prod.list > /etc/apt/sources.list.d/mssql-release.list \
&& apt-get update \
&& apt-get install --no-install-recommends libfreetype-dev freetds-dev python-dev git libpng-dev libxml2-dev \
libxslt-dev libssl-dev libopenblas-dev rsyslog supervisor tini tzdata libghc-zlib-dev libjpeg-dev cron \
libgssapi-krb5-2 unixodbc-dev -y \
&& ACCEPT_EULA=Y apt-get install -y msodbcsql18 \
&& ln -s /usr/include/locale.h /usr/include/xlocale.h \
&& pip install --no-cache-dir --upgrade pip setuptools wheel \
&& pip install matplotlib --no-cache-dir \
&& pip install --no-cache-dir -r /usr/local/requirements.txt
# Setting Up For Install
COPY conf/ /opt/project/conf/
RUN mkdir -p /var/log/project /conf \
&& cp /opt/project/conf/supervisord.conf /conf/supervisord.conf \
&& cp /opt/project/conf/rsyslog.conf /conf/rsyslog.conf
WORKDIR /opt
# Copy Over Packages
COPY project-db-migrations /opt/project/project-db-migrations
COPY infrastructure /opt/infrastructure
COPY project /opt/project/src
COPY README.md /opt/project/README.md
# Install Infrastructure
RUN cd /opt/infrastructure && python3 setup.py install
# Install Project Service
RUN cd /opt/project/src && python3 setup.py install
RUN ["chmod", "+x", "/opt/project/conf/entrypoint.sh"]
WORKDIR /
EXPOSE 80
ENTRYPOINT ["tini", "--", "/opt/project/conf/entrypoint.sh"]
CMD ["supervisord", "-c", "/conf/supervisord.conf"]
entrypoint.sh
#!/bin/bash
set -eu
echo "Setting Up Project Service"
# Adding Temp Directory
mkdir -p /opt/project/tmp
echo "Service has been setup"
exec $#
supervisord.conf
[supervisord]
nodaemon=true
logfile=/var/log/project/supervisord.log
childlogdir=/var/log/project
[program:rsyslogd]
command=/usr/sbin/rsyslogd -n -f /conf/rsyslog.conf
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:crond]
command=/usr/sbin/cron -f -l 15
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:project]
command=python -m project.run --server
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
The image is ran and deployed without changes to the user, so it should be running as root.
In this case, after some digging I found there is an issue with the Kaniko version DevOps had running. That was causing the issue. Because the image wasn't being flattened correctly, Python could not start properly.
I have a Dockerfile and it's working fine in Ubuntu VM. However, the same Dockerfile does not build in Linux Server.
Dockerfile:
FROM python:3.9.7-slim as builder-image
ARG DEBIAN_FRONTEND=noninteractive
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONFAULTHANDLER 1
RUN apt-get update && apt-get install -y --no-install-recommends python3-dev gcc libc-dev musl-dev libffi-dev g++ cargo && \
apt-get clean && rm -rf /var/lib/apt/lists/*
RUN python3.9 -m venv /home/myuser/venv
ENV PATH="/home/myuser/venv/bin:$PATH"
RUN /home/myuser/venv/bin/pip install --upgrade pip
WORKDIR /home/myuser/venv
COPY /data/requirements.txt requirements.txt
RUN pip3 install --no-cache-dir wheel
RUN pip3 install --no-cache-dir -r requirements.txt
FROM python:3.9.7-slim
RUN useradd --create-home myuser
COPY --from=builder-image /home/myuser/venv /home/myuser/venv
USER myuser
RUN mkdir /home/myuser/code
WORKDIR /home/myuser/code
ENV PYTHONUNBUFFERED=1
ENV VIRTUAL_ENV=/home/myuser/venv
ENV PATH="/home/myuser/venv/bin:$PATH"
ENTRYPOINT ["/bin/bash"]
docker build -t python-docker_14122021 .
Error:
Sending build context to Docker daemon 49.66 kB
Step 1/23 : FROM python:3.9-slim-buster as builder-image
Error parsing reference: "python:3.9-slim-buster as builder-image" is not a valid repository/tag: invalid reference format
You have a very old docker on the server. You need to have at least version 17.06 of docker to support multi-staging builds.
I have a python 3.6 app that I would like to install on a docker container. I installed the app in a virtual env (miniconda3) on my pc (windows 10) and need to do the same in a docker container (python3.6 base image (linux)). I installed the app locally using a setup.py file using python setup.py install, and an egg file as well as a separate directory with the application's code were created in site-packages. When I tried to do the same in the docker container, the installation creates only an egg file in site packages, and the app can't be imported. The other installed packages are fine.
Dockerfile:
FROM python:3.6
WORKDIR /opt
# create a virtual environment and add it to PATH so that it is
applied for all future RUN and CMD calls
ENV VIRTUAL_ENV=/opt/venv
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
# Install Mono for pythonnet.
RUN apt-get update \
&& apt-get install --yes \
apt-transport-https \
git \
dirmngr \
clang \
gnupg \
ca-certificates \
# Dependency for pyodbc.
unixodbc-dev \
&& apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-
keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF \
&& echo "deb http://download.mono-project.com/repo/debian
stretch/snapshots/5.20 main" | tee /etc/apt/sources.list.d/mono-
official-stable.list \
&& apt-get update \
&& apt-get install --yes \
&& rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
COPY src ./src
COPY setup.py ./setup.py
COPY config.json ./config.json
COPY BCUtility.dll ./BCUtility.dll
COPY settings.ini ./settings.ini
COPY redis_config.json ./redis_config.json
COPY sql_config.json ./sql_config.json
RUN python3 -m venv $VIRTUAL_ENV \
# From here on, use virtual env's python.
&& venv/bin/pip install --upgrade pip \
&& venv/bin/pip install --no-cache-dir --upgrade pip setuptools wheel \
&& venv/bin/pip install --no-cache-dir -r requirements.txt \
# Dependency for pythonnet.
&& venv/bin/pip install --no-cache-dir pycparser \
&& venv/bin/pip install -U --no-cache-dir "pythonnet==2.5.1" \
# install the app
&& venv/bin/python setup.py install
cmd /opt/venv/bin/python src/app.py
Any idea where I'm going wrong?
I want to create a docker image with specifically python 3.5 on a specific base image which is the nvidia/cuda (9.0-base image) the latter has no python environment.
The reason I need specific versions is to support running cuda10.0 python3.5 and a gcc version<7 to compile the driver all together on the same box
When I try and build the docker environments (see below) I always end up with the system update files which load python3.6
The first version I run (below) runs a system update dependencies which installs python 3.6 I have tried many variants to avoid this but always end up 3.6 in the final image.
Any suggestions for getting this running with python3.5 are welcome
Thanks
FROM nvidia/cuda
RUN apt-get update && apt-get install -y libsm6 libxext6 libxrender-dev python3.5 python3-pip
COPY . /app
WORKDIR /app
RUN pip3 install -r requirements.txt
ENTRYPOINT [ "python3" ]
CMD [ "app.py" ]
Another variant (below) I have tried is with virtualenv and here again I can't seem to force a python 3.5 environment
FROM nvidia/cuda
RUN apt-get update && apt-get install -y --no-install-recommends libsm6 libxext6 libxrender-dev python3.5 python3-pip python3-virtualenv
ENV VIRTUAL_ENV=/opt/venv
RUN python3 -m virtualenv --python=/usr/bin/python3 $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
COPY . /app
WORKDIR /app
RUN pip3 install -r requirements.txt
ENTRYPOINT [ "python3" ]
CMD [ "app.py" ]
You can try using conda. I used several stages to minimize final container and to speedup/cache local builds.
# first stage
FROM nvidia/cuda:11.1-base-ubuntu18.04 as builder
RUN apt-get update && apt-get install -y curl wget gcc build-essential
# install conda
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-4.5.12-Linux-x86_64.sh -O ~/miniconda.sh && \
/bin/bash ~/miniconda.sh -b -p /opt/conda
# create env with python 3.5
RUN /opt/conda/bin/conda create -y -n myenv python=3.5
# install requirements
WORKDIR /app
COPY requirements.txt /app
ENV PATH=/opt/conda/envs/myenv/bin:$PATH
RUN pip install -r requirements.txt
RUN pip uninstall -y pip
####################
# second stage (note: FROM container must be the same as builder)
FROM nvidia/cuda:11.1-base-ubuntu18.04 as runner
# copy environment data including python
COPY --from=builder /opt/conda/envs/myenv/bin /opt/conda/envs/myenv/bin
COPY --from=builder /opt/conda/envs/myenv/lib /opt/conda/envs/myenv/lib
# do some env settings
ENV PATH=/opt/conda/envs/myenv/bin:$PATH
ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8
####################
# final image
from runner
WORKDIR /app
COPY ./run.py /app
CMD [ "python", "run.py"]
You can install from PPA and use it as usual:
FROM nvidia/cuda
RUN apt-get update && apt-get install -y --no-install-recommends software-properties-common \
libsm6 libxext6 libxrender-dev curl \
&& rm -rf /var/lib/apt/lists/*
RUN echo "**** Installing Python ****" && \
add-apt-repository ppa:deadsnakes/ppa && \
apt-get install -y build-essential python3.5 python3.5-dev python3-pip && \
curl -O https://bootstrap.pypa.io/get-pip.py && \
python3.5 get-pip.py && \
rm -rf /var/lib/apt/lists/*
COPY requirements.txt requirements.txt
RUN pip3.5 install -r requirements.txt
CMD ["python3.5", "app.py"]