I have the following docker image
FROM ubuntu
RUN apt-get update \
&& apt-get install -y python3 \
&& apt-get install -y python3-pip \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* \
&& pip3 install boto3
ENV INSTALL_PATH /docker-flowcell-restore
RUN mkdir -p $INSTALL_PATH
WORKDIR $INSTALL_PATH
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
COPY /src/* $INSTALL_PATH/src/
ENTRYPOINT python3 src/main.py
In my python script that the ENTRYPOINT points too I have some parameters I would like to pass in. I used argparse in my python script to construct them. Example would be --key as an arg option. This --key argument will change on each run of the script. How do I pass this argument into my script so that it executes with the correct parameters?
I have tried
docker run my_image_name --key 100
but the argument is not getting to python script.
You can use CMD command to pass parameters (and set defaults ones for an entrypoint), for example:
CMD [ "python", "manage.py", "runserver", "0.0.0.0:8000" ]
Take a look here for details.
Related
This Python 3.9 project has a Dockerfile, that builds successfully. The file makes use of an ENTRYPOINT script to create some directories and handle some clean-up at run time. It is a bash script. The ENTRYPOINT script has no problem running until the very end, where it is expected to execute the CMD that is passed next. Well, I should say this behavior only happens when Kaniko builds the image. When the image is built locally, no such problem occurs. However, I am willing to chalk that up to the fact that locally is on a Windows machine. However, that shouldn't matter here because the error thrown is:
/opt/project/conf/entrypoint.sh: /usr/bin/supervisord: /usr/bin/python3: bad interpreter: No such file or directory
/opt/project/conf/entrypoint.sh: line 8: /usr/bin/supervisord: Success
Now I have looked at many "bad interpreter" questions. They all seem to revolve around the interpreter being in a custom place. I am reliant upon the default spot for the Python 3.9 interpreter. On Debian Bullseye (The OS behind the base image) that should be /usr/local/bin/python or /usr/local/bin/python3. So I am completely stumped as to why it is unable to find or use it.
Here are the implementation details:
Dockerfile:
FROM python:3.9-slim-bullseye
# Minimum Required Environment Variables
ENV SHELL=/bin/bash
ENV CC /usr/bin/gcc
ENV CXX /usr/bin/g++
ENV LANG=C.UTF-8
ENV DEBIAN_FRONTEND=noninteractive
ENV PYMSSQL_BUILD_WITH_BUNDLED_FREETDS=1
ENV PIP_CONFIG_FILE=/etc/pip.conf
ENV TZ=America/Los_Angeles
# Project Specific Environment Variables
ENV PROJECT_LOGFILE=/var/log/project/project.log
ENV PROJECT_CONFIG_DIRECTORY=/opt/project/conf
ENV PROJECT_SETTINGS_MODULE="project.settings"
# Files Needed for Dependency Installation
COPY dev/.pip.conf /etc/pip.conf
COPY dev/dev-requirements.txt /usr/local/requirements.txt
# Dependency Installation
WORKDIR /tmp
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get install musl-dev g++ bash curl gnupg -y \
&& curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \
&& curl https://packages.microsoft.com/config/debian/11/prod.list > /etc/apt/sources.list.d/mssql-release.list \
&& apt-get update \
&& apt-get install --no-install-recommends libfreetype-dev freetds-dev python-dev git libpng-dev libxml2-dev \
libxslt-dev libssl-dev libopenblas-dev rsyslog supervisor tini tzdata libghc-zlib-dev libjpeg-dev cron \
libgssapi-krb5-2 unixodbc-dev -y \
&& ACCEPT_EULA=Y apt-get install -y msodbcsql18 \
&& ln -s /usr/include/locale.h /usr/include/xlocale.h \
&& pip install --no-cache-dir --upgrade pip setuptools wheel \
&& pip install matplotlib --no-cache-dir \
&& pip install --no-cache-dir -r /usr/local/requirements.txt
# Setting Up For Install
COPY conf/ /opt/project/conf/
RUN mkdir -p /var/log/project /conf \
&& cp /opt/project/conf/supervisord.conf /conf/supervisord.conf \
&& cp /opt/project/conf/rsyslog.conf /conf/rsyslog.conf
WORKDIR /opt
# Copy Over Packages
COPY project-db-migrations /opt/project/project-db-migrations
COPY infrastructure /opt/infrastructure
COPY project /opt/project/src
COPY README.md /opt/project/README.md
# Install Infrastructure
RUN cd /opt/infrastructure && python3 setup.py install
# Install Project Service
RUN cd /opt/project/src && python3 setup.py install
RUN ["chmod", "+x", "/opt/project/conf/entrypoint.sh"]
WORKDIR /
EXPOSE 80
ENTRYPOINT ["tini", "--", "/opt/project/conf/entrypoint.sh"]
CMD ["supervisord", "-c", "/conf/supervisord.conf"]
entrypoint.sh
#!/bin/bash
set -eu
echo "Setting Up Project Service"
# Adding Temp Directory
mkdir -p /opt/project/tmp
echo "Service has been setup"
exec $#
supervisord.conf
[supervisord]
nodaemon=true
logfile=/var/log/project/supervisord.log
childlogdir=/var/log/project
[program:rsyslogd]
command=/usr/sbin/rsyslogd -n -f /conf/rsyslog.conf
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:crond]
command=/usr/sbin/cron -f -l 15
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:project]
command=python -m project.run --server
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
The image is ran and deployed without changes to the user, so it should be running as root.
In this case, after some digging I found there is an issue with the Kaniko version DevOps had running. That was causing the issue. Because the image wasn't being flattened correctly, Python could not start properly.
I am fairly new to Docker and containerisation. I am wanting to decrease the size of my_proj docker container in production.
I prefer installing packages and managing dependencies via Poetry.
How can I specify using CPU-only PyTorch in a Dockerfile?
To do this via. bash terminal, it would be:
poetry add pytorch-cpu torchvision-cpu -c pytorch
(or conda install...)
My existing Dockerfile:
FROM python:3.7-slim as base
RUN apt-get update -y \
&& apt-get -y --no-install-recommends install curl wget\
&& rm -rf /var/lib/apt/lists/*
ENV ROOT /home/worker/python/my_proj
WORKDIR $ROOT
ARG ATLASSIAN_TOKEN
ARG POETRY_HTTP_BASIC_AZURE_PASSWORD
ARG ACCESS_KEY
ENV AWS_ACCESS_KEY_ID=$ACCESS_KEY
ARG SECRET_KEY
ENV AWS_SECRET_ACCESS_KEY=$SECRET_KEY
ARG REPO
ENV REPO_URL=$REPO
ENV PYPIRC_PATH=$ROOT/.pypirc
ENV \
PYTHONFAULTHANDLER=1 \
POETRY_VERSION=1.1.4 \
POETRY_HOME=/etc/poetry \
XDG_CACHE_HOME=/home/worker/.cache \
POETRY_VIRTUALENVS_IN_PROJECT=true \
MPLCONFIGDIR=/home/worker/matplotlib \
PATH=/home/worker/python/my_proj/.venv/bin:/usr/local/bin:/etc/poetry/bin:$PATH
ADD https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py ./
RUN python get-poetry.py && chmod +x /etc/poetry/bin/poetry
RUN --mount=type=cache,target=/root/.cache pip install twine keyring artifacts-keyring
RUN --mount=type=cache,target=/root/.cache apt update && apt install gcc -y
FROM base as ws
ARG WS_APIKEY
ARG WS_PROJECTVERSION=
ARG WS_PROJECTNAME=workers-python-my_proj
ARG WS_PRODUCTNAME=HALO
COPY --chown=worker:worker . .
RUN --mount=type=cache,uid=1000,target=/home/worker/.cache poetry install --no-dev
COPY --from=openjdk:15-slim-buster /usr/local/openjdk-15 /usr/local/openjdk-15
ENV JAVA_HOME /usr/local/openjdk-15
ENV PATH $JAVA_HOME/bin:$PATH
RUN --mount=type=cache,uid=1000,target=/home/worker/.cache ./wss_agent.sh
FROM base as test
COPY . .
RUN poetry config experimental.new-installer false
RUN poetry install
RUN cd my_proj && poetry run invoke deployconfluence_server_pass=$ATLASSIAN_TOKEN
FROM base as package
COPY . .
RUN poetry build
RUN python -m pip install --upgrade pip && \
pip install twine keyring artifacts-keyring && \
twine upload -r $REPO_URL --config-file $PYPIRC_PATH dist/* --skip-existing
FROM base as build
COPY . .
RUN poetry config experimental.new-installer false
RUN poetry install --no-dev
RUN pip3 --no-cache-dir install --upgrade awscli
RUN aws s3 cp s3://....tar.gz $ROOT/my_proj # censored url
RUN mkdir $ROOT/my_proj/bert-base-cased && cd $ROOT/my_proj/bert-base-cased && \
wget https://huggingface.co/bert-base-cased/resolve/main/config.json && \
wget https://huggingface.co/bert-base-cased/resolve/main/tokenizer.json && \
wget https://huggingface.co/bert-base-cased/resolve/main/tokenizer_config.json
FROM python:3.7-slim as production
ENV ROOT=/home/worker/python/my_proj \
VIRTUAL_ENV=/home/worker/python/my_proj/.venv\
PATH=/home/worker/python/my_proj/.venv/bin:/home/worker/python/my_proj:$PATH
COPY --from=build /home/worker/python/my_proj/pyproject.toml /home/worker/python/
COPY --from=build /home/worker/python/my_proj/.venv /home/worker/python/my_proj/.venv
COPY --from=build /home/worker/python/my_proj/my_proj /home/worker/python/my_proj
WORKDIR $ROOT
ENV PYTHONPATH=$ROOT:/home/worker/python/
ENTRYPOINT [ "primary_worker", "--mongo" ]
Installing it via pip should work:
RUN pip3 install torch==1.9.0+cpu torchvision==0.10.0+cpu torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
I edit my Dockerfile from the feedback I received.
I run my python script with args python3 getappVotes.py --abc 116 --xyz 2 --processall
This is my Dockerfile. I can build successfully but not sure how I can pass above args during docker run. Arguments are optional.
FROM python:3.7.5-slim
WORKDIR /usr/src/app
RUN python -m pip install \
parse \
argparse \
datetime \
urllib3 \
python-dateutil \
couchdb \
realpython-reader
RUN mkdir -p /var/log/appvoteslog/
COPY getappVotes.py .
ENTRYPOINT ["python", "./getappVotes.py"]
CMD ["--xx 116", "--yy 2", "--processall"]
As mentioned in the comments, for your use case, you need to use both ENTRYPOINT (for the program and mandatory arguments) and CMD (for optional arguments).
More precisely, you should edit your Dockerfile in the following way:
FROM python:3.7.5-slim
WORKDIR /usr/src/app
RUN python -m pip install \
parse \
argparse \
datetime \
urllib3 \
python-dateutil \
couchdb \
realpython-reader
RUN mkdir -p /var/log/appvoteslog/
COPY getappVotes.py .
ENTRYPOINT ["python", "./getappVotes.py"]
CMD ["--xx", "116", "--yy", "2", "--processall"]
(as from a shell perspective, --xx 116 is recognized as two separated arguments, not a single argument)
For more details:
see the answer What is the difference between CMD and ENTRYPOINT in a Dockerfile?;
see also the example from this answer: Install python package in docker file − where the "manual" command python -m pip install … is replaced with a more idiomatic command, pip install --no-cache-dir -r requirements.txt
I want to create a docker image with specifically python 3.5 on a specific base image which is the nvidia/cuda (9.0-base image) the latter has no python environment.
The reason I need specific versions is to support running cuda10.0 python3.5 and a gcc version<7 to compile the driver all together on the same box
When I try and build the docker environments (see below) I always end up with the system update files which load python3.6
The first version I run (below) runs a system update dependencies which installs python 3.6 I have tried many variants to avoid this but always end up 3.6 in the final image.
Any suggestions for getting this running with python3.5 are welcome
Thanks
FROM nvidia/cuda
RUN apt-get update && apt-get install -y libsm6 libxext6 libxrender-dev python3.5 python3-pip
COPY . /app
WORKDIR /app
RUN pip3 install -r requirements.txt
ENTRYPOINT [ "python3" ]
CMD [ "app.py" ]
Another variant (below) I have tried is with virtualenv and here again I can't seem to force a python 3.5 environment
FROM nvidia/cuda
RUN apt-get update && apt-get install -y --no-install-recommends libsm6 libxext6 libxrender-dev python3.5 python3-pip python3-virtualenv
ENV VIRTUAL_ENV=/opt/venv
RUN python3 -m virtualenv --python=/usr/bin/python3 $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
COPY . /app
WORKDIR /app
RUN pip3 install -r requirements.txt
ENTRYPOINT [ "python3" ]
CMD [ "app.py" ]
You can try using conda. I used several stages to minimize final container and to speedup/cache local builds.
# first stage
FROM nvidia/cuda:11.1-base-ubuntu18.04 as builder
RUN apt-get update && apt-get install -y curl wget gcc build-essential
# install conda
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-4.5.12-Linux-x86_64.sh -O ~/miniconda.sh && \
/bin/bash ~/miniconda.sh -b -p /opt/conda
# create env with python 3.5
RUN /opt/conda/bin/conda create -y -n myenv python=3.5
# install requirements
WORKDIR /app
COPY requirements.txt /app
ENV PATH=/opt/conda/envs/myenv/bin:$PATH
RUN pip install -r requirements.txt
RUN pip uninstall -y pip
####################
# second stage (note: FROM container must be the same as builder)
FROM nvidia/cuda:11.1-base-ubuntu18.04 as runner
# copy environment data including python
COPY --from=builder /opt/conda/envs/myenv/bin /opt/conda/envs/myenv/bin
COPY --from=builder /opt/conda/envs/myenv/lib /opt/conda/envs/myenv/lib
# do some env settings
ENV PATH=/opt/conda/envs/myenv/bin:$PATH
ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8
####################
# final image
from runner
WORKDIR /app
COPY ./run.py /app
CMD [ "python", "run.py"]
You can install from PPA and use it as usual:
FROM nvidia/cuda
RUN apt-get update && apt-get install -y --no-install-recommends software-properties-common \
libsm6 libxext6 libxrender-dev curl \
&& rm -rf /var/lib/apt/lists/*
RUN echo "**** Installing Python ****" && \
add-apt-repository ppa:deadsnakes/ppa && \
apt-get install -y build-essential python3.5 python3.5-dev python3-pip && \
curl -O https://bootstrap.pypa.io/get-pip.py && \
python3.5 get-pip.py && \
rm -rf /var/lib/apt/lists/*
COPY requirements.txt requirements.txt
RUN pip3.5 install -r requirements.txt
CMD ["python3.5", "app.py"]
I have basically a docker image of a node js application.
REPOSITORY TAG IMAGE ID CREATED SIZE
abc-test 0.1 1ba85e0ca455 7 hours ago 1.37GB
I want to run npm test from folder /data/node/src but that doesn't seems to be working.
Here is the command what I am trying:
docker run -p 80:80 --entrypoint="cd /data/node/src && npm run test" abc-test:0.1
But that doesn't seems to be working.
Here is my dockerfile:
FROM python:2.7.13-slim
RUN apt-get update && apt-get install -y apt-utils curl
RUN echo 'deb http://nginx.org/packages/debian/ jessie nginx' > /etc/apt/sources.list.d/nginx.list
RUN apt-get update && apt-get install -y \
build-essential \
gcc \
git \
libcurl4-openssl-dev \
libldap-2.4-2 \
libldap2-dev \
libmysqlclient-dev \
libpq-dev \
libsasl2-dev \
nano \
nginx=1.8.* \
nodejs \
python-dev \
supervisor
ENV SERVER_DIR /data/applicationui/current/server
ADD src/application/server $SERVER_DIR
EXPOSE 14000 80
# version A: only start tornado, without nginx.
WORKDIR $SERVER_DIR/src
CMD ["npm","run","start:staging"]
Can anyone please help me here.
Pretty sure you can only run one command with ENTRYPOINT and with CMD.
From their docs:
There can only be one CMD instruction in a Dockerfile. If you list more than one CMD then only the last CMD will take effect.
Same thing with Entrypoint:
ENTRYPOINT has two forms:
ENTRYPOINT ["executable", "param1", "param2"] (exec form, preferred)
ENTRYPOINT command param1 param2 (shell form)
https://docs.docker.com/engine/reference/builder/#cmd
https://docs.docker.com/engine/reference/builder/#entrypoint
A work around that I do is the following
FROM ubuntu:16.04
WORKDIR /home/coins
RUN apt-get update
...
OTHER DOCKERFILE STUFF HERE
...
COPY ./entrypoint.sh /home/coins/
RUN chmod +x ./entrypoint.sh
ENTRYPOINT ./entrypoint.sh
entrypoint.sh:
#!/bin/bash
Can write whatever sh commands you need here..
exec sh ./some_script
EDIT:
One idea is you can add a test sh script and just trigger those 2 commands in it, and you'd be able to launch it with --entrypoint="test.sh"