Python 3 virtualenv and Docker - python-3.x

I'm trying to build a docker image with python 3 and virtualenv.
I understand that I wouldn't need to use wirtualenv in a docker image as I'm going to use only python 3, yet I see some clean isolation benefits of using virtualenv anyways.
What's the best practice? Should I avoid using virtualenv on docker?
If that's the case, how can I setup python3 and pip3 to be used as python and pip (without the 3)?
This is my Dockerfile:
FROM openjdk:8-alpine
RUN apk update && apk add bash gcc musl-dev
RUN apk add python3 python3-dev
RUN apk add py3-pip
RUN apk add libxslt-dev libxml2-dev
ENV PROJECT_HOME /opt/app
RUN mkdir -p /opt/app
RUN mkdir -p /opt/app/modules
ENV LD_LIBRARY_PATH /usr/lib/python3.6/site-packages/jep
ENV LD_PRELOAD /usr/lib/libpython3.6m.so
RUN pip3 install jep
RUN pip3 install ads
RUN pip3 install gspread
RUN pip3 list
COPY target/my-server-1.0-SNAPSHOT.jar $PROJECT_HOME/my-server-1.0-SNAPSHOT.jar
WORKDIR $PROJECT_HOME
CMD ["java", "-Dspring.data.mongodb.uri=mongodb://my-mongo:27017/mydb","-jar","./my-server-1.0-SNAPSHOT.jar"]
Thanks
=== UPDATE 1 ===
I'm trying to create a new virtual env in the WORKDIR, install some libs and then execute a shell script, even though I see it creates the whole thing when I build the image, when running the container the environment folder is empty.
This is from my Dockerfile:
RUN virtualenv ./env && source ./env/bin/activate && pip install jep \
googleads gspread oauth2client
ENTRYPOINT ["/bin/bash", "./startup.sh"]
startup.sh:
#!/bin/sh
source ./env/bin/activate
java -Dspring.data.mongodb.uri=mongodb://my-mongo:27017/mydb -jar ./my-server-1.0-SNAPSHOT.jar
It builds fine but on docker-compose up -d this is the output:
./startup.sh: source: line 2: can't open './env/bin/activate'
The env folder exists, but it's empty.
Any ideas?
Thanks!
=== UPDATE 2 ===
This is the working config:
RUN virtualenv ./my-env && source ./my-env/bin/activate \
&& pip install gspread==0.6.2 jep oauth2client googleads pandas
CMD ["/bin/bash", "-c", "./startup.sh"]
This is startup.sh:
#!/bin/sh
source ./my-env/bin/activate
java -Dspring.data.mongodb.uri=mongodb://my-mongo:27017/mydb -jar ./my-server-1.0-SNAPSHOT.jar

I don't think using virtualenv in docker is something really negative, it will slow down your container builds just a bit.
As for renaming pip3 and python3, you can create a hard link like this:
ln /usr/bin/python3 /usr/bin/python
ln /usr/bin/pip3 /usr/bin/pip
assuming python3 executable is in /usr/bin/. You can find its location by running which python3
P.S.: Your dockerfile contains loads of RUN instructions, that are creating unnecessary intermediate containers. Combine them to save space and time:
RUN apk update && apk add bash gcc musl-dev \
python3 python3-dev py3-pip \
libxslt-dev libxml2-dev
RUN mkdir -p /opt/app/modules # you don't need the first one, -p will create it for you
RUN pip3 install jep ads gspread
Or combine them even further, if you aren't planning to change them often:
RUN apk update
&& apk add bash gcc musl-dev \
python3 python3-dev py3-pip \
libxslt-dev libxml2-dev \
&& mkdir -p /opt/app/modules \
&& pip3 install jep ads gspread

The only "workaround" I've found in order to use virtualenv from my docker container is to enter to the docker by ssh, create the environment, install the libs and set its folder as a volume in the docker-compose config so it won't be deleted and I can use it afterward.
(Or to have it ready and just copy the folder at build time) which could be a good option for saving build time, isn't it?
Otherwise, If I create it on Dockerfile and install the libs there, its folder gets empty when the container runs. Don't know why.
I appreciate if anyone can suggest a better way to deal with that.

Related

Why would Python not be available to a Docker Entrypoint Script?

This Python 3.9 project has a Dockerfile, that builds successfully. The file makes use of an ENTRYPOINT script to create some directories and handle some clean-up at run time. It is a bash script. The ENTRYPOINT script has no problem running until the very end, where it is expected to execute the CMD that is passed next. Well, I should say this behavior only happens when Kaniko builds the image. When the image is built locally, no such problem occurs. However, I am willing to chalk that up to the fact that locally is on a Windows machine. However, that shouldn't matter here because the error thrown is:
/opt/project/conf/entrypoint.sh: /usr/bin/supervisord: /usr/bin/python3: bad interpreter: No such file or directory
/opt/project/conf/entrypoint.sh: line 8: /usr/bin/supervisord: Success
Now I have looked at many "bad interpreter" questions. They all seem to revolve around the interpreter being in a custom place. I am reliant upon the default spot for the Python 3.9 interpreter. On Debian Bullseye (The OS behind the base image) that should be /usr/local/bin/python or /usr/local/bin/python3. So I am completely stumped as to why it is unable to find or use it.
Here are the implementation details:
Dockerfile:
FROM python:3.9-slim-bullseye
# Minimum Required Environment Variables
ENV SHELL=/bin/bash
ENV CC /usr/bin/gcc
ENV CXX /usr/bin/g++
ENV LANG=C.UTF-8
ENV DEBIAN_FRONTEND=noninteractive
ENV PYMSSQL_BUILD_WITH_BUNDLED_FREETDS=1
ENV PIP_CONFIG_FILE=/etc/pip.conf
ENV TZ=America/Los_Angeles
# Project Specific Environment Variables
ENV PROJECT_LOGFILE=/var/log/project/project.log
ENV PROJECT_CONFIG_DIRECTORY=/opt/project/conf
ENV PROJECT_SETTINGS_MODULE="project.settings"
# Files Needed for Dependency Installation
COPY dev/.pip.conf /etc/pip.conf
COPY dev/dev-requirements.txt /usr/local/requirements.txt
# Dependency Installation
WORKDIR /tmp
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get install musl-dev g++ bash curl gnupg -y \
&& curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \
&& curl https://packages.microsoft.com/config/debian/11/prod.list > /etc/apt/sources.list.d/mssql-release.list \
&& apt-get update \
&& apt-get install --no-install-recommends libfreetype-dev freetds-dev python-dev git libpng-dev libxml2-dev \
libxslt-dev libssl-dev libopenblas-dev rsyslog supervisor tini tzdata libghc-zlib-dev libjpeg-dev cron \
libgssapi-krb5-2 unixodbc-dev -y \
&& ACCEPT_EULA=Y apt-get install -y msodbcsql18 \
&& ln -s /usr/include/locale.h /usr/include/xlocale.h \
&& pip install --no-cache-dir --upgrade pip setuptools wheel \
&& pip install matplotlib --no-cache-dir \
&& pip install --no-cache-dir -r /usr/local/requirements.txt
# Setting Up For Install
COPY conf/ /opt/project/conf/
RUN mkdir -p /var/log/project /conf \
&& cp /opt/project/conf/supervisord.conf /conf/supervisord.conf \
&& cp /opt/project/conf/rsyslog.conf /conf/rsyslog.conf
WORKDIR /opt
# Copy Over Packages
COPY project-db-migrations /opt/project/project-db-migrations
COPY infrastructure /opt/infrastructure
COPY project /opt/project/src
COPY README.md /opt/project/README.md
# Install Infrastructure
RUN cd /opt/infrastructure && python3 setup.py install
# Install Project Service
RUN cd /opt/project/src && python3 setup.py install
RUN ["chmod", "+x", "/opt/project/conf/entrypoint.sh"]
WORKDIR /
EXPOSE 80
ENTRYPOINT ["tini", "--", "/opt/project/conf/entrypoint.sh"]
CMD ["supervisord", "-c", "/conf/supervisord.conf"]
entrypoint.sh
#!/bin/bash
set -eu
echo "Setting Up Project Service"
# Adding Temp Directory
mkdir -p /opt/project/tmp
echo "Service has been setup"
exec $#
supervisord.conf
[supervisord]
nodaemon=true
logfile=/var/log/project/supervisord.log
childlogdir=/var/log/project
[program:rsyslogd]
command=/usr/sbin/rsyslogd -n -f /conf/rsyslog.conf
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:crond]
command=/usr/sbin/cron -f -l 15
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:project]
command=python -m project.run --server
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
The image is ran and deployed without changes to the user, so it should be running as root.
In this case, after some digging I found there is an issue with the Kaniko version DevOps had running. That was causing the issue. Because the image wasn't being flattened correctly, Python could not start properly.

Azure ACR Flask webapp deployment

I'm trying to deploy a Flask app on Azure by using the container registry. I used Az DevOps to pipeline building and pushing the image. Then I configure a web app to pull this image from ACR. But I'm running into one or the other error. The container is timing out and crashing and there are some errors in the application log as well. 'No module named flask'.
This is my Dockerfile
FROM python:3.7.11
ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"
RUN apt-get update
RUN apt-get install -y wget && rm -rf /var/lib/apt/lists/*
WORKDIR /app
ADD . /app/
RUN wget \
https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
&& mkdir /root/.conda \
&& bash Miniconda3-latest-Linux-x86_64.sh -b \
&& rm -f Miniconda3-latest-Linux-x86_64.sh
RUN python -m pip install --upgrade pip
RUN pip3 install Flask==2.0.2
RUN pip3 install --no-cache-dir -r /app/requirements.txt
RUN conda install python=3.7
RUN conda install -c rdkit rdkit -y
EXPOSE 5000
ENV NAME cheminformatics
CMD python app.py
I had to install miniconda because the rdkit package can be installed only from conda. I also add a PORT: 5000 key value to the configuration of the web app, but it hasn't loaded even once. I've been at this for 2 hours now. Previously, I've built images on local machine and pushed to dockerhub and ACR and was able to pull those images but it's the first time I used DevOps and I'm not sure what's going wrong here. Can someone please help?

Alpine 3.11 diff: unrecognized option: c BusyBox v1.31.1 () multi-call binary

I am using an alpine 3.11 to build my image, everything goes well during the build the dockefile is here below :
FROM alpine:3.11
LABEL version="1.0"
ARG UID="110"
ARG PYTHON_VERSION="3.8.10-r0"
ARG ANSIBLE_VERSION="5.0.1"
ARG AWSCLI_VERSION="1.22.56"
# Create jenkins user with sudo privileges
RUN adduser -u ${UID} -D -h /home/jenkins/ jenkins
RUN echo 'jenkins ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
RUN mkdir -p /tmp/.ansible
RUN chown -R jenkins:jenkins /tmp/.ansible
# Install minimal packages
RUN apk --update --no-cache add bash bind-tools curl gcc git libffi-dev libpq make mysql-client openssl postgresql-client sudo unzip wget coreutils
#RUN apk --update --no-cache add py-mysqldb
RUN apk --update --no-cache add python3=${PYTHON_VERSION} python3-dev py3-pip py3-cryptography
# Install JQ from sources
RUN wget https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64
RUN mv jq-linux64 /usr/bin/jq
RUN chmod +x /usr/bin/jq
# Install ansible and awscli with python package manager
RUN pip3 install --upgrade pip
RUN pip3 install yq --ignore-installed PyYAML
RUN pip3 install ansible==${ANSIBLE_VERSION}
RUN pip3 install awscli==${AWSCLI_VERSION} boto boto3 botocore s3cmd pywinrm pymysql 'python-dateutil<2.8.1'
# Clean cache
RUN rm -rf /var/cache/apk/*
# Display packages versions
RUN python3 --version && \
pip3 --version && \
ansible --version && \
aws --version
this image is later used to lunch some jenkins jobs nothing unusual.
But when i try to use the diff command in of these jobs I have the following error :
diff: unrecognized option: c BusyBox v1.31.1 () multi-call binary
that's why i tried to install the coreutils package but still the "-c" option is still unrecognized which is weird.
So my question is there a way to add the -c option for the diff command because in the manual of GNU this should be available automatically but apparently not on Alpine ? if there is a way could anyone please share it.
P.S : In case you are wondering why am I using the diff command it is just to compare two json files and the -c is necessary for me in this context.
Well I just had to add the diffutils package to the list after installing it everything works well
In spite of it being required in the POSIX diff specification it looks like the BusyBox implementation of diff doesn't support the -c option.
One thing you could do is change your diff invocation to use unified context diff format. Again, BusyBox diff appears to not support -u, so you need to use an explicit -U option with the number of lines of context
diff -U3 file.orig file.new
In general, the Alpine environment has many small differences like this. If you're installing the GNU versions of these tools anyways – your Dockerfile already installs GNU bash and coreutils – you'll probably find minimal to no space savings from using an Alpine base image, and using a Debian or Ubuntu base that already includes the GNU versions of these tools will be easier.
FROM ubuntu:20.04 # not Alpine
...
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive \
apt-get install --no-install-recommends --assume-yes \
bind9-utils \
build-essential \
curl \
git-core \
...
You may need to search on https://packages.debian.org/ to find equivalent Debian packages. build-essential is a metapackage that includes the entire C toolchain (gcc, make, et al.); bash, coreutils, and diffutils would typically be installed as part of the base distribution image.

docker ERROR: Could not find a version that satisfies the requirement apturl==0.5.2

I am using windows 10 OS. I want to build an container based on linux so I can replicate code and dependencies developed from ubuntu. When I try to build it outputs Error message as above.
From my understanding docker for desktop runs linux OS kernel under-the-hood therefore allowing window users to run linux based containers, not sure why it is outputting this error.
My dockerfile looks like this:
FROM ubuntu:18.04
ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"
RUN apt update \
&& apt install -y htop python3-dev wget
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
&& mkdir root/.conda \
&& sh Miniconda3-latest-Linux-x86_64.sh -b \
&& rm -f Miniconda3-latest-Linux-x86_64.sh
RUN conda create -y -n ml python=3.7
COPY . src/
RUN /bin/bash -c "cd src \
&& source activate ml \
&& pip install -r requirements.txt"
requirements.txt contains:
apturl==0.5.2
asn1crypto==0.24.0
bleach==2.1.2
Brlapi==0.6.6
certifi==2020.11.8
chardet==3.0.4
click==7.1.2
command-not-found==0.3
configparser==5.0.1
cryptography==2.1.4
cupshelpers==1.0
dataclasses==0.7
When I run docker build command it outputs:
1.649 ERROR: Could not find a version that satisfies the requirement apturl==0.5.2 1.649 ERROR: No matching distribution found for apturl==0.5.2 Deleting it and running it lead to another error. All error seem to be associated with ubuntu packages.
Am I not running a ubuntu container? why aren't I allowed to install ubuntu packages?
Thanks!
You try to install ubuntu packages with pip (which is for python packages")
try apt install -y apturl
If you want to install python packages write pip install package_name

Unable to run aliyun-cli in Docker:stable container after installing it. Errors as command not found

I am unsure if stack overflow or system fault is the right stack exchange site but I'm going with stack overflow cause the alicloud site posted to add a tag and ask a question here.
So. I'm currently building an image based on Docker:stable, that is an alpine distro, that will have aliyun-cli installed and available for use. However I am getting a weird error of Command Not Found when I'm running it. I have followed the guide here https://partners-intl.aliyun.com/help/doc-detail/139508.htm and moved the aliyun binary to /usr/sbin
Here is my Dockerfile for example
FROM docker:stable
RUN apk update && apk add curl
#Install python 3
RUN apk update && apk add python3 py3-pip
#Install AWS Cli
RUN pip3 install awscli --upgrade
# Install Aliyun CLI
RUN curl -L -o aliyun-cli.tgz https://aliyuncli.alicdn.com/aliyun-cli-linux-3.0.30-amd64.tgz
RUN tar -xzvf aliyun-cli.tgz
RUN mv aliyun /usr/bin
RUN chmod +x /usr/bin/aliyun
RUN rm aliyun-cli.tgz
However when i'm running aliyun (which can be auto-completed) I am getting this
/ # aliyun
sh: aliyun: not found
I've tried moving it to other bins. Cding into the folder and calling it explicitly but still always getting a command not found. Any suggestions would be welcome.
Did you check this Dockerfile?
Also why you need to install aws-cli in the same image and why you will need to maintain it for your self when AWS provide managed aws-cli image.
docker run --rm -it amazon/aws-cli --version
that's it for aws-cli image,but if you want in existing image then you can try
RUN pip install awscli --upgrade
DockerFile
FROM python:2-alpine3.8
LABEL com.frapsoft.maintainer="Maik Ellerbrock" \
com.frapsoft.version="0.1.0"
ARG SERVICE_USER
ENV SERVICE_USER ${SERVICE_USER:-aliyun}
RUN apk add --no-cache curl
RUN curl https://raw.githubusercontent.com/ellerbrock/docker-collection/master/dockerfiles/alpine-aliyuncli/requirements.txt > /tmp/requirements.txt
RUN \
adduser -s /sbin/nologin -u 1000 -H -D ${SERVICE_USER} && \
apk add --no-cache build-base && \
pip install aliyuncli && \
pip install --no-cache-dir -r /tmp/requirements.txt && \
apk del build-base && \
rm -rf /tmp/*
USER ${SERVICE_USER}
WORKDIR /usr/local/bin
ENTRYPOINT [ "aliyuncli" ]
CMD [ "--help" ]
build and run
docker build -t aliyuncli .
docker run -it --rm aliyuncli
output
docker run -it --rm abc aliyuncli
usage: aliyuncli <command> <operation> [options and parameters]
<aliyuncli> the valid command as follows:
batchcompute | bsn
bss | cms
crm | drds
ecs | ess
ft | ocs
oms | ossadmin
ram | rds
risk | slb
ubsms | yundun
After a lot of lookup I found a github issue in the official aliyun-cli that sort of describes that it is not compatible with alpine linux because of it's not muslc compatible.
Link here: https://github.com/aliyun/aliyun-cli/issues/54
Following the workarounds there I build a multi-stage docker file with the following that simply fixed my issue.
Dockerfile
#Build aliyun-cli binary ourselves because of issue
#in alpine https://github.com/aliyun/aliyun-cli/issues/54
FROM golang:1.13-alpine3.11 as cli_builder
RUN apk update && apk add curl git make
RUN mkdir /srv/aliyun
WORKDIR /srv/aliyun
RUN git clone https://github.com/aliyun/aliyun-cli.git
RUN git clone https://github.com/aliyun/aliyun-openapi-meta.git
ENV GOPROXY=https://goproxy.cn
WORKDIR aliyun-cli
RUN make deps; \
make testdeps; \
make build;
FROM docker:19
#Install python 3 & jq
RUN apk update && apk add python3 py3-pip python3-dev jq
#Install AWS Cli
RUN pip3 install awscli --upgrade
# Install Aliyun CLI from builder
COPY --from=cli_builder /srv/aliyun/aliyun-cli/out/aliyun /usr/bin
RUN aliyun configure set --profile default --mode EcsRamRole --ram-role-name build --region cn-shanghai

Resources