I have an image called: Image and a running container called: container.
I want to install pytorch and anaconda. What's the easiest way to do this?
Do I have to change the Dockerfile and build a new image?
Thanks a lot.
Yes, the best thing is to build your image in such a way it has the python modules are in there.
Here is an example. I build an image with the build dependencies:
$ docker build -t oz123/alpine-test-mycoolapp:0.5 - < Image
Sending build context to Docker daemon 2.56 kB
Step 1 : FROM alpine:3.5
---> 88e169ea8f46
Step 2 : ENV MB_VERSION 3.1.4
---> Running in 4587d36fa4ae
---> b7c55df49803
Removing intermediate container 4587d36fa4ae
Step 3 : ENV CFLAGS -O2
---> Running in 19fe06dcc314
---> 31f6a4f27d4b
Removing intermediate container 19fe06dcc314
Step 4 : RUN apk add --no-cache python3 py3-pip gcc python3-dev py3-cffi file git curl autoconf automake py3-cryptography linux-headers musl-dev libffi-dev openssl-dev build-base
---> Running in f01b60b1b5b9
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/community/x86_64/APKINDEX.tar.gz
(1/57) Upgrading musl (1.1.15-r5 -> 1.1.15-r6)
(2/57) Upgrading zlib (1.2.8-r2 -> 1.2.11-r0)
(3/57) Installing m4 (1.4.17-r1)
(4/57) Installing perl (5.24.0-r0)
(5/57) Installing autoconf (2.69-r0)
(6/57) Installing automake (1.15-r0)
(7/57) Installing binutils-libs (2.27-r1)
...
Note, I am installing Python's pip inside the image, so later I can download packages from pypi. Packages like numpy might require a C compiler and tool chain, so I am installing these too.
After building the packages which require the build tools chain I remove the tool chain packages:
RUN apk del file pkgconf autoconf m4 automake perl g++ libstdc++
After you have your base image, you can run your application code in
an image building on top of it:
$ cat Dockerfile
FROM oz123/alpine-test-mycoolapp
ADD . /code
WORKDIR /code
RUN pip3 install -r requirements.txt -r requirements_dev.txt
RUN pip3 install -e .
RUN make clean
CMD ["pytest", "-vv", "-s"]
I simply run this with docker.
Related
I'm trying to build a docker with python 3 and google-cloud-bigquery with the following docker file:
FROM python:3.10-alpine
RUN pip3 install google-cloud-bigquery
WORKDIR /home
COPY *.py /home/
ENTRYPOINT ["python3", "-u", "myscript.py"]
But getting errors on the pip3 install google-cloud-bigquery (too long for here)..
What's missing for installing this on python-alpine?
Looks like an incompatibility issue with the latest version of google-cloud-bigquery (>3) and numpy:
ERROR: Could not build wheels for numpy, which is required to install pyproject.toml-based projects
Try specifying a previous version, this works for me:
RUN pip3 install google-cloud-bigquery==2.34.4
Actually it seems like not a problem with numpy, which builds smoothly with all the dependency libs install, but rather with pyarrow, which does not support alpine+pip build. I've found a workaround by using alpine pre-built version of pyarrow. It is much easier than building pyarrow from source. This build works for me just fine:
FROM python:3.10.6-alpine3.16
RUN apk add --no-cache build-base linux-headers \
py3-apache-arrow=8.0.0-r0
# Copying pyarrow to site-package of actual python path. Alpine python path
# and python's docker hub path are different.
RUN mv /usr/lib/python3.10/site-packages/* \
/usr/local/lib/python3.10/site-packages/
RUN rm -rf /usr/lib/python3.10
RUN --mount=type=cache,target=/root/.cache/pip \
pip install google-cloud-bigquery==3.3.2
Update python version, alpine version and py3-apache-arrow version to install later versions. This is the latest one on the time of writing.
And make sure to remove build dependencies (build-base, linux-headers) for your release docker. I prefer multistage dockers for this.
I wrote this Dockerfile for an os
FROM randomdude/gcc-cross-x86_64-elf
RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get install -y nasm
RUN apt-get install -y xorriso
RUN apt-get install -y grup-pc-bin
RUN apt-get install -y grup-common
VOLUME /
WORKDIR /
and while running sudo docker build buildenv -t testos-buildenv
on the terminal i got this log
Sending build context to Docker daemon 2.048kB
Step 1/9 : FROM randomdude/gcc-cross-x86_64-elf
---> c7e17c42eb04
Step 2/9 : RUN apt-get update
---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
---> Running in 32e48dbf4a9c
exec /bin/sh: exec format error
The command '/bin/sh -c apt-get update' returned a non-zero code: 1
this file is inside /home/user/Desktop/os-systems/test-os/buildenv
i need help to solve it
Of course..It is for x86.
It depends on what you want to do.
If you want use it on your os. You have to build an arm64 version image. You have to replace some x86 dependences in the original Dockerfile and re-build it.
This Dockerfile is mentioned by the base image's description you use.
If you want use a x86 image but just want to build it on your OS (arm64), then you could try to use buildx.
Building a docker image, I've installed Negbio in my Dockerfile using:
RUN git clone https://github.com/ncbi-nlp/NegBio.git xyz && \
python xyz/setup.py install
When I try to run my Django application at localhost:1227 I get:
No module named 'negbio' ModuleNotFoundError exception
When I run pip list I can see negbio. What am I missing?
As per your comment, It wouldn't install with pip and hence not installing via pip.
Firstly, to make sure the https://github.com/ncbi-nlp/NegBio is properly installed via python setup.py install, you need to install it's dependencies via pip install -r requirements first. So either ways, you are doomed to have pip inside Docker.
For example, this is the sample Dockerfile that would install the negbio package properly:
FROM python:3.6-slim
RUN mkdir -p /apps
WORKDIR /apps
# Steps for installing the package via Docker:
RUN apt-get update && apt-get -y upgrade && apt-get install -y git gcc build-essential
RUN git clone https://github.com/ncbi-nlp/NegBio.git
WORKDIR /apps/NegBio
RUN pip install -r requirements.txt
RUN python setup.py install
ENV PATH=~/.local/bin:$PATH
EXPOSE 8000
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
So wouldn't harm if you actually install it via requirements.txt
I would do it like this:
requirements.txt --> have all your requirements added here
negbio==0.9.4
And make sure it's installed on the fly inside the docker using RUN pip install -r requirements.txt
I ultimately resolved my issue by going to an all Anaconda environment. Thank you for everyone's input.
I installed libpcap in my container using below docker file using docker file below. How do I make sure it was installed and working as expected?
I tried below with the hope to see libpcap
D:\work >docker exec -u 0 -it containerId sh
/app # cd /etc/apk
/etc/apk # cat repositories
http://dl-cdn.alpinelinux.org/alpine/v3.8/main
http://dl-cdn.alpinelinux.org/alpine/v3.8/community
/etc/apk #
Below is my docker file
FROM mcr.microsoft.com/dotnet/core/sdk:2.2-alpine AS build
# Install packages
RUN apk update
RUN apk -U --no-cache add libpcap
Running the apk info command has below output
WARNING: Ignoring APKINDEX.adfa7ceb.tar.gz: No such file or directory
WARNING: Ignoring APKINDEX.efaa1f73.tar.gz: No such file or directory
musl
busybox
alpine-baselayout
alpine-keys
libressl2.7-libcrypto
libressl2.7-libssl
libressl2.7-libtls
ssl_client
zlib
apk-tools
scanelf
musl-utils
libc-utils
ca-certificates
krb5-conf
libcom_err
keyutils-libs
libverto
krb5-libs
libgcc
libintl
libcrypto1.0
libssl1.0
libstdc++
userspace-rcu
lttng-ust
tzdata
Run docker exec command and try this
$ apk info
This will list all the installed packages in alpine.
I can see libcap in the output.
If you still can't see the package. Make sure you have run apk update before installing libcap
So, after building out a pipeline, I realized I will need some custom libraries for a python script I will be pulling from SCM. To install Jenkins in Docker, I used the following tutorial:
https://jenkins.io/doc/book/installing/
Like so:
docker run \
-u root \
--rm \
-d \
-p 8080:8080 \
-p 50000:50000 \
-v jenkins-data:/var/jenkins_home \
-v /var/run/docker.sock:/var/run/docker.sock \
jenkinsci/blueocean
Now, I will say I'm not a Docker guru, but I'm aware the Dockerfile allows for passing in library installs for Python. However, because I'm pulling the docker image from dockerhub, I'm not sure if it's possible to add a "RUN pip install " as an argument. Maybe there is an alternate approach someone may have.
Any help is appreciated.
EDIT 1: Here's the output of the first commenter's recommendation:
Step 1/6 : FROM jenkinsci/blueocean
---> b7eef16a711e
Step 2/6 : USER root
---> Running in 150bba5c4994
Removing intermediate container 150bba5c4994
---> 882bcec61ccf
Step 3/6 : RUN apt-get update
---> Running in 324f28f384e0
/bin/sh: apt-get: not found
The command '/bin/sh -c apt-get update' returned a non-zero code: 127
Error:
/bin/sh: apt-get: not found
The command '/bin/sh -c apt-get update' returned a non-zero code: 127
Observation:
This error comes when the container that you want to run is not Debian based, hence does not support 'apt'.
To resolve this, we need to find out which package manager it utilizes.
In my case it was: 'apk'.
Resolution:
Replace 'apt-get' with 'apk' in your Dockerfile. (If this does not work you can try 'yum' package manager as well).
Command in your Dockerfile should look like:
RUN apk update
You can create a Dockerfile
FROM jenkins:latest
USER root
RUN apt-get update
RUN apt-get install -y python-pip
# Install app dependencies
RUN pip install --upgrade pip
You can build the custom image using
docker build -t jenkinspython .
Similar to what Hemant Sing's answer, but 2 slightly different things.
First, create a unique directory: mkdir foo
"cd" to that directory and run:
docker build -f jenkinspython .
Where jenkinspython contains:
FROM jenkins:latest
USER root
RUN apt-get update
RUN apt-get install -y python-pip
# Install app dependencies
RUN pip install --upgrade pip
Notice that my change has -f, not -t. And notice that the build output does indeed contain:
Step 5/5 : RUN pip install --upgrade pip
---> Running in d460e0ebb11d
Collecting pip
Downloading https://files.pythonhosted.org/packages/5f/25/e52d3f31441505a5f3af41213346e5b6c221c9e086a166f3703d2ddaf940/pip-18.0-py2.py3-none-any.whl (1.3MB)
Installing collected packages: pip
Found existing installation: pip 9.0.1
Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-18.0
Removing intermediate container d460e0ebb11d
---> b7d342751a79
Successfully built b7d342751a79
So now that the image has been built (in my case, b7d342751a79), fire it up and verify that pip has indeed been updated:
$ docker run -it b7d342751a79 bash
root#9f559d448be9:/# pip --version
pip 18.0 from /usr/local/lib/python2.7/dist-packages/pip (python 2.7)
So now your image has pip installed, so then you can feel free to pip install whatever crazy packages you need :)