I am upgrading some images from to docker:19.03.5 to docker:20.10.22 but am hitting issues when my Push Dev job tries to install pip3 on the image being used by .gitlab-ci.yml.
Initially I was not explicitly installing pip3 as this worked fine, after an initial error saying pip3 was not installed I tried installing using apk as below.
before_script:
- apk add python3
- apk add python3-pip
- pip3 install awscli==1.18.8
- docker load --input data/image.tar
- $(aws ecr get-login --no-include-email --region us-east-1)
Push Dev:
stage: Push
script:
- docker tag proxy:latest $ECR_REPO:dev
- docker push $ECR_REPO:dev
rules:
- if: "$CI_COMMIT_BRANCH == 'main'"
However I still get this issue. Any idea what is wrong here?
$ apk add python3
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz
(1/11) Installing libbz2 (1.0.8-r4)
(2/11) Installing libexpat (2.5.0-r0)
(3/11) Installing libffi (3.4.4-r0)
(4/11) Installing gdbm (1.23-r0)
(5/11) Installing xz-libs (5.2.9-r0)
(6/11) Installing libgcc (12.2.1_git20220924-r4)
(7/11) Installing libstdc++ (12.2.1_git20220924-r4)
(8/11) Installing mpdecimal (2.5.1-r1)
(9/11) Installing readline (8.2.0-r0)
(10/11) Installing sqlite-libs (3.40.1-r0)
(11/11) Installing python3 (3.10.9-r1)
Executing busybox-1.35.0-r29.trigger
OK: 65 MiB in 34 packages
$ apk add python3-pip
ERROR: unable to select packages:
python3-pip (no such package):
required by: world[python3-pip]
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1
As per the comment made by β.εηοιτ.βε, I changed the name of the pip3 package to py3-pip to be in line with the expected package name in alpine instead of python3-pip which is Debian.
before_script:
- apk add python3
- apk add py3-pip
- pip3 install awscli==1.18.8
- docker load --input data/image.tar
- $(aws ecr get-login --no-include-email --region us-east-1)
https://pkgs.alpinelinux.org/package/v3.17/community/armv7/py3-pip
Related
I am trying to build a multi-architecture image for ARM64 using buildx command. I am using python:3.8-slim as a base image and trying to install catboost using pip, but I am getting the following error.
ERROR
ERROR: Could not find a version that satisfies the requirement catboost==1.0.4 (from versions: none)
ERROR: No matching distribution found for catboost==1.0.4
Dockerfile
FROM --platform=linux/arm64/v8 python:3.8-slim
RUN apt update && \
apt upgrade -y && \
pip install -U pip && \
pip install --upgrade setuptools
steps...
Command
docker buildx build --platform linux/arm64 -t dockerId:test-arm -f ./dockerfiles/Dockerfile .
Alternatives I have tried
In the Dockerfile I tried using the --platform=linux/amd64 which builds the arm64 successfully but still doesn't work when I deploy it on the ARM machine.
I have used anaconda for installing the packages but the error to install catboost remains the same.
#0 122.6 PackagesNotFoundError: The following packages are not available from current channels:
#0 122.6
#0 122.6 - catboost
This is a shoutout to the #catboost team. I am currently working on a deadline and would like to know if the catboost has support for ARM64 over a docker image.
I am using windows 10 OS. I want to build an container based on linux so I can replicate code and dependencies developed from ubuntu. When I try to build it outputs Error message as above.
From my understanding docker for desktop runs linux OS kernel under-the-hood therefore allowing window users to run linux based containers, not sure why it is outputting this error.
My dockerfile looks like this:
FROM ubuntu:18.04
ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"
RUN apt update \
&& apt install -y htop python3-dev wget
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
&& mkdir root/.conda \
&& sh Miniconda3-latest-Linux-x86_64.sh -b \
&& rm -f Miniconda3-latest-Linux-x86_64.sh
RUN conda create -y -n ml python=3.7
COPY . src/
RUN /bin/bash -c "cd src \
&& source activate ml \
&& pip install -r requirements.txt"
requirements.txt contains:
apturl==0.5.2
asn1crypto==0.24.0
bleach==2.1.2
Brlapi==0.6.6
certifi==2020.11.8
chardet==3.0.4
click==7.1.2
command-not-found==0.3
configparser==5.0.1
cryptography==2.1.4
cupshelpers==1.0
dataclasses==0.7
When I run docker build command it outputs:
1.649 ERROR: Could not find a version that satisfies the requirement apturl==0.5.2 1.649 ERROR: No matching distribution found for apturl==0.5.2 Deleting it and running it lead to another error. All error seem to be associated with ubuntu packages.
Am I not running a ubuntu container? why aren't I allowed to install ubuntu packages?
Thanks!
You try to install ubuntu packages with pip (which is for python packages")
try apt install -y apturl
If you want to install python packages write pip install package_name
I am trying to implement some cicd using GitLab runner,
I am very new to containers and trying to install the zip package in the container,
I was able to install awscli using pip but, I am not able to install the zip package, which is required for my shell script.
Following is the .gitlab-ci.yml file -
stages:
- build
build:
image: python:latest
stage: build
script:
- pip install awscli
- yum install zip
- bash cicdScript.sh
I'm using the python container, as my script requires awscli,
But also needs the zip package,
I tried the following -
1)
script:
- pip install awscli
- yum install zip
- bash cicdScript.sh
gives -
/bin/bash: line 82: yum: command not found
2)
script:
- pip install awscli
- apt-get install zip unzip
- bash cicdScript.sh
gives -
Reading package lists...
Building dependency tree...
Reading state information...
Package zip is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
E: Package 'zip' has no installation candidate
Try update and -y
apt-get update
apt-get install -y zip unzip
So, after building out a pipeline, I realized I will need some custom libraries for a python script I will be pulling from SCM. To install Jenkins in Docker, I used the following tutorial:
https://jenkins.io/doc/book/installing/
Like so:
docker run \
-u root \
--rm \
-d \
-p 8080:8080 \
-p 50000:50000 \
-v jenkins-data:/var/jenkins_home \
-v /var/run/docker.sock:/var/run/docker.sock \
jenkinsci/blueocean
Now, I will say I'm not a Docker guru, but I'm aware the Dockerfile allows for passing in library installs for Python. However, because I'm pulling the docker image from dockerhub, I'm not sure if it's possible to add a "RUN pip install " as an argument. Maybe there is an alternate approach someone may have.
Any help is appreciated.
EDIT 1: Here's the output of the first commenter's recommendation:
Step 1/6 : FROM jenkinsci/blueocean
---> b7eef16a711e
Step 2/6 : USER root
---> Running in 150bba5c4994
Removing intermediate container 150bba5c4994
---> 882bcec61ccf
Step 3/6 : RUN apt-get update
---> Running in 324f28f384e0
/bin/sh: apt-get: not found
The command '/bin/sh -c apt-get update' returned a non-zero code: 127
Error:
/bin/sh: apt-get: not found
The command '/bin/sh -c apt-get update' returned a non-zero code: 127
Observation:
This error comes when the container that you want to run is not Debian based, hence does not support 'apt'.
To resolve this, we need to find out which package manager it utilizes.
In my case it was: 'apk'.
Resolution:
Replace 'apt-get' with 'apk' in your Dockerfile. (If this does not work you can try 'yum' package manager as well).
Command in your Dockerfile should look like:
RUN apk update
You can create a Dockerfile
FROM jenkins:latest
USER root
RUN apt-get update
RUN apt-get install -y python-pip
# Install app dependencies
RUN pip install --upgrade pip
You can build the custom image using
docker build -t jenkinspython .
Similar to what Hemant Sing's answer, but 2 slightly different things.
First, create a unique directory: mkdir foo
"cd" to that directory and run:
docker build -f jenkinspython .
Where jenkinspython contains:
FROM jenkins:latest
USER root
RUN apt-get update
RUN apt-get install -y python-pip
# Install app dependencies
RUN pip install --upgrade pip
Notice that my change has -f, not -t. And notice that the build output does indeed contain:
Step 5/5 : RUN pip install --upgrade pip
---> Running in d460e0ebb11d
Collecting pip
Downloading https://files.pythonhosted.org/packages/5f/25/e52d3f31441505a5f3af41213346e5b6c221c9e086a166f3703d2ddaf940/pip-18.0-py2.py3-none-any.whl (1.3MB)
Installing collected packages: pip
Found existing installation: pip 9.0.1
Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-18.0
Removing intermediate container d460e0ebb11d
---> b7d342751a79
Successfully built b7d342751a79
So now that the image has been built (in my case, b7d342751a79), fire it up and verify that pip has indeed been updated:
$ docker run -it b7d342751a79 bash
root#9f559d448be9:/# pip --version
pip 18.0 from /usr/local/lib/python2.7/dist-packages/pip (python 2.7)
So now your image has pip installed, so then you can feel free to pip install whatever crazy packages you need :)
I have an image called: Image and a running container called: container.
I want to install pytorch and anaconda. What's the easiest way to do this?
Do I have to change the Dockerfile and build a new image?
Thanks a lot.
Yes, the best thing is to build your image in such a way it has the python modules are in there.
Here is an example. I build an image with the build dependencies:
$ docker build -t oz123/alpine-test-mycoolapp:0.5 - < Image
Sending build context to Docker daemon 2.56 kB
Step 1 : FROM alpine:3.5
---> 88e169ea8f46
Step 2 : ENV MB_VERSION 3.1.4
---> Running in 4587d36fa4ae
---> b7c55df49803
Removing intermediate container 4587d36fa4ae
Step 3 : ENV CFLAGS -O2
---> Running in 19fe06dcc314
---> 31f6a4f27d4b
Removing intermediate container 19fe06dcc314
Step 4 : RUN apk add --no-cache python3 py3-pip gcc python3-dev py3-cffi file git curl autoconf automake py3-cryptography linux-headers musl-dev libffi-dev openssl-dev build-base
---> Running in f01b60b1b5b9
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/community/x86_64/APKINDEX.tar.gz
(1/57) Upgrading musl (1.1.15-r5 -> 1.1.15-r6)
(2/57) Upgrading zlib (1.2.8-r2 -> 1.2.11-r0)
(3/57) Installing m4 (1.4.17-r1)
(4/57) Installing perl (5.24.0-r0)
(5/57) Installing autoconf (2.69-r0)
(6/57) Installing automake (1.15-r0)
(7/57) Installing binutils-libs (2.27-r1)
...
Note, I am installing Python's pip inside the image, so later I can download packages from pypi. Packages like numpy might require a C compiler and tool chain, so I am installing these too.
After building the packages which require the build tools chain I remove the tool chain packages:
RUN apk del file pkgconf autoconf m4 automake perl g++ libstdc++
After you have your base image, you can run your application code in
an image building on top of it:
$ cat Dockerfile
FROM oz123/alpine-test-mycoolapp
ADD . /code
WORKDIR /code
RUN pip3 install -r requirements.txt -r requirements_dev.txt
RUN pip3 install -e .
RUN make clean
CMD ["pytest", "-vv", "-s"]
I simply run this with docker.