Docker: files missing after build - python-3.x

I'm trying to build a docker container that runs a Python script. I want the code to be cloned from git when I build the image. I'm using this docker file as a base and added the following BEFORE the first line:
FROM debian:buster-slim AS intermediate
RUN apt-get update
RUN apt-get install -y git
ARG SSH_PRIVATE_KEY
RUN mkdir /root/.ssh/
RUN echo "${SSH_PRIVATE_KEY}" > /root/.ssh/id_rsa
RUN chmod 600 /root/.ssh/id_rsa
RUN touch /root/.ssh/known_hosts
RUN ssh-keyscan [git hostname] >> /root/.ssh/known_hosts
RUN git clone git#...../myApp.git
... then added the following directly after the first line:
# Copy only the repo from the intermediate image
COPY --from=intermediate /myApp /myApp
... then at the end I added this to install some dependencies:
RUN set -ex; \
apt-get update; \
apt-get install -y gcc g++ unixodbc-dev libpq-dev; \
\
pip install pyodbc; \
pip install paramiko; \
pip install psycopg2
And I changed the command to run to:
CMD ["python3 /myApp/main.py"]
If, at the end of the dockerfile before the CMD, I add the command "RUN ls -l /myApp" it lists all the files I would expect during the build. But when I use "docker run" to run the image, it gives me the following error:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: "python3 /myApp/main.py": stat python3 /myApp/main.py: no such file or directory": unknown.
My build command is:
docker build --file ./Dockerfile --tag my_app --build-arg SSH_PRIVATE_KEY="$(cat sshkey)" .
Then run with docker run my_app
There is probably some docker fundamental that I am misunderstanding, but I can't seem to figure out what it is.

This is hard to answer without your command line or your docker-compose.yml (if any). A recurrent mistake is to map a volume from the host into the container at a non empty location, in this case, your container files are hidden by the content of the host folder.
The last CMD should be like this:
CMD ["python3", "/myApp/main.py"]

Related

Unable to install package using docker : kpt not found or ambiguous repo/dir#version specify '.git' in argument

I am using below script and it giving me an error #/bin/sh: 1: kpt: not found
FROM nginx
RUN apt update
RUN apt -y install git
RUN apt -y install curl
# install kpt package
RUN mkdir -p ~/bin
RUN curl -L https://github.com/GoogleContainerTools/kpt/releases/download/v1.0.0-beta.1/kpt_linux_amd64 --output ~/bin/kpt && chmod u+x ~/bin/kpt
RUN export PATH=${HOME}/bin:${PATH}
RUN SRC_REPO=https://github.com/kubeflow/manifests
RUN kpt pkg get $SRC_REPO/tf-training#v1.1.0 tf-training
But if I create the image using
FROM nginx
RUN apt update
RUN apt -y install git
RUN apt -y install curl
and perform
docker exec -it container_name bash
and manually do the task then I am able to install kpt package. Sharing below the screenshot of the process
The error changes if I provide the full path to /bin/kpt
Error: ambiguous repo/dir#version specify '.git' in argument
FROM nginx
RUN apt update
RUN apt -y install git
RUN apt -y install curl
RUN mkdir -p ~/bin
RUN curl -L https://github.com/GoogleContainerTools/kpt/releases/download/v1.0.0-beta.1/kpt_linux_amd64 --output ~/bin/kpt && chmod u+x ~/bin/kpt
RUN export PATH=${HOME}/bin:${PATH}
# Below line of code is to ensure that kpt is installed and working fine
RUN ~/bin/kpt pkg get https://github.com/ajinkya101/kpt-demo-repo.git/Packages/Nginx
RUN SRC_REPO=https://github.com/kubeflow/manifests
RUN ~/bin/kpt pkg get $SRC_REPO/tf-training#v1.1.0 tf-training
What is happening when I am using docker and not able to install it?
First, make sure SRC_REPO is declared as a Dockerfile environment variable
ENV SRC_REPO=https://github.com/kubeflow/manifests.git
^^^ ^^^^
And make sure the URL ends with .git.
As mentioned in kpt get:
In most cases the .git suffix should be specified to delimit the REPO_URI from the PKG_PATH, but this is not required for widely recognized repo prefixes.
Second, to be sure, specify the full path of kpt, without ~ or ${HOME}.
/root/bin/kpt
For testing, add a RUN id -a && pwd to be sure who and where you are when using the nginx image.

Nextcloud docker install with SSH access enabled

I’m trying to install SSH (and enable the service) on top of my Nextcloud installation in Docker, and have it work on reboot. Having run through many Dockerfile, docker-compose combinations I can’t seem to get this to work. Ive tried using entrypoint.sh scripts with Dockerfile, but it wants a CMD at the end and then it doesn’t execute the “normal” nextcloud start up.
entrypoint.sh:
#!/bin/sh
# Start the ssh server
service ssh start
# Execute the CMD
exec "$#"
Dockerfile:
FROM nextcloud:latest
RUN apt update -y && apt-get install ssh -y
RUN apt-get install python3 -y && apt-get install sudo -y
RUN echo 'ansible ALL=(ALL:ALL) NOPASSWD:ALL' >> /etc/sudoers
RUN useradd -m ansible -s /bin/bash
RUN sudo -u ansible mkdir /home/ansible/.ssh
RUN mkdir -p /var/run/sshd
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
CMD ["/usr/sbin/sshd", "-D"]
Any help would be much appreciated. Thank you
In general I'd say - break the problem you're having down into smaller parts - it'll help isolate the source of the problem.
Here's how I'd approach the reported issue.
First - replace (in your Dockerfile)
apt-get install -y ssh
with the recommended
apt install -y openssh-server
Then - test just the required parts of your Dockerfile addressing the issue - simplify it just to the following:
FROM nextcloud:latest
RUN apt update
RUN apt install -y openssh-server
Then build a test image using this Dockerfile via the command
docker build . -t test_nextcloud
This will build the image - giving it the name (tag) of test_nextcloud.
Then run a container from this newly built image via the docker run command
docker run -p 8080:80 -d --name nextcloud test_nextcloud
This will run the container on port 8080 in detatched mode, and give the assicated container the name of nextcloud.
Then - with the container running - you should be able to enter into it using the following command
docker container exec -u 0 -it nextcloud bash
as root.
Now that you are in, you should be able to startup the ssh server via the command
service ssh start
Having followed a set of steps like this to confirm that you can indeed startup an ssh server in the nextcloud container, begin adding back in your additional logic (begining with the original Dockerfile).

Getting too many dangling images on docker-compose up

I want to integrate python in dotnet core image as I need to execute python scripts.
When I am executing this DockerFile, lots of dangling images are created.
Dangling Images
Also, is there any proper way to integrate a python interpreter? For example, I will get a URL in the .net core container and then I want to pass that URL to the python container. How can we achieve this task? I am new to Docker.
FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim AS base
RUN apt-get update && apt-get install -y --no-install-recommends wget
RUN apt-get update && apt-get install -y python3.7 \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
RUN mkdir -p tmp
WORKDIR /tmp
RUN wget https://github.com/projectdiscovery/subfinder/releases/download/v2.4.4/subfinder_2.4.4_linux_amd64.tar.gz
RUN tar -xvf subfinder_2.4.4_linux_amd64.tar.gz
RUN mv subfinder /usr/local/bin/
#Cleanup
#wget cleanup
RUN rm -f subfinder_2.4.4_linux_amd64.tar.gz
FROM base AS final
RUN mkdir app
WORKDIR /app
EXPOSE 80
EXPOSE 443
COPY Publish/ /app/
ENTRYPOINT ["dotnet", "OnePenTest.Api.dll"]
As Images are immutable, so any change you would do results in a new image is created because your compose file specifies the build command, so it will rerun the build command when you start the containers.
And if any files you're composing with a COPY change, then the current image cache is no longer used and it'll build a new image without erasing the old image.
Can use the below commands to build your file
sudo docker-compose build --force-rm [--build-arg key=val...] [SERVICE...]
docker build --rm -t <tag>
The option --force-rm removes intermediate containers after a successful build.
Just configure your docker-compose.yaml then running this with this commands:
$ docker compose down --remove-orphans # if needed to stop all running containers
$ docker compose build
$ docker compose up --no-build
Add the flag --no-build in docker compose up.
ref: https://docs.docker.com/engine/reference/commandline/compose_up/

Docker run only works after build

I can build and run a container with
docker build -t hopperweb:v5-full -f Dockerfile . &&
docker run -p 127.0.0.1:3000:8080 --rm -ti hopperweb:v5-full
However when I run the container I get this error: standard_init_linux.go:211: exec user process caused "exec format error"
docker run -p 127.0.0.1:3000:8080 --rm -ti hopperweb:v5-full
Why is it working when it's run after &&??
I can run the image with bash: docker run -p 127.0.0.1:3000:8080 --rm -ti hopperweb:v5-full bash without issue.
This is my DockerFile
FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install --yes curl
RUN apt-get install --yes sudo ## maybe not necessary, but helpful
RUN apt-get install --yes gnupg
RUN apt-get install --yes git ## not necessary, but helpful
RUN apt-get install --yes vim ## not necessary, but helpful
## INSTALL NPM
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
RUN echo 'deb https://dl.yarnpkg.com/debian/ stable main' | sudo tee /etc/apt/sources.list.d/yarn.list
RUN apt-get update
RUN apt-get install --yes yarn
RUN apt-get install --yes npm
## COPY IN APP FILES
RUN mkdir /app
COPY hopperweb/ /app/hopperweb/
RUN chmod +x /app/hopperweb/start.sh
RUN /app/hopperweb/start.sh
The contents of start.sh:
#!/bin/bash
cd /app/hopperweb/
yarn start
In your first command, the docker run is never executed, as the last command (start.sh) is run during your build and it will never terminate. So you were still running docker build.
Change the following line
RUN /app/hopperweb/start.sh
to
CMD /app/hopperweb/start.sh
Do not confuse RUN with CMD. RUN actually runs a command and commits the result; CMD does not execute anything at build time, but specifies the intended command for the image.
See: https://docs.docker.com/engine/reference/builder/#cmd

Unable to run aliyun-cli in Docker:stable container after installing it. Errors as command not found

I am unsure if stack overflow or system fault is the right stack exchange site but I'm going with stack overflow cause the alicloud site posted to add a tag and ask a question here.
So. I'm currently building an image based on Docker:stable, that is an alpine distro, that will have aliyun-cli installed and available for use. However I am getting a weird error of Command Not Found when I'm running it. I have followed the guide here https://partners-intl.aliyun.com/help/doc-detail/139508.htm and moved the aliyun binary to /usr/sbin
Here is my Dockerfile for example
FROM docker:stable
RUN apk update && apk add curl
#Install python 3
RUN apk update && apk add python3 py3-pip
#Install AWS Cli
RUN pip3 install awscli --upgrade
# Install Aliyun CLI
RUN curl -L -o aliyun-cli.tgz https://aliyuncli.alicdn.com/aliyun-cli-linux-3.0.30-amd64.tgz
RUN tar -xzvf aliyun-cli.tgz
RUN mv aliyun /usr/bin
RUN chmod +x /usr/bin/aliyun
RUN rm aliyun-cli.tgz
However when i'm running aliyun (which can be auto-completed) I am getting this
/ # aliyun
sh: aliyun: not found
I've tried moving it to other bins. Cding into the folder and calling it explicitly but still always getting a command not found. Any suggestions would be welcome.
Did you check this Dockerfile?
Also why you need to install aws-cli in the same image and why you will need to maintain it for your self when AWS provide managed aws-cli image.
docker run --rm -it amazon/aws-cli --version
that's it for aws-cli image,but if you want in existing image then you can try
RUN pip install awscli --upgrade
DockerFile
FROM python:2-alpine3.8
LABEL com.frapsoft.maintainer="Maik Ellerbrock" \
com.frapsoft.version="0.1.0"
ARG SERVICE_USER
ENV SERVICE_USER ${SERVICE_USER:-aliyun}
RUN apk add --no-cache curl
RUN curl https://raw.githubusercontent.com/ellerbrock/docker-collection/master/dockerfiles/alpine-aliyuncli/requirements.txt > /tmp/requirements.txt
RUN \
adduser -s /sbin/nologin -u 1000 -H -D ${SERVICE_USER} && \
apk add --no-cache build-base && \
pip install aliyuncli && \
pip install --no-cache-dir -r /tmp/requirements.txt && \
apk del build-base && \
rm -rf /tmp/*
USER ${SERVICE_USER}
WORKDIR /usr/local/bin
ENTRYPOINT [ "aliyuncli" ]
CMD [ "--help" ]
build and run
docker build -t aliyuncli .
docker run -it --rm aliyuncli
output
docker run -it --rm abc aliyuncli
usage: aliyuncli <command> <operation> [options and parameters]
<aliyuncli> the valid command as follows:
batchcompute | bsn
bss | cms
crm | drds
ecs | ess
ft | ocs
oms | ossadmin
ram | rds
risk | slb
ubsms | yundun
After a lot of lookup I found a github issue in the official aliyun-cli that sort of describes that it is not compatible with alpine linux because of it's not muslc compatible.
Link here: https://github.com/aliyun/aliyun-cli/issues/54
Following the workarounds there I build a multi-stage docker file with the following that simply fixed my issue.
Dockerfile
#Build aliyun-cli binary ourselves because of issue
#in alpine https://github.com/aliyun/aliyun-cli/issues/54
FROM golang:1.13-alpine3.11 as cli_builder
RUN apk update && apk add curl git make
RUN mkdir /srv/aliyun
WORKDIR /srv/aliyun
RUN git clone https://github.com/aliyun/aliyun-cli.git
RUN git clone https://github.com/aliyun/aliyun-openapi-meta.git
ENV GOPROXY=https://goproxy.cn
WORKDIR aliyun-cli
RUN make deps; \
make testdeps; \
make build;
FROM docker:19
#Install python 3 & jq
RUN apk update && apk add python3 py3-pip python3-dev jq
#Install AWS Cli
RUN pip3 install awscli --upgrade
# Install Aliyun CLI from builder
COPY --from=cli_builder /srv/aliyun/aliyun-cli/out/aliyun /usr/bin
RUN aliyun configure set --profile default --mode EcsRamRole --ram-role-name build --region cn-shanghai

Resources