Title is the question.
I'm hosting many docker containers on a rather large linux ec2 instance. One container in particular needs access to a file that gets transferred to the host before run time. The file in question is copied from a windows file server to the ec2 instance using control-m.
When the container image runs, we give it -v to specify a volume mount with a path on the host to that transferred file.
The file is not found in the container. If I make a new file in the container, the new file appears on the host. When I make a file on the host, it appears in the container. When I make a copy of the transferred file using cp -p the copied file DOES show up in the container, but the original still does not.
I don't understand why this is? My suspicion is something to do with it being on a windows server before control-m copies it to the ec2 instance.
Details:
The file lives in the path /folder_path/project_name/resources/file.txt
Its permissions are -rwxrwxr-x 1 pyadmin pyadmin where pyadmin maps to the containers root user.
It's approximately 38mb in size and when I run file file.txt I get the output ASCII text, with CRLF line terminators.
The repo also has a resources folder with files already in it when it is cloned, but none of their names conflict.
Docker Version: 20.10.13
Dockerfile:
FROM python:3.9.11-buster
SHELL ["/bin/bash", "-c"]
WORKDIR /folder_path/project_name
RUN apt-get auto-clean && apt-get update && apt-get install -y unixodbc unixodbc-dev && apt-get upgrade -y
RUN python -m pip install --upgrade pip poetry
COPY . .
RUN python -m pip install --upgrade pip poetry && \
poetry config virtualenvs.create false && \
poetry install
ENTRYPOINT [ "python" ]
Command to start container:
docker run --pull always --rm \
-v /folder_path/project_name/logs:/folder_path/project_name/logs \
-v /folder_path/project_name/extracts:/folder_path/project_name/extracts \
-v /folder_path/project_name/input:/folder_path/project_name/input \
-v /folder_path/project_name/output:/folder_path/project_name/output \
-v /folder_path/project_name/resources:/folder_path/project_name/resources \
my-registry.com/folder_path/project_name:image_tag
Related
I am having trouble with azure and docker where my local machine image is behaving differently than the image I push to ACR. while trying to deploy to web, I get this error:
ERROR - failed to register layer: error processing tar file(exit status 1): Container ID 397546 cannot be mapped to a host IDErr: 0, Message: mapped to a host ID
So in trying to fix it, I have come to find out that azure has a limit on uid numbers of 65000. Easy enough, just change ownership of the affected files to root, right?
Not so. I put the following command into my Dockerfile:
RUN chown -R root:root /usr/local/flutter/bin/cache/artifacts/gradle_wrapper/
Works great locally for changing the uids of the affected files from 397546 to 0. I do a command in the cli of the container:
find / -uid 397546
It finds none of the same files it found before. Yay! I even navigate to the directories where the affected files are, and do a quick
ls -n to double confirm they are fine, and sure enough the uids are now 0 on all of them. Good to go?
Next step, push to cloud. When I push and reset the app service, I still continue to get the same exact error above. I have confirmed on multiple fronts that it is indeed pushing the correct image to the cloud.
All of this means that somehow my local image and the cloud image are behaving differently.
I am stumped guys please help.
The Dockerfile is as below:
RUN apt-get update
RUN apt-get install -y curl git wget unzip libgconf-2-4 gdb libstdc++6 libglu1-mesa fonts-droid-fallback lib32stdc++6 python3 psmisc
RUN apt-get clean
# Clone the flutter repo
RUN git clone https://github.com/flutter/flutter.git /usr/local/flutter
# Set flutter path
ENV PATH="/usr/local/flutter/bin:/usr/local/flutter/bin/cache/dart-sdk/bin:${PATH}"
# Enable flutter web
RUN flutter upgrade
RUN flutter config --enable-web
# Run flutter doctor
RUN flutter doctor -v
# Change ownership to root of affected files
RUN chown -R root:root /usr/local/flutter/bin/cache/artifacts/gradle_wrapper/
# Copy the app files to the container
COPY ./build/web /usr/local/bin/app
COPY ./startup /usr/local/bin/app/server
COPY ./pubspec.yaml /usr/local/bin/app/pubspec.yaml
# Set the working directory to the app files within the container
WORKDIR /usr/local/bin/app
# Get App Dependencies
RUN flutter pub get
# Build the app for the web
# Document the exposed port
EXPOSE 4040
# Set the server startup script as executable
RUN ["chmod", "+x", "/usr/local/bin/app/server/server.sh"]
# Start the web server
ENTRYPOINT [ "/usr/local/bin/app/server/server.sh" ]```
So basically we have made a shell script to build web BEFORE building the docker image. we then use the static js from the build/web folder and host that on the server. No need to download all of flutter. Makes pipelines a little harder, but at least it works.
New Dockerfile:
FROM ubuntu:20.04 as build-env
RUN apt-get update && \
apt-get install -y --no-install-recommends apt-utils && \
apt-get -y install sudo
## for apt to be noninteractive
ENV DEBIAN_FRONTEND noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN true
## preesed tzdata, update package index, upgrade packages and install needed software
RUN echo "tzdata tzdata/Areas select US" > /tmp/preseed.txt; \
echo "tzdata tzdata/Zones/US select Colorado" >> /tmp/preseed.txt; \
debconf-set-selections /tmp/preseed.txt && \
apt-get update && \
apt-get install -y tzdata
RUN apt-get install -y curl git wget unzip libstdc++6 libglu1-mesa fonts-droid-fallback lib32stdc++6 python3 python3 nginx nano vim
RUN apt-get clean
# Copy files to container and build
RUN mkdir /app/
COPY . /app/
WORKDIR /app/
RUN cd /app/
# Configure nginx and remove secret files
RUN mv /app/build/web/ /var/www/html/patient
RUN cd /etc/nginx/sites-enabled
RUN cp -f /app/default /etc/nginx/sites-enabled/default
RUN cd /app/ && rm -r .dart_tool .vscode assets bin ios android google_place lib placepicker test .env .flutter-plugins .flutter-plugins-dependencies .gitignore .metadata analysis_options.yaml flutter_01.png pubspec.lock pubspec.yaml README.md
# Record the exposed port
EXPOSE 5000
# Start the python server
RUN ["chmod", "+x", "/app/server/server.sh"]
ENTRYPOINT [ "/app/server/server.sh"]
I have following problem.
Here is Dockerfile with Python app where I need to install sshfs to mount files from the sftp server via ssh.
# Set base image
FROM python:3.9
# Copy files
COPY id_rsa requirements.txt app.py /app/
COPY known_hosts /root/.ssh/
COPY ssh_config /etc/ssh
# Set working directory
WORKDIR app
# Install libraries
RUN pip install -U pip \
&& pip install -r requirements.txt \
&& mkdir source \
&& chmod 600 id_rsa \
&& apt-get -y upgrade \
&& apt-get -y update
COPY fuse.conf /etc/
RUN dpkg --configure -a \
&& apt-get install -y sshfs \
&& sshfs user#ip_address:/C:/folder_name /app/source -o IdentityFile=/app/id_rsa,auto_cache,reconnect,transform_symlinks,follow_symlinks
# Run deamon
CMD ["python", "app.py"]
When I build Docker Image, it gives me the error
Configuration file '/etc/fuse.conf'
==> File on system created by you or by a script.
==> File also in package provided by package maintainer.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** fuse.conf (Y/I/N/O/D/Z) [default=N] ? dpkg: error processing package fuse (--configure):
end of file on stdin at conffile prompt
Processing triggers for libc-bin (2.31-13+deb11u2) ...
Errors were encountered while processing:
fuse
Here is how my fuse.conf looks like:
# The file /etc/fuse.conf allows for the following parameters:
#
# user_allow_other - Using the allow_other mount option works fine as root, in
# order to have it work as user you need user_allow_other in /etc/fuse.conf as
# well. (This option allows users to use the allow_other option.) You need
# allow_other if you want users other than the owner to access a mounted fuse.
# This option must appear on a line by itself. There is no value, just the
# presence of the option.
user_allow_other
# mount_max = n - this option sets the maximum number of mounts.
# Currently (2014) it must be typed exactly as shown
# (with a single space before and after the equals sign).
mount_max = 1000
I don't know whether the problem is that somehow DEFAULT=N is not taken when the image is built?
For the evidence, I can run docker in the interactive mode, I can install sshfs inside and mount it without problems (--privileged moded).
I am new in azure portal. I push a docker image to azure container resgistry and then I create an web app where I use that image. Now want to see all my data in the image or I need that data because every time my app execute it create some new file so I need these files or I want to view these files.
can someone please tell me how can I do that?`
This is my docker file.
FROM continuumio/miniconda3:latest
RUN apt-get update && apt-get install -y \
ca-certificates \
curl
ARG NODE_VERSION=14.16.0
ARG NODE_PACKAGE=node-v$NODE_VERSION-linux-x64
ARG NODE_HOME=/opt/$NODE_PACKAGE
ENV NODE_PATH $NODE_HOME/lib/node_modules
ENV PATH $NODE_HOME/bin:$PATH
RUN curl https://nodejs.org/dist/v$NODE_VERSION/$NODE_PACKAGE.tar.gz | tar -xzC /opt/
COPY environment.yml .
COPY requirements.txt .
RUN pip install -r requirements.txt
RUN conda install -c conda-forge librosa
WORKDIR /with
RUN mkdir /app1
COPY model /app1/model
COPY assets /app1/assets
COPY build /app1/build
COPY node_modules /app1/node_modules
RUN apt-get install -y openssh \
&& echo "root:Docker!" | chpasswd
# Copy the sshd_config file to the /etc/ssh/ directory
COPY sshd_config /etc/ssh/
EXPOSE 8080 2222
# CMD ["python", "/app1/model/main.py", "/app1/assets/uploads/file.txt"]
CMD ["node", "/app1/build/server.js"]
It seems you push your image into the ACR and then deploy it to the Web App. If you want to see the data inside the image, here you'd better call it container, you need to ssh into the running container in Web App. To do this, you need to make sure that the ssh was enabled in the image. If not, follow the steps here to enable it. If the ssh already enabled, then you can follow the steps here to ssh into the container to see the data as you want.
I'm trying to build a docker container that runs a Python script. I want the code to be cloned from git when I build the image. I'm using this docker file as a base and added the following BEFORE the first line:
FROM debian:buster-slim AS intermediate
RUN apt-get update
RUN apt-get install -y git
ARG SSH_PRIVATE_KEY
RUN mkdir /root/.ssh/
RUN echo "${SSH_PRIVATE_KEY}" > /root/.ssh/id_rsa
RUN chmod 600 /root/.ssh/id_rsa
RUN touch /root/.ssh/known_hosts
RUN ssh-keyscan [git hostname] >> /root/.ssh/known_hosts
RUN git clone git#...../myApp.git
... then added the following directly after the first line:
# Copy only the repo from the intermediate image
COPY --from=intermediate /myApp /myApp
... then at the end I added this to install some dependencies:
RUN set -ex; \
apt-get update; \
apt-get install -y gcc g++ unixodbc-dev libpq-dev; \
\
pip install pyodbc; \
pip install paramiko; \
pip install psycopg2
And I changed the command to run to:
CMD ["python3 /myApp/main.py"]
If, at the end of the dockerfile before the CMD, I add the command "RUN ls -l /myApp" it lists all the files I would expect during the build. But when I use "docker run" to run the image, it gives me the following error:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: "python3 /myApp/main.py": stat python3 /myApp/main.py: no such file or directory": unknown.
My build command is:
docker build --file ./Dockerfile --tag my_app --build-arg SSH_PRIVATE_KEY="$(cat sshkey)" .
Then run with docker run my_app
There is probably some docker fundamental that I am misunderstanding, but I can't seem to figure out what it is.
This is hard to answer without your command line or your docker-compose.yml (if any). A recurrent mistake is to map a volume from the host into the container at a non empty location, in this case, your container files are hidden by the content of the host folder.
The last CMD should be like this:
CMD ["python3", "/myApp/main.py"]
Running the following docker command on mac works and on linux, running ubuntu cannot find the aws cli credentials. It returns the following message: Unable to locate credentials
Completed 1 part(s) with ... file(s) remaining
The command which runs an image and mounts a data volume and then copies a file from and s3 bucket, and starts the bash shell in the docker container.
sudo docker run -it --rm -v ~/.aws:/root/.aws username/docker-image sh -c 'aws s3 cp s3://bucketname/filename.tar.gz /home/emailer && cd /home/emailer && tar zxvf filename.tar.gz && /bin/bash'
What am I missing here?
This is my Dockerfile:
FROM ubuntu:latest
#install node and npm
RUN apt-get update && \
apt-get -y install curl && \
curl -sL https://deb.nodesource.com/setup | sudo bash - && \
apt-get -y install python build-essential nodejs
#install and set-up aws-cli
RUN sudo apt-get -y install \
git \
nano \
unzip && \
curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" && \
unzip awscli-bundle.zip
RUN sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
# Provides cached layer for node_modules
ADD package.json /tmp/package.json
RUN cd /tmp && npm install
RUN mkdir -p /home/emailer && cp -a /tmp/node_modules /home/emailer/
Mounting $HOME/.aws/ into the container should work. Make sure to mount it as read-only.
It is also worth mentioning, if you have several profiles in your ~/.aws/config -- you must also provide the AWS_PROFILE=somethingsomething environment variable. E.g. via docker run -e AWS_PROFILE=xxx ... otherwise you'll get the same error message (unable to locate credentials).
Update: Added example of the mount command
docker run -v ~/.aws:/root/.aws …
You can use environment variable instead of copying ~/.aws/credentials and config file into container for aws-cli
docker run \
-e AWS_ACCESS_KEY_ID=AXXXXXXXXXXXXE \
-e AWS_SECRET_ACCESS_KEY=wXXXXXXXXXXXXY \
-e AWS_DEFAULT_REGION=us-west-2 \
<img>
Ref: AWS CLI Doc
what do you see if you run
ls -l ~/.aws/config
within your docker instance?
the only solution that worked for me in this case is:
volumes:
- ${USERPROFILE}/.aws:/root/.aws:ro
There are a few things that could be wrong. One, as mentioned previously you should check if your ~/.aws/config file is set accordingly. If not, you can follow this link to set it up. Once you have done that you can map the ~/.aws folder using the -v flag on docker run.
If your ~/.aws folder is mapped correctly, make sure to check the permissions on the files under ~/.aws so that they are able to be accessed safely by whatever process is trying to access them. If you are running as the user process, simply running chmod 444 ~/.aws/* should do the trick. This will give full read permissions to the file. Of course, if you want write permissions you can add whatever other modifiers you need. Just make sure the read octal is flipped for your corresponding user and/or group.
The issue I had was that I was running Docker as root. When running as root it was unable to locate my credentials at ~/.aws/credentials, even though they were valid.
Directions for running Docker without root on Ubuntu are here: https://askubuntu.com/a/477554/85384
You just have to pass the credential in order to be the AWS_PROFILE, if you do not pass anything it will use the default, but if you want you can copy the default and add your desired credentials.
In Your credentials
[profile_dev]
aws_access_key_id = xxxxxxxxxxxxxxxxxxxxxxxx
aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
output = json
region = eu-west-1
In Your docker-compose
version: "3.8"
services:
cenas:
container_name: cenas_app
build: .
ports:
- "8080:8080"
environment:
- AWS_PROFILE=profile_dev
volumes:
- ~/.aws:/app/home/.aws:ro