Running ls -d to list directories, print directories in different order if trailing / is present in file name. Why is that? What sorting rules apply? and why does this happen only with docker?
With trailing /
$ docker run --rm ubuntu:16.04 /bin/bash -c "mkdir foo ; mkdir foo-bar ; ls -d foo/ foo-bar/"
foo-bar/
foo/
Without trailing /
$ docker run --rm -it ubuntu:16.04 /bin/bash -c "mkdir foo ; mkdir foo-bar ; ls -d foo foo-bar"
foo
foo-bar
I found out I get the same behavior using sort command
docker run --rm ubuntu:16.04 /bin/bash -c "echo -e 'foo/\nfoo-bar/' | sort"
But the sorting order changes when using sort -d
docker run --rm ubuntu:16.04 /bin/bash -c "echo -e 'foo/\nfoo-bar/' | sort -d"
Thanks to David for pointing me in the right direction, this is caused by the locale settings as described here
On bare ubuntu container, POSIX locale is used which has different sorting rules then en_US.
I solved my problem by installing en_US locale in the docker image, and sorting works as expected again.
Related
I have a linux system running with several Docker containers. One of them is mosquitto container which runs from mosquitto 1.6.7 docker image.
I do not have control how the Mosquitto container is created as it is given by default from a supplier/client.
I need to make changes in the mosquitto/config/mosquitto.conf file. This is the output when I run ls -l
/mosquitto/config # ls -l
total 4
-rwxrwxr-x 1 nobody nobody 210 May 24 05:35 mosquitto.conf
I tried the codes below to add a comment in the mosquitto.conf, but I am not successful.
/mosquitto/config # echo '#test' | su nobody -c 'tee -a mosquitto.conf'
nologin: this account is not available
/mosquitto/config # echo '#test' | su nobody -s sh -c 'tee -a mosquitto.conf'
su: can't execute 'sh': No such file or directory
/mosquitto/config # echo '#test' | su nobody -s bin/sh -c 'tee -a mosquitto.conf'
su: can't execute 'bin/sh': No such file or directory
/mosquitto/config # echo '#test' | su nobody -s /bin/sh -c 'tee -a mosquitto.conf'
tee: mosquitto.conf: Permission denied
#test
Is it possible to change the mosquitto.conf?
If yes, how? Thanks.
You don't.
You make a copy of it on the host machine, edit there and then mount that edited copy into the container when you start it.
e.g.
docker run -d -v /path/to/local/mosquitto.conf:/mosquitto/config/mosquitto.conf mosquitto
Basically, I need to keep the functionality of an ubuntu:18.04 image but with some environment variables set every time I execute a docker run or a docker exec this variables are dynamic, so I can't use the keyword ENV in the Dockerfile, I will need to use a script that should be sourced, for simplicity the file I will be using for this post is:
$ cat setenv.sh
#!/usr/bin/env bash
# Set some dynamic variables
export TEST="Hello World"
I have tried different approaches without success, here is my research:
Using an entrypoint
The files I used for this example:
$ cat entrypoint.sh
#!/usr/bin/env bash
echo "Setting environment"
. /setenv.sh
exec $#
$ cat Dockerfile
FROM ubuntu:18.04
COPY setenv.sh /
COPY entrypoint.sh /
ENTRYPOINT [ "/entrypoint.sh" ]
I built this Dockerfile the following command: docker build -f Dockerfile -t test_img .
This works fine except by two problems:
1. exec does not support double ampersand && nor pipes | nor escaping chars \
As I previously stated, I require my container to have the same functionality as the ubuntu image, for example, in ubuntu I can totally execute the following container:
$ docker run --rm ubuntu:18.04 bash -c "echo \"Hello World\" && ls | head -n1 "
Hello World
bin
But if I use the image I created:
$ docker run --rm test_img bash -c "echo \"Hello World\" && ls | head -n1"
Setting environment
It truncates the command every time it finds a quote (doesn't honor the escape character) a double ampersand or a pipe, here is an example of the commands in different order:
$ docker run --rm ubuntu:18.04 bash -c "ls | head -n1 && echo \"Hello World\""
bin
Hello World
$ docker run --rm test_img bash -c "ls | head -n1 && echo \"Hello World\""
Setting environment
bin
boot
dev
entrypoint.sh
etc
home
lib
lib64
media
mnt
opt
proc
root
run
sbin
setenv.sh
srv
sys
tmp
usr
var
In this case, the command truncates when finding the pipe |.
2. Entrypoint is only called for the parent shell.
If I run a ephemeral container I can see that my env variable is there:
$ docker run --rm test_img env | grep TEST
TEST=Hello World
But if I want a keep-alive container, the env var is not set:
$ docker create -ti --name=test test_img bash
e0e5278c46bdcf33195661fac5911326b701586e9a9c638f71a6e08021ee2f57
$ docker start test
test
$ docker exec test env | grep TEST
What is happening here is that the shell I create when running docker create is calling the entrypoint, but the shell I create when running docker exec is a different one.
If you login to the container you can see shells are different:
$ docker exec -ti test bash
root#e0e5278c46bd:/# ps -fe
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 15:21 pts/0 00:00:00 bash
root 15 0 0 15:29 pts/1 00:00:00 bash
root 29 15 0 15:29 pts/1 00:00:00 ps -fe
root#e0e5278c46bd:/# env | grep TEST
If instead of having an entrypoint script to set the environment variable TEST I had used the keyword ENV in my Dockerfile: ENV TEST "Hello World" this would set the variable in every shell created by the commands docker run and docker exec. Here is the example:
$ cat Dockerfile
FROM ubuntu:18.04
ENV TEST "Hello World"
$ docker build -f Dockerfile -t test_img .
Sending build context to Docker daemon 4.096kB
Step 1/2 : FROM ubuntu:18.04
---> 6526a1858e5d
Step 2/2 : ENV TEST "Hello World"
---> Using cache
---> eebe9952bb76
Successfully built eebe9952bb76
Successfully tagged test_img:latest
$ docker create -ti --name=test test_img bash
c1e508dae0f398a40c4c5534cf2811cdfe284a4f6601198f0ca97fdea100c376
$ docker start test
test
$ docker exec test env | grep TEST
TEST=Hello World
$ docker exec -ti test bash
root#c1e508dae0f3:/# env | grep TEST
TEST=Hello World
Sourcing in bashrc
I modify the Dockerfile to look like this, and built the image with the same build command:
$ cat Dockerfile
FROM ubuntu:18.04
COPY setenv.sh /
RUN echo ". /setenv.sh" >> /etc/bash.bashrc
The problem with this approach is the shell used to execute docker run, the bashrc file is not called, only on interactive bash shells, here is the output:
$ docker run --rm test_img echo $SHELL
/bin/bash
$ docker run --rm test_img env | grep TEST
$ docker run --rm test_img bash -c "env" | grep TEST
$ docker run --rm -ti test_img bash
root#1187568e1bec:/# env | grep TEST
TEST=Hello World
First I tried to add the setenv.sh to /etc/profile.d directory, but the problem with this is that /etc/profile is called only for login shells, and I will need to change the commands to explicitly use a login shell, in other words, instead of docker run test_img env I would need it to be docker run test_img bash -lc "env" (The -l is for login).
Create Dockerfile dinamically
This is the best solution so far, but is not the cleaner, I have to have a Dockerfile.pre file to create a container and save the generated variables to a file, then use this file to create a final Dockerfile and write all those ENV lines into the Dockerfile.
Combining two approaches
By using an entrypoint and sourcing in bashrc file I was able to get the variables set in all cases, the problem is the exec $# command that doesn't support full bash scripts. Is any way to modify my entrypoint script? or is there other approach for this problem?
you can create an enviroment file and just pass it to your container with the --env-file flag. This will make all the variables in the file available in the container.
ubuntu#vps-f116ed9f:~$ cat my_env_file
TEST=Hello World
ubuntu#vps-f116ed9f:~$ docker container run -it --rm --env-file my_env_file ubuntu bash -c "echo \$TEST"
Hello World
ubuntu#vps-f116ed9f:~$ docker container run -it --rm --env-file my_env_file ubuntu bash -c "echo \$TEST | wc -c"
12
here you can see i have used the latest ubuntu image, i pass my_env_file to it and then using the bash shell i print the value of this variable (Note i have to escape the $ other wise the shell will interpolate this before passing it to docker, this could be avoided by using single qoutes as the shell wont interpolate variables in single qoutes.)
I also dont see any issues using pipe or &&
ubuntu#vps-f116ed9f:~$ docker container run -it --rm --env-file my_env_file ubuntu bash -c 'ls | head -n1 && echo "$TEST"'
bin
Hello World
This also will persist in detached containers
ubuntu#vps-f116ed9f:~$ docker container run -itd --rm --name=c1 --env-file my_env_file ubuntu bash
3d7705f2f91f3f30c45e855778bd80f08a35616bbe822545c20d5a8886139693
ubuntu#vps-f116ed9f:~$ docker container exec c1 sh -c "ls | head -1 && echo \$TEST"
bin
Hello World
I would like to automatically execute tasks inside a docker container. The task that should be executed should be run inside a specific, mounted directory. To do this, I am using the this command:
docker run --rm -v /a/dir/on/my/host:/tmp some_container /bin/bash -c "cd /tmp/dir/inside/volume && echo \"$PWD\""
, followed by the actual task, which I omit, due to brevity.
PWD should give me /tmp/dir/inside/volume, but prints /a/dir/on/my/host. Why is that?
$PWD is expanded before you run your container. Use single quotes instead of double quotes to defer expansion. Also it's simpler to use --workdir or -w instead of cd .. && and subshell.
docker run --rm -v /a/dir/on/my/host:/tmp some_container /bin/bash -c 'cd /tmp/dir/inside/volume && echo "$PWD"'
or I suggest:
docker run --rm -v /a/dir/on/my/host:/tmp -w /tmp/dir/inside/volume /some_container pwd
I am trying to run a bash command inside docker from host:
$ docker exec -it -u weiss apollo_dev /bin/bash -c "rosbag"
/bin/bash: rosbag: command not found
So I tried:
$ docker exec -it -u weiss apollo_dev /bin/bash -c "echo \$PATH"
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
But when I run docker interactively:
$ docker exec -it -u weiss apollo_dev /bin/bash
weiss#docker$ echo $PATH
/usr/local/cuda-8.0/bin:/home/tmp/ros/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Any reason why I am getting different results for $PATH?
This path is most likely changed in your .bashrc file, and this file is not loaded when the shell is non interactive (see https://www.gnu.org/software/bash/manual/bash.html#Bash-Startup-Files)
So /bin/bash will load it, /bin/bash -c will not
Here you are getting de $PATH of your Host. Before you run the container the variable is replace for the host's $PATH.
$ docker exec -it -u weiss apollo_dev /bin/bash -c "echo \$PATH"
You need to pass the command without replace the variable, so when run the command in the container just invoke the $PATH variable.
$ docker exec -it -u weiss apollo_dev /bin/bash -c 'echo \$PATH'
Te 'apostrophe' is the key. Bye
I have a script with 2 ssh commands. The SSH scripts uses SSH to log into a remote server and deletes docker images.
ssh person#someserver.com 'set -x &&
echo "Stop docker images" ;
sudo docker stop $(sudo docker ps -a -q) ;
sudo docker rmi -f $(sudo docker images -q) ;
sudo docker rm -f $(sudo docker ps -a -q)'
Note use of ; to separate commands (we don't care if one or more of the commands fail).
The 2nd ssh command uses SSH to log into the same server, grab a docker compose file and run docker.
ssh person#someserver.com 'set -x &&
export AWS_CONFIG_FILE=/somelocation/myaws.conf &&
aws s3 cp s3://com.somebucket.somewhere/docker-compose/docker-compose.yml . --region us-east-1 &&
echo "Get ECR login credentials and do a docker compose up" &&
sudo $(aws ecr get-login --region us-east-1) &&
sudo /usr/local/bin/docker-compose up -d'
Note use of && to separate commands (this time we do care if one or more of the commands fail as we grab the exit code i.e exitCode=$?).
I don't like the fact I have to split this into 2 so my question is can these 2 sections of bash commands be combined into a single SSH call (with both ; and && combinations)?
Although it is possible to pass a set of commands as a simple single-quoted string, I wouldn't recommend that, because:
internal quotation marks should be escaped
it is difficult to read (and maintain!) a code that looks like a string in a text editor
I find it better to keep the scripts in separate files, then pass them to ssh as standard input:
cat script.sh | ssh -T user#host -- bash -s -
Execution of several scripts is done in the same way. Just concatenate more scripts:
cat a.sh b.sh | ssh -T user#host -- bash -s -
If you still want to use a string, use a here document instead:
ssh -T user#host -- <<'END_OF_COMMANDS'
# put your script here
END_OF_COMMANDS
Note the -T option. You don't need pseudo-terminal allocation for non-interactive scripts.
ssh person#someserver.com 'set -x;
echo "Stop docker images" ;
sudo docker stop $(sudo docker ps -a -q) ;
sudo docker rmi -f $(sudo docker images -q) ;
sudo docker rm -f $(sudo docker ps -a -q) ;
export AWS_CONFIG_FILE=/somelocation/myaws.conf &&
aws s3 cp s3://com.somebucket.somewhere/docker-compose/docker-compose.yml . --region us-east-1 &&
echo "Get ECR login credentials and do a docker compose up" &&
sudo $(aws ecr get-login --region us-east-1) &&
sudo /usr/local/bin/docker-compose up -d'