CRON bash execution result differs from manual launch [duplicate] - linux

This question already has an answer here:
Run docker command in bash file and run using crontab
(1 answer)
Closed 5 years ago.
I use GOGS as a personal Git server (using Docker image). I made a backup script which runs fine when I launch it manually, but when CRON lauches it, seems to malfunction as backups are strictly identical to previous one.
IMPORTANT PRECISION : usual problems like this are related to $PATH that might be incomplete, but script below only uses docker, ls, head or xargs commands which are under /usr/bin or /usr (so in $PATH).
Example results on 3 weeks (CRON once a week) :
root#1:~# ll /path/to/backup/
total 1724864
-rw-r--r-- 1 root root 588742750 janv. 29 11:30 gogs_2018_01_29.zip
-rw-r--r-- 1 root root 588742750 janv. 29 11:30 gogs_2018_02_02.zip
-rw-r--r-- 1 root root 588742750 janv. 29 11:30 gogs_2018_02_09.zip
Even system dates are identical... While date setup by bash are good related on execution. I don't get it.
After manual launch :
-rw-r--r-- 1 root root 588742750 janv. 29 11:30 gogs_2018_01_29.zip
-rw-r--r-- 1 root root 588742750 janv. 29 11:30 gogs_2018_02_02.zip
-rw-r--r-- 1 root root 589503781 févr. 9 11:21 gogs_2018_02_09.zip
Last backup is bigger, dates matches... Worked.
Why do I have this malfunction on CRON launch only?
Script gogs_backup:
#!/bin/bash
# Make backup
docker exec -i -t gogs_1 /app/gogs/gogs backup --target /tmp --archive-name gogs.zip
# Build file name
now=$(date +"%Y_%m_%d")
file="/path/to/backup/gogs_$now.zip"
# Copy file
docker cp gogs_1:/tmp/gogs.zip $file
# Remove everything from /tmp dir
docker exec -i -t gogs_1 rm -rf /tmp/gogs*
# Keep only 3 in destination folder
ls -drt /path/to/backup/* | head -n-3 | xargs rm
CRON line :
0 3 * * 5 bash /path/to/gogs_backup

From the exec documentation
The docker exec command runs a new command in a running container.
And from the cp documentation
The CONTAINER can be a running or stopped container.
So my guess is that docker exec fails because the container is stoped, but cp works because it works on stoped containers.
Check your logs. And check your return values, there are more bugs in this script.

Thanks to #mncl post, I decided to have a look again on docker commands documentation.
I spotted here that I runned docker exec -i -t gogs_1 /app/gogs/gogs backup --target /tmp --archive-name gogs.zip.
-i seemed useless to me, as it is a CRON launch. I removed it and the CRON launched script runned well.
Nonetheless, the Docker doc about -i puzzle me out. Why keeping STDIN attached to command would be a problem in this case?

Related

Docker command for Katalon in Linux container in Azure

As can be seen from the title, I am trying to run Katalon Studio in a Docker Linux container in Azure.
Perhaps I’m trying a tad too much regarding integration.
I integrated Docker into Azure and copied my whole Katalon project into the container through git clone from a git repository. It seems to be working, since I can run docker commands in the container and see the files in the container using ls.
However, when I try to run a test suite which is located in the container, I keep getting an error.
The command I run is
docker run -t --rm -v /home/docker/katalontest:/katalon/katalon/source katalonstudio/katalon katalon-execute.sh -browserType=“Chrome” -retry=0 -statusDelay=15 -testSuitePath=“Test Suites/Test”
The error I get is as follows:
echo 'Starting Katalon Studio'
Starting Katalon Studio
cat /katalon/version Google Chrome 72.0.3626.109 Mozilla Firefox 65.0 Katalon Studio 5.10.1
pwd
current_dir=/
workspace_dir=/tmp/katalon_execute/workspace
mkdir -p /tmp/katalon_execute/workspace
chmod -R 777 /tmp/katalon_execute/workspace
source_dir=/katalon/katalon/source
'[' -d /katalon/katalon/source ']'
project_dir=/tmp/katalon_execute/project
mkdir -p /tmp/katalon_execute/project
cp -r /katalon/katalon/source/Reports /katalon/katalon/source/report /tmp/katalon_execute/project
touch /tmp/katalon_execute/project/.classpath
chmod -R 777 /tmp/katalon_execute/project
rm -rf /tmp/katalon_execute/project/bin
rm -rf /tmp/katalon_execute/project/Libs
ls -la /tmp/katalon_execute/project total 16 drwxrwxrwx 4 root root 4096 Feb 19 07:33 . drwxr-xr-x 4 root root 4096 Feb 19 07:33 ..
-rwxrwxrwx 1 root root 0 Feb 19 07:33 .classpath drwxrwxrwx 2 root root 4096 Feb 19 07:33 Reports drwxrwxrwx 2 root root 4096 Feb 19 07:33 report
report_dir=/katalon/katalon/report
'[' -d /katalon/katalon/report ']'
mkdir -p /katalon/katalon/report
args=("$KATALON_KATALON_INSTALL_DIR/katalon" "$#")
args+=("-runMode=console")
args+=("-reportFolder=$report_dir")
args+=("-projectPath=$project_dir")
cd /tmp/katalon_execute/workspace
xvfb-run -s '-screen 0 1024x768x24' /opt/katalonstudio/katalon -testSuitePath=Test Suites/Test -g_Pass=smartlink2test -runMode=console -reportFolder=/katalon/katalon/report -projectPath=/tmp/katalon_execute/project Opening project file: /tmp/katalon_execute/project Invalid argument: Cannot find project'/tmp/katalon_execute/project'.
Request sent successfully.
ret_code=4
id -u
id -g
chown -R 0:0 /katalon/katalon/report
chmod -R 777 /katalon/katalon/report
ls /katalon/katalon/report
cd /
exit 4
It seems the project can’t be found. I can’t figure out why. Any help would be greatly appreciated.
You are right you are able to execute commands inside the container because you (clone/mount (-v /home/docker/katalontest:/katalon/katalon/source katalonstudio/katalon) the Katalon executables.
However, your issue is that your project when cloning the repo it is still inside the repository (https://github.com/katalon-studio/docker-images/tree/master/test/project - which in your case it is here (/katalon/katalon/source katalonstudio/katalon) however, you are forcing Katalon to use another location which is: -projectPath=/tmp/katalon_execute/project
Either you pass the right -projectPath or you clone/mount the repository there.
Also, I would suggest you to pay a bit more attention to the documentation.

Edit file with shell script that is in the same group but not the caller user

I would like to edit a file with a shell script which is in the same group but not the "caller" user.
-rwsr-xr-x 1 root root 4896 Oct 21 00:59 /usr/bin/luxus
-rw-rw-r-- 1 root root 4096 Oct 20 23:01 <path>/file1
lrwxrwxrwx 1 root root 0 Oct 20 23:00 <path>/dir1
/usr/bin/luxus: the shell script
file1: the file to edit
dir1: file1's parent directory (symlink)
In the shell script, this is the line where the permission issue is:
echo "string" > /usr/bin/tee <path>/file1
Output:
/usr/bin/tee: <path>/file1: Permission denied
I am trying to create an AUR package for the shell script. The latter is meant to be install on the system from a PKGBUILD. In vain, I attempted to give enough permissions to the script to edit the targeted file.
This is a PKGBUILD's sample:
install -Dm755 src/${pkgname} "${pkgdir}/usr/bin/${pkgname}"
chown root.root "${pkgdir}/usr/bin/${pkgname}"
chmod 4755 "${pkgdir}/usr/bin/${pkgname}"
After investigating, I think it is quite simply impossible because of security or because of the parent dir's rights.
Nevertheless I am asking you for a solution.
As you said "you can't do anything".
The following post quite simply explains why it won't work:
https://unix.stackexchange.com/questions/364/allow-setuid-on-shell-scripts

Can't write on hard drive owned by user and with permissions to write, read and execute

I am very puzzled by this problem I am having. I am trying to execute a file in Ubuntu 12.04 LTS via command line. I have a script that calls a program to run and write the results in a hard drive. I changed the permissions and ownership of everything to be wxr. Here is the ls -l of my script (called TEST-star):
-rwxrwxrwx 1 root root 950 Nov 15 13:16 TEST-star
Here is the ls -l of the package my script calls:
-rwxrwxrwx 1 root root 1931414 Nov 10 12:37 STAR
Finally the ls -l of the hard drive mounted in /media/CLC"
drwxrwxrwx 1 root root 8192 Nov 15 13:04 CLC
I have been trying to run it since yesterday and always get a message that I don't have permission to write the results:
EXITING because of FATAL ERROR: could not create output file ./_STARtmp//Unmapped.out.mate1.thread14
Solution: check that you have permission to write this file
I thought if I change the permissions to rwx and run my script as root it would not have a problem (using sudo). Right now I run out of options. Any suggestion would be appreciated. Please let me know what other information you would need solve this issue.
Thank you.
Here is the first line of script I am trying to run:
#!/bin/sh
cd /media/CLC/ANOPHELES-STAR-v2.4f1/; mkdir GambFemAnt1 && cd GambFemAnt1; echo $PWD && echo Starting mapping of GambFemAnt1; /home/aedes/Documents/STAR_2.4.0f1/STAR --genomeDir /media/Galaxy/Galaxy_data/Anopheles/STAR/Genome --readFilesIn /media/Galaxy/Galaxy_data/Anopheles/QC/GambFemAnt1/GambFemAnt1.fastq --runThreadN 23 --outFilterMismatchNmax 4 --outFilterMatchNminOverLread 0.75 --seedSearchLmax 30 --seedSearchStartLmax 30 --seedPerReadNmax 100000 --seedPerWindowNmax 100 --alignTranscriptsPerReadNmax 100000 --alignTranscriptsPerWindowNmax 10000 --outSAMstrandField intronMotif --outFilterIntronMotifs RemoveNoncanonical --outSAMtype BAM SortedByCoordinate --outReadsUnmapped Fastx; mv Aligned.sortedByCoord.out.bam GambFemAnt1.bam; mv Unmapped.out.mate1 GambFemAnt1-unmapped.fastq; cp *.fastq /media/CLC/ANOPHELES-STAR-v2.4f1/UNMAPED-reads/; cd /media/CLC/ANOPHELES-STAR-v2.4f1 && echo $PWD && echo GambFemAnt1 mapping finished;
I also posted a question for the authors of the package.
Turns out all the permissions were set correctly. The problem resigns within the package. I found out that it works using --runThreadN 12 instead of --runThreadN 23.

Git Install Script How to enter shell command

This is more of a git related question than openshift.
When I push files to the git server using git, I see that the file permissions of my perl index.cgi file gets changed to 700 in my openshift repo.
Hence every time I do a push, I have to log in to the server using ssh and do a chmod a+x index.cgi in my $OPENSHIFT_REPO_DIR to change it to 755.
I read in the git documentation it is possible to add a hook and I tried updating the post update script in the hooks directory with the following command
exec chmod a+x $OPENSHIFT_DATA_DIR/index.cgi
However that does not seemed to have worked.
Can you please tell me what I am doing wrong ?
How can I ensure I don't have to make the chmod change every time I push files ?
How does one execute shell commands via the git install script (I don't know what is install script, I assume it means the hooks)
based on a below suggestion i thought i would need to add the hook to the server so i tried it but i got an error
$ git add .git/hooks/post-update
error: Invalid path '.git/hooks/post-update'
error: unable to add .git/hooks/post-update to index
so i tried some options using action hooks by placing a "start" in the local .openshift/action_hooks directory.
The command there ive tried to use was
exec chmod a+x register.cgi and even chmod a+x register.cgi
Not much luck. Any inputs. I will be raising this as a separate q against actionhooks.
This is more of a git related question than openshift.
Actually, it is more related to openshift than it is to Git: Git only records 644 and 755 permissions.
But the destination system managing a checkout (and updating a working tree) would set the permission according to the default umask (which could be in your case 077).
Make sure the default umask is 022.
Regarding the hook, check its name ('post-update', no extension) and permission (it must be executable, 755 would be useful)
Bring authority in this case gitweb folder.
gitweb/
drwxr-xr-x 3 root root 4096 Sep 25 12:01 .
drwxr-xr-x 225 root root 12288 Oct 27 20:10 ..
-rwxr-xr-x 1 root root 251319 Sep 23 04:36 gitweb.cgi
lrwxrwxrwx 1 root root 10 Sep 23 04:36 index.cgi -> gitweb.cgi
drwxr-xr-x 2 root root 4096 Sep 25 12:01 static

Running app inside Docker as non-root user

After yesterday's news of Shocker, it seems like apps inside a Docker container should not be run as root. I tried to update my Dockerfile to create an app user however changing permissions on app files (while still root) doesn't seem to work. I'm guessing this is because some LXC permission is not being granted to the root user maybe?
Here's my Dockerfile:
# Node.js app Docker file
FROM dockerfile/nodejs
MAINTAINER Thom Nichols "thom#thomnichols.org"
RUN useradd -ms /bin/bash node
ADD . /data
# This next line doesn't seem to have any effect:
RUN chown -R node /data
ENV HOME /home/node
USER node
RUN cd /data && npm install
EXPOSE 8888
WORKDIR /data
CMD ["npm", "start"]
Pretty straightforward, but when I ls -l everything is still owned by root:
[ node#ed7ae33e76e1:/data {docker-nonroot-user} ]$ ls -l /data
total 64K
-rw-r--r-- 1 root root 383 Jun 18 20:32 Dockerfile
-rw-r--r-- 1 root root 862 Jun 18 16:23 Gruntfile.js
-rw-r--r-- 1 root root 1.2K Jun 18 15:48 README.md
drwxr-xr-x 4 root root 4.0K May 30 14:24 assets/
-rw-r--r-- 1 root root 416 Jun 3 14:22 bower.json
-rw-r--r-- 1 root root 930 May 30 01:50 config.js
drwxr-xr-x 4 root root 4.0K Jun 18 16:08 lib/
drwxr-xr-x 42 root root 4.0K Jun 18 16:04 node_modules/
-rw-r--r-- 1 root root 2.0K Jun 18 16:04 package.json
-rw-r--r-- 1 root root 118 May 30 18:35 server.js
drwxr-xr-x 3 root root 4.0K May 30 02:17 static/
drwxr-xr-x 3 root root 4.0K Jun 18 20:13 test/
drwxr-xr-x 3 root root 4.0K Jun 3 17:38 views/
My updated dockerfile works great thanks to #creak's clarification of how volumes work. Once the initial files are chowned, npm install is run as the non-root user. And thanks to a postinstall hook, npm runs bower install && grunt assets which takes care of the remaining install steps and avoids any need to npm install -g any node cli tools like bower, grunt or coffeescript.
Check this post: http://www.yegor256.com/2014/08/29/docker-non-root.html In rultor.com we run all builds in their own Docker containers. And every time before running the scripts inside the container, we switch to a non-root user. This is how:
adduser --disabled-password --gecos '' r
adduser r sudo
echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
su -m r -c /home/r/script.sh
r is the user we're using.
Update 2015-09-28
I have noticed this post getting a bit of attention. A word of advice for anyone who is potentially interested in doing something like this. I would try to use Python or another language as a wrapper for your script executions. Doing native bash scripts I had problems when trying to pass through a variety of arguments to my containers. Specifically there was issues with the interpretation/escaping of " and ' characters by the shell.
I was needing to change the user for a slightly different reason.
I created a docker image housing a full featured install of ImageMagick and Ffmpeg with a desire that I could do transformations on images/videos within my host OS. My problem was that these are command line tools, so it is slightly trickier to execute them via docker and then get the results back into the host OS. I managed to allow for this by mounting a docker volume. This seemed to work okay except that the image/video output was coming out as being owned by root (i.e. the user the docker container was running as), rather than the user whom executed the command.
I looked at the approach that #François Zaninotto mentioned in his answer (you can see the full make script here). It was really cool, but I preferred the option of creating a bash shell script that I would then register on my path. I took some of the concepts from the Makefile approach (specifically the user/group creation) and then I created the shell script.
Here is an example of my dockermagick shell script:
#!/bin/bash
### VARIABLES
DOCKER_IMAGE='acleancoder/imagemagick-full:latest'
CONTAINER_USERNAME='dummy'
CONTAINER_GROUPNAME='dummy'
HOMEDIR='/home/'$CONTAINER_USERNAME
GROUP_ID=$(id -g)
USER_ID=$(id -u)
### FUNCTIONS
create_user_cmd()
{
echo \
groupadd -f -g $GROUP_ID $CONTAINER_GROUPNAME '&&' \
useradd -u $USER_ID -g $CONTAINER_GROUPNAME $CONTAINER_USERNAME '&&' \
mkdir --parent $HOMEDIR '&&' \
chown -R $CONTAINER_USERNAME:$CONTAINER_GROUPNAME $HOMEDIR
}
execute_as_cmd()
{
echo \
sudo -u $CONTAINER_USERNAME HOME=$HOMEDIR
}
full_container_cmd()
{
echo "'$(create_user_cmd) && $(execute_as_cmd) $#'"
}
### MAIN
eval docker run \
--rm=true \
-a stdout \
-v $(pwd):$HOMEDIR \
-w $HOMEDIR \
$DOCKER_IMAGE \
/bin/bash -ci $(full_container_cmd $#)
This script is bound to the 'acleancoder/imagemagick-full' image, but that can be changed by editing the variable at the top of the script.
What it basically does is:
Create a user id and group within the container to match the user who executes the script from the host OS.
Mounts the current working directory of the host OS (using docker volumes) into home directory for the user we create within the executing docker container.
Sets the tmp directory as the working directory for the container.
Passes any arguments that are passed to the script, which will then be executed by the '/bin/bash' of the executing docker container.
Now I am able to run the ImageMagick/Ffmpeg commands against files on my host OS. For example, say I want to convert an image MyImage.jpeg into a PNG file, I could now do the following:
$ cd ~/MyImages
$ ls
MyImage.jpeg
$ dockermagick convert MyImage.jpeg Foo.png
$ ls
Foo.png MyImage.jpeg
I have also attached to the 'stdout' so I could run the ImageMagick identify command to get info on an image on my host, for e.g.:
$ dockermagick identify MyImage.jpeg
MyImage.jpeg JPEG 640x426 640x426+0+0 8-bit DirectClass 78.6KB 0.000u 0:00.000
There are obvious dangers about mounting the current directory and allowing any arbitrary command definition to be passed along for execution. But there are also many ways to make the script more safe/secure. I am executing this in my own non-production personal environment, so these are not of highest concern for me. But I would highly recommend you take the dangers into consideration should you choose to expand upon this script. It's also worth me mentioning that this script doesn't take an OS X host into consideration. The make file that I steal ideas/concepts from does take this into account, so you could extend this script to do so.
Another limitation to note is that I can only refer to files currently in the path for which I am executing the script. This is because of the way I am mounting the volumes, so the following would not work:
$ cd ~/MyImages
$ ls
MyImage.jpeg
$ dockermagick convert ~/DifferentDirectory/AnotherImage.jpeg Foo.png
$ ls
MyImage.jpeg
It's best just to go to the directory containing the image and execute against it directly. Of course I am sure there are ways to get around this limitation too, but for me and my current needs, this will do.
This one is a bit tricky, it is actually due to the image you start from.
If you look at the source, you notice that /data/ is a volume. So everything you do in the Dockerfile will be discarded and overridden at runtime by the volume that gets mounted then.
You can chown at runtime by changing your CMD to something like CMD chown -R node /data && npm start.
Note: I answer here because, given the generic title, this Question pops up in google when you look for a solution to "Running app inside Docker as non-root user". Hope it helps those who are stranded here.
With Alpine Linux you can create a system user like this:
RUN adduser -D -H -S -s /bin/false -u 1000 myuser
Everything in the Dockerfile after this line is executed with myuser.
myuser user has:
no password assigned
no home dir
no login shell
no root access.
This is from adduser --help:
-h DIR Home directory
-g GECOS GECOS field
-s SHELL Login shell
-G GRP Add user to existing group
-S Create a system user
-D Don't assign a password
-H Don't create home directory
-u UID User id
-k SKEL Skeleton directory (/etc/skel)
Note: This answer is given because many people looking for non-root usage will end up here. Beware, this does not address the issue that caused the problem, but is addressing the title and clarification to an answer given by #yegor256, which uses a non-root user inside the container. This answer explains how to accomplish this for non-debian/non-ubuntu use-case. This is not addressing the issue with volumes.
On Red Hat-based systems, such as Fedora and CentOS, this can be done in the following way:
RUN adduser user && \
echo "user ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/user && \
chmod 0440 /etc/sudoers.d/user
In your Dockerfile you can run commands as this user by doing:
RUN su - user -c "echo Hello $HOME"
And the command can be run as:
CMD ["su","-","user","-c","/bin/bash"]
An example of this can be found here:
https://github.com/gbraad/docker-dev/commit/644c51002f4b8e6fe5bb745638542a4c3d908b16

Resources