Docker args with whitespaces from variable. How to use it> - linux

I have searched the answer at stackoverflow(and google too) for 3 hours and I didnt find correct solution.
In my CI a need to dynamicly add docker build arguments. And I need to use it from env variables.
I have a string with docker arg:
DOCKER_ARGS="CURL_ARGS='-k https://example.invalid/'|ARG2=As|ARG3=asdas"
In my *.sh I wrote some code:
DOCKER_ARGS="CURL_ARGS='-k https://example.invalid/'|ARG2=As|ARG3=asdas"
if [ ! -z "$DOCKER_ARGS" ]; then
ARG_LIST=()
while read -d"|" ARG || [[ -n "$ARG" ]];
do
ARG_LIST+=("--build-arg $ARG")
done <<<$DOCKER_ARGS
fi
docker build -t test . ${ARG_LIST[#]}
And when I use echo ${ARG_LIST[#]}, I receive correct result:
--build-arg CURL_ARGS='-k https://example.invalid/' --build-arg ARG2=As --build-arg ARG3=asdas
But when I use docker build -t test . ${ARG_LIST[#]} it gives me an error:
"docker build" requires exactly 1 argument.
And if I use echo for command:
echo "docker build -t test . ${ARG_LIST[#]}" - I had correct result
I expect that will works as:
docker build -t image:tag . --build-arg CURL_ARGS='-k https://example.invalid/' --build-arg ARG2=As --build-arg ARG3=asdas

There are some problems with your script.
DOCKER_ARGS="CURL_ARGS='-k https://example.invalid/'|ARG2=As|ARG3=asdas"
With current handling CURL_ARGS will be set to '-k https://example.invalid/' including the quotes. Either remove the quotes from the input, or write your own un-quoting removal routine.
read -d"|" ARG
Removes trailing and leading whitespaces from input because of IFS and interprets missing slash sequences because or missing -r.
ARG_LIST+=("--build-arg $ARG")
Assigns a single element with the value --build-arg $ARG to the array. You would have to split later on the space between build-arg and $ARG. Instead assign two elements - one for --build-arg and the other for $ARG.
A better script might look like the following:
docker_args="CURL_ARGS=-k https://example.invalid/|ARG2=As|ARG3=asdas"
arg_list=()
if [[ -n "$docker_args" ]]; then
# split input on `|`
# Potentially use readarray to handle newlines
IFS='|' read -r -a args <<<"$docker_args"
# Add --built-arg to arrays
for arg in "${args[#]}"; do
# Proper quoting
arg_list+=(--build-arg "$arg")
done
fi
# proper quoting
docker build -t test . "${arg_list[#]}"
Check your scripts with http://shellcheck.net . Re-research how quoting works in shell and to use bash arrays. Prefer to use lower case variable names.

Related

Increment the title of files output by a command in a shell script

I made this simple bash script to take a full-screen screenshot and save it to the pictures folder
#!/usr/bin/bash
xfce4-screenshooter -f -s /home/rgcodes/Pictures/Screenshot_$scrshotcount.png
let "scrshotcount++"
...which runs into a problem. scrshotcount is a global variable I defined in /etc/environment to be incremented every time the script runs. However, the script fails to increment the variable globally, and causes the script to just overwrite the previous screenshot. Searches on Google and Stack Overflow revealed that the problem isn't straightforward at all (something about child shells being unable to change variables for parents), and finding some other method would be better.
Here's my question. How do we append numbers (in ascending order) to the screenshots the script throws out so that they are saved just like those taken on Windows?(Windows auto-suffixes matching filenames, rather than overwriting them, so all Screenshots have the same name 'Screenshot' and the number of times the screenshot command has been used.)
I am using #erikMD's method as a temporary stop-gap for now.
In addition to the excellent advice about using a date instead of a counter, here's a way to use a counter :/
dir=$HOME/Pictures
# find the current maximum value
current_max=$(
find "$dir" -name Screenshot_\*.png -print0 \
| sort -z -V \
| tail -z -n 1
)
if [[ ! $current_max =~ _([0-9]+)\.png ]]; then
echo "can't find the screenshot with the maximum counter value" >&2
exit 1
fi
# increment it
counter=$(( 1 + ${BASH_REMATCH[1]} ))
# and use it
xfce4-screenshooter -f -s "$dir/Screenshot_${counter}.png"
You'll have to manually create the Screenshot_1.png file.
#rgcodes below is a script that will capture screenshots with a numeric count indicator per your original post. (tested it on Ubuntu 20.04)
Script contents:
#!/bin/bash
set -uo pipefail
# add source file and line number to xtrace output
# i.e. when running: bash -x ./your_script_name.sh
export PS4='+(${BASH_SOURCE}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
capture_screenshot() {
local output_dir="${1:-/tmp/screenshot}"
local img_name="${2:-Screenshot}"
local img_ext="${3:-png}"
# create output directory (if not already existing)
mkdir -p "${output_dir}"
# get the last image in the sorted ls output
local latest_png=$(tail -n 1 \
<(sort -n -t _ -k 2 \
<(ls ${output_dir}/*.${img_ext} 2> /dev/null)))
# use the latest image to determine img_cnt value for next image
local img_cnt=0
if [[ ${latest_png} =~ _([0-9]+)\.png ]]; then
img_cnt=$((1+${BASH_REMATCH[1]}))
elif [[ ${latest_png} =~ ${img_name}.${img_ext} ]] ; then
img_cnt=1
fi
# build path to output image
local img_path="${output_dir}/${img_name}_${img_cnt}.${img_ext}"
# remove count from output image path if count == 0
if [[ "${img_cnt}" -eq "0" ]] ; then
img_path="${output_dir}/${img_name}.${img_ext}"
fi
xfce4-screenshooter -f -s "${img_path}"
}
capture_screenshot "$#"
The uses the following as defaults, but you can change them to meet your requirements.
output directory for screenshots:
/tmp/screenshot
base screenshot image name:
Screenshot
screenshot file extension:
.png
The script will attempt to create the output directory if it does not already exist (subject to user permission for creation). Below is a sample usage.
Prior to initial script execution, the output directory does not exist:
$ ls screenshot
$
Initial execution (directory is created and Screenshot.png created:
$ ./script.sh
$ ls /tmp/screenshot/
Screenshot.png
Subsequent executions:
$ ./script.sh
$ ls /tmp/screenshot/
Screenshot_1.png Screenshot.png
$ ./script.sh
$ ls /tmp/screenshot/
Screenshot_1.png Screenshot_2.png Screenshot.png
Indeed, as suggested by #j_b in the comments, you should definitely give a try to using a timestamp with the command date +"$format".
FTR, the same idea is implemented here in this project of a gnome-screenshot bash wrapper
(disclaimer: I am the author of this repo).
Example command:
date "+%Y-%m-%d_%H-%M-%S"
↓
2021-07-29_19-13-30
So the overall script could just be something like:
#!/usr/bin/env bash
xfce4-screenshooter -f -s "$HOME/Pictures/Screenshot_$(date "+%Y-%m-%d_%H-%M-%S").png"
(Note that I added missing double-quotes, and modified your shebang, as /usr/bin/env bash is more portable than /bin/bash or /usr/bin/bash.)

exporting list of variables in docker run

We have a set of variables in env file as given below
examples.env
A="/path1"
B="/path2":$A
Now, docker run cannot substitute $B for /path/path1, due to its limitations
So, I want to export the variable in launcher script and then call those variable using -e flag, as given below
mydocker.sh
input="examples.env"
while IFS= read -r line
do
export $line
done < "$input"
docker run --rm -e <Some code> centos8
Now how to create docker command to get all the variables?
Following docker command works
docker run --rm -e A -e B centos8
But If the number of variables in examples.env file is unknown, then how can we generate docker run command?
Source the variables file in your mydocker.sh script insted of export and concat each variable with --env, at the and eval the concatenated string to variable so the variables will interpreted.
Here is an example:
# Source the variables file so they will be available in current script.
. ./examples.env
# Define docker env string it will lokk like below:.
# --env A=/path1 --env B=/path1/path2
dockerenv=""
input="examples.env"
while IFS= read -r line
do
dockerenv="${dockerenv} --env $line"
done < "$input"
# Evaluate the env string so the variables in it will be interpreted
dockerenv=$(eval echo $dockerenv)
docker run --rm $dockerenv centos8
P.S.
You need the --env insted of -e becouse -e will be interpreted as echo command argument.

Bash syntax issue, 'syntax error: unexpected "do" (expecting "fi")'

I have a sh script that I am using on Windows and Mac/Linux machines, and seems to work with no issues normally.
#!/bin/bash
if [ -z "$jmxname" ]
then
cd ./tests/Performance/JMX/ || exit
echo "-- JMX LIST --"
# set the prompt used by select, replacing "#?"
PS3="Use number to select a file or 'stop' to cancel: "
# allow the user to choose a file
select jmxname in *.jmx
do
# leave the loop if the user says 'stop'
if [[ "$REPLY" == stop ]]; then break; fi
# complain if no file was selected, and loop to ask again
if [[ "$jmxname" == "" ]]
then
echo "'$REPLY' is not a valid number"
continue
fi
# now we can use the selected file, trying to get it to run the shell script
rm -rf ../../Performance/results/* && cd ../jmeter/bin/ && java -jar ApacheJMeter.jar -Jjmeter.save.saveservice.output_format=csv -n -t ../../JMX/"$jmxname" -l ../../results/"$jmxname"-reslut.jtl -e -o ../../results/HTML
# it'll ask for another unless we leave the loop
break
done
else
cd ./tests/Performance/JMX/ && rm -rf ../../Performance/results/* && cd ../jmeter/bin/ && java -jar ApacheJMeter.jar -Jjmeter.save.saveservice.output_format=csv -n -t ../../JMX/"$jmxname" -l ../../results/"$jmxname"-reslut.jtl -e -o ../../results/HTML
fi
I am now trying to do some stuff with a Docker container and have used a node:alpine image, as the rest of my project is NodeJS based, but for some reason the script will not run in the Docker container giving the following -
line 12: syntax error: unexpected "do" (expecting "fi")
How can I fix that? The script seems to be working for every system it's been run on so far, and not thrown up any issues.
The error message indicates that the script is executed as '/bin/sh', and not as /bin/bash. You can see the message with '/bin/sh -n script.sh'
Check how the script is invoked. On different systems /bin/sh is symlinked to bash or other shell that is less feature rich.
In particular, the problem is with the select statement, included in bash, but not part of the POSIX standard.
Another option is that bash on your docker is set to be POSIX compliant by default
#dash-o was correct, and adding -
RUN apk update && apk add bash
to my dockerfile added bash into the container and now it works fine :)

Run bash script with defaults to piped commands set within the script

Two questions about the same thing I think...
Question one:
Is it possible to have a bash script run with default parameters/options? ...in the sense if someone were to run the script:
./somescript.sh
it would actually run with ./somescript.sh | tee /tmp/build.txt?
Question two:
Would it also possible to prepend the script with defaults? For example, if you were to run the script ./somescript.sh
it would actually run
script -q -c "./somescript.sh" /tmp/build.txt | aha > /tmp/build.html?
Any help or guidance is very much appreciated.
You need a wrapper script that handles all such scenario for you.
For example, your wrapper script can take parameters that helps you decide.
./wrapper_script.sh --input /tmp/build.txt --ouput /tmp/build.html
By default, --input and --output can be set to values you want when they are empty.
You can use the builtin $# to know how many arguments you have and take action based on that. If you want to do your second part, for example, you could do something like
if [[ $# -eq 0 ]]; then
script -q -c "$0 /tmp/build.txt | aha /tmp/build.html
exit
fi
# do everything if you have at least one argument
Though this will have problems if your script/path have spaces, so you're probably better putting the real path to your script in the script command instead of $0
You can also use exec instead of running the command and exiting, but make sure you have your quotes in the right place:
if [[ $# -eq 0 ]]; then
exec script -q -c "$0 /tmp/build.txt | aha /tmp/build.html"
fi
# do everything when you have at least 1 argument

Triple nested quotations in shell script

I'm trying to write a shell script that calls another script that then executes a rsync command.
The second script should run in its own terminal, so I use a gnome-terminal -e "..." command. One of the parameters of this script is a string containing the parameters that should be given to rsync. I put those into single quotes.
Up until here, everything worked fine until one of the rsync parameters was a directory path that contained a space. I tried numerous combinations of ',",\",\' but the script either doesn't run at all or only the first part of the path is taken.
Here's a slightly modified version of the code I'm using
gnome-terminal -t 'Rsync scheduled backup' -e "nice -10 /Scripts/BackupScript/Backup.sh 0 0 '/Scripts/BackupScript/Stamp' '/Scripts/BackupScript/test' '--dry-run -g -o -p -t -R -u --inplace --delete -r -l '\''/media/MyAndroid/Internal storage'\''' "
Within Backup.sh this command is run
rsync $5 "$path"
where the destination $path is calculated from text in Stamp.
How can I achieve these three levels of nested quotations?
These are some question I looked at just now (I've tried other sources earlier as well)
https://unix.stackexchange.com/questions/23347/wrapping-a-command-that-includes-single-and-double-quotes-for-another-command
how to make nested double quotes survive the bash interpreter?
Using multiple layers of quotes in bash
Nested quotes bash
I was unsuccessful in applying the solutions to my problem.
Here is an example. caller.sh uses gnome-terminal to execute foo.sh, which in turn prints all the arguments and then calls rsync with the first argument.
caller.sh:
#!/bin/bash
gnome-terminal -t "TEST" -e "./foo.sh 'long path' arg2 arg3"
foo.sh:
#!/bin/bash
echo $# arguments
for i; do # same as: for i in "$#"; do
echo "$i"
done
rsync "$1" "some other path"
Edit: If $1 contains several parameters to rsync, some of which are long paths, the above won't work, since bash either passes "$1" as one parameter, or $1 as multiple parameters, splitting it without regard to contained quotes.
There is (at least) one workaround, you can trick bash as follows:
caller2.sh:
#!/bin/bash
gnome-terminal -t "TEST" -e "./foo.sh '--option1 --option2 \"long path\"' arg2 arg3"
foo2.sh:
#!/bin/bash
rsync_command="rsync $1"
eval "$rsync_command"
This will do the equivalent of typing rsync --option1 --option2 "long path" on the command line.
WARNING: This hack introduces a security vulnerability, $1 can be crafted to execute multiple commands if the user has any influence whatsoever over the string content (e.g. '--option1 --option2 \"long path\"; echo YOU HAVE BEEN OWNED' will run rsync and then execute the echo command).
Did you try escaping the space in the path with "\ " (no quotes)?
gnome-terminal -t 'Rsync scheduled backup' -e "nice -10 /Scripts/BackupScript/Backup.sh 0 0 '/Scripts/BackupScript/Stamp' '/Scripts/BackupScript/test' '--dry-run -g -o -p -t -R -u --inplace --delete -r -l ''/media/MyAndroid/Internal\ storage''' "

Resources