I'm debugging an old shell script; I want to check the values of all the variables used, it's a huge ugly script with approx more than 140 variables used. Is there anyway I can extract the variable names from the script and put them in a convenient pattern like:
#!/bin/sh
if [ ${BLAH} ....
.....
rm -rf ${JUNK}.....
to
echo ${BLAH}
echo ${JUNK}
...
Try running your script as follows:
bash -x ./script.bash
Or enable the setting in the script:
set -x
You can dump all interested variables in one command using:
set | grep -w -e BLAH -e JUNK
To dump all the variables to stdout use:
set
or
env
from inside your script.
You can extract a (sub)list of the variables declared in your script using grep:
grep -Po "([a-z][a-zA-Z0-9_]+)(?==\")" ./script.bash | sort -u
Disclaimer: why "sublist"?
The expression given will match string followed by an egal sign (=) and a double quote ("). So if you don't use syntax such as myvar="my-value" it won't work.
But you got the idea.
grep Options
-P --perl-regexp: Interpret PATTERN as a Perl regular expression (PCRE, see below) (experimental) ;
-o --only-matching: Print only the matched (non-empty) parts of a matching line, with each such part on a separate output line.
Pattern
I'm using a positive lookahead: (?==\") to require an egal sign followed by a double quote.
In bash, but not sh, compgen -v will list the names of all variables assigned (compare this to set, which has a great deal of output other than variable names, and thus needs to be parsed).
Thus, if you change the top of the script to #!/bin/bash, you will be able to use compgen -v to generate that list.
That said, the person who advised you use set -x did well. Consider this extension on that:
PS4=':$BASH_SOURCE:$LINENO+'; set -x
This will print the source file and line number before every command (or variable assignment) which is executed, so you will have a log not only of which variables are set, but just where in the source each one was assigned. This makes tracking down where each variable is set far easier.
Related
I am trying to list the names of the variables that I have declared in my script, so I have been trying with different combinations of:
( set -o posix ; set ) | less
But I am not getting the result that I expected. It gives me an ouput with all variables in addition to my script´s variables. I do not know if I am using this command in the correct way. My intention is to only list the variables´ names that I have declared in my script and get an output like this:
VAR1
VAR2
VAR3
VAR4
…
set cannot be told to list only variables declared in a script. What can be done is to temporarily store defined variables at the script's beginning and filter them out at the end, so only variables defined in-between remain:
#!/bin/bash
set -o posix
set|sed s/=.\*// >/tmp/$$
VAR4=4
VAR3=3
VAR2=2
VAR1=1
set|sed s/=.\*//|grep -vf/tmp/$$
rm /tmp/$$
Note Socowi's valuable suggestions for improvement below.
A very simple solution as long as you use the prefix for your own variables. You should do this to avoid accidentally overwriting existing variables. Set defaults returns to a very long list of all defined variables not just yours, so you need to filter it by prefix.
#!/bin/bash
my_var1=aaa
my_var2=`ls`
export my_var3=bbb
compgen -v| awk -F= '/^my_/{print $1}'
# If you use sh shell it remains
#set |awk -F= '/^my_/{print $1}'
result
my_var1
my_var2
my_var3
EDIT: #Socowi rightly suggested using compgen -v for bash instead of set. Compgen is built into bash. Thanks.
There's a local variable in netmap/LINUX/configure named BUILDDIR and its value is BUILDDIR=$PWD.It should have to be redirected to $(#D) which is netmap package build directory, /usr/local/buildroot/output/build/netmap-master in my case;otherwise, object files will be outputed to buildroot root directory.
I created a variable named NETMAP_CURRENT_BUILD and let it be /usr/local/buildroot/output/build/netmap-master,$(#D),
and then I wanna replace BUILDDIR=$PWD to
BUILDDIR=/usr/local/buildroot/output/build/netmap-master. By using sample code as following, it can't be done.
Sample Code :(sed part worked fine at terminal console)
define NETMAP_BUILD_CMDS
NETMAP_CURRENT_DIR = $(sed -e 's/\//\\\//g' <<< "$(#D)") -- empty
echo "$$(sed -e 's/\//\\\//g' <<< "$(#D)")" -- this line works fine
...
sed -e 's/BUILDDIR=$$PWD/BUILDDIR=$(NETMAP_CURRENT_DIR)/g' -i $(#D)/LINUX/configure
(double $$PWD, it has to be like this, which means string $PWD is needed rather than its value.)
...
endef
You don't actually need this, see below. However, if you do need the result of a shell command in a make variable, use $(shell ...)
Since this is a makefile, the $ are interpreted by make, not by the shell. Therefore, make will try to evaluate sed -e 's/\//\\\//g' <<< "$(#D)" as a variable name. There is of course no variable with that name, so you get an empty string.
To let make run a shell command and store the result in a make variable, use the $(shell ...) function. So
NETMAP_CURRENT_DIR = $(shell sed -e 's/\//\\\//g' <<< "$(#D)")
Note that that should be done outside of the NETMAP_BUILD_CMDS, because the contents of NETMAP_BUILD_CMDS is in fact a shell script: make first expands all variables in that script, then passes it to the shell line-by-line. The shell will actually error out on the above statement because you have spaces around the = so NETMAP_CURRENT_DIR is interpreted as a command name.
If you want to store the result in a shell variable rather than a make variable, then the easiest solution is to use backticks `...` to delimit the command instead of $(...):
define NETMAP_BUILD_CMDS
NETMAP_CURRENT_DIR=`sed -e 's/\//\\\//g' <<< "$(#D)"`; \
sed -e 's/BUILDDIR=$$PWD/BUILDDIR=$$NETMAP_CURRENT_DIR/g' -i $(#D)/LINUX/configure
endef
However, in your case you don't need this at all.
First of all, you can do simple substitutions directly in make.
NETMAP_CURRENT_DIR = $(subst /,\/,$(#D))
Even better, you can simply use a different delimiter than / in your sed expression, then there is no need to quote it:
sed -e 's#BUILDDIR=$$PWD#BUILDDIR=$(#D)#g' -i $(#D)/LINUX/configure
One final note: modifications to the code like this are better done in NETMAP_POST_PATCH_HOOKS than in NETMAP_BUILD_CMDS. So finally, the best solution is:
define NETMAP_FIX_BUILDDIR
sed -e 's#BUILDDIR=$$PWD#BUILDDIR=$(#D)#g' -i $(#D)/LINUX/configure
endef
NETMAP_POST_PATCH_HOOKS += NETMAP_FIX_BUILDDIR
I am looking to write a simple script to perform a SSH command on many hosts simultaneously, and which hosts exactly are generated from another script. The problem is that when I run the script using sometihng like sed it doesn't work properly.
It should run like sshall.sh {anything here} and it will run the {anything here} part on all the nodes in the list.
sshall.sh
#!/bin/bash
NODES=`listNodes | grep "node-[0-9*]" -o`
echo "Connecting to all nodes and running: ${#:1}"
for i in $NODES
do
:
echo "$i : Begin"
echo "----------------------------------------"
ssh -q -o "StrictHostKeyChecking no" $i "${#:1}"
echo "----------------------------------------"
echo "$i : Complete";
echo ""
done
When it is run with something like whoami it works but when I run:
[root#myhost bin]# sshall.sh sed -i '/^somebeginning/ s/$/,appendme/' /etc/myconfig.conf
Connecting to all nodes and running: sed -i /^somebeginning/ s/$/,appendme/ /etc/myconfig.conf
node-1 : Begin
----------------------------------------
sed: -e expression #1, char 18: missing command
----------------------------------------
node-1 : Complete
node-2 : Begin
----------------------------------------
sed: -e expression #1, char 18: missing command
----------------------------------------
node-2 : Complete
…
Notice that the quotes disappear on the sed command when sent to the remote client.
How do I go about fixing my bash command?
Is there a better way of achieving this?
Substitute an eval-safe quoted version of your command into a heredoc:
#!/bin/bash
# ^^^^- not /bin/sh; printf %q is an extension
# Put your command into a single string, with each argument quoted to be eval-safe
printf -v cmd_q '%q ' "$#"
while IFS= read -r hostname; do
# run bash -s remotely, with that string passed on stdin
ssh -q -o 'StrictHostKeyChecking no' "$hostname" "bash -s" <<EOF
$cmd_q
EOF
done < <(listNodes | grep -o -e "node-[0-9*]")
Why this works reliably (and other approaches don't):
printf %q knows how to quote contents to be eval'd by that same shell (so spaces, wildcards, various local quoting methods, etc. will always be supported).
Arguments given to ssh are not passed to the remote command individually!
Instead, they're concatenated into a string passed to sh -c.
However: The output of printf %q is not portable to all POSIX-derived shells! It's guaranteed to be compatible with the same shell locally in use -- ksh will always parse output from printf '%q' in ksh, bash will parse output from printf '%q' in bash, etc; thus, you can't safely pass this string on the remote argument vector, because it's /bin/sh -- not bash -- running there. (If you know your remote /bin/sh is provided by bash, then you can run ssh "$hostname" "$cmd_q" safely, but only under this condition).
bash -s reads the script to run from stdin, meaning that passing your command there -- not on the argument vector -- ensures that it'll be parsed into arguments by the same shell that escaped it to be shell-safe.
You want to pass the entire command -- with all of its arguments, spaces, and quotation marks -- to ssh so it can pass it unchanged to the remote shell for parsing.
One way to do that is to put it all inside single quotation marks. But then you'll also need to make sure the single quotation marks within your command are preserved in the arguments, so the remote shell builds the correct arguments for sed.
sshall.sh 'sed -i '"'"'/^somebeginning/ s/$/,appendme/'"'"' /etc/myconfig.conf'
It looks redundant, but '"'"' is a common Bourne trick to get a single quotation mark into a single-quoted string. The first quote ends single-quoting temporarily, the double-quote-single-quote-double-quote construct appends a single quotation mark, and then the single quotation mark resumes your single-quoted section. So to speak.
Another trick that can be helpful for troubleshooting is to add the -v flag do your ssh flags, which will spit out lots of text, but most importantly it will show you exactly what string it's passing to the remote shell for parsing and execution.
--
All of this is fairly fragile around spaces in your arguments, which you'll need to avoid, since you're relying on shell parsing on the opposite end.
Thinking outside the box: instead of dealing with all the quoting issues and the word-splitting in the wrong places, you could attempt to a) construct the script locally (maybe use a here-document?), b) scp the script to the remote end, then c) invoke it there. This easily allows more complex command sequences, with all the power of shell control constructs etc. Debugging (checking proper quoting) would be a breeze by simply looking at the generated script.
I recommend reading the command(s) from the standard input rather than from the command line arguments:
cmd.sh
#!/bin/bash -
# Load server_list with user#host "words" here.
cmd=$(</dev/stdin)
for h in ${server_list[*]}; do
ssh "$h" "$cmd"
done
Usage:
./cmd.sh <<'CMD'
sed -i '/^somebeginning/ s/$/,appendme/' /path/to/file1
# other commands
# here...
CMD
Alternatively, run ./cmd.sh, type the command(s), then press Ctrl-D.
I find the latter variant the most convenient, as you don't even need for here documents, no need for extra escaping. Just invoke your script, type the commands, and press the shortcut. What could be easier?
Explanations
The problem with your approach is that the quotes are stripped from the arguments by the shell. For example, the argument '/^somebeginning/ s/$/,appendme/' will be interpreted as /^somebeginning/ s/$/,appendme/ string (without the single quotes), which is an invalid argument for sed.
Of course, you can escape the command with the built-in printf as suggested in other answer here. But the command becomes not very readable after escaping. For example
printf %q 'sed -i /^somebeginning/ s/$/,appendme/ /home/ruslan/tmp/file1.txt'
produces
sed\ -i\ /\^somebeginning/\ s/\$/\,appendme/\ /home/ruslan/tmp/file1.txt
which is not very readable, and will look ugly, if you print it to the screen in order to show the progress.
That's why I prefer to read from the standard input and leave the command intact. My script prints the command strings to the screen, and I see them just in the form I have written them.
Note, the for .. in loop iterates $IFS-separated "words", and is generally not preferred way to traverse an array. It is generally better to invoke read -r in a while loop with adjusted $IFS. I have used the for loop for simplicity, as the question is really about invoking the ssh command.
Logging into multiple systems over SSH and using the same (or variations on the same) command is the basic use case behind ansible. The system is not without significant flaws, but for simple use cases is pretty great. If you want a more solid solution without too much faffing about with escaping and looping over hosts, take a look.
Ansible has a 'raw' module which doesn't even require any dependencies on the target hosts, and you might find that a very simple way to achieve this sort of functionality in a way that frees you from the considerations of looping over hosts, handling errors, marshalling the commands, etc and lets you focus on what you're actually trying to achieve.
I have several text files in which I have introduced shell variables ($VAR1 or $VAR2 for instance).
I would like to take those files (one by one) and save them in new files where all variables would have been replaced.
To do this, I used the following shell script (found on StackOverflow):
while read line
do
eval echo "$line" >> destination.txt
done < "source.txt"
This works very well on very basic files.
But on more complex files, the "eval" command does too much:
Lines starting with "#" are skipped
XML files parsing results in tons of errors
Is there a better way to do it? (in shell script... I know this is easily done with Ant for instance)
Kind regards
Looking, it turns out on my system there is an envsubst command which is part of the gettext-base package.
So, this makes it easy:
envsubst < "source.txt" > "destination.txt"
Note if you want to use the same file for both, you'll have to use something like moreutil's sponge, as suggested by Johnny Utahh: envsubst < "source.txt" | sponge "source.txt". (Because the shell redirect will otherwise empty the file before its read.)
In reference to answer 2, when discussing envsubst, you asked:
How can I make it work with the variables that are declared in my .sh script?
The answer is you simply need to export your variables before calling envsubst.
You can also limit the variable strings you want to replace in the input using the envsubst SHELL_FORMAT argument (avoiding the unintended replacement of a string in the input with a common shell variable value - e.g. $HOME).
For instance:
export VAR1='somevalue' VAR2='someothervalue'
MYVARS='$VAR1:$VAR2'
envsubst "$MYVARS" <source.txt >destination.txt
Will replace all instances of $VAR1 and $VAR2 (and only VAR1 and VAR2) in source.txt with 'somevalue' and 'someothervalue' respectively.
I know this topic is old, but I have a simpler working solution without exporting the variables. Can be a oneliner, but I prefer to split using \ on line end.
var1='myVar1'\
var2=2\
var3=${var1}\
envsubst '$var1,$var3' < "source.txt" > "destination.txt"
# ^^^^^^^^^^^ ^^^^^^^^^^ ^^^^^^^^^^^^^^^
# define which to replace input output
The variables need to be defined to the same line as envsubst is to get considered as environment variables.
The '$var1,$var3' is optional to only replace the specified ones. Imagine an input file containing ${VARIABLE_USED_BY_JENKINS} which should not be replaced.
Define your ENV variable
$ export MY_ENV_VAR=congratulation
Create template file (in.txt) with following content
$MY_ENV_VAR
You can also use all other ENV variables defined by your system like (in linux) $TERM, $SHELL, $HOME...
Run this command to raplace all env-variables in your in.txt file and to write the result to out.txt
$ envsubst "`printf '${%s} ' $(sh -c "env|cut -d'=' -f1")`" < in.txt > out.txt
Check the content of out.txt file
$ cat out.txt
and you should see "congratulation".
There is also this option:
define your variables in a file
$ cat variables.env
# info about what this var is
export var1=a
# info about var again
export var2=b
define a template file that uses the variables
$ cat file1-template.txt
This is var1: "${var1}"
This is var2: "${var2}"
generate the final file, with variables replaced with values
$ source variables.env
$ envsubst < file1-template.txt > file1.txt
$ cat file1.txt
This is var1: "a"
This is var2: "b"
If you want env variables to be replaced in your source files while keeping all of the non env variables as they are, you can use the following command:
envsubst "$(printf '${%s} ' $(env | sed 's/=.*//'))" < source.txt > destination.txt
The syntax for replacing only specific variables is explained here. The command above is using a sub-shell to list all defined variables and then passing it to the envsubst
So if there's a defined env variable called $NAME, and your source.txt file looks like this:
Hello $NAME
Your balance is 123 ($USD)
The destination.txt will be:
Hello Arik
Your balance is 123 ($USD)
Notice that the $NAME is replaced and the $USD is left untouched
while IFS='=' read -r name value ; do
# Print line if found variable
sed -n '/${'"${name}"'}/p' docker-compose.yml
# Replace variable with value.
sed -i 's|${'"${name}"'}|'"${value}"'|' docker-compose.yml
done < <(env)
Note: Variable name or value should not contain "|", because it is used as a delimiter.
If you really only want to use bash (and sed), then I would go through each of your environment variables (as returned by set in posix mode) and build a bunch of -e 'regex' for sed from that, terminated by a -e 's/\$[a-zA-Z_][a-zA-Z0-9_]*//g', then pass all that to sed.
Perl would do a nicer job though, you have access to the environment vars as an array and you can do executable replacements so you only match any environment variable once.
Actually you need to change your read to read -r which will make it ignore backslashes.
Also, you should escape quotes and backslashes.
So
while read -r line; do
line="${line//\\/\\\\}"
line="${line//\"/\\\"}"
line="${line//\`/\\\`}"
eval echo "\"$line\""
done > destination.txt < source.txt
Still a terrible way to do expansion though.
Export all the needed variables and then use a perl onliner
TEXT=$(echo "$TEXT"|perl -wpne 's#\${?(\w+)}?# $ENV{$1} // $& #ge;')
This will replace all the ENV variables present in TEXT with actual values.
Quotes are also preserved :)
Call the perl binary, in search and replace per line mode ( the -pi ) by running the perl code ( the -e) in the single quotes, which iterates over the keys of the special %ENV hash containing the exported variable names as keys and the exported variable values as the keys' values and for each iteration simple replace a string containing a $<<key>> with its <<value>>.
perl -pi -e 'foreach $key(sort keys %ENV){ s/\$$key/$ENV{$key}/g}' file
Caveat:
An additional logic handling is required for cases in which two or more vars start with the same string ...
envsubst seems exactly like something I wanted to use, but -v option surprised me a bit.
While envsubst < template.txt was working fine, the same with option -v was not working:
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.1 (Maipo)
$ envsubst -V
envsubst (GNU gettext-runtime) 0.18.2
Copyright (C) 2003-2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Bruno Haible.
As I wrote, this was not working:
$ envsubst -v < template.txt
envsubst: missing arguments
$ cat template.txt | envsubst -v
envsubst: missing arguments
I had to do this to make it work:
TEXT=`cat template.txt`; envsubst -v "$TEXT"
Maybe it helps someone.
I'm writing a shell script that should be somewhat secure, i.e., does not pass secure data through parameters of commands and preferably does not use temporary files. How can I pass a variable to the standard input of a command?
Or, if it's not possible, how can I correctly use temporary files for such a task?
Passing a value to standard input in Bash is as simple as:
your-command <<< "$your_variable"
Always make sure you put quotes around variable expressions!
Be cautious, that this will probably work only in bash and will not work in sh.
Simple, but error-prone: using echo
Something as simple as this will do the trick:
echo "$blah" | my_cmd
Do note that this may not work correctly if $blah contains -n, -e, -E etc; or if it contains backslashes (bash's copy of echo preserves literal backslashes in absence of -e by default, but will treat them as escape sequences and replace them with corresponding characters even without -e if optional XSI extensions are enabled).
More sophisticated approach: using printf
printf '%s\n' "$blah" | my_cmd
This does not have the disadvantages listed above: all possible C strings (strings not containing NULs) are printed unchanged.
(cat <<END
$passwd
END
) | command
The cat is not really needed, but it helps to structure the code better and allows you to use more commands in parentheses as input to your command.
Note that the 'echo "$var" | command operations mean that standard input is limited to the line(s) echoed. If you also want the terminal to be connected, then you'll need to be fancier:
{ echo "$var"; cat - ; } | command
( echo "$var"; cat - ) | command
This means that the first line(s) will be the contents of $var but the rest will come from cat reading its standard input. If the command does not do anything too fancy (try to turn on command line editing, or run like vim does) then it will be fine. Otherwise, you need to get really fancy - I think expect or one of its derivatives is likely to be appropriate.
The command line notations are practically identical - but the second semi-colon is necessary with the braces whereas it is not with parentheses.
This robust and portable way has already appeared in comments. It should be a standalone answer.
printf '%s' "$var" | my_cmd
or
printf '%s\n' "$var" | my_cmd
Notes:
It's better than echo, reasons are here: Why is printf better than echo?
printf "$var" is wrong. The first argument is format where various sequences like %s or \n are interpreted. To pass the variable right, it must not be interpreted as format.
Usually variables don't contain trailing newlines. The former command (with %s) passes the variable as it is. However tools that work with text may ignore or complain about an incomplete line (see Why should text files end with a newline?). So you may want the latter command (with %s\n) which appends a newline character to the content of the variable. Non-obvious facts:
Here string in Bash (<<<"$var" my_cmd) does append a newline.
Any method that appends a newline results in non-empty stdin of my_cmd, even if the variable is empty or undefined.
I liked Martin's answer, but it has some problems depending on what is in the variable. This
your-command <<< """$your_variable"""
is better if you variable contains " or !.
As per Martin's answer, there is a Bash feature called Here Strings (which itself is a variant of the more widely supported Here Documents feature):
3.6.7 Here Strings
A variant of here documents, the format is:
<<< word
The word is expanded and supplied to the command on its standard
input.
Note that Here Strings would appear to be Bash-only, so, for improved portability, you'd probably be better off with the original Here Documents feature, as per PoltoS's answer:
( cat <<EOF
$variable
EOF
) | cmd
Or, a simpler variant of the above:
(cmd <<EOF
$variable
EOF
)
You can omit ( and ), unless you want to have this redirected further into other commands.
Try this:
echo "$variable" | command
If you came here from a duplicate, you are probably a beginner who tried to do something like
"$variable" >file
or
"$variable" | wc -l
where you obviously meant something like
echo "$variable" >file
echo "$variable" | wc -l
(Real beginners also forget the quotes; usually use quotes unless you have a specific reason to omit them, at least until you understand quoting.)