My question is related to How to pass variables from a shell script to an expect script?
but its slightly different:
Apart from passing two runtime shell script variables, I want to pass a variable thats inside the shell script For eg:
#!/bin/sh
d=`date '+%Y%m%d_%H%M'`
expect -c '
expect "sftp>"
#use $d here(how?)
'
You don't need to pass a date variable into expect. It has a very good date command builtin:
expect -c '
# ...
set d [timestamp -format "%Y%m%d_%H%M"]
something_with $d
# ...
'
If you need to do more complicated date manipulation, read about the Tcl clock command
Another good technique to pass shell variables to expect (without having to do complicated/messy quoting/escaping) is to use the environment: export your shell variables, and then access them with expect's global env array:
export d=$(date ...)
expect -c 'puts "the date is $env(d)"'
This seems the wrong way to do things. You should set up SSH keys (with ssh-keygen and ssh-copy-id), google about this.
Anyway, try this :
#!/bin/sh
d=`date '+%Y%m%d_%H%M'`
expect -c "
something_with $d"
Note the double quotes instead of single quotes.
"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[#]}", "a & b". Use 'single quotes' for code or literal $'s: 'Costs $5 US', ssh host 'echo "$HOSTNAME"'. See http://mywiki.wooledge.org/Quotes , http://mywiki.wooledge.org/Arguments and http://wiki.bash-hackers.org/syntax/words
Related
I am trying to use a variable to store the parameters, here is the simple test:
#!/bin/bash
sed_args="-e \"s/aaaa/bbbb/g\""
echo $sed_args`
I expected the output to be
-e "s/aaaa/bbbb/g"
but it gives:
"s/aaaa/bbbb/g"
without the "-e"
I am new to bash, any comment is welcome. Thanks, maybe this is already answered somewhere.
You need an array to construct arguments dynamically:
#!/usr/bin/env bash
sed_args=('-e' 's/aaaa/bbbb/g')
echo "${sed_args[#]}"
When you use the variable without double quotes, it gets word split by the shell even before echo sees the value(s). Then, the bash's builtin echo interprets -e as a parameter for itself (which is normally used to turn on interpretation of backslash escapes).
When you double quote the variable, it won't be split and will be interpreted as a single argument to echo:
echo "$sed_args"
For strings you don't control, it's safer to use printf as it doesn't take any arguments after the format string:
printf %s "$string"
I am looking to write a simple script to perform a SSH command on many hosts simultaneously, and which hosts exactly are generated from another script. The problem is that when I run the script using sometihng like sed it doesn't work properly.
It should run like sshall.sh {anything here} and it will run the {anything here} part on all the nodes in the list.
sshall.sh
#!/bin/bash
NODES=`listNodes | grep "node-[0-9*]" -o`
echo "Connecting to all nodes and running: ${#:1}"
for i in $NODES
do
:
echo "$i : Begin"
echo "----------------------------------------"
ssh -q -o "StrictHostKeyChecking no" $i "${#:1}"
echo "----------------------------------------"
echo "$i : Complete";
echo ""
done
When it is run with something like whoami it works but when I run:
[root#myhost bin]# sshall.sh sed -i '/^somebeginning/ s/$/,appendme/' /etc/myconfig.conf
Connecting to all nodes and running: sed -i /^somebeginning/ s/$/,appendme/ /etc/myconfig.conf
node-1 : Begin
----------------------------------------
sed: -e expression #1, char 18: missing command
----------------------------------------
node-1 : Complete
node-2 : Begin
----------------------------------------
sed: -e expression #1, char 18: missing command
----------------------------------------
node-2 : Complete
…
Notice that the quotes disappear on the sed command when sent to the remote client.
How do I go about fixing my bash command?
Is there a better way of achieving this?
Substitute an eval-safe quoted version of your command into a heredoc:
#!/bin/bash
# ^^^^- not /bin/sh; printf %q is an extension
# Put your command into a single string, with each argument quoted to be eval-safe
printf -v cmd_q '%q ' "$#"
while IFS= read -r hostname; do
# run bash -s remotely, with that string passed on stdin
ssh -q -o 'StrictHostKeyChecking no' "$hostname" "bash -s" <<EOF
$cmd_q
EOF
done < <(listNodes | grep -o -e "node-[0-9*]")
Why this works reliably (and other approaches don't):
printf %q knows how to quote contents to be eval'd by that same shell (so spaces, wildcards, various local quoting methods, etc. will always be supported).
Arguments given to ssh are not passed to the remote command individually!
Instead, they're concatenated into a string passed to sh -c.
However: The output of printf %q is not portable to all POSIX-derived shells! It's guaranteed to be compatible with the same shell locally in use -- ksh will always parse output from printf '%q' in ksh, bash will parse output from printf '%q' in bash, etc; thus, you can't safely pass this string on the remote argument vector, because it's /bin/sh -- not bash -- running there. (If you know your remote /bin/sh is provided by bash, then you can run ssh "$hostname" "$cmd_q" safely, but only under this condition).
bash -s reads the script to run from stdin, meaning that passing your command there -- not on the argument vector -- ensures that it'll be parsed into arguments by the same shell that escaped it to be shell-safe.
You want to pass the entire command -- with all of its arguments, spaces, and quotation marks -- to ssh so it can pass it unchanged to the remote shell for parsing.
One way to do that is to put it all inside single quotation marks. But then you'll also need to make sure the single quotation marks within your command are preserved in the arguments, so the remote shell builds the correct arguments for sed.
sshall.sh 'sed -i '"'"'/^somebeginning/ s/$/,appendme/'"'"' /etc/myconfig.conf'
It looks redundant, but '"'"' is a common Bourne trick to get a single quotation mark into a single-quoted string. The first quote ends single-quoting temporarily, the double-quote-single-quote-double-quote construct appends a single quotation mark, and then the single quotation mark resumes your single-quoted section. So to speak.
Another trick that can be helpful for troubleshooting is to add the -v flag do your ssh flags, which will spit out lots of text, but most importantly it will show you exactly what string it's passing to the remote shell for parsing and execution.
--
All of this is fairly fragile around spaces in your arguments, which you'll need to avoid, since you're relying on shell parsing on the opposite end.
Thinking outside the box: instead of dealing with all the quoting issues and the word-splitting in the wrong places, you could attempt to a) construct the script locally (maybe use a here-document?), b) scp the script to the remote end, then c) invoke it there. This easily allows more complex command sequences, with all the power of shell control constructs etc. Debugging (checking proper quoting) would be a breeze by simply looking at the generated script.
I recommend reading the command(s) from the standard input rather than from the command line arguments:
cmd.sh
#!/bin/bash -
# Load server_list with user#host "words" here.
cmd=$(</dev/stdin)
for h in ${server_list[*]}; do
ssh "$h" "$cmd"
done
Usage:
./cmd.sh <<'CMD'
sed -i '/^somebeginning/ s/$/,appendme/' /path/to/file1
# other commands
# here...
CMD
Alternatively, run ./cmd.sh, type the command(s), then press Ctrl-D.
I find the latter variant the most convenient, as you don't even need for here documents, no need for extra escaping. Just invoke your script, type the commands, and press the shortcut. What could be easier?
Explanations
The problem with your approach is that the quotes are stripped from the arguments by the shell. For example, the argument '/^somebeginning/ s/$/,appendme/' will be interpreted as /^somebeginning/ s/$/,appendme/ string (without the single quotes), which is an invalid argument for sed.
Of course, you can escape the command with the built-in printf as suggested in other answer here. But the command becomes not very readable after escaping. For example
printf %q 'sed -i /^somebeginning/ s/$/,appendme/ /home/ruslan/tmp/file1.txt'
produces
sed\ -i\ /\^somebeginning/\ s/\$/\,appendme/\ /home/ruslan/tmp/file1.txt
which is not very readable, and will look ugly, if you print it to the screen in order to show the progress.
That's why I prefer to read from the standard input and leave the command intact. My script prints the command strings to the screen, and I see them just in the form I have written them.
Note, the for .. in loop iterates $IFS-separated "words", and is generally not preferred way to traverse an array. It is generally better to invoke read -r in a while loop with adjusted $IFS. I have used the for loop for simplicity, as the question is really about invoking the ssh command.
Logging into multiple systems over SSH and using the same (or variations on the same) command is the basic use case behind ansible. The system is not without significant flaws, but for simple use cases is pretty great. If you want a more solid solution without too much faffing about with escaping and looping over hosts, take a look.
Ansible has a 'raw' module which doesn't even require any dependencies on the target hosts, and you might find that a very simple way to achieve this sort of functionality in a way that frees you from the considerations of looping over hosts, handling errors, marshalling the commands, etc and lets you focus on what you're actually trying to achieve.
I am writing the bash script, which has to perform some of the commands on remote server via ssh. The script has two major parts:
Part 1:
Using local variables $A and $B
Part 2:
Execute commands on remote server as follows:
ssh -T user#servername << 'EOF'
...
Using local variables $A and $B
...
EOF
The problem is that local variables $A and $B are not available within scope of ssh commands on remote server. As far as I understand the variables $A and $B outside and within ssh scope are not the same.
So my question is how to pass local variables from bash script to ssh scope?
One more note, the part 2 is pretty big so I can't use "one liner" after ssh.
Thanks
The problem has nothing to do with ssh but is only related to here documents in bash or any other Posix shell.
Man page for bash says in the here document paragraph:
The format of here-documents is:
<<[-]word
here-document
delimiter
No parameter expansion, command substitution, arithmetic expansion, or pathname expansion is performed on word. If any characters in word are quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion. In the latter case, the character sequence \<newline> is ignored, and \ must be used to quote the characters \, $, and ` .
As you quote EOF, you explicitely ask the shell not to replace the $1 and $2 variables.
The most robust way is to not quote EOF and consistently quote all others special characters in the here document.
For example, if your here document contained something like:
ssh -T user#servername << 'EOF'
for f in /var/log/messages*
do echo "Filename: " $f
done
EOF
you could rewrite it with no quotes around EOF but with one inside $:
ssh -T user#servername << EOF
for f in /var/log/messages*
do echo "Filename: " \$f
done
EOF
That way all unquoted varibles would be interpolated.
Alternatively, if the server allows it, you can try to pass the 2 parameters as environment variables.
Say you want to use the names PARAM1 and PARAM2. The sshd_config file on the server should contain the line AcceptEnv PARAM1 PARAM2 because by default and for security reasons no environment variable is accepted.
You can then use:
export PARAM1=$1
export PARAM2=$2
ssh -T -o SendEnv=PARAM1 -o SenEnv=PARAM2 user#servername << 'EOF'
...
Using variables $PARAM1 and $PARAM2
...
EOF
There could be a way to directly tell ssh to use local variables but I'll quickly answer and mention that you could use a script that wraps the ssh with code that inserts the variables remotely where you will have access to them once you get control of the command prompt.
I am trying to create a script that uses the date command in bash. I am familiar with the basic syntax of the date command.
Here is the simple script:
#!/bin/bash
set -x
DATE_COMMAND="date "+%y-%m-%d %H:%M:%S""
echo "$($DATE_COMMAND)"
set +x
The thing is that the above code doesn't work.
Here is the output:
+ DATE_COMMAND='date +%y-%m-%d'
+ %H:%M:%S
onlyDate: line 3: fg: no job control
+ echo ''
+ set +x
Ok, so the problem is that the bash splits the command because of the space. I can understand that but I don't know how to avoid that. I have tried to avoid the space with \, to avoid the space and the ". Also the single quotes doesn't seem to work.
Please note that I know that this script can be written this way:
#!/bin/bash
set -x
DATE_COMMAND=$(date "+%y-%m-%d %H:%M:%S")
echo "$DATE_COMMAND"
set +x
I have tried that but I can't use this approach because I want to run the command several times in my script.
Any help will be really appreciated!
The correct approach is to define your own function inside your Bash script.
function my_date {
date "+%y-%m-%d %H:%M:%S"
}
Now you can use my_date as if it were an external program.
For example:
echo "It is now $(my_date)."
Or simply:
my_date
Why isn't your approach working?
The first problem is that your assignment is broken.
DATE_COMMAND="date "+%y-%m-%d %H:%M:%S""
This is parsed as an assignment of the string date +%y-%m-%d to the variable DATE_COMMAND. After the blank, the shell starts interpreting the remaining symbols in ways you did not intend.
This could be partially fixed by changing the quotation.
DATE_COMMAND="date '+%y-%m-%d %H:%M:%S'"
However, this doesn't really solve the problem because if we now use
echo $($DATE_COMMAND)
It will not expand the argument correctly. The date program will see the arguments '+%y-%m-%d and %H:%M:%S' (with quotes) instead of a single string. This could be solved by using eval as in
DATE_COMMAND="date '+%y-%m-%d %H:%M:%S'"
echo $(eval $DATE_COMMAND)
where the variable DATE_COMMAND is first expanded to the string date '+%y-%m-%d %H:%M:%S' that is then evaluated as if it were written like so thus invoking date correctly.
Note that I'm only showing this to explain the issue. eval is not a good solution here. Use a function instead.
PS It is better to avoid all-uppercase identifier strings as those are often in conflict with environment variables or even have a magic meaning to the shell.
Escaping the space works for me.
echo `date +%d.%m.%Y\ %R.%S.%3N`
For long scripts
declare variables section:
dateformat="%Y-%m-%d %H:%M:%S"
anywhere to get date:
datestr=`date +"${dateformat}"`
anywhere to echo date:
echo ${datestr}
For short simple scripts DaniëlGu's answer is the best:
echo `date +%d.%m.%Y\ %R.%S.%3N`
Shortest answer is
#if you want to store in a variable
now=$(date '+%F" "%T');
echo $now
#or direct output
date '+%F" "%T'
I'm using:
DF="+%m/%d/%Y %H:%M:%S";
FILE_DATE=$(date "${DF[#]}" -r "${SCHEMA_LOCAL_PATH}.gz");
Seems Bash treats space in $DF as an array's separator and 'date' receives 2 agruments instead of one so I'm sending an array.
Also works for me (check "+" sign):
DF="%m/%d/%Y %H:%M:%S";
FILE_DATE=$(date +"${DF}" -r "${SCHEMA_LOCAL_PATH}.gz");
I'm debugging an old shell script; I want to check the values of all the variables used, it's a huge ugly script with approx more than 140 variables used. Is there anyway I can extract the variable names from the script and put them in a convenient pattern like:
#!/bin/sh
if [ ${BLAH} ....
.....
rm -rf ${JUNK}.....
to
echo ${BLAH}
echo ${JUNK}
...
Try running your script as follows:
bash -x ./script.bash
Or enable the setting in the script:
set -x
You can dump all interested variables in one command using:
set | grep -w -e BLAH -e JUNK
To dump all the variables to stdout use:
set
or
env
from inside your script.
You can extract a (sub)list of the variables declared in your script using grep:
grep -Po "([a-z][a-zA-Z0-9_]+)(?==\")" ./script.bash | sort -u
Disclaimer: why "sublist"?
The expression given will match string followed by an egal sign (=) and a double quote ("). So if you don't use syntax such as myvar="my-value" it won't work.
But you got the idea.
grep Options
-P --perl-regexp: Interpret PATTERN as a Perl regular expression (PCRE, see below) (experimental) ;
-o --only-matching: Print only the matched (non-empty) parts of a matching line, with each such part on a separate output line.
Pattern
I'm using a positive lookahead: (?==\") to require an egal sign followed by a double quote.
In bash, but not sh, compgen -v will list the names of all variables assigned (compare this to set, which has a great deal of output other than variable names, and thus needs to be parsed).
Thus, if you change the top of the script to #!/bin/bash, you will be able to use compgen -v to generate that list.
That said, the person who advised you use set -x did well. Consider this extension on that:
PS4=':$BASH_SOURCE:$LINENO+'; set -x
This will print the source file and line number before every command (or variable assignment) which is executed, so you will have a log not only of which variables are set, but just where in the source each one was assigned. This makes tracking down where each variable is set far easier.