I have a command that is returning a string with a leading whitespace that gets eaten when I use command substitution
> echo " test"
test
> echo $(echo " test")
test
Why is this happening and what can I do about it?
You can double-quote command substitution to preserve that,
echo "$(echo " test")"
test
From the man bash on page,
Command Substitution
Bash performs the expansion by executing command and replacing the command substitution with the standard output of the command, with any trailing newlines
deleted. Embedded newlines are not deleted, but they may be removed during word splitting.
If the substitution appears within double quotes, word splitting and pathname expansion are not performed on the results.
Related
This question already has answers here:
How to cat <<EOF >> a file containing code?
(5 answers)
Closed 5 years ago.
I'm trying to create a script file using substitution string from ENV but want also to prevent some from escaping
export PLACEHOLDER1="myPlaceholder1Value"
sudo /bin/su -c "cat << EOF > /etc/init.d/my-script
#!/bin/bash
myvariable_1=toto$PLACEHOLDER1
myvariable_final=\"dynamicvar=\${myvariable_1},\${myvariable_2}\"
EOF
"
It results in which is not good as the myvariable_final are not escaped and substituted as the one from the init script dependencies ($remote_fs, $syslog, $network, $time)
#!/bin/bash
myvariable_1=totomyPlaceholder1Value
myvariable_2=titimyPlaceholder2Value
myvariable_final="dynamicvar=,"
If i try to put a backslash \ behind the dollars $, I manage to avoid the substitution but I getting an unwanted backslash \:
export PLACEHOLDER1="myPlaceholder1Value"
export PLACEHOLDER2="myPlaceholder2Value"
sudo /bin/su -c "cat << EOF > /etc/init.d/my-script
#!/bin/bash
myvariable_1=toto$PLACEHOLDER1
myvariable_2=titi$PLACEHOLDER2
myvariable_final=\"dynamicvar=\$\{myvariable_1},\$\{myvariable_2}\"
EOF
"
results in:
#!/bin/bash
myvariable_1=totomyPlaceholder1Value
myvariable_2=titimyPlaceholder2Value
myvariable_final="dynamicvar=$\{myvariable_1},$\{myvariable_2}"
Wanted/attended result whould have been :
#!/bin/bash
myvariable_1=totomyPlaceholder1Value
myvariable_2=titimyPlaceholder2Value
myvariable_final="dynamicvar=${myvariable_1},${myvariable_2}"
solved by putting quote around the EOF as below and using backslash to control the escaping when needed
export PLACEHOLDER1="myPlaceholder1Value"
export PLACEHOLDER2="myPlaceholder2Value"
sudo /bin/su -c "cat << 'EOF' > /etc/init.d/my-script
#!/bin/bash
myvariable_1=toto$PLACEHOLDER1
myvariable_2=titi$PLACEHOLDER2
myvariable_final=\"dynamicvar=\${myvariable_1},\${myvariable_2}\"
EOF
"
Just use 'EOF' to prevent the variable from expanding:
sudo /bin/su -c "cat << 'EOF' > /etc/init.d/my-script
# ^ ^
From man bash:
Here Documents
This type of redirection instructs the shell to read input from the
current source until a line containing only delimiter (with no
trailing blanks) is seen. All of the lines read up to that point are
then used as the standard input for a command.
The format of here-documents is:
<<[-]word
here-document
delimiter
No parameter expansion, command substitution, arithmetic expansion,
or pathname expansion is performed on word. If any characters in word
are quoted, the delimiter is the result of quote removal on word, and
the lines in the here-document are not expanded. If word is
unquoted, all lines of the here-document are subjected to parameter
expansion, command substitution, and arithmetic expansion. In the
latter case, the character sequence \<newline> is ignored, and \
must be used to quote the characters \, $, and `.
when using the su command put the command itself in sigle quotes and just escape the $ with a backslash. the placeholder variables has to set in command bash context (here after su). so you need to do sth like
su -c 'ph="ph"; cat << EOF > script
varinscript=$ph
var=\${var}
EOF'
Is it possible to create a heredoc that does not become subject to variable expansion?
e.g.
cat <<-EOF > somefile.sh
Do not print current value of $1 instead evaluate it later.
EOF
Update I am aware of escaping by \. My actual heredoc has many variables in it - and it is error prone and tedious to escape all of them.
Quote the delimiter:
cat <<-"EOF" > somefile.sh
Do not print current value of $1 instead evaluate it later.
EOF
This results in:
$ cat somefile.sh
Do not print current value of $1 instead evaluate it later.
Documentation
The format of here-documents is:
<<[-]word
here-document
delimiter
No parameter and variable expansion, command substitution, arithmetic expansion, or pathname expansion is performed on word. If
any characters in word are quoted, the delimiter is the result of
quote removal on word, and the lines in the here-document are
not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command
substitution, and arithmetic expansion, the character sequence
\ is ignored, and \ must be used to quote the characters \,
$, and `.
If the redirection operator is <<-, then all leading tab characters are stripped from input lines and the line containing
delimiter. This allows here-documents within shell scripts to be
indented in a natural fashion. [Emphasis added.]
Put backlash before the $ sign
$ VAR=XXX
$ cat << END
> dk
> \$VAR
> END
dk
$VAR
I somehow can't figure out how to start a self-written programm with arguments from a shell script. If I'm in a folder whose parent folder contains the binary, then I can start the program with
$ ../binary --opt1 arg1 --opt2 arg2
Now, say the arguments and options are listed in a file args in the current folder.
args.txt:
--opt1 arg1 --opt2 arg2
If I'm trying to execute the binary from a shell script in the current folder like:
$ ./script.sh args.txt
script.sh:
#!/bin/bash
if [ $# != 1 ]
then
exit 1;
fi
params=$(<"$1")
../binary "$params"
# ../binary <<<"$params" doesn't work either.
How can I make this work?
Edit (updated script):
#!/bin/bash
if [ $# != 1 ]
then
exit 1;
fi
params=$(<"$1")
START=$(date +%s)
../binary "$params"
# ../binary <<<"$params" doesn't work either.
END=$(date +%s)
DIFF=$(( $END - $START ))
echo "Test took $DIFF seconds"
Use command substituion:
$ ./script.sh args.txt
content of ./script.sh
#!/bin/bash
../binary $(< "$1")
Command Substitution
Command substitution allows the output of a command to replace the command name. There are two forms:
$(command)
or
`command`
Bash performs the expansion by executing command and replacing the command substitution with the standard output of the command, with any trailing newlines deleted. Embedded newlines are not deleted, but they may be removed during word splitting. The command substitution $(cat file) can be replaced by the equivalent but faster $(< file).
When the old-style backquote form of substitution is used, backslash retains its literal meaning except when followed by $, `, or \. The first back‐quote not preceded by a backslash terminates the command substitution. When using the $(command) form, all characters between the parentheses make up the command; none are treated specially.
Command substitutions may be nested. To nest when using the backquoted form, escape the inner backquotes with backslashes. If the substitution appears within double quotes, word splitting and pathname expansion are not performed on the results.
I am working on a bash script that needs to create a file in this location:
/etc/yum.repos.d/nginx.repo
with the following contents:
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1
So, I have tried to do it like this:
cat >/etc/yum.repos.d/nginx.repo <<EOL
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/\$releasever/\$basearch/
gpgcheck=0
enabled=1
EOL
When I check the contents of the file, I see the following:
As you can see, the dollar sign weren't getting escaped, so the variable was evaluated to null/empty string and the contents do not look correct. Because, when I try to install nginx, I get this error:
http://nginx.org/packages/centos///repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
Any ideas?
In principle, it suffices to use a syntax
cat >file <<EOL
$my_var
EOL
That is, use the vars as they are, without escaping $.
So instead of
baseurl=http://nginx.org/packages/centos/\$releasever/\$basearch/
^ ^
say
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
From man bash:
Here Documents
This type of redirection instructs the shell to read input from the
current source until a line containing only delimiter (with no
trailing blanks) is seen. All of the lines read up to that point are
then used as the standard input for a command.
The format of here-documents is:
<<[-]word
here-document
delimiter
No parameter expansion, command substitution, arithmetic expansion,
or pathname expansion is performed on word. If any characters in word
are quoted, the delimiter is the result of quote removal on word, and
the lines in the here-document are not expanded. If word is
unquoted, all lines of the here-document are subjected to parameter
expansion, command substitution, and arithmetic expansion. In the
latter case, the character sequence \ is ignored, and \
must be used to quote the characters \, $, and `.
See an example:
$ cat a.sh
r="hello"
cat - <<EOL
hello
$r
EOL
echo "double quotes"
cat - <<"EOL"
hello
$r
EOL
echo "single quotes"
cat - <<'EOL'
hello
$r
EOL
And let's run it:
$ bash a.sh
hello
hello <-- it expands when unquoted
double quotes
hello
$r <-- it does not expand with "EOL"
single quotes
hello
$r <-- it does not expand with 'EOL'
There's an here-doc generic syntax to prevent the content to be expanded like when you put single quotes around variables :
cat<<'EOF'
:
cat<<'EOF' > /path/to/file
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1
EOF
From
man bash | less +/here-doc
If any characters in word are quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded.
Just wrap that string into single quotes
baseurl='http://nginx.org/packages/centos/$releasever/$basearch/'
Then the dollar sign would be treated as a usual character.
[root#xxx ~]# cat test
baseurl='http://nginx.org/packages/centos/$releasever/$basearch/'
I am trying to place a line of a text file, directly in a mail function. The string has the ' ' around it in the text file.
mailx -s "New Member" "cat ../address"
A line in ../address is 'my-email#gmail.com' including the ' ' quotes.
The cat ../address is not the best way of doing it but I do not know any other way to try.
Required result should be:
mailx -s "New Member" 'my-email#gmail.com'
You need to remove the literal ' chars. from the line in order for the address to be passed correctly to mailx:
There's more than one way to do it:
mailx -s "New Member" "$(tr -d "'" < ../address)"
mailx -s "New Member" "$(xargs < ../address)"
Note: The above assumes that ../address contains only 1 line.
#chepner makes an important point:
storing the quotes in the file and then reading them in [and using the result unquoted] is not the same as quoting the command substitution.
To elaborate on this further:
The OP shows 'my-email#gmail.com' as the desired argument to pass to mailx. In this direct form of single-quoting, bash will remove the quotes (see Quote Removal section in man bash) before handing the - unexpanded - string contents, without the quotes, to mailx.
Indirect quoting does NOT work: Reading in a string that happens to contain literal quote characters around it is NOT subject to quote removal by bash, so the enclosing quotes are passed through as part of the string - mailx would see an invalid email address that starts (and ends) with a '.
Therefore, the solution is:
remove the quotes from the input string first
then use direct quoting to protect the result from (further) shell expansion; note that double-quoting is needed so as to make sure the command substitution (($...)) is evaluated.
Use backticks for command substitution. I assume there is only one line in the file.
mailx -s "New Member" `cat ../address`
An alternative/better form for backticks is $()
mailx -s "New Member" $(cat ../address)