Related
Here are a series of cases where echo $var can show a different value than what was just assigned. This happens regardless of whether the assigned value was "double quoted", 'single quoted' or unquoted.
How do I get the shell to set my variable correctly?
Asterisks
The expected output is /* Foobar is free software */, but instead I get a list of filenames:
$ var="/* Foobar is free software */"
$ echo $var
/bin /boot /dev /etc /home /initrd.img /lib /lib64 /media /mnt /opt /proc ...
Square brackets
The expected value is [a-z], but sometimes I get a single letter instead!
$ var=[a-z]
$ echo $var
c
Line feeds (newlines)
The expected value is a a list of separate lines, but instead all the values are on one line!
$ cat file
foo
bar
baz
$ var=$(cat file)
$ echo $var
foo bar baz
Multiple spaces
I expected a carefully aligned table header, but instead multiple spaces either disappear or are collapsed into one!
$ var=" title | count"
$ echo $var
title | count
Tabs
I expected two tab separated values, but instead I get two space separated values!
$ var=$'key\tvalue'
$ echo $var
key value
In all of the cases above, the variable is correctly set, but not correctly read! The right way is to use double quotes when referencing:
echo "$var"
This gives the expected value in all the examples given. Always quote variable references!
Why?
When a variable is unquoted, it will:
Undergo field splitting where the value is split into multiple words on whitespace (by default):
Before: /* Foobar is free software */
After: /*, Foobar, is, free, software, */
Each of these words will undergo pathname expansion, where patterns are expanded into matching files:
Before: /*
After: /bin, /boot, /dev, /etc, /home, ...
Finally, all the arguments are passed to echo, which writes them out separated by single spaces, giving
/bin /boot /dev /etc /home Foobar is free software Desktop/ Downloads/
instead of the variable's value.
When the variable is quoted it will:
Be substituted for its value.
There is no step 2.
This is why you should always quote all variable references, unless you specifically require word splitting and pathname expansion. Tools like shellcheck are there to help, and will warn about missing quotes in all the cases above.
You may want to know why this is happening. Together with the great explanation by that other guy, find a reference of Why does my shell script choke on whitespace or other special characters? written by Gilles in Unix & Linux:
Why do I need to write "$foo"? What happens without the quotes?
$foo does not mean “take the value of the variable foo”. It means
something much more complex:
First, take the value of the variable.
Field splitting: treat that value as a whitespace-separated list of fields, and build the resulting list. For example, if the variable
contains foo * bar then the result of this step is the 3-element
list foo, *, bar.
Filename generation: treat each field as a glob, i.e. as a wildcard pattern, and replace it by the list of file names that match this
pattern. If the pattern doesn't match any files, it is left
unmodified. In our example, this results in the list containing foo,
following by the list of files in the current directory, and finally
bar. If the current directory is empty, the result is foo, *,
bar.
Note that the result is a list of strings. There are two contexts in
shell syntax: list context and string context. Field splitting and
filename generation only happen in list context, but that's most of
the time. Double quotes delimit a string context: the whole
double-quoted string is a single string, not to be split. (Exception:
"$#" to expand to the list of positional parameters, e.g. "$#" is
equivalent to "$1" "$2" "$3" if there are three positional
parameters. See What is the difference between $* and $#?)
The same happens to command substitution with $(foo) or with
`foo`. On a side note, don't use `foo`: its quoting rules are
weird and non-portable, and all modern shells support $(foo) which
is absolutely equivalent except for having intuitive quoting rules.
The output of arithmetic substitution also undergoes the same
expansions, but that isn't normally a concern as it only contains
non-expandable characters (assuming IFS doesn't contain digits or
-).
See When is double-quoting necessary? for more details about the
cases when you can leave out the quotes.
Unless you mean for all this rigmarole to happen, just remember to
always use double quotes around variable and command substitutions. Do
take care: leaving out the quotes can lead not just to errors but to
security
holes.
In addition to other issues caused by failing to quote, -n and -e can be consumed by echo as arguments. (Only the former is legal per the POSIX spec for echo, but several common implementations violate the spec and consume -e as well).
To avoid this, use printf instead of echo when details matter.
Thus:
$ vars="-e -n -a"
$ echo $vars # breaks because -e and -n can be treated as arguments to echo
-a
$ echo "$vars"
-e -n -a
However, correct quoting won't always save you when using echo:
$ vars="-n"
$ echo "$vars"
$ ## not even an empty line was printed
...whereas it will save you with printf:
$ vars="-n"
$ printf '%s\n' "$vars"
-n
user double quote to get the exact value. like this:
echo "${var}"
and it will read your value correctly.
echo $var output highly depends on the value of IFS variable. By default it contains space, tab, and newline characters:
[ks#localhost ~]$ echo -n "$IFS" | cat -vte
^I$
This means that when shell is doing field splitting (or word splitting) it uses all these characters as word separators. This is what happens when referencing a variable without double quotes to echo it ($var) and thus expected output is altered.
One way to prevent word splitting (besides using double quotes) is to set IFS to null. See http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_06_05 :
If the value of IFS is null, no field splitting shall be performed.
Setting to null means setting to empty
value:
IFS=
Test:
[ks#localhost ~]$ echo -n "$IFS" | cat -vte
^I$
[ks#localhost ~]$ var=$'key\nvalue'
[ks#localhost ~]$ echo $var
key value
[ks#localhost ~]$ IFS=
[ks#localhost ~]$ echo $var
key
value
[ks#localhost ~]$
The answer from ks1322 helped me to identify the issue while using docker-compose exec:
If you omit the -T flag, docker-compose exec add a special character that break output, we see b instead of 1b:
$ test=$(/usr/local/bin/docker-compose exec db bash -c "echo 1")
$ echo "${test}b"
b
echo "${test}" | cat -vte
1^M$
With -T flag, docker-compose exec works as expected:
$ test=$(/usr/local/bin/docker-compose exec -T db bash -c "echo 1")
$ echo "${test}b"
1b
Additional to putting the variable in quotation, one could also translate the output of the variable using tr and converting spaces to newlines.
$ echo $var | tr " " "\n"
foo
bar
baz
Although this is a little more convoluted, it does add more diversity with the output as you can substitute any character as the separator between array variables.
Here are a series of cases where echo $var can show a different value than what was just assigned. This happens regardless of whether the assigned value was "double quoted", 'single quoted' or unquoted.
How do I get the shell to set my variable correctly?
Asterisks
The expected output is /* Foobar is free software */, but instead I get a list of filenames:
$ var="/* Foobar is free software */"
$ echo $var
/bin /boot /dev /etc /home /initrd.img /lib /lib64 /media /mnt /opt /proc ...
Square brackets
The expected value is [a-z], but sometimes I get a single letter instead!
$ var=[a-z]
$ echo $var
c
Line feeds (newlines)
The expected value is a a list of separate lines, but instead all the values are on one line!
$ cat file
foo
bar
baz
$ var=$(cat file)
$ echo $var
foo bar baz
Multiple spaces
I expected a carefully aligned table header, but instead multiple spaces either disappear or are collapsed into one!
$ var=" title | count"
$ echo $var
title | count
Tabs
I expected two tab separated values, but instead I get two space separated values!
$ var=$'key\tvalue'
$ echo $var
key value
In all of the cases above, the variable is correctly set, but not correctly read! The right way is to use double quotes when referencing:
echo "$var"
This gives the expected value in all the examples given. Always quote variable references!
Why?
When a variable is unquoted, it will:
Undergo field splitting where the value is split into multiple words on whitespace (by default):
Before: /* Foobar is free software */
After: /*, Foobar, is, free, software, */
Each of these words will undergo pathname expansion, where patterns are expanded into matching files:
Before: /*
After: /bin, /boot, /dev, /etc, /home, ...
Finally, all the arguments are passed to echo, which writes them out separated by single spaces, giving
/bin /boot /dev /etc /home Foobar is free software Desktop/ Downloads/
instead of the variable's value.
When the variable is quoted it will:
Be substituted for its value.
There is no step 2.
This is why you should always quote all variable references, unless you specifically require word splitting and pathname expansion. Tools like shellcheck are there to help, and will warn about missing quotes in all the cases above.
You may want to know why this is happening. Together with the great explanation by that other guy, find a reference of Why does my shell script choke on whitespace or other special characters? written by Gilles in Unix & Linux:
Why do I need to write "$foo"? What happens without the quotes?
$foo does not mean “take the value of the variable foo”. It means
something much more complex:
First, take the value of the variable.
Field splitting: treat that value as a whitespace-separated list of fields, and build the resulting list. For example, if the variable
contains foo * bar then the result of this step is the 3-element
list foo, *, bar.
Filename generation: treat each field as a glob, i.e. as a wildcard pattern, and replace it by the list of file names that match this
pattern. If the pattern doesn't match any files, it is left
unmodified. In our example, this results in the list containing foo,
following by the list of files in the current directory, and finally
bar. If the current directory is empty, the result is foo, *,
bar.
Note that the result is a list of strings. There are two contexts in
shell syntax: list context and string context. Field splitting and
filename generation only happen in list context, but that's most of
the time. Double quotes delimit a string context: the whole
double-quoted string is a single string, not to be split. (Exception:
"$#" to expand to the list of positional parameters, e.g. "$#" is
equivalent to "$1" "$2" "$3" if there are three positional
parameters. See What is the difference between $* and $#?)
The same happens to command substitution with $(foo) or with
`foo`. On a side note, don't use `foo`: its quoting rules are
weird and non-portable, and all modern shells support $(foo) which
is absolutely equivalent except for having intuitive quoting rules.
The output of arithmetic substitution also undergoes the same
expansions, but that isn't normally a concern as it only contains
non-expandable characters (assuming IFS doesn't contain digits or
-).
See When is double-quoting necessary? for more details about the
cases when you can leave out the quotes.
Unless you mean for all this rigmarole to happen, just remember to
always use double quotes around variable and command substitutions. Do
take care: leaving out the quotes can lead not just to errors but to
security
holes.
In addition to other issues caused by failing to quote, -n and -e can be consumed by echo as arguments. (Only the former is legal per the POSIX spec for echo, but several common implementations violate the spec and consume -e as well).
To avoid this, use printf instead of echo when details matter.
Thus:
$ vars="-e -n -a"
$ echo $vars # breaks because -e and -n can be treated as arguments to echo
-a
$ echo "$vars"
-e -n -a
However, correct quoting won't always save you when using echo:
$ vars="-n"
$ echo "$vars"
$ ## not even an empty line was printed
...whereas it will save you with printf:
$ vars="-n"
$ printf '%s\n' "$vars"
-n
user double quote to get the exact value. like this:
echo "${var}"
and it will read your value correctly.
echo $var output highly depends on the value of IFS variable. By default it contains space, tab, and newline characters:
[ks#localhost ~]$ echo -n "$IFS" | cat -vte
^I$
This means that when shell is doing field splitting (or word splitting) it uses all these characters as word separators. This is what happens when referencing a variable without double quotes to echo it ($var) and thus expected output is altered.
One way to prevent word splitting (besides using double quotes) is to set IFS to null. See http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_06_05 :
If the value of IFS is null, no field splitting shall be performed.
Setting to null means setting to empty
value:
IFS=
Test:
[ks#localhost ~]$ echo -n "$IFS" | cat -vte
^I$
[ks#localhost ~]$ var=$'key\nvalue'
[ks#localhost ~]$ echo $var
key value
[ks#localhost ~]$ IFS=
[ks#localhost ~]$ echo $var
key
value
[ks#localhost ~]$
The answer from ks1322 helped me to identify the issue while using docker-compose exec:
If you omit the -T flag, docker-compose exec add a special character that break output, we see b instead of 1b:
$ test=$(/usr/local/bin/docker-compose exec db bash -c "echo 1")
$ echo "${test}b"
b
echo "${test}" | cat -vte
1^M$
With -T flag, docker-compose exec works as expected:
$ test=$(/usr/local/bin/docker-compose exec -T db bash -c "echo 1")
$ echo "${test}b"
1b
Additional to putting the variable in quotation, one could also translate the output of the variable using tr and converting spaces to newlines.
$ echo $var | tr " " "\n"
foo
bar
baz
Although this is a little more convoluted, it does add more diversity with the output as you can substitute any character as the separator between array variables.
I'm trying to write a bash script that allows the user to pass a directory path using wildcards.
For example,
bash show_files.sh *
when executed within this directory
drw-r--r-- 2 root root 4.0K Sep 18 11:33 dir_a
-rw-r--r-- 1 root root 223 Sep 18 11:33 file_b.txt
-rw-rw-r-- 1 root root 106 Oct 18 15:48 file_c.sql
would output:
dir_a
file_b.txt
file_c.sql
The way it is right now, it outputs:
dir_a
contents of show_files.sh:
#!/bin/bash
dirs="$1"
for dir in $dirs
do
echo $dir
done
The parent shell, the one invoking bash show_files.sh *, expands the * for you.
In your script, you need to use:
for dir in "$#"
do
echo "$dir"
done
The double quotes ensure that multiple spaces etc in file names are handled correctly.
See also How to iterate over arguments in a bash shell script.
Potentially confusing addendum
If you're truly sure you want to get the script to expand the *, you have to make sure that * is passed to the script (enclosed in quotes, as in the other answers), and then make sure it is expanded at the right point in the processing (which is not trivial). At that point, I'd use an array.
names=( $# )
for file in "${names[#]}"
do
echo "$file"
done
I don't often use $# without the double quotes, but this is one time when it is more or less the correct thing to do. The tricky part is that it won't handle wild cards with spaces in very well.
Consider:
$ > "double space.c"
$ > "double space.h"
$ echo double\ \ space.?
double space.c double space.h
$
That works fine. But try passing that as a wild-card to the script and ... well, let's just say it gets to be tricky at that point.
If you want to extract $2 separately, then you can use:
names=( $1 )
for file in "${names[#]}"
do
echo "$file"
done
# ... use $2 ...
Quote the wild-card:
bash show_files.sh '*'
or make your script accept a list of arguments, not just one:
for dir in "$#"
do
echo "$dir"
done
It's better to iterate directly over "$#' rather than assigning it to another variable, in order to preserve its special ability to hold elements that themselves contain whitespace.
This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
Closed 6 years ago.
I understand the usage single quote and double quote.
but I don't know situation need to double quotes in the script.
there is no diff that statements
$ echo hello world! $1
$ echo "hello world! $1"
please show me diff between normal and double quotes.
Let's consider a directory with these files:
$ ls foo*
foo111.txt foo11.txt foo1.txt
Let's consider a minor variation on your script:
$ cat script
#!/bin/sh
echo No quotes $1
echo "Double quotes $1"
Now, let's run it:
$ bash script "foo*"
No quotes foo111.txt foo11.txt foo1.txt
Double quotes foo*
As you can see, the results are completely different. Without the double quotes, pathname expansion is performed.
To illustrate another difference:
$ bash script "long space"
No quotes long space
Double quotes long space
With double quotes, the long space between words is preserved. Without it, all contiguous whitespace is replaced with a single blank. This is an example of word splitting.
An example might demonstrate the use
To accommodate string with spaces
var=file name # Not the intended effect.
file is stored in a var and name is taken by shell as a separate cmd which gives you an error.
To prevent word splitting
var="file name"
cp $var newfile
Here $var expands to file name and in effect, the command would become
cp file name newfile
and cp would take file and name as 2 source files and newfile as the destination directory which gives you the error:
cp: target 'newfile' is not a directory
If there really exists a directory named 'newfile', it will give error:
cp: cannot stat 'file': No such file or directory
cp: cannot stat 'name': No such file or directory
The correct method is
cp "$var" newfile
In this case, the fully expanded $var is considered a single string.
Here are a series of cases where echo $var can show a different value than what was just assigned. This happens regardless of whether the assigned value was "double quoted", 'single quoted' or unquoted.
How do I get the shell to set my variable correctly?
Asterisks
The expected output is /* Foobar is free software */, but instead I get a list of filenames:
$ var="/* Foobar is free software */"
$ echo $var
/bin /boot /dev /etc /home /initrd.img /lib /lib64 /media /mnt /opt /proc ...
Square brackets
The expected value is [a-z], but sometimes I get a single letter instead!
$ var=[a-z]
$ echo $var
c
Line feeds (newlines)
The expected value is a a list of separate lines, but instead all the values are on one line!
$ cat file
foo
bar
baz
$ var=$(cat file)
$ echo $var
foo bar baz
Multiple spaces
I expected a carefully aligned table header, but instead multiple spaces either disappear or are collapsed into one!
$ var=" title | count"
$ echo $var
title | count
Tabs
I expected two tab separated values, but instead I get two space separated values!
$ var=$'key\tvalue'
$ echo $var
key value
In all of the cases above, the variable is correctly set, but not correctly read! The right way is to use double quotes when referencing:
echo "$var"
This gives the expected value in all the examples given. Always quote variable references!
Why?
When a variable is unquoted, it will:
Undergo field splitting where the value is split into multiple words on whitespace (by default):
Before: /* Foobar is free software */
After: /*, Foobar, is, free, software, */
Each of these words will undergo pathname expansion, where patterns are expanded into matching files:
Before: /*
After: /bin, /boot, /dev, /etc, /home, ...
Finally, all the arguments are passed to echo, which writes them out separated by single spaces, giving
/bin /boot /dev /etc /home Foobar is free software Desktop/ Downloads/
instead of the variable's value.
When the variable is quoted it will:
Be substituted for its value.
There is no step 2.
This is why you should always quote all variable references, unless you specifically require word splitting and pathname expansion. Tools like shellcheck are there to help, and will warn about missing quotes in all the cases above.
You may want to know why this is happening. Together with the great explanation by that other guy, find a reference of Why does my shell script choke on whitespace or other special characters? written by Gilles in Unix & Linux:
Why do I need to write "$foo"? What happens without the quotes?
$foo does not mean “take the value of the variable foo”. It means
something much more complex:
First, take the value of the variable.
Field splitting: treat that value as a whitespace-separated list of fields, and build the resulting list. For example, if the variable
contains foo * bar then the result of this step is the 3-element
list foo, *, bar.
Filename generation: treat each field as a glob, i.e. as a wildcard pattern, and replace it by the list of file names that match this
pattern. If the pattern doesn't match any files, it is left
unmodified. In our example, this results in the list containing foo,
following by the list of files in the current directory, and finally
bar. If the current directory is empty, the result is foo, *,
bar.
Note that the result is a list of strings. There are two contexts in
shell syntax: list context and string context. Field splitting and
filename generation only happen in list context, but that's most of
the time. Double quotes delimit a string context: the whole
double-quoted string is a single string, not to be split. (Exception:
"$#" to expand to the list of positional parameters, e.g. "$#" is
equivalent to "$1" "$2" "$3" if there are three positional
parameters. See What is the difference between $* and $#?)
The same happens to command substitution with $(foo) or with
`foo`. On a side note, don't use `foo`: its quoting rules are
weird and non-portable, and all modern shells support $(foo) which
is absolutely equivalent except for having intuitive quoting rules.
The output of arithmetic substitution also undergoes the same
expansions, but that isn't normally a concern as it only contains
non-expandable characters (assuming IFS doesn't contain digits or
-).
See When is double-quoting necessary? for more details about the
cases when you can leave out the quotes.
Unless you mean for all this rigmarole to happen, just remember to
always use double quotes around variable and command substitutions. Do
take care: leaving out the quotes can lead not just to errors but to
security
holes.
In addition to other issues caused by failing to quote, -n and -e can be consumed by echo as arguments. (Only the former is legal per the POSIX spec for echo, but several common implementations violate the spec and consume -e as well).
To avoid this, use printf instead of echo when details matter.
Thus:
$ vars="-e -n -a"
$ echo $vars # breaks because -e and -n can be treated as arguments to echo
-a
$ echo "$vars"
-e -n -a
However, correct quoting won't always save you when using echo:
$ vars="-n"
$ echo "$vars"
$ ## not even an empty line was printed
...whereas it will save you with printf:
$ vars="-n"
$ printf '%s\n' "$vars"
-n
user double quote to get the exact value. like this:
echo "${var}"
and it will read your value correctly.
echo $var output highly depends on the value of IFS variable. By default it contains space, tab, and newline characters:
[ks#localhost ~]$ echo -n "$IFS" | cat -vte
^I$
This means that when shell is doing field splitting (or word splitting) it uses all these characters as word separators. This is what happens when referencing a variable without double quotes to echo it ($var) and thus expected output is altered.
One way to prevent word splitting (besides using double quotes) is to set IFS to null. See http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_06_05 :
If the value of IFS is null, no field splitting shall be performed.
Setting to null means setting to empty
value:
IFS=
Test:
[ks#localhost ~]$ echo -n "$IFS" | cat -vte
^I$
[ks#localhost ~]$ var=$'key\nvalue'
[ks#localhost ~]$ echo $var
key value
[ks#localhost ~]$ IFS=
[ks#localhost ~]$ echo $var
key
value
[ks#localhost ~]$
The answer from ks1322 helped me to identify the issue while using docker-compose exec:
If you omit the -T flag, docker-compose exec add a special character that break output, we see b instead of 1b:
$ test=$(/usr/local/bin/docker-compose exec db bash -c "echo 1")
$ echo "${test}b"
b
echo "${test}" | cat -vte
1^M$
With -T flag, docker-compose exec works as expected:
$ test=$(/usr/local/bin/docker-compose exec -T db bash -c "echo 1")
$ echo "${test}b"
1b
Additional to putting the variable in quotation, one could also translate the output of the variable using tr and converting spaces to newlines.
$ echo $var | tr " " "\n"
foo
bar
baz
Although this is a little more convoluted, it does add more diversity with the output as you can substitute any character as the separator between array variables.