I have an script that echo the input given, into a file as follows:
echo $# > file.txt
When I pass a sting like "\"" I want it to exactly print "\"" to the file however it prints ".
My question is how can I print all characters of a variable containing a string without considering escapes?
When I use echo in bash like echo "\"" it only prints " while when I use echo '"\""' it prints it correctly. I thought maybe that would be the solution to use single quotes around the variable, however I cannot get the value of a variable inside single quotes.
First, note that
echo $# > file.txt
can fail in several ways. Shellcheck identifies one problem (missing quotes on $#). See the accepted, and excellent, answer to Why is printf better than echo? for others.
Second, as others have pointed out, there is no practical way for a Bash program to know exactly how parameters were specified on the command line. For instance, for all of these invocations
prog \"
prog "\""
prog '"'
the code in prog will see a $1 value that consists of one double-quote character. Any quoting characters that are used in the invocation of prog are removed by the quote removal part of the shell expansions done by the parent shell process.
Normally that doesn't matter. If variables or parameters contain values that would need to be quoted when entered as literals (e.g. "\"") they can be used safely, including passing them as parameters to other programs, by quoting uses of the variable or parameter (e.g. "$1", "$#", "$x").
There is a problem with variables or parameters that require quoting when entered literally if you need to write them in a way that they can be reused as shell input (e.g. by using eval or source/.). Bash supports the %q format specification to the printf builtin to handle this situation. It's not clear what the OP is trying to do, but one possible solution to the question is:
if (( $# > 0 )) ; then
printf -v quoted_params '%q ' "$#" # Add all parameters to 'quoted_params'
printf '%s\n' "${quoted_params% }" # Remove trailing space when printing
fi >file.txt
That creates an empty 'file.txt' when no positional parameters are provided. The code would need to be changed if that is not what is required.
If you run echo \", the function of the backslash in bash is to escape the character after it. This actually enables you to use the double quotes as an argument. You cannot use a backslash by itself; if you want to have a backslash as an argument you need to use another slash to escape that: echo \\
Now if you want to create a string where these things are not escaped, use single quotes: echo '\'
See for a better explanation this post: Difference between single and double quotes in Bash
I have a bash script which asks for two arguments with a space between them. Now I would like to automate filling out the prompt in the command line with reading from a text file. The text file contains a list with the argument combinations.
So something like this in the command line I think;
for line in 'cat text.file' ; do script.sh ; done
Can this be done? What am I missing/doing wrong?
Thanks for the help.
A while loop is probably what you need. Put the space separated strings in the file text.file :
cat text.file
bingo yankee
bravo delta
Then write the script in question like below.
#!/bin/bash
while read -r arg1 arg2
do
/path/to/your/script.sh "$arg1" "$arg2"
done<text.file
Don't use for to read files line by line
Try something like this:
#!/bin/bash
ARGS=
while IFS= read -r line; do
ARGS="${ARGS} ${line}"
done < ./text.file
script.sh "$ARGS"
This would add each line to a variable which then is used as the arguments of your script.
'cat text.file' is a string literal, $(cat text.file) would expand to output of command however cat is useless because bash can read file using redirection, also with quotes it will be treated as a single argument and without it will split at space tab and newlines.
Bash syntax to read a file line by line, but will be slow for big files
while IFS= read -r line; do ... "$line"; done < text.file
unsetting IFS for read command preserves leading spaces
-r option preserves \
another way, to read whole file is content=$(<file), note the < inside the command substitution. so a creative way to read a file to array, each element a non-empty line:
read_to_array () {
local oldsetf=${-//[^f]} oldifs=$IFS
set -f
IFS=$'\n' array_content=($(<"$1")) IFS=$oldifs
[[ $oldsetf ]]||set +f
}
read_to_array "file"
for element in "${array_content[#]}"; do ...; done
oldsetf used to store current set -f or set +f setting
oldifs used to store current IFS
IFS=$'\n' to split on newlines (multiple newlines will be treated as one)
set -f avoid glob expansion for example in case line contains single *
note () around $() to store the result of splitting to an array
If I were to create a solution determined by the literal of what you ask for (using a for loop and parsing lines from a file) I would use iterations determined by the number of lines in the file (if it isn't too large).
Assuming each line has two strings separated by a single space (to be used as positional parameters in your script:
file="$1"
f_count="$(wc -l < $file)"
for line in $(seq 1 $f_count)
do
script.sh $(head -n $line $file | tail -n1) && wait
done
You may have a much better time using sjsam's solution however.
This
STR="Hello\nWorld"
echo $STR
produces as output
Hello\nWorld
instead of
Hello
World
What should I do to have a newline in a string?
Note: This question is not about echo.
I'm aware of echo -e, but I'm looking for a solution that allows passing a string (which includes a newline) as an argument to other commands that do not have a similar option to interpret \n's as newlines.
If you're using Bash, you can use backslash-escapes inside of a specially-quoted $'string'. For example, adding \n:
STR=$'Hello\nWorld'
echo "$STR" # quotes are required here!
Prints:
Hello
World
If you're using pretty much any other shell, just insert the newline as-is in the string:
STR='Hello
World'
Bash recognizes a number of other backslash escape sequences in the $'' string. Here is an excerpt from the Bash manual page:
Words of the form $'string' are treated specially. The word expands to
string, with backslash-escaped characters replaced as specified by the
ANSI C standard. Backslash escape sequences, if present, are decoded
as follows:
\a alert (bell)
\b backspace
\e
\E an escape character
\f form feed
\n new line
\r carriage return
\t horizontal tab
\v vertical tab
\\ backslash
\' single quote
\" double quote
\nnn the eight-bit character whose value is the octal value
nnn (one to three digits)
\xHH the eight-bit character whose value is the hexadecimal
value HH (one or two hex digits)
\cx a control-x character
The expanded result is single-quoted, as if the dollar sign had not
been present.
A double-quoted string preceded by a dollar sign ($"string") will cause
the string to be translated according to the current locale. If the
current locale is C or POSIX, the dollar sign is ignored. If the
string is translated and replaced, the replacement is double-quoted.
Echo is so nineties and so fraught with perils that its use should result in core dumps no less than 4GB. Seriously, echo's problems were the reason why the Unix Standardization process finally invented the printf utility, doing away with all the problems.
So to get a newline in a string, there are two ways:
# 1) Literal newline in an assignment.
FOO="hello
world"
# 2) Command substitution.
BAR=$(printf "hello\nworld\n") # Alternative; note: final newline is deleted
printf '<%s>\n' "$FOO"
printf '<%s>\n' "$BAR"
There! No SYSV vs BSD echo madness, everything gets neatly printed and fully portable support for C escape sequences. Everybody please use printf now for all your output needs and never look back.
What I did based on the other answers was
NEWLINE=$'\n'
my_var="__between eggs and bacon__"
echo "spam${NEWLINE}eggs${my_var}bacon${NEWLINE}knight"
# which outputs:
spam
eggs__between eggs and bacon__bacon
knight
I find the -e flag elegant and straight forward
bash$ STR="Hello\nWorld"
bash$ echo -e $STR
Hello
World
If the string is the output of another command, I just use quotes
indexes_diff=$(git diff index.yaml)
echo "$indexes_diff"
The problem isn't with the shell. The problem is actually with the echo command itself, and the lack of double quotes around the variable interpolation. You can try using echo -e but that isn't supported on all platforms, and one of the reasons printf is now recommended for portability.
You can also try and insert the newline directly into your shell script (if a script is what you're writing) so it looks like...
#!/bin/sh
echo "Hello
World"
#EOF
or equivalently
#!/bin/sh
string="Hello
World"
echo "$string" # note double quotes!
The only simple alternative is to actually type a new line in the variable:
$ STR='new
line'
$ printf '%s' "$STR"
new
line
Yes, that means writing Enter where needed in the code.
There are several equivalents to a new line character.
\n ### A common way to represent a new line character.
\012 ### Octal value of a new line character.
\x0A ### Hexadecimal value of a new line character.
But all those require "an interpretation" by some tool (POSIX printf):
echo -e "new\nline" ### on POSIX echo, `-e` is not required.
printf 'new\nline' ### Understood by POSIX printf.
printf 'new\012line' ### Valid in POSIX printf.
printf 'new\x0Aline'
printf '%b' 'new\0012line' ### Valid in POSIX printf.
And therefore, the tool is required to build a string with a new-line:
$ STR="$(printf 'new\nline')"
$ printf '%s' "$STR"
new
line
In some shells, the sequence $' is a special shell expansion.
Known to work in ksh93, bash and zsh:
$ STR=$'new\nline'
Of course, more complex solutions are also possible:
$ echo '6e65770a6c696e650a' | xxd -p -r
new
line
Or
$ echo "new line" | sed 's/ \+/\n/g'
new
line
A $ right before single quotation marks '...\n...' as follows, however double quotation marks doesn't work.
$ echo $'Hello\nWorld'
Hello
World
$ echo $"Hello\nWorld"
Hello\nWorld
Disclaimer: I first wrote this and then stumbled upon this question. I thought this solution wasn't yet posted, and saw that tlwhitec did post a similar answer. Still I'm posting this because I hope it's a useful and thorough explanation.
Short answer:
This seems quite a portable solution, as it works on quite some shells (see comment).
This way you can get a real newline into a variable.
The benefit of this solution is that you don't have to use newlines in your source code, so you can indent
your code any way you want, and the solution still works. This makes it robust. It's also portable.
# Robust way to put a real newline in a variable (bash, dash, ksh, zsh; indentation-resistant).
nl="$(printf '\nq')"
nl=${nl%q}
Longer answer:
Explanation of the above solution:
The newline would normally be lost due to command substitution, but to prevent that, we add a 'q' and remove it afterwards. (The reason for the double quotes is explained further below.)
We can prove that the variable contains an actual newline character (0x0A):
printf '%s' "$nl" | hexdump -C
00000000 0a |.|
00000001
(Note that the '%s' was needed, otherwise printf will translate a literal '\n' string into an actual 0x0A character, meaning we would prove nothing.)
Of course, instead of the solution proposed in this answer, one could use this as well (but...):
nl='
'
... but that's less robust and can be easily damaged by accidentally indenting the code, or by forgetting to outdent it afterwards, which makes it inconvenient to use in (indented) functions, whereas the earlier solution is robust.
Now, as for the double quotes:
The reason for the double quotes " surrounding the command substitution as in nl="$(printf '\nq')" is that you can then even prefix the variable assignment with the local keyword or builtin (such as in functions), and it will still work on all shells, whereas otherwise the dash shell would have trouble, in the sense that dash would otherwise lose the 'q' and you'd end up with an empty 'nl' variable (again, due to command substitution).
That issue is better illustrated with another example:
dash_trouble_example() {
e=$(echo hello world) # Not using 'local'.
echo "$e" # Fine. Outputs 'hello world' in all shells.
local e=$(echo hello world) # But now, when using 'local' without double quotes ...:
echo "$e" # ... oops, outputs just 'hello' in dash,
# ... but 'hello world' in bash and zsh.
local f="$(echo hello world)" # Finally, using 'local' and surrounding with double quotes.
echo "$f" # Solved. Outputs 'hello world' in dash, zsh, and bash.
# So back to our newline example, if we want to use 'local', we need
# double quotes to surround the command substitution:
# (If we didn't use double quotes here, then in dash the 'nl' variable
# would be empty.)
local nl="$(printf '\nq')"
nl=${nl%q}
}
Practical example of the above solution:
# Parsing lines in a for loop by setting IFS to a real newline character:
nl="$(printf '\nq')"
nl=${nl%q}
IFS=$nl
for i in $(printf '%b' 'this is line 1\nthis is line 2'); do
echo "i=$i"
done
# Desired output:
# i=this is line 1
# i=this is line 2
# Exercise:
# Try running this example without the IFS=$nl assignment, and predict the outcome.
I'm no bash expert, but this one worked for me:
STR1="Hello"
STR2="World"
NEWSTR=$(cat << EOF
$STR1
$STR2
EOF
)
echo "$NEWSTR"
I found this easier to formatting the texts.
Those picky ones that need just the newline and despise the multiline code that breaks indentation, could do:
IFS="$(printf '\nx')"
IFS="${IFS%x}"
Bash (and likely other shells) gobble all the trailing newlines after command substitution, so you need to end the printf string with a non-newline character and delete it afterwards. This can also easily become a oneliner.
IFS="$(printf '\nx')" IFS="${IFS%x}"
I know this is two actions instead of one, but my indentation and portability OCD is at peace now :) I originally developed this to be able to split newline-only separated output and I ended up using a modification that uses \r as the terminating character. That makes the newline splitting work even for the dos output ending with \r\n.
IFS="$(printf '\n\r')"
On my system (Ubuntu 17.10) your example just works as desired, both when typed from the command line (into sh) and when executed as a sh script:
[bash]§ sh
$ STR="Hello\nWorld"
$ echo $STR
Hello
World
$ exit
[bash]§ echo "STR=\"Hello\nWorld\"
> echo \$STR" > test-str.sh
[bash]§ cat test-str.sh
STR="Hello\nWorld"
echo $STR
[bash]§ sh test-str.sh
Hello
World
I guess this answers your question: it just works. (I have not tried to figure out details such as at what moment exactly the substitution of the newline character for \n happens in sh).
However, i noticed that this same script would behave differently when executed with bash and would print out Hello\nWorld instead:
[bash]§ bash test-str.sh
Hello\nWorld
I've managed to get the desired output with bash as follows:
[bash]§ STR="Hello
> World"
[bash]§ echo "$STR"
Note the double quotes around $STR. This behaves identically if saved and run as a bash script.
The following also gives the desired output:
[bash]§ echo "Hello
> World"
I wasn't really happy with any of the options here. This is what worked for me.
str=$(printf "%s" "first line")
str=$(printf "$str\n%s" "another line")
str=$(printf "$str\n%s" "and another line")
This isn't ideal, but I had written a lot of code and defined strings in a way similar to the method used in the question. The accepted solution required me to refactor a lot of the code so instead, I replaced every \n with "$'\n'" and this worked for me.
I need to output a variable value to a file in a unix script. My problem it that the variable contains multiple lines. I need those to be output as '\n' literals in the file (a java options file), but I'm using echo and they always get processed into real new lines.
echo "-dmyproperty=$MULTILINE_VAR" >> jvm.options
I've tried echo options like -e o -E but they don't seem to do anything. Can anyone help?
You can use bash parameter substitution with an ANSI-C quoted newline
$ var="line1
line2
line3"
$ echo "${var//$'\n'/\\n}"
line1\nline2\nline3
I would like to copy the contents of a variable (here called var) into a file.
The name of the file is stored in another variable destfile.
I'm having problems doing this. Here's what I've tried:
cp $var $destfile
I've also tried the same thing with the dd command... Obviously the shell thought that $var was referring to a directory and so told me that the directory could not be found.
How do I get around this?
Use the echo command:
var="text to append";
destdir=/some/directory/path/filename
if [ -f "$destdir" ]
then
echo "$var" > "$destdir"
fi
The if tests that $destdir represents a file.
The > appends the text after truncating the file. If you only want to append the text in $var to the file existing contents, then use >> instead:
echo "$var" >> "$destdir"
The cp command is used for copying files (to files), not for writing text to a file.
echo has the problem that if var contains something like -e, it will be interpreted as a flag. Another option is printf, but printf "$var" > "$destdir" will expand any escaped characters in the variable, so if the variable contains backslashes the file contents won't match. However, because printf only interprets backslashes as escapes in the format string, you can use the %s format specifier to store the exact variable contents to the destination file:
printf "%s" "$var" > "$destdir"
None of the answers above work if your variable:
starts with -e
starts with -n
starts with -E
contains a \ followed by an n
should not have an extra newline appended after it
and so they cannot be relied upon for arbitrary string contents.
In bash, you can use "here strings" as:
cat <<< "$var" > "$destdir"
As noted in the comment by Ash below, #Trebawa's answer (formulated in the same room as mine!) using printf is a better approach than cat.
All of the above work, but also have to work around a problem (escapes and special characters) that doesn't need to occur in the first place: Special characters when the variable is expanded by the shell. Just don't do that (variable expansion) in the first place. Use the variable directly, without expansion.
Also, if your variable contains a secret and you want to copy that secret into a file, you might want to not have expansion in the command line as tracing/command echo of the shell commands might reveal the secret. Means, all answers which use $var in the command line may have a potential security risk by exposing the variable contents to tracing and logging of the shell.
For variables that are already exported, use this:
printenv var >file
That means, in case of the OP question:
printenv var >"$destfile"
Note: variable names are case sensitive.
Warning: It is not a good idea to export a variable just for the sake of printing it with printenv. If you have a non-exported script variable that contains a secret, exporting it will expose it to all future child processes (unless unexported, for example using export -n).
If I understood you right, you want to copy $var in a file (if it's a string).
echo $var > $destdir
When you say "copy the contents of a variable", does that variable contain a file name, or does it contain a name of a file?
I'm assuming by your question that $var contains the contents you want to copy into the file:
$ echo "$var" > "$destdir"
This will echo the value of $var into a file called $destdir. Note the quotes. Very important to have "$var" enclosed in quotes. Also for "$destdir" if there's a space in the name. To append it:
$ echo "$var" >> "$destdir"
you may need to edit a conf file in a build process:
echo "db-url-host=$POSTGRESQL_HOST" >> my-service.conf
You can test this solution with running before export POSTGRESQL_HOST="localhost"