This question already has an answer here:
Prevent "echo" from interpreting backslash escapes
(1 answer)
Closed 4 years ago.
I have a bash script that contains lines:
remote_installer_svc_args="$local_cifs_mount/eset-remote-installer.args"
svc_arg_x86="%SYSTEMROOT%\\$(basename $remote_temp_dir)\\$(basename $INSTALLER_BAT)"
svc_arg_x64="%SYSTEMROOT%\\$(basename $remote_temp_dir)\\$(basename $INSTALLER_BAT)"
echo "$svc_arg_x86" > $remote_installer_svc_args
echo "$svc_arg_x64" >> $remote_installer_svc_args
It should produce a file that looks like this (in notepad++ on windows):
instead the file looks like this:
or in vim:
What is wrong with the script? Because when I copy those lines into bash it works, only if I run the script it does produce those strange characters...
You've run into part of the mess of inconsistent behavior that plagues the echo command. Specifically, some versions of echo (in some modes) interpret escape (backslash) sequences in the string they're asked to print. Others don't. When you ask echo to print %SYSTEMROOT%\era_rd_6HbUKJTR\EraAgentInstaller.bat, it might see the \e part and think it's supposed to convert that to the ASCII escape character.
Note that there are two different characters being called "escape" here: The backslash is used by the shell as an escape character, meaning that it and the characters immediately following it have some special meaning. The ASCII escape, on the other hand, is treated as a special character by the terminal (and vim and some other things) in a somewhat similar manner. Since the ASCII escape is a nonprinting character, when notepad++ and vim have to display it, they show some sort of alternate representation ("ESC" or "^]").
Anyway, since echo is inconsistent about its treatment of the backslash character, it's best to avoid it for strings that might contain backslash. Use printf instead (see "Why is printf better than echo?" on unix.se). It's a little more complicated to use, but not too bad. The main things to realize are that the first argument to printf is a "format" string that's used to control how the rest of the arguments are printed, and that unlike echo it doesn't automatically add a newline to the end.
What you want to use is:
printf '%s\n' "$svc_arg_x86" > $remote_installer_svc_args
printf '%s\n' "$svc_arg_x64" >> $remote_installer_svc_args
Or you can simplify it to:
printf '%s\n' "$svc_arg_x86" "$svc_arg_x64" > $remote_installer_svc_args
That first argument, %s\n, says to print a plain string followed by a newline. Backslash escapes in the format string are always interpreted, but strings formatted with the %s format never have escapes interpreted. Note that in the single-command version, the format string gets applied to each of the other two arguments, so each gets a newline at the end, so each winds up on a separate line in the output file.
Related
I am trying to accept a space delimited string in place of $OPTARG while parsing an option
For example
./script -k '1 2 ad'ias'
As seen the third string can contain any special character. Is there a way that I can overlook the quote in between as I want to parse the entire string and process some options
Tried inserting the \ character but that does not work for my case because I cannot insert any character in my string.
while getopts "a:k:" option
do
echo "${option}"
case ${option} in
a)
function_a ${OPTARG} # <-- no quotes
;;
k)
function_k "${OPTARG}" # <-- quotes
;;
esac
done
I'm not sure I fully understand what the difficulty is; handling strings with special characters is a bit tricky, but (except the NUL character) basically doable. The main things to watch out for are:
When you represent a string literal (in a script, or when passing arguments to a script), you must use a valid shell representation of that string, not just the raw string. For example, suppose you want to pass/use this string:
12 34 kla#42#!' 2 M$" rtqas;::#
There are a number of ways of representing this string for use in a shell script or command line. You can leave it unquoted but escape the individual special characters, like this:
12\ 34\ kla\#42#\!\'\ 2\ M\$\"\ rtqas\;::\#
Or you could wrap it in double-quotes, and escape just those characters that retain special meaning inside double-quotes (that is, double-quotes, backquotes, and dollar signs, and if it's a bash interactive shell exclamation marks):
"12 34 kla#42#!' 2 M\$\" rtqas;::#" # For a non-interactive shell
"12 34 kla#42#\!' 2 M\$\" rtqas;::#" # For an interactive shell
If it didn't contain single-quotes, you could single-quote it; since it does, you can't use that method. But you can mix methods, e.g. using single-quotes around the parts that don't contain single-quotes and escaping or double-quoting the single quote:
'12 34 kla#42#!'\'' 2 M$" rtqas;::#' # Single-quote is escaped
'12 34 kla#42#!'"'"' 2 M$" rtqas;::#' # Single-quote is double-quoted
In bash (but not some other shells), there's also ANSI-C-escaped strings, written with $' ... ':
$'12 34 kla#42#!\' 2 M$" rtqas;::#' # Single-quote is the only character that needs escaping
Note that all of the above are just different ways of representing the exact same string; once the shell parses it, it comes out the same from any of them. You can use whatever's convenient, but you must use a syntactically valid representation of the string.
Once the string is stored in a parameter/variable, you must put double-quotes around references to that variable. In most shell contexts, when a variable is used without quotes, the shell will split it into words (based on spaces or whatever's in IFS), and try to expand anything that looks like a file wildcard; you don't want this. But if it's in double-quotes, the variable gets expanded and no further parsing is done, it's just passed through unmolested.
Actually, you should almost always double-quote variable references in shell scripts even if you don't expect them to contain special characters. We see so many shell questions here that have the answer "if you double-quoted your variable references, you wouldn't have this problem"...
Here's an example, based on your script:
#!/bin/bash
printopt() {
printf '%s value is: <<%s>>\n' "$1" "$2" # Double-quotes required here
}
while getopts "a:k:" option
do
case "${option}" in # This is one of the few places it's safe to leave off double-quotes. But they don't hurt.
a)
printopt "-a" "${OPTARG}" # Double-quotes required here
;;
k)
printopt "-k" "${OPTARG}" # Double-quotes required here
;;
esac
done
And running it with various representations of strings:
$ ./argtest.sh -a 12\ 34\ kla\#42#\!\'\ 2\ M\$\"\ rtqas\;::\# -k "1 2 ad'ias"
-a value is: <<12 34 kla#42#!' 2 M$" rtqas;::#>>
-k value is: <<1 2 ad'ias>>
$ ./argtest.sh -a '12 34 kla#42#!'"'"' 2 M$" rtqas;::#' -k $'1 2 ad\'ias'
-a value is: <<12 34 kla#42#!' 2 M$" rtqas;::#>>
-k value is: <<1 2 ad'ias>>
Ok, ok, there are a few situations where it's more complicated than that:
There are some situations where a string will be run through the shell parsing process multiple times, such as when it's being run over ssh (the command gets processed by the local shell, passed to the remote computer, then processed by that shell and executed), or used as a shell alias (the alias command gets parsed, result stored, then parsed again when you use it). In these cases, you essentially need two (or possibly more) layers of quoting/escaping: take the raw string, quote/escape by any of the above methods, then take that string and quote/escape that (probably by a different method).
Some versions of echo will parse escape (backslash) sequences in the string (using different rules than the shell itself does), which can cause confusion. I recommend using printf instead when this might be an issue; the only problem is that it's more complex than echo: it doesn't just print its arguments, it uses the first argument is a format string which controls how the rest of the arguments are printed. See my examples in the script above.
If you are passing the string to another script that doesn't use double-quotes around its parameter & variable references, you are doomed. In this case, the only thing that can be done is to fix that other script.
The problem
I have multiple property lines in a single string separated by \n like this:
LINES2="Abc1.def=$SOME_VAR\nAbc2.def=SOMETHING_ELSE\n"$LINES
The LINES variable
might contain an undefined set of characters
may be empty. If it is empty, I want to avoid the trailing \n.
I am open for any command line utility (sed, tr, awk, ... you name it).
Tryings
I tried this to no avail
sed -z 's/\\n$//g' <<< $LINES2
I also had no luck with tr, since it does not accept regex.
Idea
There might be an approach to convert the \n to something else. But since $LINES can contain arbitrary characters, this might be dangerous.
Sources
I skim read through the following questions
How can I replace a newline (\n) using sed?
sed with literal string--not input file
Here's one solution:
LINES2="Abc1.def=$SOME_VAR"$'\n'"Abc2.def=SOMETHING_ELSE${LINES:+$'\n'$LINES}"
The syntax ${name:+value} means "insert value if the variable name exists and is not empty." So in this case, it inserts a newline followed by $LINES if $LINES is not empty, which seems to be precisely what you want.
I use $'\n' because "\n" is not a newline character. A more readable solution would be to define a shell variable whose value is a single newline.
It is not necessary to quote strings in shell assignment statements, since the right-hand side of an assignment does not undergo word-splitting nor glob expansion. Not quoting would make it easier to interpolate a $'\n'.
It is not usually advisable to use UPPER-CASE for shell variables because the shell and the OS use upper-case names for their own purposes. Your local variables should normally be lower case names.
So if I were not basing the answer on the command in the question, I would have written:
lines2=Abc1.def=$someVar$'\n'Abc2.def=SOMETHING_ELSE${lines:+$'\n'$lines}
my goal is to sed the 100th line and convert it to a string, then separate the data of the sentence to word
#!/bin/bash
fid=log.txt;
sentence=`expr sed -n '100p' ${fid}`;
for word in $sentence
do
echo $word
done
but apparently this has failed.
expr: syntax error
would somebody please let me know what have I done wrong? previously for number it worked.
The expr does not seem to serve a useful purpose here, and if it did, a sed command would certainly not be a valid or useful thing to pass to it, under most circumstances. You should probably just take it out.
However, the following loop is also problematic. Unquoted variables in shell script are very frequently an error. In this case, you can't quote the thing you pass to the for loop (that would cause the loop to only run once, with the loop variable set to the quoted string) but you also cannot prevent the shell from performing wildcard expansion on the unquoted string. So if the string happened to contain *, the shell will expand that to a list of files in the current directory, for example.
Fortunately, this can all be done in an only slightly more complicated sed script.
sed '100!d;s/[ \t]\+/\n/g;q' "$fid"
That is, if the line number is not 100, delete this line and start over with the next line. Otherwise, we are at line 100; replace runs of whitespace with newlines, (print) and quit.
(The backslash escape codes \t and \n are not universally portable; and \+ for repetition is also an optional extension. I believe there are also sed variants which dislike semicolon as a command separator. Consult your sed manual page, experiment, and if everything else fails, maybe switch to Awk or Perl. Just in case, here is a version which works even on Mac OSX:
sed '100!d
s/[ ][ ]*/\
/g;q' log.txt
The stuff inside the square brackets are a space and a literal tab; in Bash, with default keybindings, type ctrl-V, tab to produce a literal tab.)
Incidentally, this also gets rid of the variable capture antipattern. There are good reasons to capture output to a variable, but if it can be avoided, you often end up with a simpler, more robust and efficient, as well as more idiomatic and elegant script. (I see no reason to put the log file name in a variable, either, in this isolated case; but in a larger script, it might make sense.)
I don't think you need expr command in this case.
expr is used to do calculations. Something like:
expr 1 + 1
Just this one is fine:
sentence=`sed -n '100p' ${fid}`;
#!/bin/bash
fid=log.txt;
sentence=$(sed -n '100p' ${fid});
for word in $sentence
do
echo $word
done
put a dollar sign and parenthesis solve the problem
I have searched for the list of metacharacters in Bash but space is not enlisted.
I wonder if I'm right by assuming that space is the "token separation character" in Bash, since it not only works as such with Shell programs or builtins but also when creating an array through compound assignment - quotes escape spaces, just like they do most other metacharacters.
They cannot be escaped by backslashes, though.
Parameters are passed to programs and functions separated by spaces, for example.
Can someone explain how (and when) bash interprets spaces? Thanks!
I've written an example:
$ a=(zero one two)
$ echo ${a[0]}
$ zero
$ a=("zero one two")
$ echo ${a[0]}
$ zero one two
From the man page:
metacharacter
A character that, when unquoted, separates words. One of the following:
| & ; ( ) < > space tab
^^^^^
According to the Posix shell specification for Token Recognition, any shell (which pretends to be Posix-compliant) should interpret whitespace as separating tokens:
If the current character is an unquoted <newline>, the current token shall be delimited.
If the current character is an unquoted <blank>, any token containing the previous character is delimited and the current character shall be discarded.
Here <blank> refers to the character class blank as defined by LC_CTYPE at the time the shell starts. In almost all cases, that character class consists precisely of the space and tab characters.
It's important to distinguish between the shell mechanism for recognizing tokens, and the use of $IFS to perform word-splitting. Word splitting is performed (in most contexts) after brace, tilde, parameter and variable, arithmetic and command expansions. Consider, for example:
$ # Setting IFS does not affect token recognition
$ bash -c 'IFS=:; arr=(foo:bar); echo "${arr[0]}"'
foo:bar
$ # But it does affect word splitting after variable expansion
$ bash -c 'IFS=: foobar=foo:bar; arr=($foobar); echo "${arr[0]}"'
foo
Yes it is. From the Bash Reference Manual's Definitions section:
blank
A space or tab character.
…
metacharacter
A character that, when unquoted, separates words. A metacharacter is a blank or one of the following characters: ‘|’, ‘&’, ‘;’, ‘(’, ‘)’, ‘<’, or ‘>’.
I'm having an extremely hard time figuring out how to echo this:
I keep getting this error:
bash: ![alt: event not found
Using double quotes around it does not work. The using single quotes around it does work, however, I also need to echo strings that have single quotes within them. I wouldn't be able to wrap the string with single quotes then.
Is there a way to echo a string of ANY content?
Thanks.
EDIT: Here is some context. I am making a Markdown renderer that grabs the content of a code editor, then appends every line of the code individually into a text file. I am doing this by doing this:
echo TheLineOfMarkdown > textfile.txt
Unlike in many programing languages, '...' and "..." in Bash do not represent "strings" per se; they quote/escape whatever they contain, but they do not create boundaries that separate arguments. So, for example, these two commands are equivalent:
echo foobar
echo "fo"ob'ar'
So if you need to quote some of an argument with single-quotes, and a different part of the argument has to contain single-quotes — no problem.
For example:
echo '![alt text](https://... "What'"'"'s up, Doc?")'
Another option is to use \, which is similar to '...' except that it only quotes a single character. It can even be used inside double-quotes:
echo "\![alt text](https://... \"What's up, Doc?\")"
For more information, see §3.1.2 "Quoting" in the Bash Reference Manual.
! is annoying. My advice: Use \!.
! invokes history completion, which is also performed inside double-quotes. So you need to single-quote the exclamation mark, but as you say that conflicts with the need to not single-quote other single-quotes.
Remember that you can mix quotes:
$ echo '!'"'"'"'
!'"
(That's just one argument.) But in this case, the backslash is easier to type and quite possibly more readable.