I have a variable, and I want to replace every occurrence of backslash ('\') with double backslash ('\\') using Shell parameter expansion. Originally, I used the following construction:
$ var='\\a\b'
$ echo "${var//\\/\\\\}"
\\\\a\\b
This works fine, but it breaks vim syntax highlighting - apparently, vim cannot handle \\} part. So, I decided to store backslash in a variable and use to to avoid syntax highlighting issues:
$ bsl='\'
$ echo "${var//$bsl/$bsl$bsl}"
\\a\b
To my surprise, it does not work, although it would work fine with any alphanumeric symbol. So, maybe I need to store 2 backslashes in a variable? Let's try it:
$ bsl='\\'
$ echo "${var//$bsl/$bsl$bsl}"
\\\\\\\\a\\\\b
Now, it went from not working to working twice the time I need. Eventually, I found that the only way to achieve desired result and preserve vim highlighting is the following:
$ bsl='\'
$ echo "${var//\\/$bsl$bsl}"
\\\\a\\b
While I already found a way to solve my issue, my question is: why parameter expansion works this way with a backslash? To me, such behavior makes no sense.
According to the Bash manual, with ${parameter/pattern/string}, "the pattern is expanded to produce a pattern just as in pathname expansion." Quoting the variable will protect it from pathname expansion and quote/backslash removal.
$ echo "${var//$bsl/$bsl$bsl}"
\\a\b
$ echo "${var//"$bsl"/$bsl$bsl}"
\\\\a\\b
For what it's worth, if you're on a GNU system you could use printf %q to achieve a similar result.
$ printf '%q\n' "$var"
\\\\a\\b
Related
I would like to escape a file path that is stored in a variable in a bash script.
I read several threads about escaping back ticks or but it seems not working as it should:
I have this variable:
The variables value is entered during the bash script execution as user parameter
CONFIG="/home/teams/blabla/blabla.yaml"
I would need to change this to: \/home\/teams\/blabla\/blabla.yaml
How can I do that with in the script via sed or so (not manually)?
With GNU bash and its Parameter Expansion:
echo "${CONFIG//\//\\/}"
Output:
\/home\/teams\/blabla\/blabla.yaml
Using the solution from this question, in your case it will look like this:
CONFIG=$(echo "/home/teams/blabla/blabla.yaml" | sed -e 's/[]\/$*.^[]/\\&/g')
echo "/home/teams/blabla/blabla.yaml" | sed 's/\//\\\//g'
\/home\/teams\/blabla\/blabla.yaml
explanation:
backslash is used to set the following letter/symbol as an regular expression or vice versa. double backslash is used when you need a backslash as letter.
Why does that need escaping? Is this an XY Problem?
If the issue is that you are trying to use that variable in a substitution regex, then the examples given should work, but you might benefit by removing some of the "leaning toothpick syndrom", which many tools can do just by using a different match delimiter. sed, for example:
$: sed "s,SOME_PLACEHOLDER_VALUE,$CONFIG," <<< SOME_PLACEHOLDER_VALUE
/home/teams/blabla/blabla.yaml
Be very careful about this, though. Commas are perfectly valid characters in a filename, as are almost anything but NULLs. Know your data.
I am using an external command to populate my bash prompt, which is run each time PS1 is evaluated. However, I have a problem when this command outputs non-printable characters (like color escape codes).
Here is an example:
$ cat green_cheese.sh
#!/bin/bash
echo -e "\033[32mcheese\033[0m"
$ export PS1="\$(./green_cheese.sh) \$"
cheese $ # <- cheese is green!
cheese $ <now type really long command>
The canonical way of dealing with non-printing characters in the PS1 prompt is to enclose them in \[ and \] escape sequences. The problem is that if you do this from the external command those escapes are not parsed by the PS1 interpreter:
$ cat green_cheese.sh
#!/bin/bash
echo -e "\[\033[32m\]cheese\[\033[0m\]"
$ export PS1="\$(./green_cheese.sh) \$"
\[\]cheese\[\] $ # <- FAIL!
Is there a particular escape sequence I can use from the external command to achieve the desired result? Or is there a way I can manually tell the prompt how many characters to set the prompt width to?
Assume that I can print anything I like from the external command, and that this command can be quite intelligent (for example, counting characters in the output). I can also make the export PS1=... command as complicated as required. However, the escape codes for the colors must come from the external command.
Thanks in advance!
I couldn't tell you exactly why this works, but replace \[ and \] with the actual characters that bash generates from them in your prompt:
echo -e "\001\033[32m\002cheese\001\033[0m\002"
[I learned this from some Stack Overflow post that I cannot find now.]
If I had to guess, it's that bash replaces \[ and \] with the two ASCII characters before executing the command that's embedded in the prompt, so that by the time green_cheese.sh completes, it's too late for bash to process the wrappers correctly, and so they are treated literally. One way to avoid this is to use PROMPT_COMMAND to build your prompt dynamically, rather than embedding executable code in the value of PS1.
prompt_cmd () {
PS1="$(green_cheese.sh)"
PS1+=' \$ '
}
PROMPT_COMMAND=prompt_cmd
This way, the \[ and \] are added to PS1 when it is defined, not when it is evaluated, so you don't need to use \001 and \002 directly.
If you can't edit the code generating the string containing ANSI color / control codes, you can wrap them after the fact.
The following will enclose ANSI control sequences in ASCII SOH (^A) and STX (^B) which are equivalent to \[ and \] respectively:
function readline_ANSI_escape() {
if [[ $# -ge 1 ]]; then
echo "$*"
else
cat # Read string from STDIN
fi | \
perl -pe 's/(?:(?<!\x1)|(?<!\\\[))(\x1b\[[0-9;]*[mG])(?!\x2|\\\])/\x1\1\x2/g'
}
Use it like:
$ echo $'\e[0;1;31mRED' | readline_ANSI_escape
Or:
$ readline_ANSI_escape "$string"
As a bonus, running the function multiple times will not re-escape already escaped control codes.
I suspect that if you echo the value of $PS1 after your first example, you’ll find that its value is the word “cheese” in green. (At least, that’s what I see when I run your example.) At first glance, this is what you want — the word “cheese” in green! Except that what you really wanted was the word cheese preceded by the escape codes that produce green. What you did by using the -e flag for echo is produce a value with the escape codes already evaluated.
That happens to work for the specification of colors, but as you’ve found, it mangles the “non-printing sequence” markers into something the $PS1 interpreter doesn’t properly understand.
Fortunately, the solution is simple: drop the -e flag. echo will then leave the escape sequences untouched, and the $PS1 interpreter will Do The Right Thing™.
I have a string variable x=tmp/variable/custom-sqr-sample/test/example
in the script, what I want to do is to replace all the “-” with the /,
after that,I should get the following string
x=tmp/variable/custom/sqr/sample/test/example
Can anyone help me?
I tried the following syntax
it didnot work
exa=tmp/variable/custom-sqr-sample/test/example
exa=$(echo $exa|sed 's/-///g')
sed basically supports any delimiter, which comes in handy when one tries to match a /, most common are |, # and #, pick one that's not in the string you need to work on.
$ echo $x
tmp/variable/custom-sqr-sample/test/example
$ sed 's#-#/#g' <<< $x
tmp/variable/custom/sqr/sample/test/example
In the commend you tried above, all you need is to escape the slash, i.e.
echo $exa | sed 's/-/\//g'
but choosing a different delimiter is nicer.
The tr tool may be a better choice than sed in this case:
x=tmp/variable/custom-sqr-sample/test/example
echo "$x" | tr -- - /
(The -- isn't strictly necessary, but keeps tr (and humans) from mistaking - for an option.)
In bash, you can use parameter substitution:
$ exa=tmp/variable/custom-sqr-sample/test/example
$ exa=${exa//-/\/}
$ echo $exa
tmp/variable/custom/sqr/sample/test/example
It is very interesting that if you intend to display 0_1 with Bash using the code
x=0
y=1
echo "$x_$y"
then it will only display
1
I tried echo "$x\_$y" and it doesn't work.
How can I echo the form $x_$y? I'm going to use it on a file name string.
Because variable names are allowed to have underscores in them, the command:
echo "$x_$y"
is trying to echo ${x_} (which is probably empty in your case) followed by ${y}. The reason for this is because parameter expansion is a greedy operation - it will take as many legal characters as possible after the $ to form a variable name.
The relevant part of the bash manpage states:
The $ character introduces parameter expansion, command substitution, or arithmetic expansion.
The parameter name or symbol to be expanded may be enclosed in braces, which are optional but serve to protect the variable to be expanded from characters immediately following it which could be interpreted as part of the name.
When braces are used, the matching ending brace is the first } not escaped by a backslash or within a quoted string, and not within an embedded arithmetic expansion, command substitution, or parameter expansion.
Hence, the solution is to ensure that the _ is not treated as part of the first variable, which can be done with:
echo "${x}_${y}"
I tend to do all my bash variables like this, even standalone ones like:
echo "${x}"
since it's more explicit, and I've been bitten so many times in the past :-)
This way:
$ echo "${x}_${y}"
0_1
wrap it in curly braces:
echo "${x}_${y}"
Just to buck the trend, you can also do this:
echo $x'_'$y
You can have quoted and unquoted parts next to each other with no space between. And since ' isn't a legal character for a variable name, bash will substitute only $x. :)
I'm writing a shell script that should be somewhat secure, i.e., does not pass secure data through parameters of commands and preferably does not use temporary files. How can I pass a variable to the standard input of a command?
Or, if it's not possible, how can I correctly use temporary files for such a task?
Passing a value to standard input in Bash is as simple as:
your-command <<< "$your_variable"
Always make sure you put quotes around variable expressions!
Be cautious, that this will probably work only in bash and will not work in sh.
Simple, but error-prone: using echo
Something as simple as this will do the trick:
echo "$blah" | my_cmd
Do note that this may not work correctly if $blah contains -n, -e, -E etc; or if it contains backslashes (bash's copy of echo preserves literal backslashes in absence of -e by default, but will treat them as escape sequences and replace them with corresponding characters even without -e if optional XSI extensions are enabled).
More sophisticated approach: using printf
printf '%s\n' "$blah" | my_cmd
This does not have the disadvantages listed above: all possible C strings (strings not containing NULs) are printed unchanged.
(cat <<END
$passwd
END
) | command
The cat is not really needed, but it helps to structure the code better and allows you to use more commands in parentheses as input to your command.
Note that the 'echo "$var" | command operations mean that standard input is limited to the line(s) echoed. If you also want the terminal to be connected, then you'll need to be fancier:
{ echo "$var"; cat - ; } | command
( echo "$var"; cat - ) | command
This means that the first line(s) will be the contents of $var but the rest will come from cat reading its standard input. If the command does not do anything too fancy (try to turn on command line editing, or run like vim does) then it will be fine. Otherwise, you need to get really fancy - I think expect or one of its derivatives is likely to be appropriate.
The command line notations are practically identical - but the second semi-colon is necessary with the braces whereas it is not with parentheses.
This robust and portable way has already appeared in comments. It should be a standalone answer.
printf '%s' "$var" | my_cmd
or
printf '%s\n' "$var" | my_cmd
Notes:
It's better than echo, reasons are here: Why is printf better than echo?
printf "$var" is wrong. The first argument is format where various sequences like %s or \n are interpreted. To pass the variable right, it must not be interpreted as format.
Usually variables don't contain trailing newlines. The former command (with %s) passes the variable as it is. However tools that work with text may ignore or complain about an incomplete line (see Why should text files end with a newline?). So you may want the latter command (with %s\n) which appends a newline character to the content of the variable. Non-obvious facts:
Here string in Bash (<<<"$var" my_cmd) does append a newline.
Any method that appends a newline results in non-empty stdin of my_cmd, even if the variable is empty or undefined.
I liked Martin's answer, but it has some problems depending on what is in the variable. This
your-command <<< """$your_variable"""
is better if you variable contains " or !.
As per Martin's answer, there is a Bash feature called Here Strings (which itself is a variant of the more widely supported Here Documents feature):
3.6.7 Here Strings
A variant of here documents, the format is:
<<< word
The word is expanded and supplied to the command on its standard
input.
Note that Here Strings would appear to be Bash-only, so, for improved portability, you'd probably be better off with the original Here Documents feature, as per PoltoS's answer:
( cat <<EOF
$variable
EOF
) | cmd
Or, a simpler variant of the above:
(cmd <<EOF
$variable
EOF
)
You can omit ( and ), unless you want to have this redirected further into other commands.
Try this:
echo "$variable" | command
If you came here from a duplicate, you are probably a beginner who tried to do something like
"$variable" >file
or
"$variable" | wc -l
where you obviously meant something like
echo "$variable" >file
echo "$variable" | wc -l
(Real beginners also forget the quotes; usually use quotes unless you have a specific reason to omit them, at least until you understand quoting.)