In Linux /etc/init.d/functions script I found the following parameter expansions that I don't quite understand:
${p//[0-9]/} replace all instances of any number to/by what?
${1##[-+]} This seems to remove all the longest left instances of minuses and pluses?
${LSB:-} This seems to say that if LSB is not set then set nothing? in other words do nothing?
These are instances of bash Shell Parameter Expansion;
see http://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html
Note: ksh and zsh support the expansions in your question, too (I'm unclear on the full extent of the overlap in functionality), whereas sh (POSIX-features-only shells), does NOT support the string-replacement expansion, ${p//[0-9]/}.
${p//[0-9]/}
Removes all digits: replaces all (//) instances of digits ([0-9]) with an empty string - i.e., it removes all digits (what comes after the last / is the replacement string, which is empty in this case).
${1##[-+]}
Strips a single leading - or +, if present: Technically, this removes the longest prefix (##) composed of a single - or + character from parameter $1. Given that the search pattern matches just a single character, there is no need to use ## for the longest prefix here, and # - for the shortest prefix - would do.
${LSB:-}
A no-op designed to prevent the script from breaking when run with the -u (nounset) shell attribute: Technically, this expansion means: In case variable $LSB is either not set or empty, it is to be replaced with the string following :-, which, in this case, is also empty.
While this may seem pointless at first glance, it has its purpose, as Sigi points out:
"
The ${LSB:-} construct makes perfect sense if the shell is invoked with the -u option (or set -u is used), and the variable $LSB might actually be unset. You then avoid the shell bailing out if you reference $LSB as ${LSB:-} instead. Since it's good practice to use set -u in complex scripts, this move comes in handy quite often.
"
${p//[0-9]/} # removes digits from anywhere in `$p`
${1##[-+]} # removes + or - from start in $1
${LSB:-} # not really doing anything
Related
I want to run a command from a bash script which has single quotes and some other commands inside the single quotes and a variable.
e.g. repo forall -c '....$variable'
In this format, $ is escaped and the variable is not expanded.
I tried the following variations but they were rejected:
repo forall -c '...."$variable" '
repo forall -c " '....$variable' "
" repo forall -c '....$variable' "
repo forall -c "'" ....$variable "'"
If I substitute the value in place of the variable the command is executed just fine.
Please tell me where am I going wrong.
Inside single quotes everything is preserved literally, without exception.
That means you have to close the quotes, insert something, and then re-enter again.
'before'"$variable"'after'
'before'"'"'after'
'before'\''after'
Word concatenation is simply done by juxtaposition. As you can verify, each of the above lines is a single word to the shell. Quotes (single or double quotes, depending on the situation) don't isolate words. They are only used to disable interpretation of various special characters, like whitespace, $, ;... For a good tutorial on quoting see Mark Reed's answer. Also relevant: Which characters need to be escaped in bash?
Do not concatenate strings interpreted by a shell
You should absolutely avoid building shell commands by concatenating variables. This is a bad idea similar to concatenation of SQL fragments (SQL injection!).
Usually it is possible to have placeholders in the command, and to supply the command together with variables so that the callee can receive them from the invocation arguments list.
For example, the following is very unsafe. DON'T DO THIS
script="echo \"Argument 1 is: $myvar\""
/bin/sh -c "$script"
If the contents of $myvar is untrusted, here is an exploit:
myvar='foo"; echo "you were hacked'
Instead of the above invocation, use positional arguments. The following invocation is better -- it's not exploitable:
script='echo "arg 1 is: $1"'
/bin/sh -c "$script" -- "$myvar"
Note the use of single ticks in the assignment to script, which means that it's taken literally, without variable expansion or any other form of interpretation.
The repo command can't care what kind of quotes it gets. If you need parameter expansion, use double quotes. If that means you wind up having to backslash a lot of stuff, use single quotes for most of it, and then break out of them and go into doubles for the part where you need the expansion to happen.
repo forall -c 'literal stuff goes here; '"stuff with $parameters here"' more literal stuff'
Explanation follows, if you're interested.
When you run a command from the shell, what that command receives as arguments is an array of null-terminated strings. Those strings may contain absolutely any non-null character.
But when the shell is building that array of strings from a command line, it interprets some characters specially; this is designed to make commands easier (indeed, possible) to type. For instance, spaces normally indicate the boundary between strings in the array; for that reason, the individual arguments are sometimes called "words". But an argument may nonetheless have spaces in it; you just need some way to tell the shell that's what you want.
You can use a backslash in front of any character (including space, or another backslash) to tell the shell to treat that character literally. But while you can do something like this:
reply=\”That\'ll\ be\ \$4.96,\ please,\"\ said\ the\ cashier
...it can get tiresome. So the shell offers an alternative: quotation marks. These come in two main varieties.
Double-quotation marks are called "grouping quotes". They prevent wildcards and aliases from being expanded, but mostly they're for including spaces in a word. Other things like parameter and command expansion (the sorts of thing signaled by a $) still happen. And of course if you want a literal double-quote inside double-quotes, you have to backslash it:
reply="\"That'll be \$4.96, please,\" said the cashier"
Single-quotation marks are more draconian. Everything between them is taken completely literally, including backslashes. There is absolutely no way to get a literal single quote inside single quotes.
Fortunately, quotation marks in the shell are not word delimiters; by themselves, they don't terminate a word. You can go in and out of quotes, including between different types of quotes, within the same word to get the desired result:
reply='"That'\''ll be $4.96, please," said the cashier'
So that's easier - a lot fewer backslashes, although the close-single-quote, backslashed-literal-single-quote, open-single-quote sequence takes some getting used to.
Modern shells have added another quoting style not specified by the POSIX standard, in which the leading single quotation mark is prefixed with a dollar sign. Strings so quoted follow similar conventions to string literals in the ANSI standard version of the C programming language, and are therefore sometimes called "ANSI strings" and the $'...' pair "ANSI quotes". Within such strings, the above advice about backslashes being taken literally no longer applies. Instead, they become special again - not only can you include a literal single quotation mark or backslash by prepending a backslash to it, but the shell also expands the ANSI C character escapes (like \n for a newline, \t for tab, and \xHH for the character with hexadecimal code HH). Otherwise, however, they behave as single-quoted strings: no parameter or command substitution takes place:
reply=$'"That\'ll be $4.96, please," said the cashier'
The important thing to note is that the single string that gets stored in the reply variable is exactly the same in all of these examples. Similarly, after the shell is done parsing a command line, there is no way for the command being run to tell exactly how each argument string was actually typed – or even if it was typed, rather than being created programmatically somehow.
Below is what worked for me -
QUOTE="'"
hive -e "alter table TBL_NAME set location $QUOTE$TBL_HDFS_DIR_PATH$QUOTE"
EDIT: (As per the comments in question:)
I've been looking into this since then. I was lucky enough that I had repo laying around. Still it's not clear to me whether you need to enclose your commands between single quotes by force. I looked into the repo syntax and I don't think you need to. You could used double quotes around your command, and then use whatever single and double quotes you need inside provided you escape double ones.
just use printf
instead of
repo forall -c '....$variable'
use printf to replace the variable token with the expanded variable.
For example:
template='.... %s'
repo forall -c $(printf "${template}" "${variable}")
Variables can contain single quotes.
myvar=\'....$variable\'
repo forall -c $myvar
I was wondering why I could never get my awk statement to print from an ssh session so I found this forum. Nothing here helped me directly but if anyone is having an issue similar to below, then give me an up vote. It seems any sort of single or double quotes were just not helping, but then I didn't try everything.
check_var="df -h / | awk 'FNR==2{print $3}'"
getckvar=$(ssh user#host "$check_var")
echo $getckvar
What do you get? A load of nothing.
Fix: escape \$3 in your print function.
Does this work for you?
eval repo forall -c '....$variable'
More precisely, why does
"`command "$variable"`"
treat the outer quotes as enclosing the inner quotes, instead of expanding the variable outside any quotes?
The exact command I used to test this is similar to an example brought up in another stackoverflow question about the correct method of quoting when using command substitution:
fileName="some path with/spaces"
echo "`dirname "$fileName"`"
which correctly echoes "some path with", instead of complaining because of an invalid number of arguments.
I read Bash's man page, where it states in chapter "EXPANSION", section "Commmand Substitution" that the new-style $() substitution preserves the meaning of any character between the parentheses, however, regarding backquotes, it only mentions that backslashes work in a limited way:
When the old-style backquote form of substitution is used, backslash retains
its literal meaning except when followed by $, `, or \. The first backquote
not preceded by a backslash terminates the command substitution.
My first thought was that backticks do the same, aside from the mentioned exception, thus "quoting" the inner double quotes, however, I got told that is not the case.
The second observation that pointed me to this direction was that
a=\$b
b=hello
echo `echo $a`
prints "$b". Had the backticks let the dollar sign get interpreted, the first variable substitution should have occurred before the subshell was invoked, with the subshell expanding the string "$b", resulting in "hello."
According to the above excerpt from the man page, I can even make sure the dollar sign is actually quoted, by using
echo `echo \$a`
and the results would still be the same.
A third observation gives me some doubts though:
echo `echo \\a`
Result: "\a"
echo \a
Result: a
Here it seems like both backslashes were retained until the subshell came into play, even though the man page states that backslashes within backquotes do not have their literal meaning when followed by another backslash.
EDIT: ^ Everything works as expected in this regard, I must have used the wrong shell (tcsh in my other terminal, and with a different character from "a").
Although I have not been able to find out what actually happens, while I was searching for the answer, I came across some people mentioning the term "quoting context" with regards to command substitution, but without any explanation as to what it means or where it is described.
I have not found any real reference to "quoting contexts" in either Bash references (gnu.org, tldp, man bash) or via DuckDuckGo.
Additionally to knowing what is going on, I'd preferably like to have some reference or guidance as to how this behavior can be discerned from it, because I think I might have failed to put some pieces together from which this naturally comes. Otherwise I'll just forget the answer.
To those recommending people to use the new-style dollar sign and parentheses substitution: on ca. 50 years old Unix machines with tens or hundreds of different proprietary environments (can't throw out a shell for a newer one), when one has to write scripts compatible between most shells that anyone might be using, it is not an option.
Thanks to anyone who can help me.
POSIX has this to say in 2.2.3 (emphasis mine):
` (backquote)
The backquote shall retain its special meaning introducing the other form of command substitution (see Command Substitution). The portion of the quoted string from the initial backquote and the characters up to the next backquote that is not preceded by a <backslash>, having escape characters removed, defines that command whose output replaces "`...`" when the word is expanded. Either of the following cases produces undefined results:
A single-quoted or double-quoted string that begins, but does not end, within the "`...`" sequence
A "`...`" sequence that begins, but does not end, within the same double-quoted string
This, to me, pretty much defines what other people might (informally?) call a quoting context comprising everything within two consecutive backquotes.
In a way, the backquote is the fourth quote in addition to single quote, double quote and backslash. Note that within double quotes, single quotes lose their quoting capability as well, so it should be no surprise that backquotes change the function of double quotes within them.
I tried your example with other shells, like the Almquist shell on FreeBSD and zsh. As expected, they output some path with.
I have a theoretical question about the syntax of Bash.
I am running Bash 4.3.11(1) in Linux Ubuntu 14.04.
In the official GNU's website: Bash official web (GNU)
in Subection 9.3.1. it says:
!string
Refer to the most recent command preceding the current position
in the history list starting with string.
In general it's understood that string is, syntactically speaking, a sequence of characters ending before the first blank or newline.
However, when describing quoting in subsection 3.1.2., we can read in paragraph 3.1.2.2. what follows:
Enclosing characters in single quotes (‘'’) preserves the literal
value of each character within the quotes.
In particular, the blanks inside single quotes are not broking the strings in separated words.
So, a expression like !'some text' would have to search in the history list of Bash for the most recent command starting by 'some text'.
However, the blank between some and text is broken when I write it in my terminal, since the following error message is shown:
bash: !'some: event not found
Is this behaviour a bug in the implementation of the shell, or well I am not understanding the expansion rules of Bash for this example?
I wouldn't call the observed behaviour a bug, because there is no specification for history expansion other than the observed behaviour of the bash shell itself. But it is certainly the case that the precise mechanics of parsing a history expansion expression is not well documented and has a lot of possibly surprising corner cases.
The bash manpage does state that history expansion "is performed immediately after a complete line is read, before the shell breaks it into words" (emphasis added), while the bash manual mentions that history expansion is provided by the History library. This is the root cause of most of the history expansion parsing oddities: history expansion works on raw unparsed input without any assistance from the bash tokenizer, and is mostly done with an external library which is not bash-specific. Since tokenizing bash input is non-trivial, it is not really surprising that the relatively simple parsing rules used during history expansion are only a rough approximation to a real bash parse.
For example, the bash manual does indicated that you can prevent a history expansion character (!) from being recognized as such by backslash-quoting it. But it is not explicitly documented that any \ which immediately precedes an ! will inhibit recognition of the history expansion, even if the backslash was itself quoted with a backslash. So the ! in \\!word does not cause the previous command starting with word to be substituted. (\\word is a common way to execute the command word instead of the alias word, so the example is not entirely contrived.)
A longer discussion of some of the corner cases of the recognition of the history expansion character can be found in this answer.
The issue raised by this question is slightly different, since it is about the next phase of the history expansion parse. Once it has been established that a particular character is a history expansion character, it is then necessary to parse the "event" which follows; as indicated by the bash manual, the event can take several forms, one of which is !string, representing the most recent command which starts with "string".
It is implied that this form will only be used if no other form applies, which means that string may not start with a digit or -, !, # or ?. It also may not start with whitespace or = (since those would inhibit history expansion) and in some circumstances ( or " (which may inhibit history expansion). And finally, it may not start with ^, $, % or *, which would be interpreted as a word designator (from the default event, which is the previous command).
The bash manual does not specify what terminates the string. It is semi-documented in the history library manual, which mentions that a history search string (or "event" as it is called in the bash manual) is terminated by whitespace, :, or any of the characters in the history configuration variable history_search_delimiter_chars. (For the record, bash currently (v4.3) sets that variable to ";&()|<>".)
As indicated earlier, quoting is taken into account when deciding whether or not to recognize a history expansion character; as it turns out, if the history expansion occurs inside a double-quoted string, then the closing double-quote is also considered a history search delimiter character. And that, as far as I know, is the entire list of characters which will delimit !string.
Nowhere in either the bash nor the history documentation does it state that a history search delimiter character can be made non-special by quoting, and indeed this does not happen. An open quote, whether double or single, or even a backslash following the ! will be treated as just part of the string to be searched for, without any special processing.
Parsing of the substring-match history expansion -- !?string? -- is completely different. That string can only be terminated by a ? or by a newline. (As the bash manual says, the trailing ? is optional if terminated by a newline.)
Once the history expansion character has been recognized and the history search string has been identified, it may then be necessary to split the retrieved history entry into words. Again, the bash manual is slightly cavalier about corner cases, when it says that "the line is broken into words in the same fashion that Bash does, so that several words surrounded by quotes are considered one word."
A pedant would observe that "in the same fashion that Bash does" is not quite the same as saying "exactly as Bash would do", and in fact the second part of the sentence is literall true: several words surrounded by quotes are considered one word even if the quotes are not really matching quotes. For example, the line:
command "$(echo " foo bar ")"
is considered by the history library to consist of the following five words:
0. command
1. "$(echo "
2. foo
3. bar
4. ")"
although the bash parse would be quite different. By contrast, bash and the history library agree on the parsing of
command "$(echo ' foo bar ')"
as two words.
Had some spam issues on my server and, after finding out and removing some Perl and PHP scripts I'm down to checking what they really do, although I'm a senior PHP programmer I have little experience with Perl, can anyone give me a hand with the script here:
http://pastebin.com/MKiN8ifp
(It was one long line of code, script was called list.pl)
The start of the script is:
$??s:;s:s;;$?::s;(.*); ]="&\%[=.*.,-))'-,-#-*.).<.'.+-<-~-#,~-.-,.+,~-{-,.<'`.{'`'<-<--):)++,+#,-.{).+,,~+{+,,<)..})<.{.)-,.+.,.)-#):)++,+#,-.{).+,,~+{+,,<)..})<*{.}'`'<-<--):)++,+#,-.{).+:,+,+,',~+*+~+~+{+<+,)..})<'`'<.{'`'<'<-}.<)'+'.:*}.*.'-|-<.+):)~*{)~)|)++,+#,-.{).+:,+,+,',~+*+~+~+{+<+,)..})
It continues with precious few non-punctuation characters until the very end:
0-9\;\\_rs}&a-h;;s;(.*);$_;see;
Replace the s;(.*);$_;see; with print to get this. Replace s;(.*);$_;see; again with print in the first half of the payload to get this, which is the decryption code. The second half of the payload is the code to decrypt, but I can't go any further with it, because as you see, the decryption code is looking for a key in an envvar or a cookie (so that only the script's creator can control it or decode it, presumably), and I don't have that key. This is actually reasonably cleverly done.
For those interested in the nitty gritty... The first part, when de-tangled looks like this:
$? ? s/;s/s;;$?/ :
s/(.*)/...lots of punctuation.../;
The $? at the beginning of the line is the pre-defined variable containing the child error, which no doubt serves only as obfuscation. It will be undefined, as there can be no child error at this point.
The questionmark following it is the start of a ternary operator
CONDITION ? IF_TRUE : IF_FALSE
Which is also added simply to obfuscate. The expression returned for true is a substitution regex, where the / slash delimiter has been replaced with colon s:pattern:replacement:. Above, I have put back slashes. The other expression, which is the one that will be executed is also a substitution regex, albeit an incredibly long one. The delimiter is semi-colon.
This substitution replaces .* in $_ - the default input and pattern-searching space - with a rather large amount of punctuation characters, which represents the bulk of the code. Since .* matches any string, even the empty string, it will simply get inserted into $_, and is for all intents and purposes identical to simply assigning the string to $_, which is what I did:
$_ = q;]="&\%[=.*.,-))'-,-# .......;;
The following lines are a transliteration and another substitution. (I inserted comments to point out the delimiters)
y; -"[%-.:<-#]-`{-}#~\$\\;{\$()*.0-9\;\\_rs}&a-h;;
#^ ^ ^ ^
#1 2 3
(1,2,3 are delimiters, the semi-colon between 2 and 3 is escaped)
The basic gist of it is that various characters and ranges -" (space to double quote), and something that looks like character classes (with ranges) [%-.:<-#], but isn't, get transliterated into more legible characters e.g. curly braces, dollar sign, parentheses,0-9, etc.
s;(.*);$_;see;
The next substitution is where the magic happens. It is also a substitution with obfuscated delimiters, but with three modifers: see. s does nothing in this case, as it only allows the wildcard character . to match newline. ee means to evaluate the expression twice, however.
In order to see what I was evaluating, I performed the transliteration and printed the result. I suspect that I somewhere along the line got some characters corrupted, because there were subtle errors, but here's the short (cleaned up) version:
s;(.*);73756220656e6372797074696f6e5f6 .....;; # very long line of alphanumerics
s;(..);chr(hex($1));eg;
s;(.*);$_;see;
s;(.*);704b652318371910023c761a3618265 .....;; # another long line
s;(..);chr(hex($1));eg;
&e_echr(\$_);
s;(.*);$_;see;
The long regexes are once again the data containers, and insert data into $_ to be evaluated as code.
The s/(..)/chr(hex($1))/eg; is starting to look rather legible. It is basically reading two characters at the time from $_ and converting it from hex to corresponding character.
The next to last line &e_echr(\$_); stumped me for a while, but it is a subroutine that is defined somewhere in this evaluated code, as hobbs so aptly was able to decode. The dollar sign is prefixed by backslash, meaning it is a reference to $_: I.e. that the subroutine can change the global variable.
After quite a few evaluations, $_ is run through this subroutine, after which whatever is contained in $_ is evaluated a last time. Presumably this time executing the code. As hobbs said, a key is required, which is taken from the environment %ENV of the machine where the script runs. Which we do not have.
Ask the B::Deparse module to make it (a little more) readable.
I want to write a wrapper bash script, and to pass all arguments to a called program. I was very sure, that this works correctly:
#!/bin/sh
someProgam $#
But when passing exotic arguments (empty, unescaped, in quotes, ...) this fails.
For example: without the wrapper script, someProgram "1 2" 3 results in the arguments
[1 2] and [3].
But called from the script, I get [1], [2], [3].
Braces are just for visualization.
NOTE: It's a Java program, which is called. But I think it doesn't matter.
#!/bin/sh
someProgram "$#"
See also the bash docs on special parameters.
BTW1, "$#" is not specific to bash. You can probably rely on "$#" in cross-platform sh scripts to be run just about anywhere.
BTW2, in case this happens to be the last line in that script, you can save your operating system a few bytes and an entry in the process table by changing the line to something like
exec someProgram "$#"
to augment ndim's answer: the behavior of "$#" is not specific to bash. it's prescribed by the Single Unix Specification:
2.2.3 Double-Quotes
Enclosing characters in double-quotes ( "" ) shall preserve the literal value of all characters within the double-quotes, with the exception of the characters dollar sign, backquote, and backslash, as follows:
The parameter '#' has special meaning inside double-quotes and is described in Special Parameters.
2.5.2 Special Parameters
Listed below are the special parameters and the values to which they shall expand. Only the values of the special parameters are listed; see Word Expansions for a detailed summary of all the stages involved in expanding words.
#
Expands to the positional parameters, starting from one. When the expansion occurs within double-quotes, and where field splitting (see Field Splitting) is performed, each positional parameter shall expand as a separate field, with the provision that the expansion of the first parameter shall still be joined with the beginning part of the original word (assuming that the expanded parameter was embedded within a word), and the expansion of the last parameter shall still be joined with the last part of the original word. If there are no positional parameters, the expansion of '#' shall generate zero fields, even when '#' is double-quoted.