I want to run a command from a bash script which has single quotes and some other commands inside the single quotes and a variable.
e.g. repo forall -c '....$variable'
In this format, $ is escaped and the variable is not expanded.
I tried the following variations but they were rejected:
repo forall -c '...."$variable" '
repo forall -c " '....$variable' "
" repo forall -c '....$variable' "
repo forall -c "'" ....$variable "'"
If I substitute the value in place of the variable the command is executed just fine.
Please tell me where am I going wrong.
Inside single quotes everything is preserved literally, without exception.
That means you have to close the quotes, insert something, and then re-enter again.
'before'"$variable"'after'
'before'"'"'after'
'before'\''after'
Word concatenation is simply done by juxtaposition. As you can verify, each of the above lines is a single word to the shell. Quotes (single or double quotes, depending on the situation) don't isolate words. They are only used to disable interpretation of various special characters, like whitespace, $, ;... For a good tutorial on quoting see Mark Reed's answer. Also relevant: Which characters need to be escaped in bash?
Do not concatenate strings interpreted by a shell
You should absolutely avoid building shell commands by concatenating variables. This is a bad idea similar to concatenation of SQL fragments (SQL injection!).
Usually it is possible to have placeholders in the command, and to supply the command together with variables so that the callee can receive them from the invocation arguments list.
For example, the following is very unsafe. DON'T DO THIS
script="echo \"Argument 1 is: $myvar\""
/bin/sh -c "$script"
If the contents of $myvar is untrusted, here is an exploit:
myvar='foo"; echo "you were hacked'
Instead of the above invocation, use positional arguments. The following invocation is better -- it's not exploitable:
script='echo "arg 1 is: $1"'
/bin/sh -c "$script" -- "$myvar"
Note the use of single ticks in the assignment to script, which means that it's taken literally, without variable expansion or any other form of interpretation.
The repo command can't care what kind of quotes it gets. If you need parameter expansion, use double quotes. If that means you wind up having to backslash a lot of stuff, use single quotes for most of it, and then break out of them and go into doubles for the part where you need the expansion to happen.
repo forall -c 'literal stuff goes here; '"stuff with $parameters here"' more literal stuff'
Explanation follows, if you're interested.
When you run a command from the shell, what that command receives as arguments is an array of null-terminated strings. Those strings may contain absolutely any non-null character.
But when the shell is building that array of strings from a command line, it interprets some characters specially; this is designed to make commands easier (indeed, possible) to type. For instance, spaces normally indicate the boundary between strings in the array; for that reason, the individual arguments are sometimes called "words". But an argument may nonetheless have spaces in it; you just need some way to tell the shell that's what you want.
You can use a backslash in front of any character (including space, or another backslash) to tell the shell to treat that character literally. But while you can do something like this:
reply=\”That\'ll\ be\ \$4.96,\ please,\"\ said\ the\ cashier
...it can get tiresome. So the shell offers an alternative: quotation marks. These come in two main varieties.
Double-quotation marks are called "grouping quotes". They prevent wildcards and aliases from being expanded, but mostly they're for including spaces in a word. Other things like parameter and command expansion (the sorts of thing signaled by a $) still happen. And of course if you want a literal double-quote inside double-quotes, you have to backslash it:
reply="\"That'll be \$4.96, please,\" said the cashier"
Single-quotation marks are more draconian. Everything between them is taken completely literally, including backslashes. There is absolutely no way to get a literal single quote inside single quotes.
Fortunately, quotation marks in the shell are not word delimiters; by themselves, they don't terminate a word. You can go in and out of quotes, including between different types of quotes, within the same word to get the desired result:
reply='"That'\''ll be $4.96, please," said the cashier'
So that's easier - a lot fewer backslashes, although the close-single-quote, backslashed-literal-single-quote, open-single-quote sequence takes some getting used to.
Modern shells have added another quoting style not specified by the POSIX standard, in which the leading single quotation mark is prefixed with a dollar sign. Strings so quoted follow similar conventions to string literals in the ANSI standard version of the C programming language, and are therefore sometimes called "ANSI strings" and the $'...' pair "ANSI quotes". Within such strings, the above advice about backslashes being taken literally no longer applies. Instead, they become special again - not only can you include a literal single quotation mark or backslash by prepending a backslash to it, but the shell also expands the ANSI C character escapes (like \n for a newline, \t for tab, and \xHH for the character with hexadecimal code HH). Otherwise, however, they behave as single-quoted strings: no parameter or command substitution takes place:
reply=$'"That\'ll be $4.96, please," said the cashier'
The important thing to note is that the single string that gets stored in the reply variable is exactly the same in all of these examples. Similarly, after the shell is done parsing a command line, there is no way for the command being run to tell exactly how each argument string was actually typed – or even if it was typed, rather than being created programmatically somehow.
Below is what worked for me -
QUOTE="'"
hive -e "alter table TBL_NAME set location $QUOTE$TBL_HDFS_DIR_PATH$QUOTE"
EDIT: (As per the comments in question:)
I've been looking into this since then. I was lucky enough that I had repo laying around. Still it's not clear to me whether you need to enclose your commands between single quotes by force. I looked into the repo syntax and I don't think you need to. You could used double quotes around your command, and then use whatever single and double quotes you need inside provided you escape double ones.
just use printf
instead of
repo forall -c '....$variable'
use printf to replace the variable token with the expanded variable.
For example:
template='.... %s'
repo forall -c $(printf "${template}" "${variable}")
Variables can contain single quotes.
myvar=\'....$variable\'
repo forall -c $myvar
I was wondering why I could never get my awk statement to print from an ssh session so I found this forum. Nothing here helped me directly but if anyone is having an issue similar to below, then give me an up vote. It seems any sort of single or double quotes were just not helping, but then I didn't try everything.
check_var="df -h / | awk 'FNR==2{print $3}'"
getckvar=$(ssh user#host "$check_var")
echo $getckvar
What do you get? A load of nothing.
Fix: escape \$3 in your print function.
Does this work for you?
eval repo forall -c '....$variable'
Related
I am trying to pass a multiline variable to a cli program from within Makefile, but not able to do so.
Makefile
.PHONY: test
test: ## Run the Tests locally
#echo $(SOME_VAR)
r=$(echo $(SOME_VAR) | sed 's/"/\\"/g')
python3 test_env_var.py "$(r)"
Where SOME_VAR is an env var containing multiline text string (in the end, I want this var to contain multiline JSON). Could you please suggest how can I achieve this from within Makefile?
Python code:
import sys
print(sys.argv[1])
If your SOME_VAR were indeed an environment variable -- that is, provided to make externally in its environment -- then you could just rely on your recipe to inherit it in its own environment:
Option 0
test:
#echo "$${SOME_VAR}"
python3 test_env_var.py "$${SOME_VAR}"
Note the doubling of the $ character to escape it from interpretation by make, so that the shell performs the interpolation.
But I suspect that you actually mean that you have the data in a make variable. In that case, your echo $(SOME_VAR) is problematic for pretty much the same reasons that python3 test_env_var.py $(SOME_VAR) would have been: not only newlines (if you really have them) but other characters that are significant to the shell, too, will be misinterpreted. What's more, make implementations typically execute each line of a recipe in a separate shell, so you cannot expect to set a shell variable in one line of a recipe and read its value back in a separate line.
Option 1
If you were prepared to assume that the variable's value did not contain any single quotes, then you could simply quote with that:
test:
#echo '$(SOME_VAR)'
python3 test_env_var.py '$(SOME_VAR)'
JSON data should use only double quotes internally, so you should be ok in that sense, but if there are any single quotes within the data themselves then that's going to break.
Do note, however, that single quotes and double quotes are not wholly analogous with each other to the shell. For example, variable expansions, command substitutions, and backslash escapes are recognized within double quotes, whereas no special characters are recognized within single quotes. As a result, you really ought to be looking to use single quotes, quite independently from those characters' differing significance in JSON.
Option 2
If you are willing to rely on GNU make, then you can use its subst function:
test:
#echo '$(subst ','\'',$(SOME_VAR))'
python3 test_env_var.py '$(subst ','\'',$(SOME_VAR))'
That converts each single-quote within the data to the four-character sequence '\''. With the overall content enclosed in single quotes, each appearance of those characters closes one single-quoted string, presents a backslash-escaped single quote (which can't appear inside a single-quoted string) and starts a new single-quoted string.
Option 3
But also consider just not. Especially if you really want multi-line data, consider instead putting it in a file and having the recipe read it from there. If necessary, it can be a built file.
test:
#echo "$$(cat test-arg)"
python3 test_env_var.py "$$(cat test-arg)"
Note again the doubling of the $ character.
More precisely, why does
"`command "$variable"`"
treat the outer quotes as enclosing the inner quotes, instead of expanding the variable outside any quotes?
The exact command I used to test this is similar to an example brought up in another stackoverflow question about the correct method of quoting when using command substitution:
fileName="some path with/spaces"
echo "`dirname "$fileName"`"
which correctly echoes "some path with", instead of complaining because of an invalid number of arguments.
I read Bash's man page, where it states in chapter "EXPANSION", section "Commmand Substitution" that the new-style $() substitution preserves the meaning of any character between the parentheses, however, regarding backquotes, it only mentions that backslashes work in a limited way:
When the old-style backquote form of substitution is used, backslash retains
its literal meaning except when followed by $, `, or \. The first backquote
not preceded by a backslash terminates the command substitution.
My first thought was that backticks do the same, aside from the mentioned exception, thus "quoting" the inner double quotes, however, I got told that is not the case.
The second observation that pointed me to this direction was that
a=\$b
b=hello
echo `echo $a`
prints "$b". Had the backticks let the dollar sign get interpreted, the first variable substitution should have occurred before the subshell was invoked, with the subshell expanding the string "$b", resulting in "hello."
According to the above excerpt from the man page, I can even make sure the dollar sign is actually quoted, by using
echo `echo \$a`
and the results would still be the same.
A third observation gives me some doubts though:
echo `echo \\a`
Result: "\a"
echo \a
Result: a
Here it seems like both backslashes were retained until the subshell came into play, even though the man page states that backslashes within backquotes do not have their literal meaning when followed by another backslash.
EDIT: ^ Everything works as expected in this regard, I must have used the wrong shell (tcsh in my other terminal, and with a different character from "a").
Although I have not been able to find out what actually happens, while I was searching for the answer, I came across some people mentioning the term "quoting context" with regards to command substitution, but without any explanation as to what it means or where it is described.
I have not found any real reference to "quoting contexts" in either Bash references (gnu.org, tldp, man bash) or via DuckDuckGo.
Additionally to knowing what is going on, I'd preferably like to have some reference or guidance as to how this behavior can be discerned from it, because I think I might have failed to put some pieces together from which this naturally comes. Otherwise I'll just forget the answer.
To those recommending people to use the new-style dollar sign and parentheses substitution: on ca. 50 years old Unix machines with tens or hundreds of different proprietary environments (can't throw out a shell for a newer one), when one has to write scripts compatible between most shells that anyone might be using, it is not an option.
Thanks to anyone who can help me.
POSIX has this to say in 2.2.3 (emphasis mine):
` (backquote)
The backquote shall retain its special meaning introducing the other form of command substitution (see Command Substitution). The portion of the quoted string from the initial backquote and the characters up to the next backquote that is not preceded by a <backslash>, having escape characters removed, defines that command whose output replaces "`...`" when the word is expanded. Either of the following cases produces undefined results:
A single-quoted or double-quoted string that begins, but does not end, within the "`...`" sequence
A "`...`" sequence that begins, but does not end, within the same double-quoted string
This, to me, pretty much defines what other people might (informally?) call a quoting context comprising everything within two consecutive backquotes.
In a way, the backquote is the fourth quote in addition to single quote, double quote and backslash. Note that within double quotes, single quotes lose their quoting capability as well, so it should be no surprise that backquotes change the function of double quotes within them.
I tried your example with other shells, like the Almquist shell on FreeBSD and zsh. As expected, they output some path with.
I have a CouchDB database that contains some daily updated status information. I can use curl to query the database from my bash prompt:
curl 'http://localhost:5984/db/_design/state/_view/stateinfo?group=true&startkey=\["2016-02-10",0\]&endkey=\["2016-02-10\u9999",\{\}\]'
Now I want to write a little function dbq in my .bash_aliases that can be used like so: dbq mm-dd where mm-dd is a date such as 02-10. I tried (among other variations):
dlq() { curl \'http://localhost:5984/db/_design/state/_view/stateinfo?group=true\&startkey=\[2016-{$1},0\]\&endkey=\[2016-{$1}\u9999,\{\}\] \' ;}
I just can't get the argument interpolation (of $1) and the escaping of brackets, quotes, slashes and ampersands in the bash function right. Any ideas on how to do this properly? (BTW, this escaping business when transitioning from a working bash prompt solution towards a reusable bash function with arguments has slowed me down several times. Any hints and pointers to some helpful resources are much appreciated.)
I find the fact that you need to pass backslashes in your original command very odd, and worry a bit that it may indicate that your backend is vulnerable to some sort of remote-code-execution attack. But putting that aside:
The thing to remember about bash quoting is that the rules around ' are very short and simple, but are so short and simple that they often surprise people. Specifically, the rule is this: absolutely nothing is special until the next ' mark. Nothing. Not backslashes, not newlines, nothing. Everything gets passed through.
This is different from most programming languages, which trips people up.
In your original command line, when you pass this:
curl 'http://localhost:5984/db/_design/state/_view/stateinfo?group=true&startkey=\["2016-02-10",0\]&endkey=\["2016-02-10\u9999",\{\}\]'
That means that the argument to curl is literally
http://localhost:5984/db/_design/state/_view/stateinfo?group=true&startkey=\["2016-02-10",0\]&endkey=\["2016-02-10\u9999",\{\}\]
with all the backslashes intact.
Now, in your function the easiest way to pass what you want is to use the fact that two adjacent strings in bash get passed together as a single argument, and do:
dlq() {
curl 'http://localhost:5984/db/_design/state/_view/stateinfo?group=true&startkey=\["2016-'"$1"'",0\]&endkey=\["2016-'"$2"'\u9999",\{\}\]'
}
Note that what I have there is a string in single quotes that ends with "2016-, then a string in double quotes that is "$1", then a string in single quotes that begins with a double quote character and continues until it also ends in "2016- and then a double quoted "$2", and then a single quoted string until the end of the line.
The body of a function is effectively quoted (one reason they are superior to aliases), so you don't need to escape the single quote or anything inside them. You do, however, need to use double quotes so that $1 is expanded.
Something like...
dlq() {
baseurl="http://localhost:5984/db/_design/state/_view/stateinfo"
curl "$baseurl?group=true&startkey=[2016-{$1},0\]&endkey=[2016-{$1}\u9999,{}]"
}
(Please double-check the parameters; I'm not sure what was escaped erroneously and what might need escaping for the API.)
I'm having an extremely hard time figuring out how to echo this:
I keep getting this error:
bash: ![alt: event not found
Using double quotes around it does not work. The using single quotes around it does work, however, I also need to echo strings that have single quotes within them. I wouldn't be able to wrap the string with single quotes then.
Is there a way to echo a string of ANY content?
Thanks.
EDIT: Here is some context. I am making a Markdown renderer that grabs the content of a code editor, then appends every line of the code individually into a text file. I am doing this by doing this:
echo TheLineOfMarkdown > textfile.txt
Unlike in many programing languages, '...' and "..." in Bash do not represent "strings" per se; they quote/escape whatever they contain, but they do not create boundaries that separate arguments. So, for example, these two commands are equivalent:
echo foobar
echo "fo"ob'ar'
So if you need to quote some of an argument with single-quotes, and a different part of the argument has to contain single-quotes — no problem.
For example:
echo '![alt text](https://... "What'"'"'s up, Doc?")'
Another option is to use \, which is similar to '...' except that it only quotes a single character. It can even be used inside double-quotes:
echo "\![alt text](https://... \"What's up, Doc?\")"
For more information, see §3.1.2 "Quoting" in the Bash Reference Manual.
! is annoying. My advice: Use \!.
! invokes history completion, which is also performed inside double-quotes. So you need to single-quote the exclamation mark, but as you say that conflicts with the need to not single-quote other single-quotes.
Remember that you can mix quotes:
$ echo '!'"'"'"'
!'"
(That's just one argument.) But in this case, the backslash is easier to type and quite possibly more readable.
I have the following command
cmd '"asdf" "a'sdf"'
I need to surround the arguments with single quotes only. cmd doesnt work with double quotes for some reason I dont know. The above command doesnt work because the single in the middle terminates the first single quote. If I escape, to the following
cmd '"asdf" "a\'sdf"'
It still doesnt work. How do I get this working?
According to the bash man page:
Enclosing characters in single quotes preserves the literal value
of each character within the quotes. A single quote may not occur
between single quotes, even when preceded by a backslash.
So the answer is, you can't include a single quote within a single-quoted string no matter how you try. But depending on how your script is set up you may be able to use '\'', which will end the first single quote, escape the second, and start another single-quoted string with the third.
A long long time ago, a mentor suggested I use constructs like '"asdf" "a'"'"'sdf"'. It works, but it's bizarre to look at.
Since you can't put single quotes inside single quotes, escaping them like '"asdf" "a'\''sdf' may be the way to go.
Note that you can also use printf and variables interactively or within a shell script. With most shells (you haven't specified what you're using), you should get the similar results to this:
$ fmt='"asdf" "a%ssdf"\n'
$ printf "$fmt" "'"
"asdf" "a'sdf"
$
or you could even include the single quote using its ASCII value:
$ fmt='"asdf" "a\047sdf"\n'
$ printf "$fmt"
"asdf" "a'sdf"
$
or in csh:
% set fmt='"asdf" "a\047sdf"\n'
% printf "$fmt"
"asdf" "a'sdf"
%
This is shell-independent because if your shell doesn't have a printf command built in (as Bash has), then the command will most likely exist as a separate binary in /bin or /usr/bin.
I don't know your use case, so it's difficult to come up with a solution that I know will be applicable.
Well, you can always use autocomplete to help you find the answer.
For instance, if I have a file with an apostrophe in it, but I want to surround the path with single quotes, I can do this (using cat as an example command where the file is named sam's file.txt):
cat 'sam[press tab here]
and it autocompletes to this, for me:
cat 'sam'\''s\ file.txt
So, apparently it is possible. Another answerer mentioned the '\'' thing first, but I figured I'd tell you one way to try to figure it out if that's not working for you (and I thought I'd be another witness to tell you the other answer seems to work).