Writing a small script in bash (MacOS in fact) and I want to use find, with multiple sources. Not normally a problem, but the list of source directories to search is held as a string in a variable. Again, not normally a problem, but some of them contain spaces in their name.
I can construct the full command string and if entered directly at the command prompt (copy and paste in fact) it works as required and expected. But when I try and run it within the script, it flunks out on the spaces in the name and I have been unable to get around this.
I cannot quote the entire source string as that is then just seen as one single item which of course does not exist. I escape each space with a backslash within the string held in the variable and it is simply lost. If I use double backslash, they both remain in place and again it fails. Any method of quoting I have tried is basically ignored, the quotes are seen as normal characters and splitting is done at each space.
I have so far only been able to use eval on the whole command string to get it to work but I felt there ought to be a better solution than this.
Ironically, if I use AppleScript I CAN create a suitable command string and run it perfectly with doShellScript (ok, that's using JXA, but it's the same with actual AppleScript). However, I have so far been unable to find the correct escape mechanism just in a bash script, without resorting to eval.
Anyone suggest a solution to this?
If possible, don't store all paths in one string. An array is safer and more convenient:
paths=("first path" "second path" "and so on")
find "${paths[#]}"
The find command will expand to
find "first path" "second path" "and so on"
If you have to use the string and don't want to use eval, split the string into an array:
string="first\ path second\ path and\ so\ on"
read -a paths <<< "$string"
find "${paths[#]}"
Paths inside string should use \ to escape spaces; wraping paths inside"" or '' will not work. eval might be the better option here.
Related
I want to run a command from a bash script which has single quotes and some other commands inside the single quotes and a variable.
e.g. repo forall -c '....$variable'
In this format, $ is escaped and the variable is not expanded.
I tried the following variations but they were rejected:
repo forall -c '...."$variable" '
repo forall -c " '....$variable' "
" repo forall -c '....$variable' "
repo forall -c "'" ....$variable "'"
If I substitute the value in place of the variable the command is executed just fine.
Please tell me where am I going wrong.
Inside single quotes everything is preserved literally, without exception.
That means you have to close the quotes, insert something, and then re-enter again.
'before'"$variable"'after'
'before'"'"'after'
'before'\''after'
Word concatenation is simply done by juxtaposition. As you can verify, each of the above lines is a single word to the shell. Quotes (single or double quotes, depending on the situation) don't isolate words. They are only used to disable interpretation of various special characters, like whitespace, $, ;... For a good tutorial on quoting see Mark Reed's answer. Also relevant: Which characters need to be escaped in bash?
Do not concatenate strings interpreted by a shell
You should absolutely avoid building shell commands by concatenating variables. This is a bad idea similar to concatenation of SQL fragments (SQL injection!).
Usually it is possible to have placeholders in the command, and to supply the command together with variables so that the callee can receive them from the invocation arguments list.
For example, the following is very unsafe. DON'T DO THIS
script="echo \"Argument 1 is: $myvar\""
/bin/sh -c "$script"
If the contents of $myvar is untrusted, here is an exploit:
myvar='foo"; echo "you were hacked'
Instead of the above invocation, use positional arguments. The following invocation is better -- it's not exploitable:
script='echo "arg 1 is: $1"'
/bin/sh -c "$script" -- "$myvar"
Note the use of single ticks in the assignment to script, which means that it's taken literally, without variable expansion or any other form of interpretation.
The repo command can't care what kind of quotes it gets. If you need parameter expansion, use double quotes. If that means you wind up having to backslash a lot of stuff, use single quotes for most of it, and then break out of them and go into doubles for the part where you need the expansion to happen.
repo forall -c 'literal stuff goes here; '"stuff with $parameters here"' more literal stuff'
Explanation follows, if you're interested.
When you run a command from the shell, what that command receives as arguments is an array of null-terminated strings. Those strings may contain absolutely any non-null character.
But when the shell is building that array of strings from a command line, it interprets some characters specially; this is designed to make commands easier (indeed, possible) to type. For instance, spaces normally indicate the boundary between strings in the array; for that reason, the individual arguments are sometimes called "words". But an argument may nonetheless have spaces in it; you just need some way to tell the shell that's what you want.
You can use a backslash in front of any character (including space, or another backslash) to tell the shell to treat that character literally. But while you can do something like this:
reply=\”That\'ll\ be\ \$4.96,\ please,\"\ said\ the\ cashier
...it can get tiresome. So the shell offers an alternative: quotation marks. These come in two main varieties.
Double-quotation marks are called "grouping quotes". They prevent wildcards and aliases from being expanded, but mostly they're for including spaces in a word. Other things like parameter and command expansion (the sorts of thing signaled by a $) still happen. And of course if you want a literal double-quote inside double-quotes, you have to backslash it:
reply="\"That'll be \$4.96, please,\" said the cashier"
Single-quotation marks are more draconian. Everything between them is taken completely literally, including backslashes. There is absolutely no way to get a literal single quote inside single quotes.
Fortunately, quotation marks in the shell are not word delimiters; by themselves, they don't terminate a word. You can go in and out of quotes, including between different types of quotes, within the same word to get the desired result:
reply='"That'\''ll be $4.96, please," said the cashier'
So that's easier - a lot fewer backslashes, although the close-single-quote, backslashed-literal-single-quote, open-single-quote sequence takes some getting used to.
Modern shells have added another quoting style not specified by the POSIX standard, in which the leading single quotation mark is prefixed with a dollar sign. Strings so quoted follow similar conventions to string literals in the ANSI standard version of the C programming language, and are therefore sometimes called "ANSI strings" and the $'...' pair "ANSI quotes". Within such strings, the above advice about backslashes being taken literally no longer applies. Instead, they become special again - not only can you include a literal single quotation mark or backslash by prepending a backslash to it, but the shell also expands the ANSI C character escapes (like \n for a newline, \t for tab, and \xHH for the character with hexadecimal code HH). Otherwise, however, they behave as single-quoted strings: no parameter or command substitution takes place:
reply=$'"That\'ll be $4.96, please," said the cashier'
The important thing to note is that the single string that gets stored in the reply variable is exactly the same in all of these examples. Similarly, after the shell is done parsing a command line, there is no way for the command being run to tell exactly how each argument string was actually typed – or even if it was typed, rather than being created programmatically somehow.
Below is what worked for me -
QUOTE="'"
hive -e "alter table TBL_NAME set location $QUOTE$TBL_HDFS_DIR_PATH$QUOTE"
EDIT: (As per the comments in question:)
I've been looking into this since then. I was lucky enough that I had repo laying around. Still it's not clear to me whether you need to enclose your commands between single quotes by force. I looked into the repo syntax and I don't think you need to. You could used double quotes around your command, and then use whatever single and double quotes you need inside provided you escape double ones.
just use printf
instead of
repo forall -c '....$variable'
use printf to replace the variable token with the expanded variable.
For example:
template='.... %s'
repo forall -c $(printf "${template}" "${variable}")
Variables can contain single quotes.
myvar=\'....$variable\'
repo forall -c $myvar
I was wondering why I could never get my awk statement to print from an ssh session so I found this forum. Nothing here helped me directly but if anyone is having an issue similar to below, then give me an up vote. It seems any sort of single or double quotes were just not helping, but then I didn't try everything.
check_var="df -h / | awk 'FNR==2{print $3}'"
getckvar=$(ssh user#host "$check_var")
echo $getckvar
What do you get? A load of nothing.
Fix: escape \$3 in your print function.
Does this work for you?
eval repo forall -c '....$variable'
I have a folder that was created automatically. The user unintentionally provided smart (curly) quotes as part of the name, and the process that sanitizes the inputs did not catch these. As a result, the folder name contains the smart quotes. For example:
this-is-my-folder’s-name-“Bob”
I'm now trying to rename/remove said folder on the command line, and none of the standard tricks for dealing with files/folders with special characters (enclosing in quotes, escaping the characters, trying to rename it by inode, etc.) are working. All result in:
mv: cannot move this-is-my-folder’s-name-“Bob” to this-is-my-folders-name-BOB: No such file or directory
Can anyone provide some advice as to how I can achieve this?
To get the name in a format you can copy-and-paste into your shell:
printf '%q\n' this*
...will print out the filename in a manner the shell will accept as valid input. This might look something like:
$'this-is-my-folder200\231s-name-200\234Bob200\235'
...which you can then use as an argument to mv:
mv $'this-is-my-folder200\231s-name-200\234Bob200\235' this-is-my-folders-name-BOB
Incidentally, if your operating system works the same way mine does (when running the test above), this would explain why using single-character globs such as ? for those characters didn't work: They're actually more than one byte long each!
You can use shell globbing token ? to match any single character, so matching the smart quotes using ? should do:
mv this-is-my-folder?s-name-?Bob? new_name
Here replacing the smart quotes with ? to match the file name.
There are several possibilities.
If an initial substring of the file name ending before the first quote is unique within the directory, then you can use filename completion to help you type an appropriate command. Type "mv" (without the quotes) and the unique initial substring, then press the TAB key to request filename completion. Bash will complete the filename with the correct characters, correctly escaped.
Use a graphical file browser. Then you can select the file to rename by clicking on it. (Details of how to proceed from there depend on the browser.) If you don't have a graphical terminal and can't get one, then you may be able to do the same with a text-mode browser such as Midnight Commander.
A simple glob built with the ? or * wildcard should be able to match the filename
Use a more complex glob to select the filename, and perhaps others with the same problem. Maybe something like *[^a-zA-Z0-9-]* would do. Use a pattern substitution to assign a new name. Something like this:
for f in *[^a-zA-Z0-9-]*; do
mv "$f" "${f//[^a-zA-Z0-9-]/}"
done
The substitution replaces all appearances of a characters that are not decimal digits, appercase or lowercase Latin letters, or hyphens with nothing (i.e. it strips them). Do take care before you use this, though, to make sure you're not going to make more changes than you intend to do.
I'm having an extremely hard time figuring out how to echo this:
I keep getting this error:
bash: ![alt: event not found
Using double quotes around it does not work. The using single quotes around it does work, however, I also need to echo strings that have single quotes within them. I wouldn't be able to wrap the string with single quotes then.
Is there a way to echo a string of ANY content?
Thanks.
EDIT: Here is some context. I am making a Markdown renderer that grabs the content of a code editor, then appends every line of the code individually into a text file. I am doing this by doing this:
echo TheLineOfMarkdown > textfile.txt
Unlike in many programing languages, '...' and "..." in Bash do not represent "strings" per se; they quote/escape whatever they contain, but they do not create boundaries that separate arguments. So, for example, these two commands are equivalent:
echo foobar
echo "fo"ob'ar'
So if you need to quote some of an argument with single-quotes, and a different part of the argument has to contain single-quotes — no problem.
For example:
echo '![alt text](https://... "What'"'"'s up, Doc?")'
Another option is to use \, which is similar to '...' except that it only quotes a single character. It can even be used inside double-quotes:
echo "\![alt text](https://... \"What's up, Doc?\")"
For more information, see §3.1.2 "Quoting" in the Bash Reference Manual.
! is annoying. My advice: Use \!.
! invokes history completion, which is also performed inside double-quotes. So you need to single-quote the exclamation mark, but as you say that conflicts with the need to not single-quote other single-quotes.
Remember that you can mix quotes:
$ echo '!'"'"'"'
!'"
(That's just one argument.) But in this case, the backslash is easier to type and quite possibly more readable.
I need to parse the $PATH environment variable in my application.
So I was wondering what escape characters would be valid in $PATH.
I created a test directory called /bin:d and created a test script called funny inside it. It runs if I call it with an absolute path.
I just can't figure out how to escape : in $PATH I tried escaping the colon with \ and wrapping it into single ' and double " quotes. But always when I run which funny it can't find it.
I'm running CentOS 6.
This is impossible according to the POSIX standard. This is not a function of a specific shell, PATH handling is done within the execvp function in the C library. There is no provision for any kind of quoting.
This is the reason why including certain characters (anything not in the "portable filename character set" - colon is specifically called out as an example.) is strongly recommended against.
From SUSv7:
Since <colon> is a separator in this context, directory names that might be used in PATH should not include a <colon> character.
See also source of GLIBC execvp. We can see it uses the strchrnul and memcpy functions for processing the PATH components, with absolutely no provision for skipping over or unescaping any kind of escape character.
Looking at the function
extract_colon_unit
it seems to me that this is impossible. The : is unconditionally and
inescapably used as the path separator.
Well, this is valid at least for bash. Other shells may vary.
You could try mounting it
mount /bin:d /bind
PATH=/bind
According to http://tldp.org/LDP/abs/html/special-chars.html single quotes should preserve all special characters, so without trying it, I would think that '/bin:d' would work (with)in $PATH.
I have the following command
cmd '"asdf" "a'sdf"'
I need to surround the arguments with single quotes only. cmd doesnt work with double quotes for some reason I dont know. The above command doesnt work because the single in the middle terminates the first single quote. If I escape, to the following
cmd '"asdf" "a\'sdf"'
It still doesnt work. How do I get this working?
According to the bash man page:
Enclosing characters in single quotes preserves the literal value
of each character within the quotes. A single quote may not occur
between single quotes, even when preceded by a backslash.
So the answer is, you can't include a single quote within a single-quoted string no matter how you try. But depending on how your script is set up you may be able to use '\'', which will end the first single quote, escape the second, and start another single-quoted string with the third.
A long long time ago, a mentor suggested I use constructs like '"asdf" "a'"'"'sdf"'. It works, but it's bizarre to look at.
Since you can't put single quotes inside single quotes, escaping them like '"asdf" "a'\''sdf' may be the way to go.
Note that you can also use printf and variables interactively or within a shell script. With most shells (you haven't specified what you're using), you should get the similar results to this:
$ fmt='"asdf" "a%ssdf"\n'
$ printf "$fmt" "'"
"asdf" "a'sdf"
$
or you could even include the single quote using its ASCII value:
$ fmt='"asdf" "a\047sdf"\n'
$ printf "$fmt"
"asdf" "a'sdf"
$
or in csh:
% set fmt='"asdf" "a\047sdf"\n'
% printf "$fmt"
"asdf" "a'sdf"
%
This is shell-independent because if your shell doesn't have a printf command built in (as Bash has), then the command will most likely exist as a separate binary in /bin or /usr/bin.
I don't know your use case, so it's difficult to come up with a solution that I know will be applicable.
Well, you can always use autocomplete to help you find the answer.
For instance, if I have a file with an apostrophe in it, but I want to surround the path with single quotes, I can do this (using cat as an example command where the file is named sam's file.txt):
cat 'sam[press tab here]
and it autocompletes to this, for me:
cat 'sam'\''s\ file.txt
So, apparently it is possible. Another answerer mentioned the '\'' thing first, but I figured I'd tell you one way to try to figure it out if that's not working for you (and I thought I'd be another witness to tell you the other answer seems to work).