I have a file1.txt with the below contents:
time="2016-04-25T17:43:11Z" level=info msg="SHA1 Fingerprint=9F:AD:D4:FD:22:24:20:A2:1E:0C:7F:D0:19:C5:80:42:66:56:AC:6F"
I want the file to look as below:
9F:AD:D4:FD:22:24:20:A2:1E:0C:7F:D0:19:C5:80:42:66:56:AC:6F
Actually, I need to pass the command as a string. That is why the bash command needs to be encapsulated in a string with double quotes. However, when i include
" grep -Po '(?<=Fingerprint=)[^"]*' "
I don't get the desired output. It seems that i need to escape the double quotes correctly.
To answer the literal question, you can use a backslash to escape literal double quotes in your command. However, for the reasons given in BashFAQ #50, this is exceedingly poor practice:
# Avoid this absent a very good reason
grep_cmd_str="grep -Po '(?<=Fingerprint=)[^\"]*'"
eval "$grep_cmd_str" <file1.txt # eval is necessary, with all the issues that implies
Better practice when you need to store a simple command (with no redirections or other shell constructs)[1] in a variable is to use an array[2], not a scalar variable, to hold its arguments:
# Use this principally if you need to dynamically build up an argument list
grep_args=( grep -Po '(?<=Fingerprint=)[^"]*' )
"${grep_args[#]}" <file1.txt
If you don't have any constraints that require you to use either of the above, consider a function (which does allow redirections and shell constructs so long as they're hardcoded):
# Use this whenever possible, in preference to the above
grep_fp() { grep -Po '(?<=Fingerprint=)[^"]*' "$#"; }
grep_fp <file1.txt
[1] - Not evaluating shell constructs, in this context, is a security feature: it protects you against malicious filenames or similar content in values which have been substituted into the value you're using as a command.
[2] - Note that arrays are not available in POSIX sh, which your question is also tagged for. That said, similar functionality is available by overriding "$#" (presumably within a limited scope, such as a function, such that its original value is preserved).
Related
I want to run a command from a bash script which has single quotes and some other commands inside the single quotes and a variable.
e.g. repo forall -c '....$variable'
In this format, $ is escaped and the variable is not expanded.
I tried the following variations but they were rejected:
repo forall -c '...."$variable" '
repo forall -c " '....$variable' "
" repo forall -c '....$variable' "
repo forall -c "'" ....$variable "'"
If I substitute the value in place of the variable the command is executed just fine.
Please tell me where am I going wrong.
Inside single quotes everything is preserved literally, without exception.
That means you have to close the quotes, insert something, and then re-enter again.
'before'"$variable"'after'
'before'"'"'after'
'before'\''after'
Word concatenation is simply done by juxtaposition. As you can verify, each of the above lines is a single word to the shell. Quotes (single or double quotes, depending on the situation) don't isolate words. They are only used to disable interpretation of various special characters, like whitespace, $, ;... For a good tutorial on quoting see Mark Reed's answer. Also relevant: Which characters need to be escaped in bash?
Do not concatenate strings interpreted by a shell
You should absolutely avoid building shell commands by concatenating variables. This is a bad idea similar to concatenation of SQL fragments (SQL injection!).
Usually it is possible to have placeholders in the command, and to supply the command together with variables so that the callee can receive them from the invocation arguments list.
For example, the following is very unsafe. DON'T DO THIS
script="echo \"Argument 1 is: $myvar\""
/bin/sh -c "$script"
If the contents of $myvar is untrusted, here is an exploit:
myvar='foo"; echo "you were hacked'
Instead of the above invocation, use positional arguments. The following invocation is better -- it's not exploitable:
script='echo "arg 1 is: $1"'
/bin/sh -c "$script" -- "$myvar"
Note the use of single ticks in the assignment to script, which means that it's taken literally, without variable expansion or any other form of interpretation.
The repo command can't care what kind of quotes it gets. If you need parameter expansion, use double quotes. If that means you wind up having to backslash a lot of stuff, use single quotes for most of it, and then break out of them and go into doubles for the part where you need the expansion to happen.
repo forall -c 'literal stuff goes here; '"stuff with $parameters here"' more literal stuff'
Explanation follows, if you're interested.
When you run a command from the shell, what that command receives as arguments is an array of null-terminated strings. Those strings may contain absolutely any non-null character.
But when the shell is building that array of strings from a command line, it interprets some characters specially; this is designed to make commands easier (indeed, possible) to type. For instance, spaces normally indicate the boundary between strings in the array; for that reason, the individual arguments are sometimes called "words". But an argument may nonetheless have spaces in it; you just need some way to tell the shell that's what you want.
You can use a backslash in front of any character (including space, or another backslash) to tell the shell to treat that character literally. But while you can do something like this:
reply=\”That\'ll\ be\ \$4.96,\ please,\"\ said\ the\ cashier
...it can get tiresome. So the shell offers an alternative: quotation marks. These come in two main varieties.
Double-quotation marks are called "grouping quotes". They prevent wildcards and aliases from being expanded, but mostly they're for including spaces in a word. Other things like parameter and command expansion (the sorts of thing signaled by a $) still happen. And of course if you want a literal double-quote inside double-quotes, you have to backslash it:
reply="\"That'll be \$4.96, please,\" said the cashier"
Single-quotation marks are more draconian. Everything between them is taken completely literally, including backslashes. There is absolutely no way to get a literal single quote inside single quotes.
Fortunately, quotation marks in the shell are not word delimiters; by themselves, they don't terminate a word. You can go in and out of quotes, including between different types of quotes, within the same word to get the desired result:
reply='"That'\''ll be $4.96, please," said the cashier'
So that's easier - a lot fewer backslashes, although the close-single-quote, backslashed-literal-single-quote, open-single-quote sequence takes some getting used to.
Modern shells have added another quoting style not specified by the POSIX standard, in which the leading single quotation mark is prefixed with a dollar sign. Strings so quoted follow similar conventions to string literals in the ANSI standard version of the C programming language, and are therefore sometimes called "ANSI strings" and the $'...' pair "ANSI quotes". Within such strings, the above advice about backslashes being taken literally no longer applies. Instead, they become special again - not only can you include a literal single quotation mark or backslash by prepending a backslash to it, but the shell also expands the ANSI C character escapes (like \n for a newline, \t for tab, and \xHH for the character with hexadecimal code HH). Otherwise, however, they behave as single-quoted strings: no parameter or command substitution takes place:
reply=$'"That\'ll be $4.96, please," said the cashier'
The important thing to note is that the single string that gets stored in the reply variable is exactly the same in all of these examples. Similarly, after the shell is done parsing a command line, there is no way for the command being run to tell exactly how each argument string was actually typed – or even if it was typed, rather than being created programmatically somehow.
Below is what worked for me -
QUOTE="'"
hive -e "alter table TBL_NAME set location $QUOTE$TBL_HDFS_DIR_PATH$QUOTE"
EDIT: (As per the comments in question:)
I've been looking into this since then. I was lucky enough that I had repo laying around. Still it's not clear to me whether you need to enclose your commands between single quotes by force. I looked into the repo syntax and I don't think you need to. You could used double quotes around your command, and then use whatever single and double quotes you need inside provided you escape double ones.
just use printf
instead of
repo forall -c '....$variable'
use printf to replace the variable token with the expanded variable.
For example:
template='.... %s'
repo forall -c $(printf "${template}" "${variable}")
Variables can contain single quotes.
myvar=\'....$variable\'
repo forall -c $myvar
I was wondering why I could never get my awk statement to print from an ssh session so I found this forum. Nothing here helped me directly but if anyone is having an issue similar to below, then give me an up vote. It seems any sort of single or double quotes were just not helping, but then I didn't try everything.
check_var="df -h / | awk 'FNR==2{print $3}'"
getckvar=$(ssh user#host "$check_var")
echo $getckvar
What do you get? A load of nothing.
Fix: escape \$3 in your print function.
Does this work for you?
eval repo forall -c '....$variable'
I have a file whose name is -q (It looks like this is made by accident)
I want to see it’s content, so I tried these
$ cat '-q'
$ cat "-q"
$ cat $'-q'
But nothing worked.(All gives same error cat: invalid option -- 'q')
Is there any way to see it’s content?
sjsam's helpful answer offers viable solutions, but it's worth digging deeper:
Arguments passed to Unix utilities (such as cat) can be divided into two groups (using POSIX terminology):
options (e.g., -l; depending on the option's definition, possibly followed by an option-argument) - switches that modify the utility's basic behavior.
operands (e.g., a filename) - the data that the utility operates on.
POSIX requires all options, if any, to come before any operands, but GNU utilities relax this requirement to allow intermingling options and operands for convenience.
Either way, a - followed by a letter or digit generally implies an option.
So as to allow passing operands that happen to start with -, special option -- can be used to tell any utility: treat any subsequent arguments as operands, even if they look like options.
Since operand -q in this case looks like an option, preceding it with -- to mark it as an operand is a must.
As for how to pass string -q itself as an argument:
-q contains no shell metacharacters, so the shell passes it as-is to cat - no quoting needed.
Quoting ('-q', "-q", $'-q', \-q) does no harm in this case, but has absolutely no effect here, because the shell - after having applied any expansions - simply removes any quoting characters (a process called quote removal) before passing the word to cat (as Charles Duffy notes in comments on the question).
to create a file named -q do
touch -- "-q"
and to view its contents
cat -- '-q' # Used single quotes to treat anything inside as literal characters
should do it. -- means what follows should be treated as a positional parameter.
Pls take time to look at #rici's [ answer ] which reminds some substle aspects regarding single vs double quotes.
I'm having an extremely hard time figuring out how to echo this:
I keep getting this error:
bash: ![alt: event not found
Using double quotes around it does not work. The using single quotes around it does work, however, I also need to echo strings that have single quotes within them. I wouldn't be able to wrap the string with single quotes then.
Is there a way to echo a string of ANY content?
Thanks.
EDIT: Here is some context. I am making a Markdown renderer that grabs the content of a code editor, then appends every line of the code individually into a text file. I am doing this by doing this:
echo TheLineOfMarkdown > textfile.txt
Unlike in many programing languages, '...' and "..." in Bash do not represent "strings" per se; they quote/escape whatever they contain, but they do not create boundaries that separate arguments. So, for example, these two commands are equivalent:
echo foobar
echo "fo"ob'ar'
So if you need to quote some of an argument with single-quotes, and a different part of the argument has to contain single-quotes — no problem.
For example:
echo '![alt text](https://... "What'"'"'s up, Doc?")'
Another option is to use \, which is similar to '...' except that it only quotes a single character. It can even be used inside double-quotes:
echo "\![alt text](https://... \"What's up, Doc?\")"
For more information, see §3.1.2 "Quoting" in the Bash Reference Manual.
! is annoying. My advice: Use \!.
! invokes history completion, which is also performed inside double-quotes. So you need to single-quote the exclamation mark, but as you say that conflicts with the need to not single-quote other single-quotes.
Remember that you can mix quotes:
$ echo '!'"'"'"'
!'"
(That's just one argument.) But in this case, the backslash is easier to type and quite possibly more readable.
I would like to extract a string as below:
setenv VARIABLE /opt/Application/File/Version22
I want to display only
File/Version22
on screen. It means /opt/Application/ is always the prefix. I want to display the rest of the string. How do I do it in csh?
Regards
Now that you've updated the question (thank you for that), it's clear that you always want to remove the /opt/Application prefix.
The most straightforward way to do that, which will work in any shell, is:
echo $VARIABLE | sed 's|^/opt/Application/||'
(It's usual to use / as a delimiter for replacements like this, but you can use any punctuation character; I'm using | to avoid conflicting with the / characters in the pattern.)
A more tcsh-specific, and possibly more efficient, way to do it is:
echo $VARIABLE:s|/opt/Application/||
It's likely to be more efficient because it's done within the shell and avoids invoking the external sed command. On the other hand, the overhead of executing sed is unlikely to be significant.
Note carefully that the :s syntax is not supported in the original csh. It is supported in tcsh and in some newer versions of csh. If you want to do this portably, just use sed.
C-Shell has the built-in string modifiers that can do this with very little code:
echo $VARIABLE:h:t/$VARIABLE:t
string modifiers:
:h = remove the last directory (aka head)
:t = remove every directory except the last one (aka tail)
I assume that you want to get the 2 last names in a filename:
setenv VARIABLE /opt/Application/File/Version22
set lastfile=`basename $VARIABLE`
set prevpath=`dirname $VARIABLE`
set previousfile=`basename $prevpath`
set file=$previousfile/$lastfile
echo $file
What idiom should one use in Bash scripts (no Perl, Python, and such please) to build up a command line for another program out of the script's arguments while handling filenames correctly?
By correctly, I mean handling filenames with spaces or odd characters without inadvertently causing the other program to handle them as separate arguments (or, in the case of < or > — which are, after all, valid if unfortunate filename characters if properly escaped — doing something even worse).
Here's a made-up example of what I mean, in a form that doesn't handle filenames correctly: Let's assume this script (foo) builds up a command line for a command (bar, assumed to be in the path) by taking all of foo's input arguments and moving anything that looks like a flag to the front, and then invoking bar:
#!/bin/bash
# This is clearly wrong
FILES=
FLAGS=
for ARG in "$#"; do
echo "foo: Handling $ARG"
if [ x${ARG:0:1} = "x-" ]; then
# Looks like a flag, add it to the flags string
FLAGS="$FLAGS $ARG"
else
# Looks like a file, add it to the files string
FILES="$FILES $ARG"
fi
done
# Call bar with the flags and files (we don't care that they'll
# have an extra space or two)
CMD="bar $FLAGS $FILES"
echo "Issuing: $CMD"
$CMD
(Note that this just an example; there are lots of other times one needs to do this and that to a bunch of args and then pass them onto other programs.)
In a naive scenario with simple filenames, that works great. But if we assume a directory containing the files
one
two
three and a half
four < five
then of course the command foo * fails miserably in its task:
foo: Handling four < five
foo: Handling one
foo: Handling three and a half
foo: Handling two
Issuing: bar four < five one three and a half two
If we actually allow foo to issue that command, well, the results won't be what we're expecting.
Previously I've tried to handle this through the simple expedient of ensuring that there are quotes around each filename, but I've (very) quickly learned that that is not the correct approach. :-)
So what is? Constraints:
I want to keep the idiom as simple as possible (not least so I can remember it).
I'm looking for a general-purpose idiom, hence my making up the bar program and the contrived example above instead of using a real scenario where people might easily (and reasonably) go down the route of trying to use features in the target program.
I want to stick to Bash script, I don't want to call out to Perl, Python, etc.
I'm fine with relying on (other) standard *nix utilities, like xargs, sed, or tr provided we don't get too obtuse (see #1 above). (Apologies to Perl, Python, etc. programmers who think #3 and #4 combine to draw an arbitrary distinction.)
If it matters, the target program might also be a Bash script, or might not. I wouldn't expect it to matter...
I don't just want to handle spaces, I want to handle weird characters correctly as well.
I'm not bothered if it doesn't handle filenames with embedded nul characters (literally character code 0). If someone's managed to create one in their filesystem, I'm not worried about handling it, they've tried really hard to mess things up.
Thanks in advance, folks.
Edit: Ignacio Vazquez-Abrams pointed me to Bash FAQ entry #50, which after some reading and experimentation seems to indicate that one way is to use Bash arrays:
#!/bin/bash
# This appears to work, using Bash arrays
# Start with blank arrays
FILES=()
FLAGS=()
for ARG in "$#"; do
echo "foo: Handling $ARG"
if [ x${ARG:0:1} = "x-" ]; then
# Looks like a flag, add it to the flags array
FLAGS+=("$ARG")
else
# Looks like a file, add it to the files array
FILES+=("$ARG")
fi
done
# Call bar with the flags and files
echo "Issuing (but properly delimited, not exactly as this appears): bar ${FLAGS[#]} ${FILES[#]}"
bar "${FLAGS[#]}" "${FILES[#]}"
Is that correct and reasonable? Or am I relying on something environmental above that will bite me later. It seems to work and it ticks all the other boxes for me (simple, easy to remember, etc.). It does appear to rely on a relatively recent Bash feature (FAQ entry #50 mentions v3.1, but I wasn't sure whether that was arrays in general of some of the syntax they were using with it), but I think it's likely I'll only be dealing with versions that have it.
(If the above is correct and you want to un-delete your answer, Ignacio, I'll accept it provided I haven't accepted any others yet, although I stand by my statement about link-only answers.)
Why do you want to "build up" a command? Add the files and flags to arrays using proper
quoting and issue the command directly using the quoted arrays as arguments.
Selected lines from your script (omitting unchanged ones):
if [[ ${ARG:0:1} == - ]]; then # using a Bash idiom
FLAGS+=("$ARG") # add an element to an array
FILES+=("$ARG")
echo "Issuing: bar \"${FLAGS[#]}\" \"${FILES[#]}\""
bar "${FLAGS[#]}" "${FILES[#]}"
For a quick demo of using arrays in this manner:
$ a=(aaa 'bbb ccc' ddd); for arg in "${a[#]}"; do echo "..${arg}.."; done
Output:
..aaa..
..bbb ccc..
..ddd..
Please see BashFAQ/050 regarding putting commands in variables. The reason that your script doesn't work is because there's no way to quote the arguments within a quoted string. If you were to put quotes there, they would be considered part of the string itself instead of as delimiters. With the arguments left unquoted, word splitting is done and arguments that include spaces are seen as more than one argument. Arguments with "<", ">" or "|" are not a problem in any case since redirection and piping is performed before variable expansion so they are seen as characters in a string.
By putting the arguments (filenames) in an array, spaces, newlines, etc., are preserved. By quoting the array variable when it's passed as an argument, they are preserved on the way to the consuming program.
Some additional notes:
Use lowercase (or mixed case) variable names to reduce the chance that they will collide with the shell's builtin variables.
If you use single square brackets for conditionals in any modern shell, the archaic "x" idiom is no longer necessary if you quote the variables (see my answer here). However, in Bash, use double brackets. They provide additional features (see my answer here).
Use getopts as Let_Me_Be suggested. Your script, though I know it's only an example, will not be able to handle switches that take arguments.
This for ARG in "$#" can be shortened to this for ARG (but I prefer the readability of the more explicit version).
See BashFAQ #50 (and also maybe #35 on option parsing). For the scenario you describe, where you're building a command dynamically, the best option is to use arrays rather than simple strings, as they won't lose track of where the word boundaries are. The general rules are: to create an array, instead of VAR="foo bar baz", use VAR=("foo" "bar" "baz"); to use the array, instead of $VAR, use "${VAR[#]}". Here's a working version of your example script using this method:
#!/bin/bash
# This is clearly wrong
FILES=()
FLAGS=()
for ARG in "$#"; do
echo "foo: Handling $ARG"
if [ x${ARG:0:1} = "x-" ]; then
# Looks like a flag, add it to the flags array
FLAGS=("${FLAGS[#]}" "$ARG") # FLAGS+=("$ARG") would also work in bash 3.1+, as Dennis pointed out
else
# Looks like a file, add it to the files string
FILES=("${FILES[#]}" "$ARG")
fi
done
# Call bar with the flags and files (we don't care that they'll
# have an extra space or two)
CMD=("bar" "${FLAGS[#]}" "${FILES[#]}")
echo "Issuing: ${CMD[*]}"
"${CMD[#]}"
Note that in the echo command I used "${VAR[*]}" instead of the [#] form because there's no need/point to preserving word breaks here. If you wanted to print/record the command in unambiguous form, this would be a lot messier.
Also, this gives you no way to build up redirections or other special shell options in the built command -- if you add >outfile to the FILES array, it'll be treated as just another command argument, not a shell redirection. If you need to programmatically build these, be prepared for headaches.
getopts should be able to handle spaces in arguments correctly ("file name.txt"). Weird characters should work as well, assuming they are correctly escaped (ls -b).