I need to use a system command (grep) which has a variable concatenated with a string as the regex to search a file.
Is it possible to concatenate a regex for grep between a variable and string in Perl??
I have tried using the . operator but it doesn't work.
if(`grep -e "$fubname._early_exit_indicator = 1" $golden_path/early_exit_information.tsv`){
print "-D- The golden data indicates early exit should occur\n";
$golden_early_exit_indicator=1;
}
Expected to match the regex, "$fubname._early_exit_indicator = 1" but it doesn't match as required.
The expected result should be:
-D- The golden data indicates early exit should occur
But in the present code, it doesn't print this.
Output link: (https://drive.google.com/open?id=1N0SaZ-r3bYPlljKUgTOH5AbxCAaHw7zD)
The problem is that the . operator is not recognized as an operator inside quotes. Dot operators are use between strings, not inside strings. Using the dot inside a string, inserts it literally. This literal dot in the pattern, causes the grep command in your code to fail.
Also note that inside quotes, Perl tries to interpolates variable using certain identifier parsing rules.
See perldoc perlop for the different types of quoting that are used in Perl, and see perldoc perldata for information about the identifier parsing rules.
In summary, in order to interpolate the variable $fubname in the backticks argument, use
"${fubname}_early_exit_indicator = 1"
Note that we need braces around the identifier, since the following underscore is a valid identifier character. (To the contrary a literal dot is not a valid identifier character, so if following character was a literal dot, you would not need the braces around the identifier.)
The . operator will not work inside the quotes. use something like this-
if(`grep -e "${fubname}_early_exit_indicator = 1" ...
I hope this works
Related
I want to run a command from a bash script which has single quotes and some other commands inside the single quotes and a variable.
e.g. repo forall -c '....$variable'
In this format, $ is escaped and the variable is not expanded.
I tried the following variations but they were rejected:
repo forall -c '...."$variable" '
repo forall -c " '....$variable' "
" repo forall -c '....$variable' "
repo forall -c "'" ....$variable "'"
If I substitute the value in place of the variable the command is executed just fine.
Please tell me where am I going wrong.
Inside single quotes everything is preserved literally, without exception.
That means you have to close the quotes, insert something, and then re-enter again.
'before'"$variable"'after'
'before'"'"'after'
'before'\''after'
Word concatenation is simply done by juxtaposition. As you can verify, each of the above lines is a single word to the shell. Quotes (single or double quotes, depending on the situation) don't isolate words. They are only used to disable interpretation of various special characters, like whitespace, $, ;... For a good tutorial on quoting see Mark Reed's answer. Also relevant: Which characters need to be escaped in bash?
Do not concatenate strings interpreted by a shell
You should absolutely avoid building shell commands by concatenating variables. This is a bad idea similar to concatenation of SQL fragments (SQL injection!).
Usually it is possible to have placeholders in the command, and to supply the command together with variables so that the callee can receive them from the invocation arguments list.
For example, the following is very unsafe. DON'T DO THIS
script="echo \"Argument 1 is: $myvar\""
/bin/sh -c "$script"
If the contents of $myvar is untrusted, here is an exploit:
myvar='foo"; echo "you were hacked'
Instead of the above invocation, use positional arguments. The following invocation is better -- it's not exploitable:
script='echo "arg 1 is: $1"'
/bin/sh -c "$script" -- "$myvar"
Note the use of single ticks in the assignment to script, which means that it's taken literally, without variable expansion or any other form of interpretation.
The repo command can't care what kind of quotes it gets. If you need parameter expansion, use double quotes. If that means you wind up having to backslash a lot of stuff, use single quotes for most of it, and then break out of them and go into doubles for the part where you need the expansion to happen.
repo forall -c 'literal stuff goes here; '"stuff with $parameters here"' more literal stuff'
Explanation follows, if you're interested.
When you run a command from the shell, what that command receives as arguments is an array of null-terminated strings. Those strings may contain absolutely any non-null character.
But when the shell is building that array of strings from a command line, it interprets some characters specially; this is designed to make commands easier (indeed, possible) to type. For instance, spaces normally indicate the boundary between strings in the array; for that reason, the individual arguments are sometimes called "words". But an argument may nonetheless have spaces in it; you just need some way to tell the shell that's what you want.
You can use a backslash in front of any character (including space, or another backslash) to tell the shell to treat that character literally. But while you can do something like this:
reply=\”That\'ll\ be\ \$4.96,\ please,\"\ said\ the\ cashier
...it can get tiresome. So the shell offers an alternative: quotation marks. These come in two main varieties.
Double-quotation marks are called "grouping quotes". They prevent wildcards and aliases from being expanded, but mostly they're for including spaces in a word. Other things like parameter and command expansion (the sorts of thing signaled by a $) still happen. And of course if you want a literal double-quote inside double-quotes, you have to backslash it:
reply="\"That'll be \$4.96, please,\" said the cashier"
Single-quotation marks are more draconian. Everything between them is taken completely literally, including backslashes. There is absolutely no way to get a literal single quote inside single quotes.
Fortunately, quotation marks in the shell are not word delimiters; by themselves, they don't terminate a word. You can go in and out of quotes, including between different types of quotes, within the same word to get the desired result:
reply='"That'\''ll be $4.96, please," said the cashier'
So that's easier - a lot fewer backslashes, although the close-single-quote, backslashed-literal-single-quote, open-single-quote sequence takes some getting used to.
Modern shells have added another quoting style not specified by the POSIX standard, in which the leading single quotation mark is prefixed with a dollar sign. Strings so quoted follow similar conventions to string literals in the ANSI standard version of the C programming language, and are therefore sometimes called "ANSI strings" and the $'...' pair "ANSI quotes". Within such strings, the above advice about backslashes being taken literally no longer applies. Instead, they become special again - not only can you include a literal single quotation mark or backslash by prepending a backslash to it, but the shell also expands the ANSI C character escapes (like \n for a newline, \t for tab, and \xHH for the character with hexadecimal code HH). Otherwise, however, they behave as single-quoted strings: no parameter or command substitution takes place:
reply=$'"That\'ll be $4.96, please," said the cashier'
The important thing to note is that the single string that gets stored in the reply variable is exactly the same in all of these examples. Similarly, after the shell is done parsing a command line, there is no way for the command being run to tell exactly how each argument string was actually typed – or even if it was typed, rather than being created programmatically somehow.
Below is what worked for me -
QUOTE="'"
hive -e "alter table TBL_NAME set location $QUOTE$TBL_HDFS_DIR_PATH$QUOTE"
EDIT: (As per the comments in question:)
I've been looking into this since then. I was lucky enough that I had repo laying around. Still it's not clear to me whether you need to enclose your commands between single quotes by force. I looked into the repo syntax and I don't think you need to. You could used double quotes around your command, and then use whatever single and double quotes you need inside provided you escape double ones.
just use printf
instead of
repo forall -c '....$variable'
use printf to replace the variable token with the expanded variable.
For example:
template='.... %s'
repo forall -c $(printf "${template}" "${variable}")
Variables can contain single quotes.
myvar=\'....$variable\'
repo forall -c $myvar
I was wondering why I could never get my awk statement to print from an ssh session so I found this forum. Nothing here helped me directly but if anyone is having an issue similar to below, then give me an up vote. It seems any sort of single or double quotes were just not helping, but then I didn't try everything.
check_var="df -h / | awk 'FNR==2{print $3}'"
getckvar=$(ssh user#host "$check_var")
echo $getckvar
What do you get? A load of nothing.
Fix: escape \$3 in your print function.
Does this work for you?
eval repo forall -c '....$variable'
I have something like this:
projectName= echo $tempPBXProjFilePath | sed "s/.*\/\(.*\)\.xcodeproj.*$/\1/g";
I want to extract substring from $tempPBXProjFilePath. And this is correct. However, if I write it like this:
projectName=echo $tempPBXProjFilePath | sed "s/.*\/\(.*\)\.xcodeproj.*$/\1/g";
It is wrong. The difference is the whitespace after the variable.
I know there is no whitespace after variable directly. But what's the meaning of the whitespace after equal-sign. Is there any place whitespace has special meaning?
Variable Assignment
The syntax for variable assignment is:
name=value
Note, there are no spaces around the = sign. If the value has spaces, or special characters, it should be quoted with single quotes:
name='value with spaces or special characters'
or with double quotes for variable expansion:
name="stringA $variable stringB"
If quotes are missing, the second word in the value part is interpreted as a command. Actually, this is a way to pass environment variables to a command (see below).
If the value is missing, a variable with an empty value is created.
Environment Variables
There is another syntax that allows to assign environment variables for a command:
nameA=valueA nameB=valueB nameC=valueC command arguments
The name-value pairs are separated with space characters.
Example
LD_PRELOAD=/path/to/my/malloc.so /bin/ls
The command assigns LD_PRELOAD environment variable to /path/to/my/malloc.so before invoking /bin/ls.
Your Commands
Thus, your command:
projectName= echo $tempPBXProjFilePath
actually means that you call echo command with arguments expanded from $tempPBXProjFilePath, and set projectName environment variable to an empty value.
And this command:
projectName=echo $tempPBXProjFilePath
sets projectName environment variable to echo string, and calls a command expanded from $tempPBXProjFilePath variable.
Note, if a variable is not enclosed in double quotes, the special characters that present in its value are interpreted by the shell. In order to prevent reinterpretation of the special characters, you should use weak quoting: "$variable". And if you want to prevent even variable expansion in a string value, use single quotes: 'some value'.
Bash divides each line into words at each whitespace (spaces or tabs).
The first word it finds is the name of the command to be executed and the remaining words become arguments to that command.
so when you pass
projectName=echo
bash understand projectName=echo as a variable assignment, and
$tempPBXProjFilePath | sed "s/.*\/\(.*\)\.xcodeproj.*$/\1/g";
as a command! (as pointed by Chris Dodd)
Whitespace
Putting spaces on either or both sides of the equal-sign (=) when assigning a value to a variable will fail.
INCORRECT 1
example = Hello
INCORRECT 2
example= Hello
INCORRECT 3
example =Hello
The only valid form is no spaces between the variable name and assigned value:
CORRECT 1
example=Hello
CORRECT 2
example=" Hello"
You can see more at:
http://wiki.bash-hackers.org/scripting/newbie_traps
I have folders in a directory with names giving specific information. For example:
[allied]_remarkable_points_[treatment]
[nexus]_advisory_plans_[inspection]
....
So I have a structure similar to this: [company]_title_[topic]. The script has to match the file naming structure to variables in a script in order to extract the information:
COMPANY='[allied]';
TITLE='remarkable points'
TOPIC='[treatment]'
The folders do not contain a constant number of characters, so I can't use indexed matching in the script. I managed to extract $TITLE and $TOPIC, but I can't manage to match the first string since the variable brings me back the complete folders name.
FOLDERNAME=${PWD##*/}
This is the line is giving me grief:
COMPANY=`expr $FOLDERNAME : '\(\[.*\]\)'`
I tried to avoid the greedy behaviour by placing ? in the regular expression:
COMPANY=`expr $FOLDERNAME : '\(\[.*?\]\)'`
but as soon as I do that, it returns nothing
Any ideas?
expr isn't needed for regular-expression matching in bash.
[[ $FOLDERNAME =~ (\[[^]]*\]) ]] && COMPANY=${BASH_REMATCH[1]}
Use [^]]* instead of .* to do a non-greedy match of the bracketed portion. An bigger regular expression can capture all three parts:
[[ $FOLDERNAME =~ (\[[^]]*\])_([^_]*)_(\[[^]]*\]) ]] && {
COMPANY=${BASH_REMATCH[1]}
TITLE=${BASH_REMATCH[2]}
TOPIC=${BASH_REMATCH[3]}
}
Bash has built-in string manipulation functionality.
for f in *; do
company=${f%%\]*}
company=${company#\[} # strip off leading [
topic=${f##\[}
topic=${f%\]} # strip off trailing ]
:
done
The construct ${variable#wildcard} removes any prefix matching wildcard from the value of variable and returns the resulting string. Doubling the # obtains the longest possible wildcard match instead of the shortest. Using % selects suffix instead of prefix substitution.
If for some reason you do want to use expr, the reason your non-greedy regex attempt doesn't work is that this syntax is significantly newer than anything related to expr. In fact, if you are using Bash, you should probably not be using expr at all, as Bash provides superior built-in features for every use case where expr made sense, once in the distant past when the sh shell did not have built-in regex matching and arithmetic.
Fortunately, though, it's not hard to get non-greedy matching in this isolated case. Just change the regex to not match on square brackets.
COMPANY=`expr "$FOLDERNAME" : '\(\[[^][]*\]\)'`
(The closing square bracket needs to come first within the negated character class; in any other position, a closing square bracket closes the character class. Many newbies expect to be able to use backslash escapes for this, but that's not how it works. Notice also the addition of double quotes around the variable.)
If you're not adverse to using grep, then:
COMPANY=$(grep -Po "^\[.*?\]" $FOLDERNAME)
I'm trying to write an expect script, which is based on Tcl. (actually I'm trying to generate expect scripts in python, and no, I can't use pyexpect)
How can I send a string without special characters (like [) being interpreted as commands?
In bash shell, I can use single quotes to write raw strings.
$ echo 'hello $world'
hello $world
But with expect, I have many problems with special characters. For example, brackets are interpreted as commands.
expect1.1> send_user "[hello]"
invalid command name "hello"
while executing
"hello"
How can I just print the string "[hello]"?
EDIT: just enclosing a string in curly braces doesn't always work.
For example
expect1.4> send_user {h}ello}
doesn't print the string "h}ello". It gives an error:
extra characters after close-brace
while executing
"send {h}ello}
And there is no way to escape the curly braces according to the doc.
Or what if I want to send a string starting with *} ? Rule 5 in http://tcl.tk/man/tcl8.5/TclCmd/Tcl.htm#M10 will interfere and no way to escape the * or the }.
Curly braces inhibit all expansion*:
echo {hello $world}
You could also just escape the brackets:
Echo "hello \$world"
Curly braces inhibit all expansion when the statement is parsed. However, some commands may choose to do their own expansion internally. The if statement is one such command -- even with curly braces variables will get expanded. Not many commands have this behavior, though.
This is all documented in the Tcl man page. Here's a link to the section on curly braces: http://tcl.tk/man/tcl8.5/TclCmd/Tcl.htm#M10
You might also be interested in this question on stackoverflow: General string quoting for Tcl
Note that curly braces aren't magic -- you still have to apply the quoting rules. For example, you can't have unbalanced curly braces, etc. the precise quoting mechanism you need will vary depending on the data. For some strings that means curly braces, for some it means quotes, for others you might need the backslash. For still others you may need a combination along with string concatenation.
I want to write a wrapper bash script, and to pass all arguments to a called program. I was very sure, that this works correctly:
#!/bin/sh
someProgam $#
But when passing exotic arguments (empty, unescaped, in quotes, ...) this fails.
For example: without the wrapper script, someProgram "1 2" 3 results in the arguments
[1 2] and [3].
But called from the script, I get [1], [2], [3].
Braces are just for visualization.
NOTE: It's a Java program, which is called. But I think it doesn't matter.
#!/bin/sh
someProgram "$#"
See also the bash docs on special parameters.
BTW1, "$#" is not specific to bash. You can probably rely on "$#" in cross-platform sh scripts to be run just about anywhere.
BTW2, in case this happens to be the last line in that script, you can save your operating system a few bytes and an entry in the process table by changing the line to something like
exec someProgram "$#"
to augment ndim's answer: the behavior of "$#" is not specific to bash. it's prescribed by the Single Unix Specification:
2.2.3 Double-Quotes
Enclosing characters in double-quotes ( "" ) shall preserve the literal value of all characters within the double-quotes, with the exception of the characters dollar sign, backquote, and backslash, as follows:
The parameter '#' has special meaning inside double-quotes and is described in Special Parameters.
2.5.2 Special Parameters
Listed below are the special parameters and the values to which they shall expand. Only the values of the special parameters are listed; see Word Expansions for a detailed summary of all the stages involved in expanding words.
#
Expands to the positional parameters, starting from one. When the expansion occurs within double-quotes, and where field splitting (see Field Splitting) is performed, each positional parameter shall expand as a separate field, with the provision that the expansion of the first parameter shall still be joined with the beginning part of the original word (assuming that the expanded parameter was embedded within a word), and the expansion of the last parameter shall still be joined with the last part of the original word. If there are no positional parameters, the expansion of '#' shall generate zero fields, even when '#' is double-quoted.