I'm facing a problem with awk in Linux. I would like to do make this script work :
awk -v var="$MYVAR" "{gsub(/export OTHER_VAR=\$OTHER_VAR:/, "var")}1" /etc/myfile
The problem here is that I want the variable "var" to be interpreted (it works) and the variable $OTHERVAR not to be interpreted, and this I what I can't manage to do.
In the end, I want to do this:
I have a variable
MYVAR=export OTHER_VAR=\$OTHER_VAR:some_text
I want to replace, in /etc/myfile, the following pattern :
export OTHER_VAR=$OTHER_VAR:/folder/bin by export OTHER_VAR=$OTHER_VAR:some_text:/folder/bin.
I hope I made myself clear ...
Thanks in advance !
Sylvain
test_document='export OTHER_VAR=$OTHER_VAR:whatever'
search_regex='^export OTHER_VAR=[$]OTHER_VAR:'
replace_str='export OTHER_VAR=$OTHER_VAR:some_text:'
awk -v search_regex="$search_regex" \
-v replace_str="$replace_str" \
'{gsub(search_regex, replace_str)} {print}' <<<"$test_document"
...properly emits as output:
export OTHER_VAR=$OTHER_VAR:some_text:whatever
Note some changes:
We're escaping the $ in the regex as [$]. Unlike \$, this is parsed consistently across all quoting contexts: It is explicitly generating a regex character class, rather than having any other potential meaning.
Using single quotes for literal strings ensures that no shell interpolation takes place within them.
Using {print} is a bit easier for readers to understand than a bare 1 in awk.
Excluding variable names with meaning to the OS or shell, use of lower-case characters in variable names is in line with POSIX-specified convention. See http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html, fourth paragraph.
Related
What I want:
There is a file /scripts/backup/config.cfg which contains variables. In my specific case the important ones are:
BACKUPLOCATION=""
ROOTLOCATION="/backup"
Then there is a script /scripts/backup/performBackup.sh
For a specific reason I want a part of the script do the following operations:
read the value of the variable ROOTLOCATION
add a ("/" and) timestamp (Date&Time)
safe the new created value to BACKUPLOCATION (by replacing its current value)
Example
If this is the previous state of the config.cfg:
BACKUPLOCATION="dummy"
ROOTLOCATION="/backup"
After the script ran it should be:
BACKUPLOCATION="/backup/2020-05-02-23-00"
ROOTLOCATION="/backup/"
What I tried
First of all the config file gets "loaded" using
source /scripts/backup/config.cfg
I then tried to use the sed command but the quotes are messing with me. Here is one try (which didn't work):
sed -i 's/BACKUPLOCATION\=.*/BACKUPLOCATION="'$ROOTLOCATION/$(date +%Y-%m-%d-%H-%M)'"/' /scripts/backup/config.cfg
Try this:
source /scripts/backup/config.cfg
sed -i 's|BACKUPLOCATION=.*|BACKUPLOCATION="'"$ROOTLOCATION/$(date +%Y-%m-%d-%H-%M)"'"|' /scripts/backup/config.cfg
The problem with your sed is that you use / as delimiter, which is present in $ROOTLOCATION after expansion, therefore sed fails. I used |, which is usually is not present in filenames. If you ever create a file with |, that sed will fail too! So, "know your data" :)
This question already has answers here:
How can I store a command in a variable in a shell script?
(12 answers)
Closed 4 years ago.
These work as advertised:
grep -ir 'hello world' .
grep -ir hello\ world .
These don't:
argumentString1="-ir 'hello world'"
argumentString2="-ir hello\\ world"
grep $argumentString1 .
grep $argumentString2 .
Despite 'hello world' being enclosed by quotes in the second example, grep interprets 'hello (and hello\) as one argument and world' (and world) as another, which means that, in this case, 'hello will be the search pattern and world' will be the search path.
Again, this only happens when the arguments are expanded from the argumentString variables. grep properly interprets 'hello world' (and hello\ world) as a single argument in the first example.
Can anyone explain why this is? Is there a proper way to expand a string variable that will preserve the syntax of each character such that it is correctly interpreted by shell commands?
Why
When the string is expanded, it is split into words, but it is not re-evaluated to find special characters such as quotes or dollar signs or ... This is the way the shell has 'always' behaved, since the Bourne shell back in 1978 or thereabouts.
Fix
In bash, use an array to hold the arguments:
argumentArray=(-ir 'hello world')
grep "${argumentArray[#]}" .
Or, if brave/foolhardy, use eval:
argumentString="-ir 'hello world'"
eval "grep $argumentString ."
On the other hand, discretion is often the better part of valour, and working with eval is a place where discretion is better than bravery. If you are not completely in control of the string that is eval'd (if there's any user input in the command string that has not been rigorously validated), then you are opening yourself to potentially serious problems.
Note that the sequence of expansions for Bash is described in Shell Expansions in the GNU Bash manual. Note in particular sections 3.5.3 Shell Parameter Expansion, 3.5.7 Word Splitting, and 3.5.9 Quote Removal.
When you put quote characters into variables, they just become plain literals (see http://mywiki.wooledge.org/BashFAQ/050; thanks #tripleee for pointing out this link)
Instead, try using an array to pass your arguments:
argumentString=(-ir 'hello world')
grep "${argumentString[#]}" .
In looking at this and related questions, I'm surprised that no one brought up using an explicit subshell. For bash, and other modern shells, you can execute a command line explicitly. In bash, it requires the -c option.
argumentString="-ir 'hello world'"
bash -c "grep $argumentString ."
Works exactly as original questioner desired. There are two restrictions to this technique:
You can only use single quotes within the command or argument strings.
Only exported environment variables will be available to the command
Also, this technique handles redirection and piping, and other shellisms work as well. You also can use bash internal commands as well as any other command that works at the command line, because you are essentially asking a subshell bash to interpret it directly as a command line. Here's a more complex example, a somewhat gratuitously complex ls -l variant.
cmd="prefix=`pwd` && ls | xargs -n 1 echo \'In $prefix:\'"
bash -c "$cmd"
I have built command processors both this way and with parameter arrays. Generally, this way is much easier to write and debug, and it's trivial to echo the command you are executing. OTOH, param arrays work nicely when you really do have abstract arrays of parameters, as opposed to just wanting a simple command variant.
please explain the line below used in shell scripts,
awk -F\| -v src=$storekey 'src==$41' $SRC_Path >> $DST_Path
Thanks!
Ok first ${variable} is a shell variable, so those would be defined higher in your script i.e.
storekey = "1234" or something
you can try this on your shell (linux or command line terminal)
type:
$ storekey="foo"
$ echo $storekey
So most of your question is pertaining to the variables and the command line which confuses how they are used, if you replaced the variables on a command line to test, you could work test it out to find out what they are doing.
In essence Awk is a stream parsing tool, so if you had a file of say 10 columns with a known delimiter such as "," or "|" you could ask awk for a specific column to be printed or output. This is what is happening below, but it is being confused by the presence of custom shell variables.
then to break down the command line awk is parsing a "|" delimited input (-F\| ) defined by $storekey variable, taking the column where src== $41 (this has some reference to the data being input), from $SRC_PATH (a directory) to $DST_PATH (another directory or path).
If you could share more of the shell script I could provide a more in depth answer.
btw, you could also find out more information, using the commands
man awk
info awk
from your command line, however these are a bit arcane for those not so familiar with *nix variants.
I want to get some data from a HTTP server. What it sends me depends on what I put in a POST request.
What I put in the INPUT_TEXT field is a sequence of words. When I run the following command, I get good looking output.
$ curl http://localhost:59125/process -d INPUT_TEXT="here are some words"
I want a bash script to take some string as a command line argument, and pass it appropriately to curl. The first thing I tried was to put the following in a script:
sentence=$1
command="curl http://localhost:59125/process -d INPUT_TEXT=\"${sentence}\""
$command
I then run the script like so:
$ ./script "here are some words"
But then I get a curl Couldn't resolve host error for each of "are", "some", and "words". It would seem that "here" got correctly treated as the INPUT_TEXT, but the rest of the words were then considered to be hosts, and not part of the option.
So I tried:
command=("curl" "http://localhost:59125/process" "-d" "INPUT_TEXT='$sentence'")
${command[#]}
I got the same output as the first script. I finally got what I wanted with:
result=$(curl http://localhost:59125/process -d INPUT_TEXT="${sentence}")
echo $result
I'm still unsure as to what the distinction is. In the first two cases, when I echoed out the contents of command, I get exactly what I input from the interactive Bash prompt, which had worked fine. What caused the difference?
The following will work:
command=("curl" "http://localhost:59125/process"
"-d" "INPUT_TEXT=$sentence")
"${command[#]}"
That has two changes from yours:
I removed the incorrect quotes around $sentence since you don't want to send quotes to the server (as far as I can see).
I put double-quotes around the use of "${command[#]}". Without the double quotes, the array's elements are concatenated with spaces between them and then the result is word-split. With double quotes, the individual array elements are used as individual words.
The second point is well-explained in the bash FAQ and a bunch of SO answers dealing with quotes.
The important thing to understand is that quotes only quote when a command is parsed. A quote which is a character in a variable is just a character; it is not reinterpreted when the value of the variable expanded. Whitespace in the variable is used for word-splitting if the variable expansion is unquoted; the fact that the whitespace was quoted in the the command which defined the variable is completely irrelevant. In this sense, bash is just the same as any other programming language.
I saw a code in The Bash command :(){ :|:& };: will spawn processes to kernel death. Can you explain the syntax? as follows
user#host$ :(){ :|:& };:
Here colon used as identifier for function name.
Can colon be used as identifier?
Yes, it can.
$ :()
> {
> echo "hello from : :)"
> }
$ :
hello from : :)
According to the documentation:
name
A word consisting solely of letters, numbers, and underscores, and beginning with a letter or underscore. Names are used as shell variable and function names. Also referred to as an identifier.
No, the colon is not valid in function names. So either the bomb doesn't work in bash, or the documentation is failing.
I shortly thought that the colon might refer to the built-in operator, but I don't see how that could get the expected semantics.
The implementation seems to be inconsistent. You can define and call a function containing colons and Google even suggests this for packages in their style guide.
Though I noticed you can not export nor unset these functions.
#!/usr/bin/env bash
foo:bar() {
echo "foo:bar called"
}
foo:bar
export -f foo:bar
unset foo:bar
The export won't complain but if you call another bash script afterwards foo:bar is not available.
The unset will even trigger an error:
/foo/bar: line 11: unset: `foo:bar': not a valid identifier
$ bash --version
GNU bash, version 4.2.46(1)-release (x86_64-redhat-linux-gnu)