I have a variable in a linux bash ".sh" script
$data="test_1"
now I want to create a new variable ($name) that contains only the part of $data before the underscore, so
$name="test"
I thought of doing this with sed
name=$(echo "$dataset" | sed 's/_.*//');
but this doesn't seem to work. What am I doing wrong?
No need to call an external process(sed). Instead you can use shell's parameter substitution like this:
$ data="test_1"
$ echo "${data%%_*}"
test
${var%%Pattern} Remove from $var the longest part of Pattern that matches the back end(from the right) of $var.
${var%Pattern} for removing shortest pattern
More info on parameter substitution can be found here.
You can store it in a variable like this:
$ name="${data%%_*}"
$ echo "$name"
test
Related
lets say I have a string variable variable
var="This line is with OldText"
I want to find and replace the text inside this variable
The method that I tried is,
echo $var | sed -i "s/OldText/NewText/g" >> result.log
in this case this gives an error saying "no input files".
The expected output is ,
"This line is with NewText"
what is the correct method to do this using sed, awk or any other method.
Using sed
$ var=$(sed "s/OldText/NewText/" <<< $var)
$ echo $var
This line is with NewText
You don't use -i, as that's for changing a file in place. If you want to replace the value in the variable, you need to reassign to it.
If you are using Bash shell, you could:
$ echo ${var/OldText/NewText}
This line is with NewText
so
$ var=${var/OldText/NewText}
See this for more.
I am working on a script that involves paths with wildcards. I know the wildcards will match one file only, I just don't know what the file extension will be ahead of time so I am using a wildcard.
The goal here is to find the path to the appropriate file, and then add that path to line 16 of a script.
I have something like this:
path=/path/to/somewhere/fileName*
sed "16 a file=$path" myScript.sh
What I expect to get is this (on line 16):
file=/path/to/somewhere/fileName.extension
But what I get is:
file=/path/to/somewhere/fileName*
For some reason sed is not expanding the wildcard when it adds the contents of $path and I can't figure out how to make sed do such a thing. I'm looking for a solution that either a) has sed properly expand $path or b) a way to get $path to contain the fully expanded string before being passed to sed.
Your variable contains just a string and you then interpolate that string. sed can't know that's not what you mean. If you want the shell (not sed!) to expand the wildcard, probably use a loop.
for path in /path/to/somewhere/fileName*; do
if [ -e "$path" ]; then # handle wildcard possibly not matching
sed "16 a file=$path" myScript.sh
fi
done
It's unclear what should happen if the wildcard matches multiple files; perhaps you want to add a break before fi to only substitute the first one if that happens.
This might work for you (GNU sed, echo and bash):
export path='/path/toSomeWhere/filename*'
sed '16{s/$/\na file=$(echo $path)/;s/.*/echo "&"/e}' file
Export the variable path which has been set to /path/toSomeWhere/filename* (notice the single quotes which prevents interpolation).
On line 16 of file append a line a file=$(echo $path) and then surround both lines by double quotes and prepend the echo command and evaluate (the e flag on the second substitution command) the expression.
Alternative:
sed '17e echo "a file=$(echo /path/toSomeWhere/filename*)"' file
I need to extract path from a string. I found examples in another post, but missing additional steps.
I have a string as below:
title="test test good dskgkdh hdfyr /rlsmodules/svnrepo/SOURCE/CBL/MQ/BASELINE/MQO000.CBL kdlkfg nsfgf trhrnrt"
cobsrc=$(awk '{match($0,/\/[^"]*/,a);print a[0]}' <<< $title)
echo $cobsrc
Output is
/rlsmodules/svnrepo/SOURCE/CBL/MQ/BASELINE/MQO000.CBL kdlkfg nsfgf trhrnrt
I need only
/rlsmodules/svnrepo/SOURCE/CBL/MQ/BASELINE/MQO000.CBL
What modification is required?
An existing post on similar query:
how to extract path from string in shell script
Four solutions, in order of my own preference.
First option would be simple parameter expansion, in two steps:
$ title="/${title#*/}"
$ title="${title%% *}"
$ echo "$title"
/rlsmodules/svnrepo/SOURCE/CBL/MQ/BASELINE/MQO000.CBL
The first line removes everything up to the first slash (while prepending a slash to replace the one that's stripped", the second line removes everything from the first bit of whitespace that remains.
Or, if you prefer, use a regex:
$ [[ $title =~ ^[^/]*(/[^ ]+)\ ]]
$ echo ${BASH_REMATCH[1]}
/rlsmodules/svnrepo/SOURCE/CBL/MQ/BASELINE/MQO000.CBL
The regex translates as:
null at the beginning of the line,
a run of zero or more non-slashes,
an atom:
a slash followed by non-space characters
a space, to end the previous atom.
The $BASH_REMATCH array contains the content of the bracketed atom.
Next option might be grep -o:
$ grep -o '/[^ ]*' <<<"$title"
(Result redacted -- you know what it'll be.)
You could of course assign this output to a variable using command substitution, which you already know about.
Last option is another external tool...
$ sed 's:^[^/]*::;s/ .*//' <<<"$title"
This is the same functionality as is handled by the parameter expansion (at the top of the answer) only in a sed script, which requires a call to an external program. Included only for pedantry. :)
Could you please try following.
echo "$title" | awk 'match($0,/\/.*\/[^ ]*/){print substr($0,RSTART,RLENGTH)}'
Output will be as follows.
/rlsmodules/svnrepo/SOURCE/CBL/MQ/BASELINE/MQO000.CBL
Solution 2nd: Considering that your variable don't have space in between its value then following may help you too.
echo "$title" | awk '{sub(/[^/]* /,"");sub(/ .*/,"")} 1'
Is there a way to prevent envsubst from substituting a $VARIABLE? For example, I would expect something like:
export THIS=THAT
echo "dont substitute \\\$THIS" | envsubst
and have it return
dont substitute $THIS
but instead I get
dont substitute \THAT
is there any escape character for doing this?
If you give envsubst a list of variables, it only substitutes those variables, ignoring other substitutions. I'm not exactly sure how it works, but something like the following seems to do what you want:
$ export THIS=THAT FOO=BAR
$ echo 'dont substitute $THIS but do substitute $FOO' | envsubst '$FOO'
dont substitute $THIS but do substitute BAR
Note that $THIS is left alone, but $FOO is replaced by BAR.
export DOLLAR='$'
export THIS=THAT
echo '${DOLLAR}THIS' | envsubst
Or more clear:
export THIS=THAT
echo '${DOLLAR}THIS' | DOLLAR='$' envsubst
My workaround is as follows:
Original template:
$change_this
$dont_change_this
Editted template:
$change_this
§dont_change_this
Now you can process:
envsubst < $template | sed -e 's/§/$/g'
This relies on the character § not occurring anywhere else on your template. You can use any other character.
$ echo $SHELL
/bin/bash
$ echo \$SHELL
$SHELL
$ echo \$SHELL | envsubst
/bin/bash
$ echo \$\${q}SHELL | envsubst
$SHELL
So doing $$ allows you to add a $ character. Then just "substitute" non-existent variable (here I used ${q} but can be something more meaningful like ${my_empty_variable} and you'll end up with what you need.
Just as with the paragraph solution - you need something special - here... a non-existent variable, which I like a bit more than performing additional sed on templates.
If there's only one or two variables you don't want to expand, you can sort of whitelist them by temporarily setting them to their own name, like this:
$ echo 'one $two three $four' | four='$four' envsubst
one three $four
Here, the $four variable gets replaced with $four, effectively leaving it unchanged.
In my case I wanted to only escape vars that aren't already defined. To do so run:
envsubst "$(env | sed -e 's/=.*//' -e 's/^/\$/g')"
Another way to "escape" some environment variable substitution is to use default value assignment (or any other variable processing) as envsubst will not substitute these:
$ export two=2
$ echo 'one $two three ${four:-}' | envsubst
one 2 three ${four:-}
The fourth envvar is not substituted, while in its output the processing to use defaulkt value is still there. This does not matter though, as processing this line later on will still deliver nothing if the variable is not set and its value when set.
Here's an alternative that I use, as it saves installing the entire gettext package for just one program. I have this awk script, I call envtmpl, it will swap any environment variable that looks like {{ENV-VAR}} for the value of ENV-VAR
#! /usr/bin/awk -f
{ for (a in ENVIRON) gsub("{{" _ a _ "}}",ENVIRON[a]); print }
So
$ echo "My shell '{{SHELL}}' is cool" | envtmpl
My shell '/bin/bash' is cool
As you can see, if {{ & }} aren't what you prefer, its really each to change and this script works fine with busybox's awk.
It's not going to be the world's fastest solution, but it's really easy to implement and I mostly run it to prepare config files, so speed is pretty irrelevant.
WARNING: The only major difference between this and envsubst is that this will NOT alter variables where no value exists. That is {{HAS-NO-VALUE}} will be left exactly as that, where as envsubst will remove those (replace them with blank).
You can fix this by adding more code into the awk, if you want.
The way I did it is
export DONT_CHANGE_THIS=\${DONT_CHANGE_THIS}
envsubst < some-template.yml > changed.yml
So it will try to replace ${var} with \${var} and as output, you will get ${var} printed as it is
I used escape character for this
MYENVVAR="\${MYENVVAR}"
export MYENVVAR
envsubst #whatever you want
then reset it to what actually I want
MYENVVAR="my value"
export MYENVVAR
I just connected parts of other answers to create one-liner that substitutes vars prefixed with $, but ignores $$:
echo "\$TEST ; \$\$l" > TEST_FILE
cat TEST_FILE
# $TEST ; $$l
export TEST=1
cat TEST_FILE | sed -e 's/\$\$/§/g' | envsubst | sed -e 's/§/\$/g'
# 1 ; $l
I'm trying to use bash string operators on a constant string. For instance, you can do the following on variable $foo:
$ foo=a:b:c; echo ${foo##*:}
c
Now, if the "a:b:c" string is constant, I would like to have a more concise solution like:
echo ${"a:b:c"##*:}
However, this is not valid bash syntax. Is there any way to perform this?
[The reason I need to do this (rather than hardcoding the result of the substitution, ie. "c" here) is because I have a command template where a "%h" placeholder is replaced by something before running the command; the result of the substitution is seen as a constant by bash.]
That's not possible using parameter expansion.
You could use other commands for this like sed/awk/expr.
but I don't see the requirement for this.
You could just do:
tmp=%h
echo ${tmp##*:}
Or if speed is not an issue, and you don't want to clutter the current environment with unneeded variables:
(tmp=%h; echo ${tmp##*:})
Anyway, you'd be better off using the command template to do the string manipulation or using something simple like cut:
# get third filed delimited by :
$ cut -d: -f3<<<'a:b:c'
c
Or more sophisticated like awk or sed:
#get last field separated by ':'
$ awk -F: '{print $NF}'<<<'a:b:c'
c
$ sed 's/.*:\([^:]*\)/\1/'<<<'a:b:c'
c
Depends on what you need.
You could use expr to get a similar result:
$ expr match "a:b:c" '.*:\(.*\)'
c
You may be able to use Bash regex matching:
pattern='.*:([^:]+)$'
[[ "a:b:c" =~ $pattern ]]
echo "${BASH_REMATCH[1]}"
But why can't you do your template substitution into a variable assignment, then use the variable in the parameter expansion?