Lets say i have a nested for loop:
for i in $test
do
name=something
for j in $test2
do
name2=something
jj=$j | sed s/'tRap\/tRapTrain'/'BEEML\/BEEMLTrain'/g
if [ name == name2 ]
then
qsub scrip.sh $i $j $jj
fi
done
done
Now the problem occurs when i try to rename the variable $j into variable $jj. I only get empty values back for submitting the script within the if statement. Is there another way to rename variables so that i can pass them through to the if part of the code?
PS. i tried 3 for loops but this makes the script awfully slow.
Your problem is piping the assignment into sed. Try something like
jj=$(echo $j | sed s/'tRap\/tRapTrain'/'BEEML\/BEEMLTrain'/g)
This uses command substitution to assign jj.
This is not correct:
jj=$j | sed s/'tRap\/tRapTrain'/'BEEML\/BEEMLTrain'/g
In order to assign the output of a command to a variable you need to use command substitution like this:
jj=$(sed s/'tRap\/tRapTrain'/'BEEML\/BEEMLTrain'/g <<< "$j")
You may not even have to use sed because bash has in-built string replacement. For example, the following will replace foo with bar in the j variable and assign it to jj:
jj=${j//foo/bar}
There is also a problem with your if-statement. It should be:
if [ "$name" == "$name2" ]
A tiny little thing:
Sed treats the first character after the action selector as the field separator.
Knowing this you can translate your expresion:
sed s/'tRap\/tRapTrain'/'BEEML\/BEEMLTrain'/g
into:
sed s%'tRap/tRapTrain'%'BEEML/BEEMLTrain'%g
So you don't have to worry about scaping your slashes when substituting paths. I normally use '%', but feel free to use any other character. I think the optimal approach would be using a non printable character:
SEP=$'\001' ; sed s${SEP}'tRap/tRapTrain'${SEP}'BEEML/BEEMLTrain'${SEP}g
Related
I am extracting a part from an existing file and storing it as a string in a variable.The string looks something like this.
var="*a<br>*b<br>*c"
Now as * is a special character in unix it doesnot work in further operations(like sed,grep) until I put an escape character infront of every *
Thats why,I am doing something like this -
echo $var | sed 's/\*/\\*/g'
On running this command in bash we get
echo $var | sed 's/\*/\\*/g'
\*a<br>\*b<br>\*c
which is the desired output,but when I try to store this in a variable, I am getting back my original variable like so
var=`echo $var | sed 's/\*/\\*/g'`
echo $var
*a<br>*b<br>*c
I am assuming this happens because the variable ignores the backslashes interpreting them as escape characters. How can I retain the backslashes and store them as in a variable?
The problem is caused by backticks. Use $( ) instead, and it goes away:
var="*a<br>*b<br>*c"
var=$(printf '%s\n' "$var" | sed 's/\*/\\*/g')
printf '%s\n' "$var"
(Why is this problem caused by backticks? Because the only way to nest them is to escape the inner ones with backslashes, so they necessarily change how backslashes behave; whereas $( ), because it uses different starting and ending sigils, can be nested natively).
That said, if your shell is one (like bash) with ksh-inspired extensions, you don't need sed at all here, as the shell can perform simple string replacements natively via parameter expansion:
var="*a<br>*b<br>*c"
printf '%s\n' "${var//'*'/'\*'}"
For background on why this answer uses printf instead of echo, see Why is printf better than echo? at [unix.se], or the APPLICATION USAGE section of the POSIX specification for echo.
I am working on a script. I have a file called test.txt whose contents are as follows:
a. parent = 192.168.1.2
b. child1 = 192.168.1.21
c. child2 = 192.154.1.2
I need to store the values in three different variables called parent, child1and child2 as follows and then my script will use these values:
parent = 192.168.1.2
child1= 192.168.1.21
child2= 192.154.1.2
How can I do that using sed or awk? I know there is a way to extract substrings using awk function substr but my particular requirement is tostore them in variables as mentioned above. Thanks
Try this if you're using bash:
$ declare $(awk '{print $2"="$4}' file)
$ echo "$parent"
192.168.1.2
If the file contained white space in the values you want to init the variables with then you'd just have to set IFS to a newline before invoking declare, e.g. (simplified the input file to highlight the important part of white space on the right of the = signs):
$ cat file
parent=192.168.1.2 is first
child1=192.168.1.21 comes after it
child2=and then theres 192.154.1.2
$ IFS=$'\n'; declare $(awk -F'=' '{print $1"="$2}' file)
$ echo "$parent"
192.168.1.2 is first
$ echo "$child1"
192.168.1.21 comes after it
Ed Morton's answer is the way to go for the specific problem at hand - elegant and concise.
Update: Ed has since updated his answer to also provide a solution that correctly deals with variable value values with embedded spaces - the original lack of which prompted this answer.
His solution is superior to this one - more concise and more efficient (the only caveat is that you may have to restore the previous $IFS value afterward).
This solution may still be of interest if you need to process variable definitions one by one, e.g., in order to transform variable values based on other shell functions or variables before assigning them.
The following uses bash with process substitution on a simplified problem to process variable definitions one by one:
#!/usr/bin/env bash
while read -r name val; do # read a name-value pair
# Assign the value after applying a transformation to it; e.g.:
# 'value of' -> 'value:'
declare $name="${val/ of /: }" # `declare "$name=${val/ of /: }"` would work too.
done < <(awk -F= '{print $1, $2}' <<<$'v1=value of v1\nv2= value of v2')
echo "v1=[$v1], v2=[$v2]" # -> 'v1=[value: v1], v2=[value: v2]'
awk's output lines are read line by line, split into name and value, and declared as shell variables individually.
Since read, which trims by whitespace, is only given 2 variable names to read into, the 2nd one receives everything from the 2nd token _through the end of the line, thus preserving interior whitespace (and, as written, will trim leading and trailing whitespace in the process).
Note that declare normally does not require a variable reference on the RHS of the assignment (the value) to be double-quoted (e.g. a=$b; though it never hurts). In this particular case, however - seemingly because the LHS (the name) is also a variable reference - the double quotes are needed.
I also got it done finally . Thanks everyone for helping.
counter=0
while read line
do
declare $(echo $line | awk '{print $2"="$4}')
#echo "$parent"
if [ $counter = 0 ]
then
parent=$(echo $parent)
fi
if [ $counter = 1 ]
then
child1=$(echo $child)
else
child2=$(echo $child)
fi
counter=$((counter+1))
done < "/etc/cluster_info.txt"
eval "$( sed 's/..//;s/ *//g' YourFile )"
just a sed equivalent to Ed solution and with an eval instead of declare.
I'd like to convert a list separated with '\n' in another one separated with space.
Ex:
Get a dictionary like ispell english dictionary. http://downloads.sourceforge.net/wordlist/ispell-enwl-3.1.20.zip
My initial idea was using a variable as accumulator:
a=""; cat american.0 | while read line; do a="$a $line"; done; echo $a
... but it results '\n' string!!!
Questions:
Why is it not working?
What is the correct way to do that?
Thanks.
The problem is that when you have a pipeline:
command_1 | command_2
each command is run in a separate subshell, with a separate copy of the parent environment. So any variables that the command creates, or any modifications it makes to existing variables, will not be perceived by the containing shell.
In your case, you don't really need the pipeline, because this:
cat filename | command
is equivalent, in every way that you need, to this:
command < filename
So you can write:
a=""; while read line; do a="$a $line"; done < american.0; echo $a
to avoid creating any subshells.
That said, according to this StackOverflow answer, you can't really rely on a shell variable being able to hold more than about 1–4KB of data, so you probably need to rethink your overall approach. Storing the entire word-list in a shell variable likely won't work, and even if it does, it likely won't work well.
Edited to add: To create a temporary file named /tmp/american.tmp that contains what the variable $a would have, you can write:
while IFS= read -r line; do
printf %s " $line"
done < american.0 > /tmp/american.tmp
If you want to replace '\n' with a space, you can simply use tr as follows:
a=$(tr '\n' ' ' < american.0)
I have a variable in a shell script in which I'd like to format the data. The variable stores new data during every iteration of a loop. Each time the new data is stored, I'd like to insert a new line character. Here is how I'm trying to store the data into the variable.
VARIABLE="$VARIABLE '\n' SomeData"
Unfortunately, the output includes the literal '\n' Any help would be appreciative.
Try $'\n':
VAR=a
VAR="$VAR"$'\n'b
echo "$VAR"
gives me
a
b
A common technique is:
nl='
'
VARIABLE="PreviousData"
VARIABLE="$VARIABLE${nl}SomeData"
echo "$VARIABLE"
PreviousData
SomeData
Also common, to prevent inadvertently having your string start with a newline:
VARIABLE="$VARIABLE${VARIABLE:+$nl}SomeData"
(The expression ${VARIABLE:+$nl} will expand to a newline if and only if VARIABLE is set and non-empty.)
VAR="one"
VAR="$VAR.\n.two"
echo -e $VAR
Output:
one.
.two
Other than $'\n' you can use printf also like this:
VARIABLE="Foo Bar"
VARIABLE=$(printf "${VARIABLE}\nSomeData")
echo "$VARIABLE"
OUTPUT:
Foo Bar
SomeData
I had a problem with all the other solutions: when using a # followed by SPACE (quite common when writing in Markdown) both would get split onto a new line.
So, another way of doing it would involve using single quotes so that the "\n" get rendered.
FOO=$'# Markdown Title #\n'
BAR=$'Be *brave* and **bold**.'
FOOBAR="$FOO$BAR"
echo "$FOOBAR"
Output:
# Markdown Title #
Be *brave* and **bold**.
Single quote All special characters between these quotes lose their
special meaning.https://www.tutorialspoint.com/unix/unix-quoting-mechanisms.htm
So the syntax you use does something different that you want to achieve.
This is what you need:
The $'\X' construct makes the -e option in echo unnecessary.
https://linux.die.net/abs-guide/escapingsection.html
echo -e "something\nsomething"
or
echo "something"$'\n'"something"
It's a lot simpler than you think:
VARIABLE="$VARIABLE
SomeData"
Building upon the first two solutions, I'd do like shown below. Concatenating strings with the '+=' operator, somehow looks clearer to me.
Also rememeber to use printf as opposed to echo, you will save yourself so much trouble
sometext="This is the first line"
sometext+=$'\n\n'
sometext+="This is the second line AFTER the inserted new lines"
printf '%s' "${sometext}"
Outputs:
This is the first line
This is the third line AFTER the inserted new line
Your problem is in the echo command, in ash you have to use the option -e to expand special characters. This should work for you:
VAR="First line"
VAR="$VAR\nSecond line"
echo -e $VAR
This outputs
First line
Second line
I'm trying to use bash string operators on a constant string. For instance, you can do the following on variable $foo:
$ foo=a:b:c; echo ${foo##*:}
c
Now, if the "a:b:c" string is constant, I would like to have a more concise solution like:
echo ${"a:b:c"##*:}
However, this is not valid bash syntax. Is there any way to perform this?
[The reason I need to do this (rather than hardcoding the result of the substitution, ie. "c" here) is because I have a command template where a "%h" placeholder is replaced by something before running the command; the result of the substitution is seen as a constant by bash.]
That's not possible using parameter expansion.
You could use other commands for this like sed/awk/expr.
but I don't see the requirement for this.
You could just do:
tmp=%h
echo ${tmp##*:}
Or if speed is not an issue, and you don't want to clutter the current environment with unneeded variables:
(tmp=%h; echo ${tmp##*:})
Anyway, you'd be better off using the command template to do the string manipulation or using something simple like cut:
# get third filed delimited by :
$ cut -d: -f3<<<'a:b:c'
c
Or more sophisticated like awk or sed:
#get last field separated by ':'
$ awk -F: '{print $NF}'<<<'a:b:c'
c
$ sed 's/.*:\([^:]*\)/\1/'<<<'a:b:c'
c
Depends on what you need.
You could use expr to get a similar result:
$ expr match "a:b:c" '.*:\(.*\)'
c
You may be able to use Bash regex matching:
pattern='.*:([^:]+)$'
[[ "a:b:c" =~ $pattern ]]
echo "${BASH_REMATCH[1]}"
But why can't you do your template substitution into a variable assignment, then use the variable in the parameter expansion?