How to find and remove multiple sub strings in a string - linux

I have an input as follows
Input File Name : a.txt
-- Some Comment
delete from tb.Test WHERE id = 'abxd1';
delete from tb1.Test WHERE id = 'abxd2';
-- Some Comment
delete from tb1.Table1 WHERE id = 'abxd3';
Expected output file : b.txt
-- Some Comment
delete from Test WHERE id = 'abxd1';
delete from Test WHERE id = 'abxd2';
-- Some Comment
delete from Table1 WHERE id = 'abxd2';
The following code will just replace the value "tb.". I am trying to make this as a generic script.
while read line
do
str=$line
echo "${str/tb./}" >>b.txt
done <$1
Thanks for you help


You could use sed:
sed 's/delete from \(tb[0-9]\?\).\([[:alnum:]]\+\)/delete from \2/g' input.file


You can do it using sed:
echo $str | sed -r 's/tb[a-z0-9_]*.//g'
HTH


Assuming that you want to remove X. in delete from X.Y (everything before the DOT and the DOT), it is even simpler:
sed 's/delete from .*\./delete from /' a.txt
and you can use the -i option to overwrite the same file
sed -i 's/delete from .*\./delete from /' a.txt
HTH, Marcello

Related

remove character on the last line that specific word appears

we have the following file example
we want to remove the , character on the last line that topic word exists
more file
{"topic":"life_is_hard","partition":84,"replicas":[1006,1003]},
{"topic":"life_is_hard","partition":85,"replicas":[1001,1004]},
{"topic":"life_is_hard","partition":86,"replicas":[1002,1005]},
{"topic":"life_is_hard","partition":87,"replicas":[1003,1006]},
{"topic":"life_is_hard","partition":88,"replicas":[1004,1001]},
{"topic":"life_is_hard","partition":89,"replicas":[1005,1002]},
{"topic":"life_is_hard","partition":90,"replicas":[1006,1004]},
{"topic":"life_is_hard","partition":91,"replicas":[1001,1005]},
{"topic":"life_is_hard","partition":92,"replicas":[1002,1006]},
{"topic":"life_is_hard","partition":93,"replicas":[1003,1001]},
{"topic":"life_is_hard","partition":94,"replicas":[1004,1002]},
{"topic":"life_is_hard","partition":95,"replicas":[1005,1003]},
{"topic":"life_is_hard","partition":96,"replicas":[1006,1005]},
{"topic":"life_is_hard","partition":97,"replicas":[1001,1006]},
{"topic":"life_is_hard","partition":98,"replicas":[1002,1001]},
{"topic":"life_is_hard","partition":99,"replicas":[1003,1002]},
expected output
{"topic":"life_is_hard","partition":84,"replicas":[1006,1003]},
{"topic":"life_is_hard","partition":85,"replicas":[1001,1004]},
{"topic":"life_is_hard","partition":86,"replicas":[1002,1005]},
{"topic":"life_is_hard","partition":87,"replicas":[1003,1006]},
{"topic":"life_is_hard","partition":88,"replicas":[1004,1001]},
{"topic":"life_is_hard","partition":89,"replicas":[1005,1002]},
{"topic":"life_is_hard","partition":90,"replicas":[1006,1004]},
{"topic":"life_is_hard","partition":91,"replicas":[1001,1005]},
{"topic":"life_is_hard","partition":92,"replicas":[1002,1006]},
{"topic":"life_is_hard","partition":93,"replicas":[1003,1001]},
{"topic":"life_is_hard","partition":94,"replicas":[1004,1002]},
{"topic":"life_is_hard","partition":95,"replicas":[1005,1003]},
{"topic":"life_is_hard","partition":96,"replicas":[1006,1005]},
{"topic":"life_is_hard","partition":97,"replicas":[1001,1006]},
{"topic":"life_is_hard","partition":98,"replicas":[1002,1001]},
{"topic":"life_is_hard","partition":99,"replicas":[1003,1002]}
we try to removed the character , from the the last line that contain topic word as the following sed cli but this syntax not renewed the ,
sed -i '${s/,[[:blank:]]*$//}' file
sed (GNU sed) 4.2.2

In case you have control M characters in your Input_file then remove them by doing:
tr -d '\r' < Input_file > temp && mv temp Input_file
Could you please try following once. From your question what I understood is you want to remove comma from very last line which has string topic in it, if this is the case then I am coming up with tac + awk solution here.
tac Input_file |
awk '/topic/ && ++count==1{sub(/,$/,"")} 1' |
tac
Once you are happy with above results then append > temp && mv temp Input_file to above command too, to save output into Input_file itself.
Explanation:
Atac will read Input_file from bottom line to first line then passing it's output to awk where I am checking if first occurrence of topic is coming remove comma from last and rest of lines simply print then passing this output to tac again to make Input_file in original form again.

You should use the address $ (last line):
sed '$s/,$//' file

Using awk:
$ awk '{if(NR>1)print p;p=$0}END{sub(/,$/,"",p);print p}' file
Output:
...
{"topic":"life_is_hard","partition":98,"replicas":[1002,1001]},
{"topic":"life_is_hard","partition":99,"replicas":[1003,1002]}

bash string manipulation - Display the value, not the variable name

I'm writing a script to process inbound data files. The inbound file names all follow the same pattern:
word1_word2_word3_YYYYMMDD.txt
My script takes the name of the inbound file, strips the file extension, strips out the date, replaces all underscores with spaces and appends the resulting string to each line in the original file. I can succesfully create the desired string and have assigned it to a variable "STR"
The last step is to append the value of $STR to each line in the file so that the data lines within the file end up looking like this:
casenumber1"|"word1 word2 word3
casenumber2"|"word1 word2 word3
casenumber3"|"word1 word2 word3
My problem is that for the life of me I cannot get bash to display the variable value, it always displays the variable name.
This is the line I use to create the string needed from the file name:
STR=`echo $DATAFILENAME | cut -d '.' -f 1 | sed 's/[0-9]*//g'|sed 's/_/ /g' | sed 's/[[:blank:]]*$//'`
I'm trying to use a typical sed replace command:
sed 's/$/`echo "$STR"`/g' inputfile > outputfile
But keep getting the variable name instead of the variable value:
example output:
1000056|$"STR"
1000057|$"STR"
...
desired output:
1000056|Closed With Notification
1000057|Closed With Notification
What am I doing wrong? Thanks, Vic

The gist of your question is that you need to add a string to a file using sed and the value of that string is contained in a variable, which you call "a", as we read in the final list.
Then you need use this combination, which is missing from your list above:
sed "s/$/| $a/g" $DATAFILE > datfile99
The problem is that the single quotes around your command prevent the interpolation of the variable $a.
If you wrap the command in double quotes the whole string will be passed to sed after that the shell replaces $a to its current value.

Try replacing your ' with " this will tell your shell to substitute any shell variables
sed -i "s/$/echo $STR/g"
Note -i option will make actual changes to your file, hence it is wise to backup.
EDIT: instead of using this
STR=`echo $DATAFILENAME | cut -d '.' -f 1 | sed 's/[0-9]*//g'|sed 's/_/ /g' | sed 's/[[:blank:]]*$//'`
You can try this
sed -i -r "s/(.*)[.][a-zA-Z]+$/\\1/g;s/[._]/ /g" <<< "$DATAFILENAME"

Extracting part of a string to a variable in bash

noob here, sorry if a repost. I am extracting a string from a file, and end up with a line, something like:
abcdefg:12345:67890:abcde:12345:abcde
Let's say it's in a variable named testString
the length of the values between the colons is not constant, but I want to save the number, as a string is fine, to a variable, between the 2nd and 3rd colons. so in this case I'd end up with my new variable, let's call it extractedNum, being 67890 . I assume I have to use sed but have never used it and trying to get my head around it...
Can anyone help? Cheers
On a side-note, I am using find to extract the entire line from a string, by searching for the 1st string of characters, in this case the abcdefg part.

Pure Bash using an array:
testString="abcdefg:12345:67890:abcde:12345:abcde"
IFS=':'
array=( $testString )
echo "value = ${array[2]}"
The output:
value = 67890

Here's another pure bash way. Works fine when your input is reasonably consistent and you don't need much flexibility in which section you pick out.
extractedNum="${testString#*:}" # Remove through first :
extractedNum="${extractedNum#*:}" # Remove through second :
extractedNum="${extractedNum%%:*}" # Remove from next : to end of string
You could also filter the file while reading it, in a while loop for example:
while IFS=' ' read -r col line ; do
# col has the column you wanted, line has the whole line
# # #
done < <(sed -e 's/\([^:]*:\)\{2\}\([^:]*\).*/\2 &/' "yourfile")
The sed command is picking out the 2nd column and delimiting that value from the entire line with a space. If you don't need the entire line, just remove the space+& from the replacement and drop the line variable from the read. You can pick any column by changing the number in the \{2\} bit. (Put the command in double quotes if you want to use a variable there.)

You can use cut for this kind of stuff. Here you go:
VAR=$(echo abcdefg:12345:67890:abcde:12345:abcde |cut -d":" -f3); echo $VAR
For the fun of it, this is how I would (not) do this with sed, but I'm sure there's easier ways. I guess that'd be a question of my own to future readers ;)
echo abcdefg:12345:67890:abcde:12345:abcde |sed -e "s/[^:]*:[^:]*:\([^:]*\):.*/\1/"

this should work for you: the key part is awk -F: '$0=$3'
NewVar=$(getTheLineSomehow...|awk -F: '$0=$3')
example:
kent$ newVar=$(echo "abcdefg:12345:67890:abcde:12345:abcde"|awk -F: '$0=$3')
kent$ echo $newVar
67890
if your text was stored in var testString, you could:
kent$ echo $testString
abcdefg:12345:67890:abcde:12345:abcde
kent$ newVar=$(awk -F: '$0=$3' <<<"$testString")
kent$ echo $newVar
67890

how to edit a line using sed or awk in linux containing a certain number or string

My Stress.k file is as follows
180.4430
*INCLUDE
$# filename
*STRESS_INITIALIZATION
*END
I want it to be like
180.4430
*INCLUDE
$# filename
*STRESS_INITIALIZATION
*/home/hassan/534.k
*END
for that I used sed as follows
a="$(cat flow.k)"
sed -i -e '/*END/i \*/home/hassan/$a.k ' Stress.k
where flow.k has only a single number like 534.k or something . Here sed put the line before END but it doesn't take the value of a , instead it puts the same alphabet and it doesn't understand $a.k.
Please also tell me how to delete the second last line or the line with a string hassan for example so that I can delete it first and the for the next step I use it to enter my required line.
if possible please also suggest the alternatives.
best regards

bash variables are only replaced when in double quotes, e.g.
sed -i -e "/*END/i \*/home/hassan/$a.k " Stress.k

Use double quotes to allow the variable to be expanded.
sed -i -e "/*END/i \*/home/hassan/$a.k " Stress.k
To replace the string, do it as you read in the file:
a=$(sed 's/534/100/' flow.k)
To delete a line:
sed '/hassan/d' inputfile
To read a file into the stream after the current line:
sed '/foo/r filename' inputfile

how to replace specific record on a line containg a string with a number from another file using inplace editing sed in linux

I have an input file like following.
R sfst 1000.0000
$ new time step for mass scaled calculation
R dt2ms -4.000E-7
$ friction value for blank
R mue 0.120000
$ blankholder force
R bhf 2.0000E+5
$ simulation time
R endtime 0.150000
i want to change the value on the line containing 'mue'
with following I can read it but cant change it.
awk ' /mue/ { print $3 } ' input.txt
The value is to be taken from another file fric.txt.
fric.txt contains only numbers, one on each line .
fric.txt has data like
0.1234
0.234
0.0234
.
.
Blockquote
It should be noted that ONLY the FIRST instance need to be replaced and the format i.e. white spacing be kept cosntant.
Blockquote
Can anybody guide me doing this using sed or awk?

Try this command:
$ awk '/mue/ && !seen {getline $3 <"fric.txt"; seen=1} 1' input.txt

This might work for you:
sed '/\<mue\>/!d;=;s/.* \([^ ]\+\).*/\1/;R fric.txt' input.txt |
sed 'N;N;s|\n|s/|;s|\n|/|;s|$|/|;q' >temp.sed
sed -i -f temp.sed input.txt

You can do it with a sed in the sed (assuming you like to take line 1 from fric.txt):
sed -ir 's/(.*mue[ \t]+)[0-9.]+(.*)/\1'$(sed -n '1{p;q}' fric.txt)'\2/' input.txt

Resources