I'm writing a script to process inbound data files. The inbound file names all follow the same pattern:
word1_word2_word3_YYYYMMDD.txt
My script takes the name of the inbound file, strips the file extension, strips out the date, replaces all underscores with spaces and appends the resulting string to each line in the original file. I can succesfully create the desired string and have assigned it to a variable "STR"
The last step is to append the value of $STR to each line in the file so that the data lines within the file end up looking like this:
casenumber1"|"word1 word2 word3
casenumber2"|"word1 word2 word3
casenumber3"|"word1 word2 word3
My problem is that for the life of me I cannot get bash to display the variable value, it always displays the variable name.
This is the line I use to create the string needed from the file name:
STR=`echo $DATAFILENAME | cut -d '.' -f 1 | sed 's/[0-9]*//g'|sed 's/_/ /g' | sed 's/[[:blank:]]*$//'`
I'm trying to use a typical sed replace command:
sed 's/$/`echo "$STR"`/g' inputfile > outputfile
But keep getting the variable name instead of the variable value:
example output:
1000056|$"STR"
1000057|$"STR"
...
desired output:
1000056|Closed With Notification
1000057|Closed With Notification
What am I doing wrong? Thanks, Vic
The gist of your question is that you need to add a string to a file using sed and the value of that string is contained in a variable, which you call "a", as we read in the final list.
Then you need use this combination, which is missing from your list above:
sed "s/$/| $a/g" $DATAFILE > datfile99
The problem is that the single quotes around your command prevent the interpolation of the variable $a.
If you wrap the command in double quotes the whole string will be passed to sed after that the shell replaces $a to its current value.
Try replacing your ' with " this will tell your shell to substitute any shell variables
sed -i "s/$/echo $STR/g"
Note -i option will make actual changes to your file, hence it is wise to backup.
EDIT: instead of using this
STR=`echo $DATAFILENAME | cut -d '.' -f 1 | sed 's/[0-9]*//g'|sed 's/_/ /g' | sed 's/[[:blank:]]*$//'`
You can try this
sed -i -r "s/(.*)[.][a-zA-Z]+$/\\1/g;s/[._]/ /g" <<< "$DATAFILENAME"
Related
I have a directory /user/reports under which many files are there, one of them is :
report.active_user.30092018.77325.csv
I need output as number after date i.e. 77325 from above file name.
I created below command to find a value from file name:
ls /user/reports | awk -F. '/report.active_user.30092018/ {print $(NF-1)}'
Now, I want current date to be passed in above command as variable and get result:
ls /user/reports | awk -F. '/report.active_user.$(date +'%d%m%Y')/ {print $(NF-1)}'
But not getting required output.
Tried bash script:
#!/usr/bin/env bash
_date=`date +%d%m%Y`
active=$(ls /user/reports | awk -F. '/report.active_user.${_date}/ {print $(NF-1)}')
echo $active
But still output is blank.
Please help with proper syntax.
As #cyrus said you must use double quotes in your variable assignment because simple quote are use only for string and not for containing variables.
Bas use case
number=10
string='I m sentence with or wihtout var $number'
echo $string
Correct use case
number=10
string_with_number="I m sentence with var $number"
echo $string_with_number
You can use simple quote but not englobe all the string
number=10
string_with_number='I m sentence with var '$number
echo $string_with_number
Don't parse ls
You don't need awk for this: you can manage with the shell's capabilities
for file in report.active_user."$(date "+%d%m%Y")"*; do
tmp=${file%.*} # remove the extension
number=${tmp##*.} # remove the prefix up to and including the last dot
echo "$number"
done
See https://www.gnu.org/software/bash/manual/bashref.html#Shell-Parameter-Expansion
execute PKG_SP_MAINTENANCE.MoveAccount(91, 129031, 958408630); Lowes
From the above statement I am trying to get the content between the first comma and second comma i.e., 129031 and replace it with a new string which is passed as a parameter to the script. For now let's replace with N . I tried the following sed command ended up getting an error. Could someone please help?
04:24:01 Tue Sep 19 [serviceb#LQASRDSBMST002V:~/isops/tmp] cat Navya | sed 's/,^.\{*\},/N/'
sed: -e expression #1, char 14: Invalid content of \{\}
$: echo "start,middle,end" | sed 's/,[^,]*,/,NEW,/g'
start,NEW,end
Is this what you mean? This simply matches the inner-most commas and replaces the text.
Depending how you want to handle strings with more than two commas, you could do something like this to match the outer-most instead:
$: echo "start,middle,end" | sed 's/,.*,/,NEW,/g'
start,NEW,end
How to insert string to the beginning of the last line?
I want to add a time stamp to a text file which contains multiple lines
var1 = `date`
LINE1
LINE2
LINE3
...
(INSERT var1 here) LASTLINE
sed 's/^/test line /g' textfile inserts characters to the beginning of every line but how can I specifically modify the last line only?
Thanks
Going forward:
sed '$s/^/sample text /' textfile works, but only when inserting regular strings. If I try
var1 = "sample text"
and use substition, here are the problems I encounter
using single quotes in sed does not expand variables, so sed '$s/^/$var1/' textfile will insert the string $var1 into the beginning of the last line.
To enable variable substitution I tried using double quotes. It works when I specify the exact line number. something like:
sed "5s/^/$var1/" textfile
But when I try sed "$s/^/$var1" text file, it returns an error:
sed: -e expression #1, char 5: extra characters after command
Can someone help me please?
Like this:
sed '$s/^/test line /' textfile
$ indicates last line. Similarly, you can insert into a any specific line by putting the line number in place of $
But when I try sed "$s/^/$var1" text file, it returns an error:
It returns an error because the shell attempts to expand $s since you've used double quotes. You need to escape the $ in $s.
sed "\$s/^/$var1/" filename
sedshould be the best tool, but awk can do this too:
awk '{a[++t]=$0} END {for (i=1;i<t;i++) print a[i];print v$0}' v="$var1" file
It will insert value of var1 in front of last line
Another variation
awk 'NR==f {$0=v$0}1' v="$var1" f=$(wc -l file)
PS you do not need to specify file after awk, not sure why. If you do so, it reads it double.
This command would work for you:
sed -i "5s/^/$var1 /" text file
I'm trying to retrieve a memory value from file, and compare it to reference value. But one thing at a time....
I've attempted using set/source/grep/substring to variable but non of them actually worked. Then I found a way to do it using a for loop (see code).
The issue: I'm receiving the entire string from the file, but I can't manage to get rid of the last character in it.
#!/bin/bash
#source start_params.properties
#mem_val= "$default.default.minmaxmemory.main"
#mem_val= grep "default.default.minmaxmemory.main" start_params.properties
for mLine in $(grep 'default.default.minmaxmemory.main' start_params.properties)
do
echo "$mLine"
done
echo "${mLine:4:5}" # didn't get rid of the last `m` in `-max4095m`
v1="max"
v2="m"
echo "$mLine" | sed -e "s/.*${v1}//;s/${v2}.*//" #this echo the right value.
The loop iterates twice:
First output: default.default.minmaxmemory.main=-min512m
Second output: -max4096m
Then the sed command output is 4096,but how can I change the last line in the code S.T. it'll store the value in a variable?
Thank you for your suggestions,
You could use grep to filter the max part and then another a grep -o to extract the numbers:
echo "$mLine" | grep "$max" | grep -o '[[:digit:]]*'
$ sed '/max[0-9]/!d; s/.*max//; s/m//' start_params.properties
4096
remove lines not matching max[0-9]
remove first part of line until max
remove final m
My Stress.k file is as follows
180.4430
*INCLUDE
$# filename
*STRESS_INITIALIZATION
*END
I want it to be like
180.4430
*INCLUDE
$# filename
*STRESS_INITIALIZATION
*/home/hassan/534.k
*END
for that I used sed as follows
a="$(cat flow.k)"
sed -i -e '/*END/i \*/home/hassan/$a.k ' Stress.k
where flow.k has only a single number like 534.k or something . Here sed put the line before END but it doesn't take the value of a , instead it puts the same alphabet and it doesn't understand $a.k.
Please also tell me how to delete the second last line or the line with a string hassan for example so that I can delete it first and the for the next step I use it to enter my required line.
if possible please also suggest the alternatives.
best regards
bash variables are only replaced when in double quotes, e.g.
sed -i -e "/*END/i \*/home/hassan/$a.k " Stress.k
Use double quotes to allow the variable to be expanded.
sed -i -e "/*END/i \*/home/hassan/$a.k " Stress.k
To replace the string, do it as you read in the file:
a=$(sed 's/534/100/' flow.k)
To delete a line:
sed '/hassan/d' inputfile
To read a file into the stream after the current line:
sed '/foo/r filename' inputfile