I have a directory /user/reports under which many files are there, one of them is :
report.active_user.30092018.77325.csv
I need output as number after date i.e. 77325 from above file name.
I created below command to find a value from file name:
ls /user/reports | awk -F. '/report.active_user.30092018/ {print $(NF-1)}'
Now, I want current date to be passed in above command as variable and get result:
ls /user/reports | awk -F. '/report.active_user.$(date +'%d%m%Y')/ {print $(NF-1)}'
But not getting required output.
Tried bash script:
#!/usr/bin/env bash
_date=`date +%d%m%Y`
active=$(ls /user/reports | awk -F. '/report.active_user.${_date}/ {print $(NF-1)}')
echo $active
But still output is blank.
Please help with proper syntax.
As #cyrus said you must use double quotes in your variable assignment because simple quote are use only for string and not for containing variables.
Bas use case
number=10
string='I m sentence with or wihtout var $number'
echo $string
Correct use case
number=10
string_with_number="I m sentence with var $number"
echo $string_with_number
You can use simple quote but not englobe all the string
number=10
string_with_number='I m sentence with var '$number
echo $string_with_number
Don't parse ls
You don't need awk for this: you can manage with the shell's capabilities
for file in report.active_user."$(date "+%d%m%Y")"*; do
tmp=${file%.*} # remove the extension
number=${tmp##*.} # remove the prefix up to and including the last dot
echo "$number"
done
See https://www.gnu.org/software/bash/manual/bashref.html#Shell-Parameter-Expansion
Related
I'm a beginner to bash scripting and been writing a script to check different log files and I'm bit stuck here.
clientlist=/path/to/logfile/which/consists/of/client/names
#i will grep only the client name from the file which has multiple log lines
clients=$(grep --color -i 'list of client assets:' $clientlist | cut -d":" -f1 )
echo "Clients : $clients"
#For example "Clients: Apple
# Samsung
# Nokia"
#number of clients may vary from time to time
assets=("$clients".log)
echo assets: "$assets"
The code above greps the client name from the log file and i'm trying to use the grepped client name (each) to construct a logfile with each client name.
The number of clients is indefinite and may vary from time to time.
The code I have returns the client name as a whole
assets: Apple
Samsung
Nokia.log
and I'm bit unsure of how to cut the string and pass it on one by one to return the assets which has .log for each client name. How can i do this ?
Apple.log
Samsung.log
Nokia.log
(Apologies if I have misunderstood the task)
Using awk
if your input file (I'll call it clients.txt) is:
Clients: Apple
Samsung
Nokia
The following awk step:
awk '{print $NF".log"}' clients.txt
outputs:
Apple.log
Samsung.log
Nokia.log
(You can pipe straight into awk and omit the file name if the pipe stream is as the file contents in the above example).
It is highly likely that a simple awk procedure can perform the entire task beginning with the 'clientlist' you process with grep (awk has all the functionality of grep built-in) but I'd need to know the structure of the origial file to extract the client names.
One awk idea:
assets=( $(awk -F: '/list of client assets:/ {print $2".log"}' "${clientlist}") )
# or
mapfile -t assets < <(awk -F: '/list of client assets:/ {print $2".log"}' "${clientlist}")
Where:
-F: - define input field delimiter as :
/list of client assets:/ - for lines that contain the string list of clients assets: print the 2nd :-delimited field and append the string .log on the end
One sed idea:
assets=( $(sed 's/.*://; s/$/.log/' "${clientlist}") )
# or
mapfile -t assets < <(sed 's/.*://; s/$/.log/' "${clientlist}")
Where:
s/.*:// - strip off everything up to the :
s/$/.log/ - replace end of line with .log
Both generate:
$ typeset -p assets
declare -a assets=([0]="Apple.log" [1]="Samsung.log" [2]="Nokia.log")
$ echo "${assets[#]}"
Apple.log Samsung.log Nokia.log
$ printf "%s\n" "${assets[#]}"
Apple.log
Samsung.log
Nokia.log
$ for i in "${!assets[#]}"; do echo "assets[$i] = ${assets[$indx]}"; done
assets[0] = Apple.log
assets[1] = Samsung.log
assets[2] = Nokia.log
NOTE: the alternative answers using mapfile address the issue referenced in Charles Duffy comment (see bash pitfall #50); readarray is a synonym for mapfile
remove from the string 89dde7.rqsnhq34h.fmu8s1vn0i94hl.tgz.tar.gz only the .tar.gz part and the result should be 89dde7.rqsnhq34h.fmu8s1vn0i94hl.tgz.
It can also happen some files with this extension:
91xhq8vkxlkdfpmfg566qahrwkh01c7n0scpdsr4p4vf6.tbz.tar.bz2 and others with double extension tar.tbz tar.zip and so on ...
In case .tar.zip the result must be nomearchivio.tar in the case 91xhq8vkxlkdfpmfg566qahrwkh01c7n0scpdsr4p4vf6.tbz.tar.bz2 must be 91xhq8vkxlkdfpmfg566qahrwkh01c7n0scpdsr4p4vf6.tbz
I use this :
nameFile= "89dde7.rqsnhq34h.fmu8s1vn0i94hl.tgz.tar.gz"
name=${nameFile%.*}
and the result is :
echo $name
89dde7.rqsnhq34h.fmu8s1vn0i94hl.tgz.tar
can you help me? Thanks
P.S. note that there are also other points within the file name.
Since you know exactly what you want to remove, just write it in full:
name=${nameFile%.tar.gz}
Or to remove the last two "extensions" .*.*:
name=${nameFile%.*.*}
You could use sed and remove the last 7 characters
echo $nameFile |sed 's/.\{7\}$//'
You could give a try to awk, for example:
echo 89dde7.rqsnhq34h.fmu8s1vn0i94hl.tgz.tar.gz | awk -F '\.tgz' '{print $1".tgz"}'
It will output:
89dde7.rqsnhq34h.fmu8s1vn0i94hl.tgz
For other files:
echo "01c7n0scpdsr4p4vf6.tbz.tar.bz2" | awk -F '\.tbz' '{print $1".tbz"}'
It will output:
01c7n0scpdsr4p4vf6.tbz
In this case, awk is using as a delimiter -F '\.tbz' your pattern, .tgz or tbz and then prints all items found at the left + your desired extension.
Given a list of paths separated by a single space:
/home/me/src/test /home/me/src/vendor/a /home/me/src/vendor/b
I want to remove the prefix /home/me/src/ so that the result is:
test vendor/a vendor/b
For a single path I would do: ${PATH#/home/me/src/} but how do I apply it to this series?
You can use // to replace all occurrences of substring. Replace it with null string to remove them.
$ path="/home/me/src/test /home/me/src/vendor/a /home/me/src/vendor/b"
$ echo ${path//\/home\/me\/src\/}
test vendor/a vendor/b
Reference: ${parameter/pattern/string} in Bash reference manual
Using shell parameter expansion doesn't seem to be the solution for this, since it would remove everything up to / from a given point is useful, as nu11p01n73R's answer reveals.
For clarity, I would use sed with the syntax sed 's#pattern#replacement#g':
$ str="/home/me/src/test /home/me/src/vendor/a /home/me/src/vendor/b"
$ sed 's#/home/me/src/##g' <<< "$str"
test vendor/a vendor/b
Like always a grep solution from my side :
echo 'your string' | grep -Po '^/([^ /]*/)+\K.+'
Please note that the above regex do this for any string like /x/y/z/test ... But if you are interested only in replacing /home/me/src/, try the following :
echo 'your string' | grep -Po '^/home/me/src/\K.+' --color
I'm writing a script to process inbound data files. The inbound file names all follow the same pattern:
word1_word2_word3_YYYYMMDD.txt
My script takes the name of the inbound file, strips the file extension, strips out the date, replaces all underscores with spaces and appends the resulting string to each line in the original file. I can succesfully create the desired string and have assigned it to a variable "STR"
The last step is to append the value of $STR to each line in the file so that the data lines within the file end up looking like this:
casenumber1"|"word1 word2 word3
casenumber2"|"word1 word2 word3
casenumber3"|"word1 word2 word3
My problem is that for the life of me I cannot get bash to display the variable value, it always displays the variable name.
This is the line I use to create the string needed from the file name:
STR=`echo $DATAFILENAME | cut -d '.' -f 1 | sed 's/[0-9]*//g'|sed 's/_/ /g' | sed 's/[[:blank:]]*$//'`
I'm trying to use a typical sed replace command:
sed 's/$/`echo "$STR"`/g' inputfile > outputfile
But keep getting the variable name instead of the variable value:
example output:
1000056|$"STR"
1000057|$"STR"
...
desired output:
1000056|Closed With Notification
1000057|Closed With Notification
What am I doing wrong? Thanks, Vic
The gist of your question is that you need to add a string to a file using sed and the value of that string is contained in a variable, which you call "a", as we read in the final list.
Then you need use this combination, which is missing from your list above:
sed "s/$/| $a/g" $DATAFILE > datfile99
The problem is that the single quotes around your command prevent the interpolation of the variable $a.
If you wrap the command in double quotes the whole string will be passed to sed after that the shell replaces $a to its current value.
Try replacing your ' with " this will tell your shell to substitute any shell variables
sed -i "s/$/echo $STR/g"
Note -i option will make actual changes to your file, hence it is wise to backup.
EDIT: instead of using this
STR=`echo $DATAFILENAME | cut -d '.' -f 1 | sed 's/[0-9]*//g'|sed 's/_/ /g' | sed 's/[[:blank:]]*$//'`
You can try this
sed -i -r "s/(.*)[.][a-zA-Z]+$/\\1/g;s/[._]/ /g" <<< "$DATAFILENAME"
noob here, sorry if a repost. I am extracting a string from a file, and end up with a line, something like:
abcdefg:12345:67890:abcde:12345:abcde
Let's say it's in a variable named testString
the length of the values between the colons is not constant, but I want to save the number, as a string is fine, to a variable, between the 2nd and 3rd colons. so in this case I'd end up with my new variable, let's call it extractedNum, being 67890 . I assume I have to use sed but have never used it and trying to get my head around it...
Can anyone help? Cheers
On a side-note, I am using find to extract the entire line from a string, by searching for the 1st string of characters, in this case the abcdefg part.
Pure Bash using an array:
testString="abcdefg:12345:67890:abcde:12345:abcde"
IFS=':'
array=( $testString )
echo "value = ${array[2]}"
The output:
value = 67890
Here's another pure bash way. Works fine when your input is reasonably consistent and you don't need much flexibility in which section you pick out.
extractedNum="${testString#*:}" # Remove through first :
extractedNum="${extractedNum#*:}" # Remove through second :
extractedNum="${extractedNum%%:*}" # Remove from next : to end of string
You could also filter the file while reading it, in a while loop for example:
while IFS=' ' read -r col line ; do
# col has the column you wanted, line has the whole line
# # #
done < <(sed -e 's/\([^:]*:\)\{2\}\([^:]*\).*/\2 &/' "yourfile")
The sed command is picking out the 2nd column and delimiting that value from the entire line with a space. If you don't need the entire line, just remove the space+& from the replacement and drop the line variable from the read. You can pick any column by changing the number in the \{2\} bit. (Put the command in double quotes if you want to use a variable there.)
You can use cut for this kind of stuff. Here you go:
VAR=$(echo abcdefg:12345:67890:abcde:12345:abcde |cut -d":" -f3); echo $VAR
For the fun of it, this is how I would (not) do this with sed, but I'm sure there's easier ways. I guess that'd be a question of my own to future readers ;)
echo abcdefg:12345:67890:abcde:12345:abcde |sed -e "s/[^:]*:[^:]*:\([^:]*\):.*/\1/"
this should work for you: the key part is awk -F: '$0=$3'
NewVar=$(getTheLineSomehow...|awk -F: '$0=$3')
example:
kent$ newVar=$(echo "abcdefg:12345:67890:abcde:12345:abcde"|awk -F: '$0=$3')
kent$ echo $newVar
67890
if your text was stored in var testString, you could:
kent$ echo $testString
abcdefg:12345:67890:abcde:12345:abcde
kent$ newVar=$(awk -F: '$0=$3' <<<"$testString")
kent$ echo $newVar
67890