I have a script and trying to append the content from my variables to a .log file. I saw on another post(How can I append a variable text to last line of file via command line?) that you can do this using echo "$(cat $FILE)$APPEND" > $FILE. I tried doing that and it’s able to run my script without errors and give the answers I need through terminal but it doesn’t append it to my log file. Can anyone please give me any pointers?
#!/bin/bash
mdate=echo date
mcpu=echo $[100-$(vmstat 1 2 |tail -1|awk ‘{print $15}’)]%
mmem=free | grep Mem | awk ‘{print $3/$2 * 100.0}’
sudo cat /dev/null > /home/daniel/systemstatus.log
echo “$(cat $systemstatus.log)$mdate” >> $systemstatus.log
echo “$(cat $systemstatus.log)$mcpu” >> $systemstatus.log
echo “$(cat $systemstatus.log)$mmem” >> $systemstatus.log
Your code requires several modifications, you cannot assign variables with the output of commands the way you tried to, you need to use command substitutions (var=$(command)) :
mdate=$(date)
mcpu=$(( 100 - $(vmstat 1 2 | tail -1| awk '{print $15}') ))
mmem=$(free | grep Mem | awk '{print $3/$2 * 100.0}')
Your post contains quotes that are not valid for the shell. Use single quotes ' or double quotes " (each has its purpose).
But the logging part is simpler than you thought
logfile=/home/daniel/systemstatus.log
echo "$mdate" >"$logfile" # Overwrites the previous log file
echo "$mcpu" >>"$logfile" # Appends to the log file
echo "$mmem" >>"$logfile" # Appends to the log file
If the only reason for capturing the output of your commands in variables is for logging you can simplify further by redirecting to your log file without capturing the output.
date >"$logfile" # Overwrites the previous log file
echo $(( 100 - $(vmstat 1 2 | tail -1| awk '{print $15}') )) >>"$logfile"
free | grep Mem | awk '{print $3/$2 * 100.0}' >>"$logfile"
You can even put the redirection first, which may be cleaner to the eye depending on your taste :
>"$logfile" date # Overwrites the previous log file
>>"$logfile" echo $(( 100 - $(vmstat 1 2 | tail -1| awk '{print $15}') ))
>>"$logfile" free | grep Mem | awk '{print $3/$2 * 100.0}'
This answer assumes your commands actually work (I have not tested them).
The following should work:
#!/bin/bash
mdate=$(echo date)
mcpu=$(echo $[100-$(vmstat 1 2 |tail -1|awk '{print $15}')]%)
mmem=$(free | grep Mem | awk '{print $3/$2 * 100.0}')
>/home/daniel/systemstatus.log
echo “$(cat $systemstatus.log)$mdate” >> systemstatus.log
echo “$(cat $systemstatus.log)$mcpu” >> systemstatus.log
echo “$(cat $systemstatus.log)$mmem” >> systemstatus.log
Note that if you would like to store the the output of a command in bash you should round the command in $() or `` However, the usage of grave accents is now deprecated.
Furthermore, using cat /dev/null > to empty a file in Linux would basically have the same result as simply using >
As for the issue that you are experiencing, note that appending $ before any word basically makes the script/command to look for a variable with such name instead of the file name.
Regards
Try removing $ before systemstatus
When you type $systemstatus, it means a variable named "systemstatus", not the systemstatus file.
Related
I have the below requirement. I am trying to run the condition in loop and it's taking more time. Is there a one time command anything which will not take more time to process a 70 MB file.
Requirement:
if #pRECTYPE="SBSB" line contains #pSBEL_MCTR_RSN="XXX" tag then we need to copy and append that to next #pRECTYPE="SBEL record at the end of the line
File :note : in file there will be no blank lines. I have given enter to avoid line continuation
#pRUKE=dfgt#pRECTYPE="SMDR", #pCONFIG="Y" XXXXXXX
#pRUKE=dfgt#pRECTYPE="SBSB", #pGWID="1234", #pSBEL_MCTR_RSN="KX28", #pSBSB_9000_COLL=""
#pRUKE=dfgt#pRECTYPE="KBSG", #pKBSG_UPDATE_CD="IN", XXXXXXXXXXX
#pRUKE=dfgt#pRECTYPE="SBEL", #pSBEL_EFF_DT="01/01/2017", #pCSPI_ID="JKOX0001", #pSBEL_FI="A"
#pRUKE=dfgt#pRECTYPE="SBEK", #pSBEK_UPDATE_CD="IN",XXXXXXXXXXXXXXXXXXX
#pRUKE=dfgt#pRECTYPE="DBCS", #pDBCS_UPDATE_CD="IN",XXXXXXXXXXXXXXXXXXXXXXXXXX
#pRUKE=dfgt#pRECTYPE="MEME", #pMEME_REL="18", #pMEEL_MCTR_RSN="KX28"
#pRUKE=dfgt#pRECTYPE="ATT0", #pATT0_UPDATE_CD="AP",XXXXXXXXX
#pRUKE=dfgt#pRECTYPE="SBSB", #pGWID="1234", #pSBEL_MCTR_RSN="KX28", #pSBSB_9000_COLL=""
#pRUKE=dfgt#pRECTYPE="KBSG", #pKBSG_UPDATE_CD="IN", XXXXXXXXXXX
example :
Before :
#pRUKE=dfgt#pRECTYPE="SMDR", #pCONFIG="Y" XXXXXXX
#pRUKE=dfgt#pRECTYPE="SBSB", #pGWID="1234", #pSBEL_MCTR_RSN="KX28", #pSBSB_9000_COLL=""
#pRUKE=dfgt#pRECTYPE="KBSG", #pKBSG_UPDATE_CD="IN", XXXXXXXXXXX
#pRUKE=dfgt#pRECTYPE="SBEL", #pSBEL_EFF_DT="01/01/2017", #pCSPI_ID="JKOX0001", #pSBEL_FI="A"
After:
#pRUKE=dfgt#pRECTYPE="SMDR", #pCONFIG="Y" XXXXXXX
#pRUKE=dfgt#pRECTYPE="SBSB", #pGWID="1234", #pSBEL_MCTR_RSN="KX28", #pSBSB_9000_COLL=""
#pRUKE=dfgt#pRECTYPE="KBSG", #pKBSG_UPDATE_CD="IN", XXXXXXXXXXX
#pRUKE=dfgt#pRECTYPE="SBEL", #pSBEL_EFF_DT="01/01/2017", #pCSPI_ID="JKOX0001", #pSBEL_FI="A", #pSBEL_MCTR_RSN="KX28"
After SBSB, if there is no SBEL, then that SBSB can be ignored.
What I did is:
egrep -n "pRECTYPE=\"SBSB\"|pRECTYPE=\"SBEL\"" filename | sed '$!N;/pRECTYPE=\"SBEL\"/P;D' | awk -F\: '{print $1}' | awk 'NR%2{printf "%s,",$0;next;}1' > 4.txt;
by this I will get the line number, eg:
2,4
17,19
Line 9 12 14 will be ignored
while read line
do
echo "$line";
SBSB=`echo "$line" | awk -F, '{print $1}'`;
SBEL=`echo "$line" | awk -F, '{print $2}'`;
echo $SBSB;
echo $SBEL;
SBSB_Fetch=`sed -n "$SBSB p" $fil | grep -Eo '(#pSBEL_MCTR_RSN)=[^ ]+' | sed 's/,$//' | sed 's/^/, /g'`;
echo $SBSB_Fetch;
if [[ "$SBSB_Fetch" == "" ]];then
echo "blank";
s=blank;
else
echo "value";
sed -i "${SBEL}s/.*/&${SBSB_Fetch}/" $fil;
fi
done < 4.txt;
Since I am ready and updating each line ,it's taking more time, is there any way to reduce the run time?
For 70 Mb it's taking 4 .5 hours now.
For performance, you need to really limit how many external tools you invoke inside a loop in a shell script.
This requires GNU awk:
gawk '
/#pRECTYPE="SBSB"/ {match($0, /#pSBEL_MCTR_RSN="[^"]*"/, m)}
/#pRECTYPE="SBEL"/ && isarray(m) {$0 = $0 ", " m[0]; delete m}
1
' file
This should be pretty quick:
only invoking one external command
no shell loops
only have to read the input file once.
I have the following script:
#!/bin/bash
TotalMem=$(top -n 1 | grep Mem | awk 'NR==1{print $4}') #integer
UsadoMem=$(top -n 1 | grep Mem | awk 'NR==1{print $8}') #integer
PorcUsado='scale=2;UsadoMem/TotalMem'|bc -l
echo $PorcUsado
The variable PorcUsado returns empty. I search for the use of bc, but something is wrong...
You're assigning PorcUsado to scale=2;UsadoMem/TotalMem and then piping the output of that assignment (nothing) into bc. You probably want the pipe inside a command substitution, e.g. (using a here string instead of a pipe):
PorcUsado=$(bc -l <<<'scale=2;UsadoMem/TotalMem')
But you'll also need to evaluate those shell variables - bc can't do it for you:
PorcUsado=$(bc -l <<<"scale=2;$UsadoMem/$TotalMem")
Notice the use of " instead of ' and the $ prefix to allow Bash to evaluate the variables.
Also, if this is the whole script, you can just skip the PorcUsado variable at all and let bc write directly to stdout.
#!/bin/bash
TotalMem=$(top -n 1 | grep Mem | awk 'NR==1{print $4}') #integer
UsadoMem=$(top -n 1 | grep Mem | awk 'NR==1{print $8}') #integer
bc -l <<<"scale=2;$UsadoMem/$TotalMem"
Why pipe top output at all? Seems too costly.
$ read used buffers < <(
awk -F':? +' '
{a[$1]=$2}
END {printf "%d %d", a["MemTotal"]-a["MemFree"], a["Buffers"]}
' /proc/meminfo
)
Of course, it can easily be a one-liner if you value brevity over readability.
I think the pipe is the problem try something like this:
PorcUsado=$(echo "scale=2;$UsadoMem/$TotalMem" | bc -l)
i haven't tested it yet but you have to echo the string and pipe the result from echo to bc.
EDIT: Correcting the variable names
You don't need grep or bc, since awk can search and do math all by itself:
top -n 1 -l 1 | awk '/Mem/ {printf "%0.2f\n",$8/$4;exit}'
I want to print the longest and shortest username found in /etc/passwd. If I run the code below it works fine for the shortest (head -1), but doesn't run for (sort -n |tail -1 | awk '{print $2}). Can anyone help me figure out what's wrong?
#!/bin/bash
grep -Eo '^([^:]+)' /etc/passwd |
while read NAME
do
echo ${#NAME} ${NAME}
done |
sort -n |head -1 | awk '{print $2}'
sort -n |tail -1 | awk '{print $2}'
Here the issue is:
Piping finishes with the first sort -n |head -1 | awk '{print $2}' command. So, input to first command is provided through piping and output is obtained.
For the second command, no input is given. So, it waits for the input from STDIN which is the keyboard and you can feed the input through keyboard and press ctrl+D to obtain output.
Please run the code like below to get desired output:
#!/bin/bash
grep -Eo '^([^:]+)' /etc/passwd |
while read NAME
do
echo ${#NAME} ${NAME}
done |
sort -n |head -1 | awk '{print $2}'
grep -Eo '^([^:]+)' /etc/passwd |
while read NAME
do
echo ${#NAME} ${NAME}
done |
sort -n |tail -1 | awk '{print $2}
'
All you need is:
$ awk -F: '
NR==1 { min=max=$1 }
length($1) > length(max) { max=$1 }
length($1) < length(min) { min=$1 }
END { print min ORS max }
' /etc/passwd
No explicit loops or pipelines or multiple commands required.
The problem is that you only have two pipelines, when you really need one. So you have grep | while read do ... done | sort | head | awk and sort | tail | awk: the first sort has an input (i.e., the while loop) - the second sort doesn't. So the script is hanging because your second sort doesn't have an input: or rather it does, but it's STDIN.
There's various ways to resolve:
save the output of the while loop to a temporary file and use that as an input to both sort commands
repeat your while loop
use awk to do both the head and tail
The first two involve iterating over the password file twice, which may be okay - depends what you're ultimately trying to do. But using a small awk script, this can give you both the first and last line by way of the BEGIN and END blocks.
While you already have good answers, you can also use POSIX shell to accomplish your goal without any pipe at all using the parameter expansion and string length provided by the shell itself (see: POSIX shell specifiction). For example you could do the following:
#!/bin/sh
sl=32;ll=0;sn=;ln=; ## short len, long len, short name, long name
while read -r line; do ## read each line
u=${line%%:*} ## get user
len=${#u} ## get length
[ "$len" -lt "$sl" ] && { sl="$len"; sn="$u"; } ## if shorter, save len, name
[ "$len" -gt "$ll" ] && { ll="$len"; ln="$u"; } ## if longer, save len, name
done </etc/passwd
printf "shortest (%2d): %s\nlongest (%2d): %s\n" $sl "$sn" $ll "$ln"
Example Use/Output
$ sh cketcpw.sh
shortest ( 2): at
longest (17): systemd-bus-proxy
Using either pipe/head/tail/awk or the shell itself is fine. It's good to have alternatives.
(note: if you have multiple users of the same length, this just picks the first, you can use a temp file if you want to save all names and use -le and -ge for the comparison.)
If you want both the head and the tail from the same input, you may want something like sed -e 1b -e '$!d' after you sort the data to get the top and bottom lines using sed.
So your script would be:
#!/bin/bash
grep -Eo '^([^:]+)' /etc/passwd |
while read NAME
do
echo ${#NAME} ${NAME}
done |
sort -n | sed -e 1b -e '$!d'
Alternatively, a shorter way:
cut -d":" -f1 /etc/passwd | awk '{ print length, $0 }' | sort -n | cut -d" " -f2- | sed -e 1b -e '$!d'
For example, I have below log files from the 16th-20th of Feb 2015. Now I want to create a single file named, mainentrywatcherReport_2015-02-16_2015-02-20.log. So in other words, I want to extract the date format from the first and last file of week (Mon-Fri) and create one output file every Saturday. I will be using cron to trigger the script every Saturday.
$ ls -l
mainentrywatcher_2015-02-16.log
mainentrywatcher_2015-02-17.log
mainentrywatcher_2015-02-18.log
mainentrywatcher_2015-02-19.log
mainentrywatcher_2015-02-20.log
$ cat *.log >> mainentrywatcherReport_2015-02-16_2015-02-20.log
$ mv *.log archive/
Can anybody help on how to rename the output file to above format?
Perhaps try this:
parta=`ls -l | head -n1 | cut -d'_' -f2 | cut -d'.' -f1`
partb=`ls -l | head -n5 | cut -d'_' -f2 | cut -d'.' -f1`
filename=mainentrywatcherReport_${parta}_${partb}.log
cat *.log >> ${filename}
"ls -l" output is described in the question
"head -nX" takes the Xth line of the output
"cut -d'_' -f2" takes everything (that remains) after the first underscore
"cut -d'.' -f1" times everything (that remains) before the first period
both commands are surrounded by ` marks (above tilde ~) to capture the output of the command to a variable
file name assembles the two dates stripped of the unnecessary with the other formatting desired for the final file name.
the cat command demonstrates one possible way to use the resulting filename
Happy coding! Leave a comment if you have any questions.
You can try this if you want to introduce simple looping...
FROM=ls -lrt mainentrywatcher_* | awk '{print $9}' | head -1 | cut -d"_" -f2 | cut -d"." -f1
TO=ls -lrt mainentrywatcher_* | awk '{print $9}' | tail -1 | cut -d"_" -f2 | cut -d"." -f1
FINAL_LOG=mainentrywatcherReport_${FROM}_${TO}.log
for i in ls -lrt mainentrywatcher_* | awk '{print $9}'
do
cat $i >> $FINAL_LOG
done
echo "All Logs Stored in $FINAL_LOG"
Another approach given your daily files and test contents as follows:
mainentrywatcher_2015-02-16.log -> a
mainentrywatcher_2015-02-17.log -> b
mainentrywatcher_2015-02-18.log -> c
mainentrywatcher_2015-02-19.log -> d
mainentrywatcher_2015-02-20.log -> e
That utilizes bash parameter expansion/substring extraction would be a simple loop:
#!/bin/bash
declare -i cnt=0 # simple counter to determine begin
for i in mainentrywatcher_2015-02-*; do # loop through each matching file
tmp=${i//*_/} # isolate date
tmp=${tmp//.*/}
[ $cnt -eq 0 ] && begin=$tmp || end=$tmp # assign first to begin, last to end
((cnt++)) # increment counter
done
cmbfname="${i//_*/}_${begin}_${end}.log" # form the combined logfile name
cat ${i//_*/}* > $cmbfname # cat all into combined name
## print out begin/end/cmbfname & contents to verify
printf "\nbegin: %s\nend : %s\nfname: %s\n\n" $begin $end $cmbfname
printf "contents: %s\n\n" $cmbfname
cat $cmbfname
exit 0
use/output:
alchemy:~/scr/tmp/stack/tmp> bash weekly.sh
begin: 2015-02-16
end : 2015-02-20
fname: mainentrywatcher_2015-02-16_2015-02-20.log
contents: mainentrywatcher_2015-02-16_2015-02-20.log
a
b
c
d
e
You can, of course, modify the for loop to accept a positional parameter containing the partial filename and pass the partial file name from the command line.
Something like this:
#!/bin/sh
LOGS="`echo mainentrywatcher_2[0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].log`"
HEAD=
TAIL=
for logs in $LOGS
do
TAIL=`echo $logs | sed -e 's/^.*mainentrywatcher_//' -e 's/\.log$//'`
test -z "$HEAD" && HEAD=$TAIL
done
cat $LOGS >mainentrywatcherReport_${HEAD}_${TAIL}.log
mv $LOGS archive/
That is:
get a list of the existing logs (which happen to be sorted) in a variable $LOGS
walk through the list, getting just the date according to the example
save the first date as $HEAD
save the last date as $TAIL
after the loop, cat all of those files into the new output file
move the used-up log-files into the archive directory.
consider todays date as 24/02/14
I have set of files in mention directory "/apps_kplus/KplusOpenReport/crs_scripts/rconvysya" and file names as
INTTRADIVBMM20142402
INTTRADIVBFX20142402
INTTRADIVBFI20142402
INTTRADIVBDE20142402
INTPOSIVBIR20142402
INTPOSIVBIR20142302
INTTRADIVBDE20142302
INTTRADIVBFI20142302
INTTRADIVBFX20142302
INTTRADIVBMM20142302
I wanted to get count of file with current date. for that i am using below query
#! /bin/bash
tm=$(date +%y%d%m)
x=$(ls /apps_kplus/KplusOpenReport/crs_scripts/rconvysya/INT*$tm.txt 2>/dev/null | wc -l)
if [ $x -eq 5 ]
then
exit 4
else
exit 3
fi
But not getting desired output.
what is wrong.
You're searching for files with a .txt extension, and the files you list have none. If you remove the .txt from the pathname, it works.
With Awk, anything is possible!
tm=$(date +%Y%d%m) # Capital Y is for the four digit year!
ls -a | awk -v tm=$tm '$0 ~ tm' | wc -l
This is the Great and Awful Awk Command. (Awful as in both full of awe and as in Starbucks Coffee).
The -v parameter allows me to set Awk variables. The $0 represents the entire line which is the file name from the ls command. The $0 ~ tm means I'm looking for files that contain the time string you specified.
I could do this too:
ls -a | awk "/$tm\$/"
Which lets the shell interpolate $tm into the Awk program. This is looking only for files that end in your $tm stamp. The /.../ by itself means matching the entire line. It's an even shorter Awk shortcut than I had before. Plus, it makes sure you're only matching the timestamp on the end of the file.
You can try the following awk:
awk -v d="$date" '{split(d,ary,/\//);if($0~"20"ary[3]ary[1]ary[2]) cnt++}END{print cnt}' file
where $date is your shell variable containing the date you wish to search for.
$ cat file
INTTRADIVBMM20142402
INTTRADIVBFX20142402
INTTRADIVBFI20142402
INTTRADIVBDE20142402
INTPOSIVBIR20142402
INTPOSIVBIR20142302
INTTRADIVBDE20142302
INTTRADIVBFI20142302
INTTRADIVBFX20142302
INTTRADIVBMM20142302
$ awk -v d="$date" '{split(d,ary,/\//);if($0~"20"ary[3]ary[1]ary[2]) cnt++}END{print cnt}' file
5
ls /apps_kplus/KplusOpenReport/crs_scripts/rconvysya | grep "`date +%Y%m%d`"| wc -l
This worked for me and it seems like the simplest solution, I hope it fits you needs
To get exit status 4 if there are 0k files present use code bellow:
0=`ls /apps_kplus/KplusOpenReport/crs_scripts/rconvysya | grep "`date +%Y%m%d`"| xargs -n 1 stat -c%s | grep "^0$" | wc -l`
if [ "$0" -eq "0" ]
then
exit 4
else
exit 3
fi