new line in shell script - linux

I have the below command in my shell script to send all the four echo statements in one email
{
echo "Data Successfully loaded into LIP table"
echo "Total Items Purchased: `echo $QUERY1 | awk '{print $1}'`"
echo "Total Items MissingorMismatch: `echo $QUERY1 | awk '{print $2}'`"
echo "Error Percentage: $QUERY2"
} | mailx -s "Report for $DATE_YEST_FORMAT1" -r rj#host.com user2#host.com
But when I see my email, I get the output like below which I don't want-
Data Successfully loaded into LIP table Total Items Purchased: 3956391
Total Items MissingorMismatch: 975825 Error Percentage: 24.66452380464924
I need output something like below as line by line.
Data Successfully loaded into LIP table
Total Items Purchased: 3956391
Total Items MissingorMismatch: 975825
Error Percentage: 24.66452380464924
Any suggestion why is it happening like this? I am running SunOS (Solaris).
Update:-
After trying the suggestion give by Kevin
QUERY1=`hive -e "
set mapred.job.queue.name=hdmi-technology;
SELECT SUM(total_items_purchased), SUM(total_items_missingormismatch) from lip_data_quality where dt='$DATE_YEST_FORMAT2';"`
QUERY2=`hive -e "
set mapred.job.queue.name=hdmi-technology;
SELECT 100 * SUM(total_items_missingormismatch*1.0) / SUM(total_items_purchased) FROM lip_data_quality where dt='$DATE_YEST_FORMAT2';"`
mailx -s "LIP Data Quality Report for $DATE_YEST_FORMAT1" -r rj#host.com rj#host.com uname#host.com <<EOF
Data Successfully loaded into LIP_DATA_QUALITY table
Total Items Purchased: $(echo $QUERY1 | awk '{print $1}')
Total Items MissingorMismatch: $(echo $QUERY1 | awk '{print $2}')
Error Percentage: $QUERY2
EOF
Output that I got in an email-
Data Successfully loaded into LIP_DATA_QUALITY table
Total Items Purchased: $(echo 3712928 393455 | awk '{print }')
Total Items MissingorMismatch: $(echo 3712928 393455 | awk '{print }')
Error Percentage: 10.596892802661404
Which is not right.. I should be getting output like this-
Data Successfully loaded into LIP_DATA_QUALITY table
Total Items Purchased: 3712928
Total Items MissingorMismatch: 393455
Error Percentage: 10.596892802661404
I am running SunOS
bash-3.00$ uname -a
SunOS lvsaishdc3in0001 5.10 Generic_142901-02 i86pc i386 i86pc

It's possible that mailx ignores single newlines and separates paragraphs by an empty line, similar to tex and our own SO.
That combined with a heredoc
mailx -s "Report for $DATE_YEST_FORMAT1" -r rj#host.com user2#host.com <<EOF
Data Successfully loaded into LIP table
Total Items Purchased: `echo $QUERY1 | awk '{print $1}'`
Total Items MissingorMismatch: `echo $QUERY1 | awk '{print $2}'`
Error Percentage: $QUERY2
EOF
[It appears that your bash does not recognize the $(command) syntax, so I have removed that.]
[N.B. I just tested, and Solaris's mailx sent an email formatted precisely as I typed it. If your mail reader is displaying in HTML mode, however, it may collapse whitespace and show it all as one line.]

Try to do it using \n as it is usually the standard escape sequence for a newline in a printing function., like so:
{
echo -e "Data Successfully loaded into LIP table\n"
echo -e "Total Items Purchased: `echo $QUERY1 | awk '{print $1}'\n"
echo -e "Total Items MissingorMismatch: `echo $QUERY1 | awk '{print $2}'\n"
echo -e "Error Percentage: $QUERY2\n"
} | mailx -s "Report for $DATE_YEST_FORMAT1" -r rj#host.com user2#host.com
more info can be gotten here.

Related

script to check when a directory is last updated

I am trying to write a script that takes the last update date of a hadoop path subtracts it from current date and sends out an email if the difference is more than a certain number of days (variable).It needs to loop through a pipe delimited config file which has the number of days, table name,hadoop path and email
example of config file
30|db.big_leasing_center|/TST/DL/EDGE_BASE/GFDCP-52478/BIG_LEASING_CENTER/Data/|abcd#gmail.com
2|db.event|/TST/DL/EDGE_BASE/GFDCP-52478/GFDCP01P-ITG_FIN_DB/EVENTS/Data/|cab#gmail.com
below is what i tried
#!/bin/ksh
check_last_refresh() {
DAYS=$1
TABLE=$2
HDP_PATH=$3
EMAIL_DL=$4
last_refresh_date=$(hdfs dfs -ls $HDP_PATH | grep '^-' | awk '{print $6}' | sort -rh | head -1)
echo $last_refresh_date
diff=$(( ( $(date '+%s') - $(date '+%s' -d "$last_refresh_date") ) / 86400 ))
echo $diff
if [ "$diff" -gt "$DAYS" ]; then
echo "HI for $TABLE has an issue" | mail -s "HI for $TABLE has an issue, Please check" -b $EMAIL_DL
fi
return 0
}
cat /data/scratchSpace/bznhd9/CONFIG.txt | while read line; do
DAYS=$(echo $line|awk -F'|' '{print $1}')
TABLE=$(echo $line|awk -F'|' '{print $2}')
HDP_PATH=$(echo $line|awk -F'|' '{print $3}')
EMAIL_DL=$(echo $line|awk -F'|' '{print $4}')
echo $TABLE
r=$(check_last_refresh $DAYS $TABLE $HDP_PATH $EMAIL_DL)
echo $r
done
Please help

Linux script shows unnecessary files

I have written following script and it shows some unnecessary files when i'm running it. I just want to execute only the command and receive the alerts only.
the script as follows
df -h | nawk '/backup/ {print $5 " " $6}' | while read line;
do
usep=$(echo $line | nawk '{printf "%d", $1}' )
partition=$(echo $line | nawk '{print $2}')
if (( $usep >= 90)); then
echo "$partition ($usep%)" | mailx -s "172.27.68.101" 0234343456#alert.abc.lk;
echo "$partition ($usep%)" | mailx -s "172.27.68.101" amalw#abc.lk;
echo "$partition ($usep%)" | mailx -s "172.27.68.101" amalw#abc.lk;
fi
done
Follwing image shows the output problem
How can i add multiple recipient to this script without opening such directories?
To paste a multi-line bash code into terminal, add parenthesis around the lines otherwise each line gets run as a separate command as soon as it gets pasted:
(df -h | nawk '/backup/ {print $5 " " $6}' | while read line; do
usep=$(echo "$line" | nawk '{printf "%d", $1}')
partition=$(echo $line | nawk '{print $2}')
if(("$usep" >= 90)) ; then echo "$partition ($usep%)" | mailx -s "172.27.68.101" 0234343456#alert.abc.lk;
echo "$partition ($usep%)" | mailx -s "172.27.68.101" amalw#abc.lk;
echo "$partition ($usep%)" | mailx -s "172.27.68.101" amalw#abc.lk;
fi
done)
Firstly command df -h|grep backup|sed 's/\%//g'|awk '$5 >= 90 {print $5"% "$6}' for having FS(Partition) more than 90% usage.
the rest of command to alert trought mail. Then :
df -h|grep backup|sed 's/\%//g'|awk '$5 >= 90 {print $5"% "$6}'|while read USAGE PARTITION
do
echo "$PARTITION ($USAGE)"|mailx -s "172.27.68.101" "0234343456#alert.abc.lk,amalw#abc.lk"
done

awk - send sum to global variable

I have a line in a bash script that calculates the sum of unique IP requests to a certain page.
grep $YESTERDAY $ACCESSLOG | grep "$1" | awk -F" - " '{print $1}' | sort | uniq -c | awk '{sum += 1; print } END { print " ", sum, "total"}'
I am trying to get the value of sum to a variable outside the awk statement so I can compare pages to each other. So far I have tried various combinations of something like this:
unique_sum=0
grep $YESTERDAY $ACCESSLOG | grep "$1" | awk -F" - " '{print $1}' | sort | uniq -c | awk '{sum += 1; print ; $unique_sum=sum} END { print " ", sum, "total"}'
echo "${unique_sum}"
This results in an echo of "0". I've tried placing __$unique_sum=sum__ in the END, various combinations of initializing the variable (awk -v unique_sum=0 ...) and placing the variable assignment outside of the quoted sections.
So far, my Google-fu is failing horribly as most people just send the whole of the output to a variable. In this example, many lines are printed (one for each IP) in addition to the total. Failing a way to capture the 'sum' variable, is there a way to capture that last line of output?
This is probably one of the most sophisticated things I've tried in awk so my confidence that I've done anything useful is pretty low. Any help will be greatly appreciated!
You can't assign a shell variable inside an awk program. In general, no child process can alter the environment of its parent. You have to have the awk program print out the calculated value, and then shell can grab that value and assign it to a variable:
output=$( grep $YESTERDAY $ACCESSLOG | grep "$1" | awk -F" - " '{print $1}' | sort | uniq -c | awk '{sum += 1; print } END {print sum}' )
unique_sum=$( sed -n '$p' <<< "$output" ) # grab the last line of the output
sed '$d' <<< "$output" # print the output except for the last line
echo " $unique_sum total"
That pipeline can be simplified quite a lot: awk can do what grep can do, so first
grep $YESTERDAY $ACCESSLOG | grep "$1" | awk -F" - " '{print $1}'
is (longer, but only one process)
awk -F" - " -v date="$YESTERDAY" -v patt="$1" '$0 ~ date && $0 ~ patt {print $1}' "$ACCESSLOG"
And the last awk program just counts how many lines and can be replaced with wc -l
All together:
unique_output=$(
awk -F" - " -v date="$YESTERDAY" -v patt="$1" '
$0 ~ date && $0 ~ patt {print $1}
' "$ACCESSLOG" | sort | uniq -c
)
echo "$unique_output"
unique_sum=$( wc -l <<< "$unique_output" )
echo " $unique_sum total"

Bash formatting for nice ascii logs

I've written a function to do some logging on different aspects of the files, folder, etc and I 'm going to have this automatically email members of my staff. All this is fine but I wouldn't mind some pointers and help to make the formation nicer - this will get sent out via mailutils as part of the script.
Just looking for some nice spacing tips,etc
function report(){
lsb_release -a
echo "OS: $(uname -s)"
echo "ARCH: $(uname -m)"
echo "VER: $(uname -r)"
echo "Apache running "
ps aux|grep -i apache2
echo "Showing if SSL is open"
apache2ctl -M | grep ssl
echo "Space on local disk"
df -h
echo "Showing permissions for the web folders"
echo $'*** /var/www// ***'
ls -l /var/www | awk '{print $1 " " $3 " " $4 " " $9}'
ls -l /var/www/user1 | awk '{print $1 " " $3 " " $4 " " $9}'
ls -l /var/www/user2 | awk '{print $1 " " $3 " " $4 " " $9}'
echo "Showing network status"
ifconfig eth0
echo " DNS "
tail /etc/resolv.conf
echo "Current workspaces set up on the local server "
grep user2 /var/www/temp/text.txt | grep -E -o '[0-9]+[0-9]'
}
Try piping the "column" command instead of or after using awk or cut.
ls -l | awk '{print $1" "$3" "$4" "$9}' | tail -n +2 | column -t
Check out the difference between "mount" and "mount | column -t" or "df -PH and df -PH | column -t"
If you want to concatenate columns of info with the same number or relational fields you can use process redirection and "paste."
paste -d ' ' <(cat /sys/class/scsi_host/host*/device/fc_host:host*/port_name) \
<(cat /sys/class/scsi_host/host*/device/fc_host:host*/speed) \
<(cat /sys/class/scsi_host/host*/device/fc_host:host*/port_state)
0x218000e05a0001aa 4 Gbit Online
0x218000e05a2001aa 4 Gbit Online

Send an email to a email group using shell script

Below is my shell script that is working fine.
#!/bin/bash
DATE_YEST_FORMAT2=`perl -e 'use POSIX qw(strftime); print strftime "%Y%m%d",localtime(time()- 3600*96);'`
echo $DATE_YEST_FORMAT2
QUERY1=`hive -e "
set mapred.job.queue.name=hdmi-technology;
SELECT SUM(total_items_purchased), SUM(total_items_missingormismatch) from lip_data_quality where dt='$DATE_YEST_FORMAT2';`
QUERY2=`hive -e "
set mapred.job.queue.name=hdmi-technology;
SELECT 100 * SUM(total_items_missingormismatch*1.0) / SUM(total_items_purchased) FROM lip_data_quality where dt='$DATE_YEST_FORMAT2';"`
echo "Total items purchased: `echo $QUERY1 | awk '{print $1}'`"
echo "Total Items MissingorMismatch: `echo $QUERY1 | awk '{print $2}'`"
echo "Error Percentage: $QUERY2"
I am running the above shell script like this below-
sh -x test.sh
Problem Statement:-
From the above shell script, I am getting below three things from the last three echo statements-
Total items purchased
Total Items MissingorMismatch
Error Percentage
I need to send all these above three things in an email to our email group we have DL-host#company.com by using that shell script above or suppose if I have 10 email list to which I need to send email with the same contents and subjects, then I think, I can store all these 10 emails list in some variable and just read it from there and keep on sending, right?. Is this possible to do it? I was thinking email structure like this below, just making very simple.
Subject
Test Data
Mail Body
Total items purchased:- Some Number
Total Items MissingorMismatch:- Some Number
Error Percentage:- Some Number
Any thoughts will be appreciated.
Update, After trying larsks suggestions, I am getting only last echo statement in an email meaning only the error percentage one, not all three in a single email:-
#!/bin/bash
DATE_YEST_FORMAT2=`perl -e 'use POSIX qw(strftime); print strftime "%Y%m%d",localtime(time()- 3600*96);'`
echo $DATE_YEST_FORMAT2
QUERY1=`hive -e "
set mapred.job.queue.name=hdmi-technology;
SELECT SUM(total_items_purchased), SUM(total_items_missingormismatch) from lip_data_quality where dt='$DATE_YEST_FORMAT2';`
QUERY2=`hive -e "
set mapred.job.queue.name=hdmi-technology;
SELECT 100 * SUM(total_items_missingormismatch*1.0) / SUM(total_items_purchased) FROM lip_data_quality where dt='$DATE_YEST_FORMAT2';"`
echo "Total items purchased: `echo $QUERY1 | awk '{print $1}'`"
echo "Total Items MissingorMismatch: `echo $QUERY1 | awk '{print $2}'`"
echo "Error Percentage: $QUERY2" | mail -s "Test Data" rj#host.com
You typically use the /bin/mail program to send email from a shell script. You provide a subject and recipients on the command line and the message body on stdin. For example, inside your script you could do something like this:
{
echo "Total items purchased: `echo $QUERY1 | awk '{print $1}'`"
echo "Total Items MissingorMismatch: `echo $QUERY1 | awk '{print $2}'`"
echo "Error Percentage: $QUERY2"
} | mail -s "Test Data" DL-host#company.com
You can also pipe the output of an existing script into mail (if you don't want to modify the script, or if you only want to send mail somtimes):
<your script> | mail -s "Test Data" DL-host#company.com
You can specify multiple recipients on the command line, so:
... | mail -s "Test Data" address1#company.com address2#company.com
and so forth.

Resources