I am having a output file called result from a script as below:
|first_t | PASS |
|second_t | PASS |
|third_t | File_Error |
And I used the below column command to get the output in sequence as below:
column -t -s " " result > result_new
|first_t | PASS |
|second_t | PASS |
|third_t | File_Error |
And after sending the result_new file to the mail using below mail command as in the below script I am not getting the expected output.
#!/bin/bash
mail -s "COLUMN CHECK_1" name#gmail.com < result_new
I am getting this output in my mail:
|first_t | PASS |
|second_t | PASS |
|third_t | File_Error |
My expected output in mail should be this:
|first_t | PASS |
|second_t | PASS |
|third_t | File_Error |
Can someone tell me what should I do here?
I'm assuming the issue is with the (receiving) email client and that it's displaying the message using a variable width font (eg, spaces aren't as wide as letters); to get around this issue I can think of a couple options:
send the output as an ascii text file attachment (user would need to open the attachment with an ascii text reader - eg, vim, notepad, etc)
format the outgoing email (eg, MIME + 'text/html') so the (receiving) email client knows to display the content as fixed-width text (though at this point you now need to look at your options on the sending end as to how to send such a formatted email - eg, sendmail, mailx, mutt, whatever mail points to on your system, etc; this also assumes the (receiving) email client is capable of displaying MIME encoded messages)
One simple sendmail example I've used when sending emails (from linux to Windows/Mac users) where I want the contents of the email to be displayed as fixed-width text:
$ subject="this is the subject of my email"
$ emlist="list_of_email_addresses"
$ mailfile=$(mktemp)
$ echo "Subject: ${subject}
MIME-Version: 1.0
Content-Type: text/html
<html><body><pre>" > "${mailfile}"
cat result_new >> "${mailfile}"
echo "</pre></body></html>" >> "${mailfile}"
$ cat ${mailfile} | /usr/sbin/sendmail ${emlist}
$ 'rm' -rf "${mailfile}"
If you don't have access to sendmail then you may need to do some testing with your mail (sending) program to see how to send a MIME encoded message ...
Related
I have a folder with a mixture of files types (.bam, .bam.bai, and .log). I created a for loop to perform two commands on each of the .bam files. My current code direct the output of each command into a separate csv files, because I could not figure out how to direct the outputs to separate columns.
TYIA!
Question 1
I want to export the output from the commands into the same csv. How can I alter my code so that the output from my first command is saved as the first column of a csv, and the output from my second command is saved as the second column of the same csv.
Question 2
What is the name of the syntax used to select files in a for loop? For instance, the * in *.bam represents a wildcard. Is this regex? I had a tough time trying to alter this so that only *.bam files were selected for the for loop (and .bam.bai were excluded). I ended up with *[.bam] by guessing and empirically testing my outputs. Are there any websites that do a good job of explaining this syntax and provide lots of examples (coder level: newbie)
Current Code
> ~/Desktop/Sample_Names.csv
> ~/Desktop/Read_Counts.csv
echo "Sample" | cat - > ~/Desktop/Sample_Names.csv
echo "Total_Reads" | cat - > ~/Desktop/Read_Counts.csv
for file in *[.bam]
do
samtools view -c $file >> ~/Desktop/Read_Counts.csv
gawk -v RS="^$" '{print FILENAME}' $file >> ~/Desktop/Sample_Names.csv
done
Current Outputs (truncated)
>Sample_Names.csv
| Sample |
|--------------|
| B40-JV01.bam |
| B40-JV02.bam |
| B40-JV03.bam |
>Read_Counts.csv
| Total_Reads |
|-------------|
| 3835555 |
| 4110463 |
| 144558 |
Desired Output
>Combined_Outputs.csv
| Sample | Total_Reads |
|--------------|-------------|
| B40-JV01.bam | 3835555 |
| B40-JV02.bam | 4110463 |
| B40-JV03.bam | 144558 |
Something like
echo "Sample,Total_Reads" > Combined_Outputs.csv
for file in *.bam; do
printf "%s,%s\n" "$file" "$(samtools view -c "$file")"
done >> Combined_Outputs.csv
Print one line for each file, and move the output redirection outside of the loop for efficiency.
I have live raw json stream data from the virtual radar server I'm using.
I use Netcat to fetch the data and jq to save it on my kali linux. using the following command.
nc 127.0.0.1 30006 | jq > A7.json
But i want to filter specific content from the data stream.
i use the following command to extract the data.
cat A7.json | jq '.acList[] | select(.Call | contains("QTR"))' - To fetch Selected airline
But i realized later that the above command only works once. in other words, it does not refresh. as the data is updating every second. i have to execute the command over and over again to extract the filter data which is causing to generate duplicate data.
Can someone help me how to filter the live data without executing the command over and over .
As you don't use the --stream option, I suppose your document is a regular JSON document.
To execute your command every second, you can implement a loop that sleeps for 1 second:
while true; do sleep 1; nc 127.0.0.1 30006 | jq '.acList[] | select(…)'; done
To have the output on the screen and also save to a file (like you did with A7.json), you can add a call to tee:
# saves the document as returned by `nc` but outputs result of `jq`
while true; do sleep 1; nc 127.0.0.1 30006 | tee A7.json | jq '.acList[] | …'; done
# saves the result of `jq` and outputs it
while true; do sleep 1; nc 127.0.0.1 30006 | jq '.acList[] | …' | tee A7.json; done
Can you try this ?
nc localhost 30006 | tee -a A7.json |
while true; do
stdbuf -o 0 jq 'try (.acList[] | select(.Call | contains("QTR")))' 2>/dev/null
done
Assuming that no other process is competing for the port, I'd suggest trying:
nc -k -l localhost 30006 | jq --unbuffered ....
Or if you want to keep a copy of the output of the netcat command:
nc -k -l localhost 30006 | tee A7.json | jq --unbuffered ....
You might want to use tee -a A7.json instead.
Break Down
Why I did what I did?
I have live raw json stream data from the virtual radar server, which is running on my laptop along with Kali Linux WSL on the background.
For those who don't know virtual radar server is a Modes-s transmission decoder that is used to decode different ADS-B Formats. Also, it rebroadcast the data in a variety of formats.. one of them is the Json stream. And I want to save select aircraft data in json format on Kali Linux.
I used the following commands to save the data before
$ nc 127.0.0.1 30001 | jq > A7.json - To save the stream.
$ cat A7.json | jq '.acList[] | select(.Call | contains("QTR"))' - To fetch Selected airline
But I realized two things after using the above. One I'm storing unwanted data which is consuming my storage. Two, when I used the second command it just goes through the json file once and produces the data which is saved at that moment and that moment alone. which caused me problems as I have to execute the command over and over again to extract the filter data which is causing me to generate duplicate data.
Command worked for me
The following command worked flawlessly for my problem.
$ nc localhost 30001 | sudo jq --unbuffered '.acList[] | select (.Icao | contains("800CB8"))' > A7.json
The following also caused me some trouble which i explain clearly down below.
Errors X Explanation
This error was resulted in the missing the Field name & key in the json object.
$ nc localhost 30001 | sudo jq --unbuffered '.acList[] | select (.Call | contains("IAD"))' > A7.json
#OUTPUT
jq: error (at <stdin>:0): null (null) and string ("IAD") cannot have their containment checked
If you see the below JSON data you'll see the missing Field name & key which caused the error message above.
{
"Icao": "800CB8",
"Alt": 3950,
"GAlt": 3794,
"InHg": 29.7637787,
"AltT": 0,
"Call": "IAD766",
"Lat": 17.608658,
"Long": 83.239166,
"Mlat": false,
"Tisb": false,
"Spd": 209,
"Trak": 88.9,
"TrkH": false,
"Sqk": "",
"Vsi": -1280,
"VsiT": 0,
"SpdTyp": 0,
"CallSus": false,
"Trt": 2
}
{
"Icao": "800CB8",
"Alt": 3950,
"GAlt": 3794,
"AltT": 0,
"Lat": 17.608658,
"Long": 83.239166,
"Mlat": false,
"Spd": 209,
"Trak": 88.9,
"Vsi": -1280
}
{
"Icao": "800CB8",
"Alt": 3800,
"GAlt": 3644,
"AltT": 0,
"Lat": 17.608795,
"Long": 83.246155,
"Mlat": false,
"Spd": 209,
"Trak": 89.2,
"Vsi": -1216
}
Commands that didn't work for me.
Command #1
When i used jq with --stream with the filter it produced the below output. --Stream with output filter worked without any errors.
$ nc localhost 30001 | sudo jq --stream '.acList[] | select (.Icao | contains("800"))' > A7.json
#OUTPUT
jq: error (at <stdin>:0): Cannot index array with string "acList"
jq: error (at <stdin>:0): Cannot index array with string "acList"
jq: error (at <stdin>:0): Cannot index array with string "acList"
jq: error (at <stdin>:0): Cannot index array with string "acList"
jq: error (at <stdin>:0): Cannot index array with string "acList"
jq: error (at <stdin>:0): Cannot index array with string "acList"
Command #2
For some reason -k -l didn't work to listen to the data. but the other command worked perfectly. i think it didn't work because of the existing port outside of the wsl.
$ nc -k -l localhost 30001
$ nc localhost 30001
Thank you to everyone who helped me to solve my issue. I'm very grateful to you guys
I am trying to output the size of an ARP table from a FW using an Expect script so it can be graphed. After the below code the output displayed to screen is shown:
/usr/bin/expect -f -<< EOD
spawn ssh test#1.2.3.4
sleep 1
expect {
"*word:" {send "password\r"}
}
sleep 1
expect {
"*>" {send "show arp all | match \"total ARP entries in table\"\r"}
}
sleep 1
expect {
"*>" {send "exit\r"}
}
expect eof
EOD
spawn ssh test#1.2.3.4
FW-Active
Password:
Number of failed attempts since last successful login: 0
test#FW-Active(active)> show arp all | match "total ARP entries in table"
total ARP entries in table : 2861
What I am trying to do is be able to output only the numeric value indicated from the total ARP entries in table. I am assuming I need to some how do a "cut" or "awk" or something to extract only the numbers but I am not having any luck. Any help is greatly appreciated.
You store the output of that whole command in a variable, let's say a.
Something like this will probably work. Since you're using expect, you might want to figure out how to store that output as a variable that way you can manipulate it. I stored the output as $a in my example.
$ echo $a
total ARP entries in table : 2861
$ echo ${a% *}
total ARP entries in table :
$ echo ${a% *}-
total ARP entries in table : -
$ echo ${a##* }
2861
Logic explanation (Parameter/Variable Substituion in BASH):
1) To removing/stripping the left hand side part, use # for reaching the first matching character value (reading / parsing from left side), ## for reaching the last matching character/value. It works by giving *<value> within the { } braces.
2) To removing/stripping the right hand side part, use % for reaching the first matching character value (reading / parsing from right side), %% for reaching the last matching character/value. It works by giving <value>* within the { } braces.
Or if you don't want to store the output or anything, then simply do this:
show arp all | match "total ARP entries in table" | grep -o "[0-9][0-9]*"
Or (the following assumes that you don't change
show arp all | match "total ARP entries in table" | sed "s/ *//g"|cut -d':' -f2
I've written a script to get the browser version of users but I need to clean up the output. What the script does is looks at the apache logs for # and IE8 then emails me the information. The problem I have is the output as when the grep finds a email address and IE8 it gives me the full output - i.e. /page/code/user#foobar.com/home.php whereas the output i'm looking is the just the email address and to only have this information recorded once a day:
Example:
user#foobar IE8
Thanks
#!/bin/bash
#Setting date and time (x and y and z aren't being used at the moment)
x="$(date +'%d/%b/%Y')"
y="$(date +'%T')"
z="$(date +'%T' | awk 'BEGIN { FS =":"} ; {print $1}')"
#Human readable for email title
emaildate=$(date +"%d%b%Y--Hour--%H")
#Setting date and time for grep and filename
beta="$(date +'%d/%b/%Y:%H')"
sigma="$(date +'%d-%b-%Y-%H')"
#CurrentAccess logs
log='/var/logs/access.log'
#Set saved log location
newlogs=/home/user/Scripts/browser/logs
#Prefrom the grep for the current day
grep # $log | grep $beta | awk 'BEGIN { FS = " " } ; { print $7 }' | sort -u >> $newlogs/broswerusage"$sigma".txt
mail -s "IE8 usage for $emaildate" user#exmaple.com < $newlogs/broswernusage"$sigma".txt
I've encountered some problem when doing the procmail recipe.
Here what I have get so far :
:0
* ^X-Loop: myemail#gmail\.com
/dev/null
:0
# filtering email by number 60
* ^Subject:.*(60)
{
:0c:
${DEFAULT}
#trying to take out input from the body
:0fb
| head -10
#Forward it to the other folder
:0
mytest/
}
The problem occur when procmail reading the body of the email.It will show output like this :
+96szV6aBDlD/F7vuiK8fUYVknMQPfPmPNikB+fdYLvbwsv9duz6HQaDuwhGn6dh9w2U
1sABcykpdyfWqWhLt5RzCqppYr5I4yCmB1CNOKwhlzI/w8Sx1QTzGT32G/ERTlbr91BM VmNQ==
MIME-Version: 1.0
Received: by 10.52.97.41 with SMTP id dx9mr14500007vdb.89.1337845760664; Thu,
24 May 2012 00:49:20 -0700 (PDT)
Received: by 10.52.34.75 with HTTP; Thu, 24 May 2012 00:49:20 -0700 (PDT)
Date: Thu, 24 May 2012 15:49:20 +0800
Message-ID: <CAE1Fe-r4Lid+YSgFTQdpsniE_wzeGjETWLLJJxat+HK94u1=AQ#mail.gmail.com>
Subject: 60136379500
From: my email <my email#gmail.com>
To: your email <your email#gmail.com>
Content-Type: multipart/alternative; boundary=20cf307f380654240604c0c37d07
--20cf307f380654240604c0c37d07
Content-Type: text/plain; charset=ISO-8859-1
hi
there
how
are
you
--20cf307f380654240604c0c37d07
+96szV6aBDlD/F7vuiK8fUYVknMQPfPmPNikB+fdYLvbwsv9duz6HQaDuwhGn6dh9w2U
1sABcykpdyfWqWhLt5RzCqppYr5I4yCmB1CNOKwhlzI/w8Sx1QTzGT32G/ERTlbr91BM VmNQ==
I have manage to get the output but it is not working if the sender send fewer than 3 lines as the output will print out the footer of the email as well (because it is between the range of head -10).
I only want the body of the email to be filter (print out in text file) in the procmail.
Is it possible?Can anyone show me the way?I'm in my wits ends.Thanks
Attempting to treat a MIME multipart as just a lump of text is fraught with peril. In order to properly process the body, you should use a MIME-aware tool. But if you just want to assume that the first part is a text part and drop all other parts, you can create something fairly simple and robust.
# Truncate everything after first body part:
# Change second occurrence of --$MATCH to --$MATCH--
# and trim anything after it
:0fb
* ^Content-type: multipart/[a-z]+; boundary="\/[^"]+
| sed -e "1,/^--$MATCH$/b" -e "/^--$MATCH$/!b" -e 's//&--/' -eq
For elegance points, you might be able to develop the script to implement your 10-line body truncation action at the same time, but at least, this should hopefully get you started. (I would switch to awk or Perl at this point.)
:0fb
* ^Content-type: multipart/[a-z]+; boundary="\/[^"]+
| awk -v "b=--$MATCH" ' \
($0 == b || $0 == b "--") && seen++ { printf "%s--\n", $0; exit } \
!seen || p++ < 10'
Properly, the MIME part's headers should not count towards the line count.
This is slightly speculative; I assume by "footer" you mean the ugly base64-encoded attachment after the first body part, and of course, this recipe will do nothing at all for single-part messages. Maybe you want to fall back to your original recipe for those.
Recently had a similar issue and solved it with this (adapted to OP)...
#trying to take out input from the body
:0fb
| sed -n '/^Content-Type/,/^--/ { /^Content-Type/b; /^--/b; p }'
Explanations: in general form....
sed -n '/begin/,/end/ { /begin/b; /end/b; p }'
-n: --> turn printing off
/begin/ --> begin of pattern range (remainder commands only apply inside range)
,/end/ --> , end of sed pattern range
{ /begin/b; --> /b branch causes lines with pattern /begin/ to skip remaining commands
/end/b; --> (same as above), these lines will skip the upcoming (p)rint command
p }' --> prints lines that in pattern that made it to this command