I've problem in searching and storing specific part of a file into a variable in bash shell.
Here it's one sample of my files:
From root#machine2.com Mon Jan 7 16:56:50 2013
Return-Path: <root#machine2.com>
X-Original-To: smsto+9121403571#machine2.com
Delivered-To: smsto+9121403571#machine2.com
Received: by machine2.com (Postfix, from userid 0)
id 43C191A1ECE; Mon, 7 Jan 2013 16:56:50 +0330 (IRST)
Date: Mon, 07 Jan 2013 16:56:50 +0330
To: smsto+9121403571#machine2.com
Subject: =?us-ascii?Q?Testing\=08?=
User-Agent: Heirloom mailx 12.5 7/5/10
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-Id: <20130107132650.43C191A1ECE#machine2.com>
From: root#machine2.com (root)
My note ..
blah blah ...
What i need to do is to Storing some of these fields into variables (parameter like FROM, SUBJECT and EMAIL BODY)
for FROM and SUBJECT fields it was easy to search and get the data.
But for EMAIL BODY, as you see there is no any Labels to search for it ... so i was thinking one of possible ways to get the email body would be searching for FROM label and then using its line number to get EMAIL BODY from next line to end of file.
unfortunately I'm not that familiar with linux command to do such thing.
Please help me.
You can use sed to print from the blank line to the end of the file:
$ sed -n '/^\s*$/,$p' file
My note ..
blah blah ...
# Command substitution to store into a variable
$ body=$(sed -n '/^\s*$/,$p' file)
$ echo $body
My note .. blah blah ...
# Remember to quote variables to respect newlines
$ echo "$body"
My note ..
blah blah ...
If you don't want to include the first blank line use:
$ sed -n '/^\s*$/,$ {/^.*[^ ]\+.*/,$p}' file
Or strip all blank lines in the body:
$ sed -n '/^\s*$/,$ {/^.*[^ ]\+.*/p}' file
Another way to approach the problem is to look for the first empty line (which occurs right after the 'From:' line you talk about) and print everything after that. You can do this using awk and setting a null record separator. For example:
BODY=$(awk 'NR>1' RS= file)
However, the advantage/problem of the above is that blank lines will be discarded. If this is undesirable, here's a method that should satisfy:
BODY=$(awk 'i==1; /^$/ { i=1 }' file)
Then:
echo "$BODY"
Results:
My note ..
blah blah ...
Related
The proposal is to be a pure bash function for splitting strings that accepts any string as a delimiter and any string as an input.
QUESTION: How to create a function for splitting strings that accepts any string as input and as delimiter?
!!!REASON FOR QUESTION!!! There are many, many proposals (see this example) for string splitting with bash commands, but almost all of them only work in specific cases and not according to our proposal.
NOTES: We consider the following Linux distributions in their latest versions to be eligible as compatible plataforms -> Debiam, Ubuntu (server and desktop), Arch, RedHat, CentOS, SUSE (server and desktop).
Thanks and be kind! 🤗
SOME INPUT TO TEST:
read -r -d '' FILE_CONTENT << 'HEREDOC'
BEGIN
§\\§[+][.][-]
A literal backslash, ‘\’.°
°\a
The “alert” character, Ctrl-g, ASCII code 7 (BEL). (This often makes some sort of audible noise.)
\b
Backspace, Ctrl-h, ASCII code 8 (BS).
\f
Formfeed, Ctrl-l, ASCII code 12 (FF).
\n
Newline, Ctrl-j, ASCII code 10 (LF).
\r
Carriage return, Ctrl-m, ASCII code 13 (CR).
\t
Horizontal TAB, Ctrl-i, ASCII code 9 (HT).
\v
Vertical TAB, Ctrl-k, ASCII code 11 (VT).-
\nnn
The octal value nnn, where nnn stands for 1 to 3 digits between ‘0’ and ‘7’. For example, the code for the ASCII ESC (escape) character is ‘\033’.
15
It may also be helpful to note (though understandably you had no room to do so) that the -d option to readarray first appears in Bash 4.4. –
fbicknel
Aug 18, 2017 at 15:57
4
Great answer (+1). If you change your awk to awk '{ gsub(/,[ ]+|$/,"\0"); print }' ./ and eliminate that concatenation of the final ", " then you don't have to go through the gymnastics on eliminating the final record. So: readarray -td '' a < <(awk '{ gsub(/,[ ]+/,"\0"); print; }' <<<"$string") on Bash that supports readarray. Note your method is Bash 4.4+ I think because of the -d in readarray –
dawg
Nov 26, 2017 at 22:28
10
Wow, what a brilliant answer! Hee hee, my response: ditched the bash script and fired up python! –
artfulrobot
May 14, 2018 at 11:32
11
I'd move your right answers up to the top, I had to scroll through a lot of rubbish to find out how to do it properly :-) –
paxdiablo
Jan 9, 2020 at 12:31
44
This is exactly the kind of thing that will convince you to never code in bash. An astoundingly simple task that has 8 incorrect solutions. Btw, this is without a design constraint of, "Make it as obscure and finicky as possible"§$
END
HEREDOC
F_MS_STR_TO_SPLIT="${FILE_CONTENT:6:-3}"
F_MS_DELIMITER_P="int }' ./ and eliminate"
f_my_answer "$F_MS_STR_TO_SPLIT" "$F_MS_DELIMITER_P"
f_my_answer "$F_MS_STR_TO_SPLIT" "."
f_my_answer "$F_MS_STR_TO_SPLIT" "+"
f_my_answer "$F_MS_STR_TO_SPLIT" "'"
f_my_answer "$F_MS_STR_TO_SPLIT" "\\"
f_my_answer "$F_MS_STR_TO_SPLIT" "-"
f_my_answer "a.+b.+c" "[.][+]"
f_my_answer "a[.][+]b[.][+]c" "[.][+]"
f_my_answer "a.+b.+c" ".+"
There are many, many proposals for string splitting with bash commands, but almost all of them only work in specific cases and not accepts any string as input and as delimiter.
The function below, created by jhnc and modified by me, accepts any string as input and as delimiter. 😉
FUNCTION
declare -a F_MASTER_SPLITTER_R;
f_master_splitter(){
: 'Split a given string and returns an array.
Args:
F_MS_STR_TO_SPLIT (str): String to split.
F_MS_DELIMITER_P (Optional[str]): Delimiter used to split. If not informed
the split will be done by spaces.
Returns:
F_MASTER_SPLITTER_R (array): Array with the provided string separated by
the informed delimiter.
'
local F_MS_STR_TO_SPLIT="$1"
local F_MS_DELIMITER_P="$2"
if [ -z "$F_MS_DELIMITER_P" ] ; then
F_MS_DELIMITER_P=" "
fi
F_MASTER_SPLITTER_R=();
local F_MS_ITEM=""
while
F_MS_ITEM="${F_MS_STR_TO_SPLIT%%"$F_MS_DELIMITER_P"*}"
F_MASTER_SPLITTER_R+=("$F_MS_ITEM")
F_MS_STR_TO_SPLIT="${F_MS_STR_TO_SPLIT:${#F_MS_ITEM}}"
((${#F_MS_STR_TO_SPLIT}))
do
F_MS_STR_TO_SPLIT="${F_MS_STR_TO_SPLIT:${#2}}"
done
}
USAGE
f_master_splitter "<STR_INPUT>" "<STR_DELIMITER>"
NOTE: The f_master_splitter above was made available completely free as part of this project ez_i - Create shell script installers easily!.
TO TEST (MORE ELABORATE)
read -r -d '' FILE_CONTENT << 'HEREDOC'
BEGIN
§\\§[+][.][-]
A literal backslash, ‘\’.°
°\a
The “alert” character, Ctrl-g, ASCII code 7 (BEL). (This often makes some sort of audible noise.)
\b
Backspace, Ctrl-h, ASCII code 8 (BS).
\f
Formfeed, Ctrl-l, ASCII code 12 (FF).
\n
Newline, Ctrl-j, ASCII code 10 (LF).
\r
Carriage return, Ctrl-m, ASCII code 13 (CR).
\t
Horizontal TAB, Ctrl-i, ASCII code 9 (HT).
\v
Vertical TAB, Ctrl-k, ASCII code 11 (VT).-''%s
\nnn
The octal value nnn, where nnn stands for 1 to 3 digits between ‘0’ and ‘7’. For example, the code for the ASCII ESC (escape) character is ‘\033’.
15
It may also be helpful to note (though understandably you had no room to do so) that the -d option to readarray first appears in Bash 4.4. –
fbicknel
Aug 18, 2017 at 15:57
4
Great answer (+1). If you change your awk to awk '{ gsub(/,[ ]+|$/,"\0"); print }' %s./ \"and eliminate+.-°\a“\b\f\n\r\t\v\nnn‘’\033`` that concatenation of the final ", " then you don't have to go through the gymnastics on eliminating the final record. So: readarray -td '' a < <(awk '{ gsub(/,[ ]+/,"\0"); print; }' <<<"$string") on Bash that supports readarray. Note your method is Bash 4.4+ I think because of the -d in readarray –
dawg
Nov 26, 2017 at 22:28
10
Wow, what a brilliant answer! Hee hee, my response: ditched the bash script and fired up python! –
artfulrobot
May 14, 2018 at 11:32
11
I'd move your right answers up to the top, I had to scroll through a lot of rubbish to find out how to do it properly :-) –
paxdiablo
Jan 9, 2020 at 12:31
44
This is exactly the kind of thing that will convince you to never code in bash. An astoundingly simple task that has 8 incorrect solutions. Btw, this is without a design constraint of, "Make it as obscure and finicky as possible"§$
END
HEREDOC
F_MS_STR_TO_SPLIT="${FILE_CONTENT:6:-3}"
F_MS_DELIMITER_P="int }' %s./ \\\"and eliminate+.-°\a“\b\f\n\r\t\v\nnn‘’\033\`\`"
f_print_my_array() {
LENGTH=${#F_MASTER_SPLITTER_R[*]}
for ((i=0;i<=$(($LENGTH-1));i++)); do
echo ">>>>>>>>>>"
echo "${F_MASTER_SPLITTER_R[$i]}"
echo "<<<<<<<<<<"
done
}
echo ">>>>>>>>>>>>>>>>>>>>"
f_master_splitter "$F_MS_STR_TO_SPLIT" "$F_MS_DELIMITER_P"
f_print_my_array
echo "<<<<<<<<<<<<<<<<<<<<"
echo ">>>>>>>>>>>>>>>>>>>>"
f_master_splitter "$F_MS_STR_TO_SPLIT" "."
f_print_my_array
echo "<<<<<<<<<<<<<<<<<<<<"
echo ">>>>>>>>>>>>>>>>>>>>"
f_master_splitter "$F_MS_STR_TO_SPLIT" "+"
f_print_my_array
echo "<<<<<<<<<<<<<<<<<<<<"
echo ">>>>>>>>>>>>>>>>>>>>"
f_master_splitter "$F_MS_STR_TO_SPLIT" "'"
f_print_my_array
echo "<<<<<<<<<<<<<<<<<<<<"
echo ">>>>>>>>>>>>>>>>>>>>"
f_master_splitter "$F_MS_STR_TO_SPLIT" "\\"
f_print_my_array
echo "<<<<<<<<<<<<<<<<<<<<"
echo ">>>>>>>>>>>>>>>>>>>>"
f_master_splitter "$F_MS_STR_TO_SPLIT" "-"
f_print_my_array
echo "<<<<<<<<<<<<<<<<<<<<"
echo ">>>>>>>>>>>>>>>>>>>>"
f_master_splitter "a.+b.+c" "[.][+]"
f_print_my_array
echo "<<<<<<<<<<<<<<<<<<<<"
echo ">>>>>>>>>>>>>>>>>>>>"
f_master_splitter "a[.][+]b[.][+]c" "[.][+]"
f_print_my_array
echo "<<<<<<<<<<<<<<<<<<<<"
echo ">>>>>>>>>>>>>>>>>>>>"
f_master_splitter "a.+b.+c" ".+"
f_print_my_array
echo "<<<<<<<<<<<<<<<<<<<<"
A very special thanks to jhnc! You rock! 🤗
I've done my requisite 20 searches but I can't quite find an example that includes the 'ignore null' part of what I'm trying to do. Working on a Linux-ish system that uses bash and has grep/awk/sed/perl and the other usual suspects. Output from a job is in the format:
Some Field I Dont Care About = Nothing Interesting
Another Field That Doesnt Matter = 216
Name = The_Job_name
More Useless Stuff = Blah blah
Estimated Completion = Aug 13, 2015 13:30 EDT
Even Yet Still More Nonsense = Exciting value
...
Jobs not currently active will have a null value for estimated completion time. The field names are long, and multi-word names contain spaces as shown. The delimiter is always "=" and it always appears in the same column, padded with spaces on either side. There may be dozens of jobs listed, and there are about 36 fields for each job. At any given time there are only one or two active, and those are the ones I care about.
I am trying to get the value for the 'Name' field and the value of the 'Estimated Completion' field on a single line for each record that is currently active, hence ignoring nulls, like this:
Job_04 Aug 13, 2015 13:30 EDT
Job_21 Aug 09, 2015 10:10 EDT
...
I started with <command> | grep '^Name\|^Estimated' which got me the lines I care about.
I have moved on to awk -F"=" '/^Name|^Estimated/ {print $2}' which gets the values by themselves. This is where is starts to go awry - I tried to join every other line using awk -F"=" '/^Name|^Estimated/ {print $2}'| sed 'N;s/\n/ /' but the output from that is seriously wonky. Add to this I am not sure whether I should be looking for blank lines and eliminating them (and the preceding line) to get rid of the nulls at this point, or if it is better to read the values into variables and printf them.
I'm not a Perl guy, but if that would be a better approach I'd be happy to shift gears and go in that direction. Any thoughts or suggestions appreciated, Thanks!
Some Field I Dont Care About = Nothing Interesting
Another Field That Doesnt Matter = 216
Name = Job_4119
More Useless Stuff = Blah blah
Estimated Completion =
Even Yet Still More Nonsense = Exciting value
...
I can't comment, not enough reputation...
But I think something like this will work, in your print command
{printf "%s,",$2;next}{print;}
Or use paste command?
paste -s -d",\n" file
You can do something like:
awk -F"=" '/^Name/ {name=$2} /^Estimated/ { print name, $2}' file
if they always come in the same order: name first, estimate next.
You can then add a NULL check to the last field and don't print the line if it matches like:
awk -F"=" '/^Name/ {name=$2} /^Estimated/ { if($2 != "") {print name, $2}}' file
$ awk -F'\\s*=\\s*' '{a[$1]=$2} /^Estimated/ && $2{print a["Name"], $2}' file
The_Job_name Aug 13, 2015 13:30 EDT
Replace \\s with [[:space:]] if you aren't using gawk, i.e.:
$ awk -F'[[:space:]]*=[[:space:]]*' '{a[$1]=$2} /^Estimated/ && $2{print a["Name"], $2}' file
and if your awk doesn't even support character classes then GET A NEW AWK but in the meantime:
$ awk -F'[ \t]*=[ \t]*' '{a[$1]=$2} /^Estimated/ && $2{print a["Name"], $2}' file
I have a text file with the following format:
Wind River Linux glibc_cgl (cgl) 3.0.3
Build label: NDPGSN_5_0_SRC_GSN_LINUX_GPBOS_2
Build host: eselnvlx1114
Build date: Mon Mar 18 23:24:08 CET 2013
Installed: Fri Jun 20 02:22:08 EEST 2014
Last login: Fri Aug 8 11:37:08 2014 from 172
gsh list_imsins
=== sysadm#eqm01s14p2 ANCB ~ # gsh list_imsin
ps Class Identifiers |
---------------------------------------
A imsins -imsi 20201
A imsins -imsi 20205
A imsins -imsi 20210
A imsins -imsi 204045
I want to extract the numbers next to -imsi. The output would look like:
20201
20205
202210
204045
And after that process the output further, which I've already done. At first I was informed that the text format was static, so I wrote the following script:
for (( r=1; r<5; r++));
do
awk 'NR>12' IMSI$r.txt | awk '{print $NF "\r"}' > N$r
awk 'NR>12' IMSI$r.txt | awk '{print $NF "\r"}' >> out
done
I had 2 files as output because I needed to use both for other purposes.
Is there any way to make the script more flexible, to deal with dynamic text files?
As a possible solution, is it possible to make the script look for the phrase -imsi and grab the record after it? And continue doing so until it finds the end of file?
I tried doing that using grep and awk but I never got the right output. If you have any other ideas to do that please share.
I would go for something like:
$ awk '/-imsi/ {print $NF}' file
20201
20205
20210
204045
This prints the last word on those lines containing -imsi.
You can also use grep with a look-behind, to print the numbers after -imsi.
$ grep -Po '(?<=-imsi )[0-9]*' file
20201
20205
20210
204045
I've encountered some problem when doing the procmail recipe.
Here what I have get so far :
:0
* ^X-Loop: myemail#gmail\.com
/dev/null
:0
# filtering email by number 60
* ^Subject:.*(60)
{
:0c:
${DEFAULT}
#trying to take out input from the body
:0fb
| head -10
#Forward it to the other folder
:0
mytest/
}
The problem occur when procmail reading the body of the email.It will show output like this :
+96szV6aBDlD/F7vuiK8fUYVknMQPfPmPNikB+fdYLvbwsv9duz6HQaDuwhGn6dh9w2U
1sABcykpdyfWqWhLt5RzCqppYr5I4yCmB1CNOKwhlzI/w8Sx1QTzGT32G/ERTlbr91BM VmNQ==
MIME-Version: 1.0
Received: by 10.52.97.41 with SMTP id dx9mr14500007vdb.89.1337845760664; Thu,
24 May 2012 00:49:20 -0700 (PDT)
Received: by 10.52.34.75 with HTTP; Thu, 24 May 2012 00:49:20 -0700 (PDT)
Date: Thu, 24 May 2012 15:49:20 +0800
Message-ID: <CAE1Fe-r4Lid+YSgFTQdpsniE_wzeGjETWLLJJxat+HK94u1=AQ#mail.gmail.com>
Subject: 60136379500
From: my email <my email#gmail.com>
To: your email <your email#gmail.com>
Content-Type: multipart/alternative; boundary=20cf307f380654240604c0c37d07
--20cf307f380654240604c0c37d07
Content-Type: text/plain; charset=ISO-8859-1
hi
there
how
are
you
--20cf307f380654240604c0c37d07
+96szV6aBDlD/F7vuiK8fUYVknMQPfPmPNikB+fdYLvbwsv9duz6HQaDuwhGn6dh9w2U
1sABcykpdyfWqWhLt5RzCqppYr5I4yCmB1CNOKwhlzI/w8Sx1QTzGT32G/ERTlbr91BM VmNQ==
I have manage to get the output but it is not working if the sender send fewer than 3 lines as the output will print out the footer of the email as well (because it is between the range of head -10).
I only want the body of the email to be filter (print out in text file) in the procmail.
Is it possible?Can anyone show me the way?I'm in my wits ends.Thanks
Attempting to treat a MIME multipart as just a lump of text is fraught with peril. In order to properly process the body, you should use a MIME-aware tool. But if you just want to assume that the first part is a text part and drop all other parts, you can create something fairly simple and robust.
# Truncate everything after first body part:
# Change second occurrence of --$MATCH to --$MATCH--
# and trim anything after it
:0fb
* ^Content-type: multipart/[a-z]+; boundary="\/[^"]+
| sed -e "1,/^--$MATCH$/b" -e "/^--$MATCH$/!b" -e 's//&--/' -eq
For elegance points, you might be able to develop the script to implement your 10-line body truncation action at the same time, but at least, this should hopefully get you started. (I would switch to awk or Perl at this point.)
:0fb
* ^Content-type: multipart/[a-z]+; boundary="\/[^"]+
| awk -v "b=--$MATCH" ' \
($0 == b || $0 == b "--") && seen++ { printf "%s--\n", $0; exit } \
!seen || p++ < 10'
Properly, the MIME part's headers should not count towards the line count.
This is slightly speculative; I assume by "footer" you mean the ugly base64-encoded attachment after the first body part, and of course, this recipe will do nothing at all for single-part messages. Maybe you want to fall back to your original recipe for those.
Recently had a similar issue and solved it with this (adapted to OP)...
#trying to take out input from the body
:0fb
| sed -n '/^Content-Type/,/^--/ { /^Content-Type/b; /^--/b; p }'
Explanations: in general form....
sed -n '/begin/,/end/ { /begin/b; /end/b; p }'
-n: --> turn printing off
/begin/ --> begin of pattern range (remainder commands only apply inside range)
,/end/ --> , end of sed pattern range
{ /begin/b; --> /b branch causes lines with pattern /begin/ to skip remaining commands
/end/b; --> (same as above), these lines will skip the upcoming (p)rint command
p }' --> prints lines that in pattern that made it to this command
I have a program that accepts HTTP post of files and write all the POST result into a file, I want to write a script to delete the HTTP headers, only leave the binary file data, how to do it?
The file content is below (the data between Content-Type: application/octet-stream and ------------KM7cH2GI3cH2Ef1Ij5gL6GI3Ij5GI3 is what I want:
POST /?user_name=vvvvvvvv&size=837&file_name=logo.gif& HTTP/1.1^M
Accept: text/*^M
Content-Type: multipart/form-data; boundary=----------KM7cH2GI3cH2Ef1Ij5gL6GI3Ij5GI3^M
User-Agent: Shockwave Flash^M
Host: 192.168.0.198:9998^M
Content-Length: 1251^M
Connection: Keep-Alive^M
Cache-Control: no-cache^M
Cookie: cb_fullname=ddddddd; cb_user_name=cdc^M
^M
------------KM7cH2GI3cH2Ef1Ij5gL6GI3Ij5GI3^M
Content-Disposition: form-data; name="Filename"^M
^M
logo.gif^M
------------KM7cH2GI3cH2Ef1Ij5gL6GI3Ij5GI3^M
Content-Disposition: form-data; name="Filedata"; filename="logo.gif"^M
Content-Type: application/octet-stream^M
^M
GIF89an^#I^^M
------------KM7cH2GI3cH2Ef1Ij5gL6GI3Ij5GI3^M
Content-Disposition: form-data; name="Upload"^M
^M
Submit Query^M
------------KM7cH2GI3cH2Ef1Ij5gL6GI3Ij5GI3-
You want to do this as the file is going over, or is this something you want to do after the file comes over?
Almost any scripting language should work. My AWK is a bit rusty, but...
awk '/^Content-Type: application\/octet-stream/,/^--------/'
That should print everything between application/octet-stream and the ---------- lines. It might also include both those lines too which means you'll have to do something a bit more complex:
BEGIN {state = 0}
{
if ($0 ~ /^------------/) {
state = 0;
}
if (state == 1) {
print $0
}
if ($0 ~ /^Content-Type: application\/octet-stream/) {
state = 1;
}
}
The application\/octet-stream line is after the print statement because you want to set state to 1 after you see application/octet-stream.
Of course, being Unix, you could pipe the output of your program through awk and then save the file.
If you use Python, email.parser.Parser will allow you to parse a multipart MIME document.
This may be a crazy idea, but I would try stripping the headers with procmail.
Look at the Mime::Tools suite for Perl. It has a rich set of classes; I’m sure you could put something together in just a few lines.
This probably contains some typos or something, but bear with me anyway. First determine the boundary (input is the file containing the data - pipe if necessary):
boundary=`grep '^Content-Type: multipart/form-data; boundary=' input|sed 's/.*boundary=//'`
Then filter the Filedata part:
fd='Content-Disposition: form-data; name="Filedata"'
sed -n "/$fd/,/$boundary/p"
The last part is filter a few extra lines - header lines before and including the empty line and the boundary itself, so change the last line from previous to:
sed -n "/$fd/,/$boundary/p" | sed '1,/^$/d' | sed '$d'
sed -n "/$fd/,/$boundary/p" filters the lines between the Filedata header and the boundary (inclusive),
sed '1,/^$/d' is deleting everything up to and including the first line (so removes the headers) and
sed '$d' removes the last line (the boundary).
After this, you wait for Dennis (see comments) to optimize it and you get this:
sed "1,/$fd/d;/^$/d;/$boundary/,$d"
Now that you've come here, scratch all this and do what Ignacio suggested. Reason - this probably won't work (reliably) for this, as GIF is binary data.
Ah, it was a good exercise! Anyway, for the lovers of sed, here's the excellent page:
http://sed.sourceforge.net/sed1line.txt
Outstanding information.