extracting location specific record using sed based on given criteria using bash script - linux

Question

I want to extract data from a ASCII file that looks like the one provided here the block starting 1NAME. The block starting 1NAME can repeat any number of times - I have files where there is only one block and in some files as many as 744:
AVERAGE MODELNAME -- RUNNAME
0 1 11121 0. 11122 24.
-9700000 4000000 0 -241200000000 -1620000
1.00000 1000.00000 10 10 1 2 0 15. 11. 0.
1 1 500 400
NAME
11121 0.00 11121 1.00
1NAME
0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00
0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00
0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00
NAME
11121 1.00 11121 2.00
1NAME
1.0000000E+00 45.0000000E+00 01.0000000E+00 115.0000000E+00 5.0000000E+00
2.0000000E+00 66.0000000E+00 09.0000000E+00 180.0000000E+00 4.0000000E+00
3.0000000E+00 80.0000000E+00 70.0000000E+00 130.0000000E+00 5.0000000E+00
I would like to extract values from (1) given recurring location in the file, starting after "1NAME", (2) pipe outputs to a text file and create header that identifies what location it was pulled from and (3) create a custom code that can take in input for multiple locations (say record 1, 5, 8) after 1NAME and output them into separate outputs (example: one output for all records at location 1, one output file for location 5, ...).
As an example say I want to grab record 1, 5 and 8 after 1NAME in the given input file. Outputs for each record should be output as follows in separate record specific text file labeled as GRID#.txt:
GRID 1
0.0000000E+00
00.0000000E+00
GRID 5
0.0000000E+00
5.0000000E+00
GRID 8
0.0000000E+00
09.0000000E+00
I was able to extract data on at a time using sed. However I need to extract data from multiple locations from the input file. So I tried to put all the information in a script. Following are the steps I took.
Input file has multiple whitespaces and inconsistent blank lines. So I used sed to remove multiple spaces and replace with one space. And then using the piped output from this step, removed all blank lines. This resulted in all the data in the file arranged as one value per row.
sed 's/\s\+/\n/g' <input.txt>| sed '/^$/d
To extract data, I then used sed command (format as follows) from piped output from step 1.
sed -n -e 11p -e 50p
I tried to put all these commands as a bash (or csh, either option) script with custom row number. I tried using foreach (naively) and then learned that it cannot be used within bash. I will be using fellow user recommended scripts instead.
#!/bin/bash
set FILE=$cwd/sample_or_2day
foreach GRID (23729)
foreach GRIDTIME(28 41)
sed 's/\s\+/\n/g' $FILE | sed '/^$/d' | sed '1,36d' > temp_out
sed -n -e "$GRIDTIME" temp_out | tee $cwd/out_$GRID
Thanks for your patience. I am a nervous programmer and trying to master basics. I spent time looking at sed instruction pages, and user support forums. Any recommendations are welcome - especially with explicit instructions. Thanks!

Answer 1

You provided your attempt at a csh script, but tagged your question as bash. I am answering with a bash script.
The core of your question is how to extract information from a formatted printout. In general, one should avoid such situations: one should use programming environments which are aware of the data structures being manipulated, in order to avoid re-parsing at each step. In the real world, however, such situations arise very often, and one has to cope with them.
Your approach to transform all white spaces into newlines works in your case. Instead of multiple sed commands, the fastest way to achieve it is by
tr -s ' ' '\n'
(the -s option squeezes multiple occurrences of the target character into one, eliminating blank lines)
Then, you are interested in the 7th and 14th lines after each occurrence of a line containing 1NAME. This is done in sed by
sed -n -e '/^1NAME$/{n;n;n;n;n;n;n;p;n;n;n;n;n;n;n;p}'
which means: when you see 1NAME, execute the nextline command seven times, then execute the print command. Do this twice.
You could use a shell variable:
next7='n;n;n;n;n;n;n;p'
And
cat ./sample_or_2day | tr -s ' ' '\n' | sed -n -e '/^1NAME$/'"{$next7;$next7}"
would produce
0.0000000E+00
0.0000000E+00
66.0000000E+00
130.0000000E+00
correct, the first block was also taken in. To skip it, let’s add the sed instruction you had already figured out, -e1,36d.
$ cat ./sample_or_2day | tr -s ' ' '\n' | sed -n -e1,36d -e'/^1NAME$/'"{$next7;$next7}"
66.0000000E+00
130.0000000E+00
You might also want bash to construct the sed command line for you: for instance, the command
sed -n -e{7..29..7}p
would be expanded by the shell as
sed -n -e7p -e14p -e21p -e28p
which, as you know, means that sed would print those input lines only.
You might also want to learn about for loops in bash, which come in two different flavors, for example:
for var in word1 word2 word3 ...; do ... ; done
for (( i=0; i<10; i++ )); do ...; done
Now, it is not clear to me how you want to manage your output files. I provide a bash version of your script (providing a list of values for GRID, instead of just one), which shows another possible brace expansion in bash.
#!/bin/bash
FILE=./sample_or_2day
for GRID in 23729 23755 23768; do
cat "$FILE" | tr -s ' ' '\n' | sed -n -e{28,41}p >> "./out_$GRID"
done

Answer 2

This is what has worked for me so far, but not as a bash script:
sed 's/\s\+/\n/g' ./sample_or_2day | sed '/^$/d' | sed '1,36d'| sed -n -e{23724..194842..97421}p > './out'
In the above script:
sed 's/\s+/\n/g' -> replaces multiple whitespaces with one space
sed '/^$/d' -> deletes blank line from piped output
sed '1,36d' -> removes line 1-36 from piped output
sed -n -e{23724..194842..97421}p -> prints record starting line
23724 and at intervals of 97421, until line 194842
'./out' -> outputs to file labeled out

Answer 3

If you're open to adding a python dependency, you may find this'll help:
http://stromberg.dnsalias.org/~strombrg/context-split.html
Or try awk, replacing /^BEGIN/ and /^END/ with your own regex's:
#!/bin/sh
awk '
BEGIN { show=0 }
/^END/ { show=0 }
{ if (show==1) print $0 }
/^BEGIN/ { show=1 }' $#