Is it possible to do simple arithmetic in sed addresses? - linux

Question

Is it possible to do simple arithmetic in sed addresses?
Judging by the "addresses" manual section, the answer seems no. But maybe there is a workaround?
For example, how can I print the second last line of a file? It would be cool something like:
sed -n '$-1 p' file
But it obviously does not work... so I usually have to do multiple sed calls, first for identifying the line, then do the arithmetic using the shell $((expr)) and then finally call sed again. Like this:
sed -n "$(($(sed -n '$ =' file)-1)) p" file
Is there a "better", more compact, more readable way for doing arithmetics with sed addresses?
In a serious moment of procrastination, I decided to write a small script that quickly changes the xterm colorscheme. The idea is that you have the .Xresources a file with a start marker and an end marker:
...
START_MARKER
...
END_MARKER
...
and you want to delete everything that is between the markers, but not the markers themselves. Again, it would be great to do something like:
sed '/START_MARKER/+1,/END_MARKER/-1 d' file
...but you can't!

Answer 1

You're right, one can't directly do math in sed1, even addresses. But you can use some trickery to do what you want:
Second-last row:
$ seq 5 | sed -n -e '${ # On the last line
> g # Replace the buffer with the hold space
> p # and print it
> }
> h' # All lines, store the current line in the hold space.
4
Between START and END:
$ cat test.in
1
START
2
3
END
4
$ cat test.in | sed '/^START$/,/^END$/{
> /^START$/d
> /^END$/d
> p
> }
> d'
2
3
$ cat test.in | sed -n -e '/^START$/,/^END$/!d' -e '/^START/d' -e '/^END$/d' -e p
2
3
I'm using a BSD (mac) sed; on GNU systems you can use ; between lines instead of a newline. Or stick it in a script.
1: Sed is Turing complete, so you can do math, but it's unwieldy at best: http://rosettacode.org/wiki/A%2BB#sed
Yes, I know, UUOC; it's for illustration only

Answer 2

Delete the second last line:
sed ':r;$!{N;br};s/\n[^\n]*\(\n[^\n]*\)$/\1/' file
Delete everything inside markers:
sed ':r;$!{N;br};s/START_MARKER.*END_MARKER/START_MARKER\nEND_MARKER/' file
Far from being elegant, but kinda works.
As it was mentioned in the comments, sed operates on lines. However, you can read another line into the pattern space with N command. The two lines will now both be in the pattern space and will be separated with a \n. sed also has means of execution flow control, namely labels and conditional/unconditional branches. Everything is documented in man sed, also here is a full reference with examples. In the code above r is a label; $!{..} means "everywhere except last line, do ..; N;br reads another line and branches unconditionally to r again. So with :r;$!{N;br} you read all the input into the pattern space and then you operate on it as a single line with \n separating lines of the input.

Answer 3

This might work for you (GNU sed);
sed '$!N;$s/.*\n//;P;D' file
and this works and should be easy to understand:
sed '/start/,/end/!d;//d' file
These are solutions to your questions but as for arithmetic best use awk or perl.

Answer 4

You have some good sed suggestions, here's one based on GNU awk:
awk -v RS='START_MARKER|END_MARKER' 'RT == "END_MARKER"' infile
RS='START_MARKER|END_MARKER' splits input with the markers as separators.
RT is set to the matched separator, when it matches "END" the default block {print $0} is executed.
So for example if you wanted to print all but the last three lines, set FS to \n and apply the appropriate loop:
awk -v RS='START_MARKER|END_MARKER' -v FS='\n' 'RT == "END" { for(i=1; i<NF-3; i++) print $i }' infile

Answer 5

You can use simple method to show second last line of the file.
TOTAL_LENGTH=$(cat file_name | wc -l)
SECOND_LAST_LINE=`expr $TOTAL_LENGTH - 1`
head -$SECOND_LAST_LINE | tail -1
If you want to delete the second last line from the file:
sed -i "$SECOND_LAST_LINE"d file_name

Answer 6

A more comprehensive treatment for doing arithmetic in sedis given in solution #2. An introduction to using sed to `sed' its own script is here.
As the brain pain strain incurred in solution #2 from the quixotic comment demands of too much "hand waving" actually is too much "hand waving" of code, in juxtaposition, this is solution #3:
echo -e 'a\nb\nc\nd\ne' | sed -n '1!G;h;$p' | sed -n 3p
which still uses piping ("But maybe there is a workaround?"), where the numeral 3 must be replaced "by hand" for the desired line from the end of the file ala $-3.
Suppose the sed script is '$-4 p; $-6p; $-8 p;'
echo -e 'a\nb\nc\nd\ne\nf\ng\nh\ni' |
sed -n '1!G;h;$p' |
sed -n '4 p; 6p; 8 p;' |
sed -n '1!G;h;$p'
does the job via
echo '$-4 p; $-6p; $-8 p;' | sed s/$-//
Caveats:
The sed commands must be as simple as print.
The "simple arithmetic" can only be of the form '$-n'.
The arithmetic is not calculated "normally".
A "single" 'sed' command string (a "line" if the previous piping is considered as such) would embed and combine these two commands as outlined in the next answer #2.
The coup de grâce.
Given the perfunctory dismissal of the first answer here is #2:
As this is only the 2nd or 3rd time writing a substantial sed script, serious syntax subtlety (s)circumvention scuppering solutions seemed sufficient: ala
# file prep
echo -e ' a\n b\n c\n d\n e\n f' >test
The following strikeout is not incorrect but after playing and "messing about" with sed with an SO problem over here the sed execute can be simpler w/o IO redirection if run from the pattern buffer to get the file length line count $ via:
sed -e '1{h; s/.*/sed -n "$=" test /e' -e 'p;x}; ${p;x;}' test
The $= enumeration is held in the hold buffer from the get go and printed again at the end.
# get "sed -n $= test" command output into sed script
sed -n '1esed -n "$=" test >sedr' test
# see where this is headed? so far "sed -n ... test" is irrelevant
# a pedantic "sed" only solution would keep it this way with
# all the required "sed"'ng as part of an 'e' command or '$e'
# where the 'sedr' file is itself "sed"'d ultimately to a final
# command 'sed -n /<the calculated line number>/p'
# one could quibble whether '>sedr' io redirection is "pure sed"
# modify 'sedr'with [the sed RPN][1] to get <the calculated line number>
# with judicious use of "sed"'s 'r' command and buffering will
# realize the effective script to compute the desired result
# this is left as an exercise needing perverse persistence with
# a certain amount of masochistic agony
As a hint as to how to proceed; using the technique of solution #3 the sed script $- addresses are now replaced by the $= value and -. So sed is again used to edit its own script.
Parsing the sed script must accurately modify just the $- in addresses only.
Also, to use the RPN calculator the infix arithmetic must have post fixed operators. It is a conventional paradigm in theories of automata and formal languages to convert Polish Notation or its Reverse to Infix and vice versa.
Hopefully, this establishes the answer in the afirmative that it can be done (mais, pas par moi) and in the negative that it is not a trivial exercice (c'est par moi).
Excruciating rationale for an arbitrary solution is at the end.
The environment used for the empirical tests:
linuxuser#ubuntu:~$ sed --version
sed (GNU sed) 4.4
Copyright (C) 2017 Free Software Foundation, Inc.
linuxuser#ubuntu:~$ uname -a
Linux ubuntu 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:00 UTC 2019 i686 i686 i686 GNU/Linux
linuxuser#ubuntu:~$ lsbname -a
lsbname: command not found
linuxuser#ubuntu:~$ apropos lsb
lsb_release (1) - print distribution-specific information
lsblk (8) - list block devices
linuxuser#ubuntu:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic
Solution #1
A technique thinking outside inside the box:
seq 60 | sed -n '$!p' | sed -n '$!p' | sed -n '$!p' | sed -n '$p'
which prints:
57
specifically, for the second last line:
sed -n '$!p' file | sed -n '$p'
More generally, a script can iterate over sed -n '$!p' to "count backwards" from the end of a file.
Well, the answer to:
Is it possible to do simple arithmetic in sed addresses?
rhetorically is, it depends on one's abilities, wishes and desires as well as a realistic assessment of practicality. As well, the implication is that a single sed invocation should be used for this task exclusively. But yes it is possible.
A firm grounding in the studies of Automata, Formal Languages and Recursive Function Theory does not hurt.
As stated in previous answers: Not only can sed do simple arithmetic but also any computable function which includes complex arithmetic. To do so however requires implementing the Primitive Recursive Functions (PRF) (which of course sed does) of Recursive Function Theory (RFT). Of course the finite size of machine architecture does limit the computation without infinite tape resources as a Turing machine proves. In any case not wishing to demonstrate this the precedents are to be found in the sed manual.
Specifically, to do arithmetic (finitely) an RPN calculator:
https://www.gnu.org/software/sed/manual/html_node/Increment-a-number.html#FOOT9
Now then, using such a tool can create a sed script that precomputes the arithmetic that is then embedded in a sed script to print the desired output. A simple demonstration is given by the OP noting now that the shell arithmetic computation can be done using the RPN sed script.
This reduces to a form such as (very crude)
sed '/$(sed RPN($= - 3*4) file)/;p;' file
but still requires feeding sed a sed'd script.
Also, there is arguably the quibbling over the use of bash $() but it could be argued bash is already used to execute the first 'sed' so no harm no foul.
Recognizing that sed implements the PRF or equivalently is Turing complete means that yes, a single invocation of sed is adequate.
The paradigm can therefore do this.
Some commands that could expedite this task are:
e, e command, r, R, w, W
in addition to the usual hold and pattern buffer commands.
The r, R, w, W commands are particularly advantageous as temporary buffer space.
e [command] [3.7 Commands Specific to GNU sed][2]
This command allows one to pipe input from a shell command into
pattern space. Without parameters, the e command executes the
command that is found in pattern space ...
More abstractly, it is completely possible, though highly impractical, to write a sed script to execute the sed paradigm itself that also includes arithmetic calculations even in addresses.
A sed peculiarity. The expression /\n/ will not match any address and matches in pattern space only if a sed command like 'N'ext or s/.*/\n/ introduces one.
Confirmed via:
echo -e '\n\n' | sed -n ' /\n/ {s//hello/;p}'
But
echo -e '\n\n' | sed -n '0,/\n\n\n/ {s//hello/;p}'
outputs 3 blank lines and
echo -e '\n\n' | sed -n '0,/\n/ {s/.*/hello/;p}'
echo -e '\n\n' | sed -n '0,/\n\n\n/ {s/.*/hello/;p}'
each output 3 hello's
hello
hello
hello
while this is well-behaved:
echo -e '\n\n' | sed -n '0,/^$/ {s//hello/;p}'