Real-time monitoring of intermittent binary data - linux

Question

Context: monitor a low-volume, intermittent stream from a program
When debugging some program, one sometimes have to monitor some output. When output is ascii, no problem, just run in terminal (the program itself, or nc if it uses a TCP or UDP interface, or cat /dev/somedevice, or socat ..., whatever).
Need: monitor a binary stream real-time... and half-solutions
But sometimes output is binary. One can pipe it into various incantations of od, hd, e.g. od -t d1 for decimal numbers, od -t a1 for augmented ascii display (explicit non-printable characters using printable ones), etc.
The trouble is: those buffer the input until they have a complete line to print (a line often fitting 16 input characters). So basically until the program sends 16 character, the monitoring does not show anything. When the stream is low volume and/or intermittent, this defeats the purpose of a real-time monitoring. Many protocols indeed only send a handful of bytes at a time.
It would be nice to tell it "ok buffer the input if you wish, but don't wait more than delay x before printing it, even if it won't fill one line".
man od, man hd don't mention any related option.
Non-solution
Heavy programs like wireshark are not really an option: they cover only part of the needs and are not combinable. I often do things like this:
{ while read a ; do { echo -n -e 'something' ; } | tee >(od -t d1 >&2) ; done ; } | socat unix-connect:/somesocket stdio | od -t d1
This monitors the output, and each time I press enter in the terminal, injects the sequence "something". It works very well but terminal output is buffered by 16-byte chunks and thus delayed a lot.
Summary
How do you simply monitor binary output from a program without byte-alignment-dependent delay ?

Answer 1

I don't know which distribution you're using, but check to see whether you have, or can install, most. From the manpage:
OPTIONS
-b Binary mode. Use this switch when you want
to view files containing 8 bit characters.
most will display the file 16 bytes per line
in hexadecimal notation. A typical line
looks like:
01000000 40001575 9C23A020 4000168D ....#..u.#. #...
When used with the -v option, the same line
looks like:
^A^#^#^# #^#^U u 9C #A0 #^#^V8D ....#..u.#. #...
Not in the manpage, but essential for your task, is the keystroke F (N.B. upper-case), which puts most into 'tail mode'. In this mode, most updates whenever new input is present.
On the downside, most can't be told to begin in tail mode, so you can't just pipe to its stdin (it will try to read it all before showing anything). So you'll need to
<your_command> >/tmp/output
in the background, or in its own terminal, as appropriate. Then
most -b /tmp/output
and press F.

Answer 2

Here's the best thing I found so far, hope someone knows something better.
Use -w1 option in od, or one of the examples format strings in man hd that eats one byte at a time. Kind of works, though makes a column-based display, which does not use terminal area efficiently.
{ while read a ; do { echo -n -e 'something' ; } | tee >(od -t d1 >&2) ; done ; } | socat unix-connect:/somesocket stdio | hexdump -v -e '/1 "%_ad# "' -e '/1 " _%_u\_\n"'
This displays thing like this:
0# _nul_ (plenty of wasted space on the right... )
1# _1_ (no overview of what's happening...)
2# _R_
3# _ _
4# _+_
5# _D_
6# _w_
7# _d8_
8# _ht_
9# _nak_
The good thing is one can configure it to their context and taste:
{ while read a ; do { echo -n -e 'something' ; } | tee >(od -t d1 >&2) ; done ; } | socat unix-connect:/somesocket stdio | hexdump -v -e '/1 "%_ad# "' -e '/1 "%02X "' -e '/1 " _%_u\_\n"'
This displays thing like this:
0# 6B _k_
1# 21 _!_
2# F6 _f6_
3# 7D _}_
4# 07 _bel_
5# 07 _bel_
6# 60 _`_
7# CA _ca_
8# CC _cc_
9# AB _ab_
But still, a regular hd display would use screen area more efficiently:
hd
00000000 f2 76 5d 82 db b6 88 1b 43 bf dd ab 53 cb e9 19 |.v].....C...S...|
00000010 3b a8 12 01 3c 3b 7a 18 b1 c0 ef 76 ce 28 01 07 |;...<;z....v.(..|