merge sorted files, with minimal buffering - linux

Question

I have two log files that are prefixed with a sortable timesetamp.
I'd like to see them, in order, while the processes generating the log files are still running. This is a pretty faithful simulation of the situation:
slow() {
# print stdout at 30bps
exec pv -qL 30
}
timestamp() {
# prefix stdin with a sortable timestamp
exec tai64n
}
# Simulate two slowly-running batch jobs:
seq 000 099 | slow | timestamp > seq.1 &
seq1=$!
seq 100 199 | slow | timestamp > seq.2 &
seq2=$!
# I'd like to see the combined output of those two logs, in timestamp-sorted order
try1() {
# this shows me the output as soon as it's available,
# but it's badly interleaved and not necessarily in order
tail -f seq.1 --pid=$seq1 &
tail -f seq.2 --pid=$seq2 &
}
try2() {
# this gives the correct output,
# but outputs nothing till both jobs have stopped
sort -sm <(tail -f seq.1 --pid=$seq1) <(tail -f seq.2 --pid=$seq2)
}
try2
wait

Answer 1

The solution using tee (to write the files so that the standard output still goes to the console) won't work because tee introduces unwanted latency and doesn't solve the problem. Similarly, I couldn't get solutions using tail -f -s 0.01 (which alters the polling to 100/s), and/or some kind of call like split --filter='sort -sm' to sort small batches.
I also don't have tai64n, so my test code actually used this functionally identical perl code:
tai64n() {
perl -MTime::HiRes=time -pe '
printf "\#4%015x%x%n", split(/\./,time), $c; print 0 x(25-$c) . " "'
}
After failing to solve this with sh and bash, I exercised my standard failover, perl:
slow() {
# print stdout at 30bps
pv -qL 30
}
tai64n_and_tee() {
# prefix stdin with a sortable timestamp and copy to given file
perl -MTime::HiRes=time -e '
$_ = shift;
open(TEE, "> $_") or die $!;
while (<>) {
$_ = sprintf("\#4%015x%x%n", split(/\./,time), $c) . 0 x(25-$c) . " $_";
print TEE $_;
print $_;
}
' "$1"
}
# Simulate two slowly-running batch jobs:
seq 000 099 | slow | tai64n_and_tee seq.1 &
seq 100 199 | slow | tai64n_and_tee seq.2 &
wait
This was convenient for me because I had already used perl for the timestamp. I failed to do this with perl acting as tai64n and a separate perl call to act as tee, but it might work with the real tai64n.