Run MacOS sed on Linux 64 using nix-shell or nix shell - linux

Question

Answering sed questions on SO I often come across the problem of "works on linux but not MacOS" is it possible to load a version of sed that runs on MacOS into a nix shell?
Currently using flakes I can obtain the latest gnused using:
nix shell nixpkgs#gnused
Is there a way to temporarily install a BSD/MacOS version of sed?

Answer 1

Not so long ago I looked into this problem; perhaps you can make use
of what I found.
Web searches show little demand for FreeBSD sed outside BSD so I
decided to get it (well, GET it) from
github
and build and test it on a recent Debian system. To do so I created
3 files (listed below):
adapt/local.c - support for non-GNU functions
adapt/local.h - ditto header file
Makefile - downloads, generates required files, runs 1 or 2 of
the test suites; refer to comments in header and near target test
On my system the executable builds without errors. To test it using
tests/multi_test.sh I ran
make test | tee multi_test.log | grep '^not ok' | tee multi_test.err.log
with 127 of 130 tests succeeding and 3 failing:
not ok 69 7.1 # Print and file routines
not ok 75 7.7 # w results
not ok 97 8.21 # \ in y command
These discrepancies, I think, are no more than what can be expected: #69
is triggered by differences in output by sed -n l, #75 by differences
in /usr/share/dict/words contents (testcase of limited portability),
and #97 goes away if using SHELL := /bin/bash in the makefile or using
printf '%s\n' 'a\b(c' instead of echo 'a\b(c' in the test script.
UPDATE 2021-11-24: #69 goes away with e.g.
LANG=en make test | grep '^not ok', causing the locale-dependent
library function iswprint()
in process.c#lputs() to return zero for characters 0xA0..0xFF. #75
can only succeed if the first 200 lines of /usr/share/dict/words are
identical to those used by the testcase author; I fail to see the
reasoning behind this. (end UPDATE)
Still on the to-do list:
Makefile's *.names variables are hard-coded, better to create
them dynamically; however, the names seem to change rarely for a
program with sed's history
support for the ATF
tests by tests/sed2_test.sh (currently ignored)
support for embedded ident strings in the executable (__FBSDID
macro currently ignored) rather than rely on a date in the man page for
version
info
./Makefile
Note that recipes aren't prefixed with the usual tab character here
but with > (at beginning of line) acting as a make operator
(.RECIPEPREFIX = >).
# desc:
# Download, build, test FreeBSD sed (dated 2020-06-10) on GNU/Linux
# compat:
# dash 0.5.10 GNU make 4.2.1 GNU wget 1.20 GNU gcc 9.3.0 man 2.9
# ref:
# https://www.freebsd.org/cgi/man.cgi?sed
# https://github.com/freebsd/freebsd-src/tree/main/usr.bin/sed
# https://github.com/joshuarubin/wcwidth9
# files:
# Makefile adapt/local.c adapt/local.h
# howto:
# make download
# make all
# (optional) make download.tests download.regress.multitest.out test
# (optional) cp $(exe) /usr/local/bin/bsdsed
# (optional) cp $(man.1) /usr/local/man/man1/bsdsed.1
# note:
# Mind the $(wgetFlags) and $(CFLAGS)
SHELL := /bin/sh
wgetFlags ?= --no-verbose --wait=1
# $(call wgetCmd,subtarget-name)
define wgetCmd =
wget $(wgetFlags) --no-host-directories --directory-prefix=$($1.ldir) \
-- $(addprefix $($1.url),$($1.names))
endef
#
hdrs := defs.h extern.h
srcs := misc.c compile.c process.c main.c
hdrx := wcwidth9.h
srcx := local.c
objs := $(patsubst %.c,%.o,$(srcs) $(srcx))
# test scripts expect an executable named 'sed'
exe := sed
man.1 := $(exe).1
man.ps := $(exe).ps
binaries := $(exe) $(man.1) $(man.ps)
#
subtargets := wcwidth9 sed tests regress.multitest.out
dnldtargets := $(addprefix download.,$(subtargets))
#
wcwidth9.url := https://github.com/joshuarubin/wcwidth9/raw/master/
wcwidth9.names := $(hdrx)
wcwidth9.ldir := ./
#
sed.url := https://github.com/freebsd/freebsd-src/raw/master/usr.bin/sed/
sed.names := POSIX sed.1 $(srcs) $(hdrs)
sed.ldir := ./
tests.url := $(sed.url)tests/
tests.names := \
hanoi.sed inplace_race_test.sh legacy_test.sh math.sed \
multi_test.sh regress.G.out regress.P.out regress.b2a.out \
regress.bcb.out regress.c0.out regress.c1.out regress.c2.out \
regress.c3.out regress.hanoi.out regress.icase1.out \
regress.icase2.out regress.icase3.out regress.icase4.out \
regress.in regress.math.out regress.not.out regress.psl.out \
regress.s3.out regress.s4.out regress.s5.out regress.sg.out \
regress.sh regress.y.out sed2_test.sh
tests.ldir := ./tests/
regress.multitest.out.url := $(sed.url)tests/regress.multitest.out/
regress.multitest.out.names := \
1.1 1.2 1.3 1.4 1.4.1 1.5 1.6 1.7 1.8 1.9 1.10 1.11 1.12 1.13 \
1.14 1.15 1.16 1.17 1.18 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 \
2.10 2.11 2.12 2.13 2.14 2.15 2.16 2.17 2.18 2.19 2.20 2.21 \
2.22 2.23 3.1 3.2 3.3 3.4 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 5.1 \
5.2 5.3 5.4 5.5 5.6 5.7 5.8 6.1 6.2 6.3 6.4 6.5 6.6 7.1 7.2 \
7.3 7.4 7.5 7.6 7.7 7.8 8.1 8.2 8.3 8.4 8.5 8.6 8.7 8.8 8.9 \
8.10 8.11 8.12 8.13 8.14 8.15 8.16 8.17 8.18 8.19 8.20 8.21 \
8.22 8.23 9.1 9.2 9.3 9.4 9.5 9.6 9.7 9.8 9.9 9.10 9.11 9.12 \
9.13 9.14 9.15 9.16 9.17 9.18 9.19 9.20 9.21 9.22 9.23 9.24 \
9.25 9.26 9.27 9.28 9.29 9.30 9.31
regress.multitest.out.ldir := ./tests/regress.multitest.out/
.RECIPEPREFIX = >
.DELETE_ON_ERROR:
.PHONY: all clean realclean
all : $(binaries)
clean : ; rm -f -- $(objs) $(binaries)
realclean : clean
> rm -f -- local.c manpage.1 $(foreach T,$(subtargets),$(addprefix $($(T).ldir),$($(T).names)))
> rmdir --ignore-fail-on-non-empty -- \
$(patsubst %/,%,$(foreach T,$(subtargets),$(filter-out %. %./,$($(T).ldir))))
# notonbsd: enable modifications in extern.h and $(srcx)
# __FBSDID: don't embed RCS ID
$(objs) : CFLAGS += -Dnotonbsd -D__FBSDID\(s\)=
$(objs) : $(srcs) $(hdrs)
extern.h : adapt/local.h
> grep -q '^.ifdef\s*notonbsd' $# || { cat $# $< > $#.tmp && mv -f -- $#.tmp $# ; }
local.c : adapt/local.c
> cp $< $#
local.o : $(hdrx)
main.o : CFLAGS += -D__unreachable=__builtin_unreachable
$(exe) : $(objs)
> $(LINK.c) -o $# $^
> $(if $(DEBUG),,strip $#)
$(man.1) : manpage.1
> cp $< $#
%.ps : %.1
> man -l -t $< > $#
.PHONY : download.all $(dnldtargets) download
download.all : $(dnldtargets)
$(dnldtargets) :
> $(call wgetCmd,$(patsubst download.%,%,$#))
download : download.wcwidth9 download.sed
> mv -f sed.1 manpage.1
> touch adapt/local.h
.PHONY: test
# ! run after: make download.tests download.regress.multitest.out
# ! hint: make test | tee multi_test.log | grep '^not ok' | tee multi_test.err.log
# - set PATH so scripts invoke the new sed executable
# - run multi_test.sh (requires /usr/share/dict/words regress.multitest.out/*)
# - optionally run inplace_race_test.sh
# - ignore legacy_test.sh regress.* (require m4 regress.m4 hanoi.sed math.sed)
# - ignore sed2_test.sh (requires ATF, cf. https://github.com/freebsd/freebsd-src/tree/main/contrib/atf)
test : | /usr/share/dict/words
> cd $(patsubst %/,%,$(tests.ldir)); \
PATH="..:$$PATH"; \
$(SHELL) multi_test.sh \
$(if $(racetest),; $(SHELL) inplace_race_test.sh && rm -f file[0-9] file[0-9].prev)
./adapt/local.c
/* Support for non-GNU functions, cf. man.freebsd.org */
#ifdef notonbsd
#include <err.h>
#include <limits.h>
#include <regex.h>
#include <stdio.h>
#include <stddef.h>
#include <string.h>
#include <stdarg.h>
#include <errno.h>
#include "defs.h"
#include "extern.h"
#include "wcwidth9.h" /* https://github.com/joshuarubin/wcwidth9 */
void
errc(int eval, int code, const char *fmt, ...)
{
va_list args;
va_start(args, fmt);
errno = code;
err(eval, fmt, args);
va_end(args);
}
char*
getprogname()
{
return (program_invocation_short_name);
}
int
wcwidth(wchar_t wc)
{
return wcwidth9(wc);
}
size_t
strlcpy(char *dst, const char *src, size_t dstsize)
{
return snprintf(dst, dstsize, "%s", src);
}
size_t
strlcat(char *dst, const char *src, size_t dstsize)
{
int dlen, slen, dslen, addlim;
dlen = strlen(dst);
slen = strlen(src);
dslen = dlen + slen;
addlim = (dstsize > dslen ? slen : dstsize - dlen - 1);
if ( addlim > 0 )
strncat(dst, src, addlim);
return dslen;
}
#endif /* notonbsd */
./adapt/local.h
/* To be appended to extern.h */
#ifdef notonbsd
extern char *program_invocation_short_name;
char *getprogname();
void errc(int eval, int code, const char *fmt, ...);
int wcwidth(wchar_t wc);
size_t strlcpy(char *dst, const char *src, size_t dstsize);
size_t strlcat(char *dst, const char *src, size_t dstsize);
#endif /* notonbsd */
UPDATE on 2022-01-07
Following Makefile to build macOS sed on a Debian system uses the
same ./adapt/local.c and ./adapt/local.h as listed above. At my end
make download all builds the executable without errors but warns about
an ignored return value of fchown if $(CFLAGS) contains -O3.
make download.TEST test downloads test files, generates and runs
test-edited.sh (a modified TEST/sed.test) which adjusts test
parameters and fixes a redirection issue. Test logs show that 113
(30+83) out of 115 (30+85) test cases succeed (30 in test_error(),
the rest numbered):
test 2.8 fails: sed -n -e '0p' lines1 is run by macOS sed (and
FreeBSD sed) but GNU sed says "invalid usage of line address 0"
test 7.1 fails, even when line continuation char.s are eliminated:
l command of macOS sed (and FreeBSD sed) outputs backslash as \
but POSIX.1-2017
(and GNU sed) says \\
Test 7.7 (involving /usr/share/dict/words) is an indication that
the equivalent FreeBSD sed testcase #75 (see above) should be run
against the local file, not one produced by the test author.
./Makefile
Note: recipe prefix is > (at beginning of line), not tab.
# desc:
# Download, build, test macOS sed (dated 2017-03-27) on GNU/Linux.
# compat:
# dash 0.5.10 GNU make 4.2.1 GNU wget 1.20.3 GNU gcc 9.3.0
# GNU sed 4.7 GNU coreutils 8.30 man 2.9.1
# ref:
# https://opensource.apple.com/source/text_cmds/text_cmds-106/sed/
# https://github.com/joshuarubin/wcwidth9
# see:
# Last sed manpage (dated May 10, 2005) captured by archive.org on Aug 8, 2017
# https://web.archive.org/web/20170808213955/https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/sed.1.html
# https://leancrew.com/all-this/2021/03/apple-and-links/
# https://opensource.apple.com/source/text_cmds/text_cmds-106/text_cmds.plist
# https://unix.stackexchange.com/questions/13711/differences-between-sed-on-mac-osx-and-other-standard-sed
# files:
# Makefile adapt/local.c adapt/local.h
# howto:
# make download
# make all
# (optional) make download.TEST test
# note:
# Mind the $(wgetFlags) and $(CFLAGS)
SHELL := /bin/sh
refsed ?= /usr/bin/sed --posix
wgetFlags ?= --no-verbose --wait=1
# $(call wgetCmd,subtarget-name)
define wgetCmd =
wget $(wgetFlags) --no-host-directories --directory-prefix=$($1.ldir) \
-- $(addprefix $($1.url),$($1.names))
endef
#
hdrs := defs.h extern.h
srcs := misc.c compile.c process.c main.c
hdrx := wcwidth9.h
srcx := local.c
objs := $(patsubst %.c,%.o,$(srcs) $(srcx))
exe := sed
#
sed.genfiles := $(exe) $(exe).1.gz $(exe).pdf
test.gendirs := ./sed.out/ ./nsed.out/
test.genfiles := test-edited.sh test.log test-diff.log $(addsuffix *,$(test.gendirs))
#
subtargets := wcwidth9 sed TEST
dnldtargets := $(addprefix download.,$(subtargets))
#
wcwidth9.url := https://github.com/joshuarubin/wcwidth9/raw/master/
wcwidth9.names := $(hdrx)
wcwidth9.ldir := ./
sed.url := https://opensource.apple.com/source/text_cmds/text_cmds-106/sed/
sed.names := POSIX sed.1 $(srcs) $(hdrs)
sed.ldir := ./
TEST.url := $(sed.url)TEST/
TEST.names := hanoi.sed math.sed sed.test
TEST.ldir := ./TEST/
.RECIPEPREFIX = >
.DELETE_ON_ERROR:
.PHONY: all testclean clean realclean
all : $(sed.genfiles)
testclean :
> rm -f -d -- $(test.genfiles) $(patsubst %/,%,$(test.gendirs))
clean : testclean
> rm -f -- $(objs) $(sed.genfiles)
realclean : clean
> rm -f -- local.c $(foreach T,$(subtargets),$(addprefix $($(T).ldir),$($(T).names)))
> rmdir --ignore-fail-on-non-empty -- $(patsubst %/,%, \
$(foreach T,$(subtargets),$(filter-out %. %./,$($(T).ldir))))
# notonbsd: enable modifications in extern.h and $(srcx)
# __FBSDID: don't embed RCS ID
$(objs) : CFLAGS += $(if $(DEBUG),,-s -O3) -Dnotonbsd -D__FBSDID\(s\)=
$(objs) : $(srcs) $(hdrs)
extern.h : adapt/local.h
> grep -q '^.ifdef\s*notonbsd' $# || { cat $# $< > $#.tmp && mv -f -- $#.tmp $# ; }
local.c : adapt/local.c
> cp $< $#
local.o : $(hdrx)
main.o : CFLAGS += -D__unreachable=__builtin_unreachable
$(exe) : $(objs)
> $(LINK.c) $^ -o $#
%.1.gz : %.1
> gzip -fk $<
%.ps : %.1
> man -l -t $< > $#
%.pdf : %.ps
> ps2pdf $< $#
.PHONY : download.all $(dnldtargets) download
download.all : $(dnldtargets)
$(dnldtargets) :
> $(call wgetCmd,$(patsubst download.%,%,$#))
download : download.wcwidth9 download.sed
> touch adapt/local.h
.PHONY: test
# run after: make download.TEST
# notes:
# test 2.8: GNU sed says "invalid usage of line address 0"
# test 7.1 fails, even when line continuation char.s are eliminated:
# `l` command of macOS sed (and BSD sed) outputs backslash as '\'
# but POSIX.1-2017 (and GNU sed) says `\\`
test : test-diff.log
> printf '## test results in "%s"\n' '$<' 1>&2
test-diff.log : test.log
> - diff -c $(test.gendirs) > $#
# notes:
# `LANG=en`: cause test 7.1 not to output unicode/widechar
# creates $(test.gendirs)
# all tests in test_error() are supposed to produce error output
test.log : test-edited.sh
> LANG=en $(SHELL) $< 1>$# 2>&1
> rm -f -- lines[1-4] script[1-2]
test-edited.sh : $(TEST.ldir)sed.test $(exe)
> $(refsed) \
-e '# main() : adjust test parameters' \
-e '/^.BASE=/ s,.*,BASE="$(refsed)",' \
-e '/^.TEST=/ s,.*,TEST="./$(exe)",' \
-e '# be silent' \
-e '/^.BSD=/ s,.*,BSD=0,' \
-e '/^.GNU=/ s,.*,GNU=0,' \
-e '/^.SUN=/ s,.*,SUN=0,' \
-e '# allow "sed --posix" as exename' \
-e '/^.tests / s,\(\$$[A-Z][A-Z]*\),"\1",g' \
-e '# comment out "diff -c"' \
-e '/^.diff -c/ s,^,#,' \
-e '# test_error(): fix stdin redirection' \
-e '/^.exec 0>&3 4>&1 5>&2/ s,0>&3,3<\&0,' \
-e '/^.exec 0>&3 1>&4 2>&5/ s,0>&3,0<\&3,' \
$< > $#
.PHONY : test-7.1-compare
# special case
test-7.1-compare : $(addsuffix .nolinecontinuation,$(wildcard $(addsuffix *_7.1,$(test.gendirs))))
> - diff -c $^
%.nolinecontinuation : %
> $(refsed) -e ':a' -e '/[^\\]\\$$/N; s/\\\n//; ta' < $< > $#
Possibly of interest:
$ apt-cache search '^freebsd'
ctfutils - FreeBSD CTF utilities
freebsd-buildutils - Utilities for building FreeBSD sources
freebsd-glue - Emulate a FreeBSD build environment
freebsd-manpages - Manual pages for a GNU/kFreeBSD system
freebsd-mk - FreeBSD makefile templates for bmake
libarchive-tools - FreeBSD implementations of 'tar' and 'cpio' and other archive tools
libfreebsd-glue-0 - FreeBSD glue environment (shared objects)
libipx2 - FreeBSD IPX address conversion support library
libsbuf6 - FreeBSD string buffer library
libutil-freebsd-9 - FreeBSD utility library