Answering sed questions on SO I often come across the problem of "works on linux but not MacOS" is it possible to load a version of sed that runs on MacOS into a nix shell?
Currently using flakes I can obtain the latest gnused using:
nix shell nixpkgs#gnused
Is there a way to temporarily install a BSD/MacOS version of sed?
Not so long ago I looked into this problem; perhaps you can make use
of what I found.
Web searches show little demand for FreeBSD sed outside BSD so I
decided to get it (well, GET it) from
github
and build and test it on a recent Debian system. To do so I created
3 files (listed below):
adapt/local.c - support for non-GNU functions
adapt/local.h - ditto header file
Makefile - downloads, generates required files, runs 1 or 2 of
the test suites; refer to comments in header and near target test
On my system the executable builds without errors. To test it using
tests/multi_test.sh I ran
make test | tee multi_test.log | grep '^not ok' | tee multi_test.err.log
with 127 of 130 tests succeeding and 3 failing:
not ok 69 7.1 # Print and file routines
not ok 75 7.7 # w results
not ok 97 8.21 # \ in y command
These discrepancies, I think, are no more than what can be expected: #69
is triggered by differences in output by sed -n l, #75 by differences
in /usr/share/dict/words contents (testcase of limited portability),
and #97 goes away if using SHELL := /bin/bash in the makefile or using
printf '%s\n' 'a\b(c' instead of echo 'a\b(c' in the test script.
UPDATE 2021-11-24: #69 goes away with e.g.
LANG=en make test | grep '^not ok', causing the locale-dependent
library function iswprint()
in process.c#lputs() to return zero for characters 0xA0..0xFF. #75
can only succeed if the first 200 lines of /usr/share/dict/words are
identical to those used by the testcase author; I fail to see the
reasoning behind this. (end UPDATE)
Still on the to-do list:
Makefile's *.names variables are hard-coded, better to create
them dynamically; however, the names seem to change rarely for a
program with sed's history
support for the ATF
tests by tests/sed2_test.sh (currently ignored)
support for embedded ident strings in the executable (__FBSDID
macro currently ignored) rather than rely on a date in the man page for
version
info
./Makefile
Note that recipes aren't prefixed with the usual tab character here
but with > (at beginning of line) acting as a make operator
(.RECIPEPREFIX = >).
# desc:
# Download, build, test FreeBSD sed (dated 2020-06-10) on GNU/Linux
# compat:
# dash 0.5.10 GNU make 4.2.1 GNU wget 1.20 GNU gcc 9.3.0 man 2.9
# ref:
# https://www.freebsd.org/cgi/man.cgi?sed
# https://github.com/freebsd/freebsd-src/tree/main/usr.bin/sed
# https://github.com/joshuarubin/wcwidth9
# files:
# Makefile adapt/local.c adapt/local.h
# howto:
# make download
# make all
# (optional) make download.tests download.regress.multitest.out test
# (optional) cp $(exe) /usr/local/bin/bsdsed
# (optional) cp $(man.1) /usr/local/man/man1/bsdsed.1
# note:
# Mind the $(wgetFlags) and $(CFLAGS)
SHELL := /bin/sh
wgetFlags ?= --no-verbose --wait=1
# $(call wgetCmd,subtarget-name)
define wgetCmd =
wget $(wgetFlags) --no-host-directories --directory-prefix=$($1.ldir) \
-- $(addprefix $($1.url),$($1.names))
endef
#
hdrs := defs.h extern.h
srcs := misc.c compile.c process.c main.c
hdrx := wcwidth9.h
srcx := local.c
objs := $(patsubst %.c,%.o,$(srcs) $(srcx))
# test scripts expect an executable named 'sed'
exe := sed
man.1 := $(exe).1
man.ps := $(exe).ps
binaries := $(exe) $(man.1) $(man.ps)
#
subtargets := wcwidth9 sed tests regress.multitest.out
dnldtargets := $(addprefix download.,$(subtargets))
#
wcwidth9.url := https://github.com/joshuarubin/wcwidth9/raw/master/
wcwidth9.names := $(hdrx)
wcwidth9.ldir := ./
#
sed.url := https://github.com/freebsd/freebsd-src/raw/master/usr.bin/sed/
sed.names := POSIX sed.1 $(srcs) $(hdrs)
sed.ldir := ./
tests.url := $(sed.url)tests/
tests.names := \
hanoi.sed inplace_race_test.sh legacy_test.sh math.sed \
multi_test.sh regress.G.out regress.P.out regress.b2a.out \
regress.bcb.out regress.c0.out regress.c1.out regress.c2.out \
regress.c3.out regress.hanoi.out regress.icase1.out \
regress.icase2.out regress.icase3.out regress.icase4.out \
regress.in regress.math.out regress.not.out regress.psl.out \
regress.s3.out regress.s4.out regress.s5.out regress.sg.out \
regress.sh regress.y.out sed2_test.sh
tests.ldir := ./tests/
regress.multitest.out.url := $(sed.url)tests/regress.multitest.out/
regress.multitest.out.names := \
1.1 1.2 1.3 1.4 1.4.1 1.5 1.6 1.7 1.8 1.9 1.10 1.11 1.12 1.13 \
1.14 1.15 1.16 1.17 1.18 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 \
2.10 2.11 2.12 2.13 2.14 2.15 2.16 2.17 2.18 2.19 2.20 2.21 \
2.22 2.23 3.1 3.2 3.3 3.4 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 5.1 \
5.2 5.3 5.4 5.5 5.6 5.7 5.8 6.1 6.2 6.3 6.4 6.5 6.6 7.1 7.2 \
7.3 7.4 7.5 7.6 7.7 7.8 8.1 8.2 8.3 8.4 8.5 8.6 8.7 8.8 8.9 \
8.10 8.11 8.12 8.13 8.14 8.15 8.16 8.17 8.18 8.19 8.20 8.21 \
8.22 8.23 9.1 9.2 9.3 9.4 9.5 9.6 9.7 9.8 9.9 9.10 9.11 9.12 \
9.13 9.14 9.15 9.16 9.17 9.18 9.19 9.20 9.21 9.22 9.23 9.24 \
9.25 9.26 9.27 9.28 9.29 9.30 9.31
regress.multitest.out.ldir := ./tests/regress.multitest.out/
.RECIPEPREFIX = >
.DELETE_ON_ERROR:
.PHONY: all clean realclean
all : $(binaries)
clean : ; rm -f -- $(objs) $(binaries)
realclean : clean
> rm -f -- local.c manpage.1 $(foreach T,$(subtargets),$(addprefix $($(T).ldir),$($(T).names)))
> rmdir --ignore-fail-on-non-empty -- \
$(patsubst %/,%,$(foreach T,$(subtargets),$(filter-out %. %./,$($(T).ldir))))
# notonbsd: enable modifications in extern.h and $(srcx)
# __FBSDID: don't embed RCS ID
$(objs) : CFLAGS += -Dnotonbsd -D__FBSDID\(s\)=
$(objs) : $(srcs) $(hdrs)
extern.h : adapt/local.h
> grep -q '^.ifdef\s*notonbsd' $# || { cat $# $< > $#.tmp && mv -f -- $#.tmp $# ; }
local.c : adapt/local.c
> cp $< $#
local.o : $(hdrx)
main.o : CFLAGS += -D__unreachable=__builtin_unreachable
$(exe) : $(objs)
> $(LINK.c) -o $# $^
> $(if $(DEBUG),,strip $#)
$(man.1) : manpage.1
> cp $< $#
%.ps : %.1
> man -l -t $< > $#
.PHONY : download.all $(dnldtargets) download
download.all : $(dnldtargets)
$(dnldtargets) :
> $(call wgetCmd,$(patsubst download.%,%,$#))
download : download.wcwidth9 download.sed
> mv -f sed.1 manpage.1
> touch adapt/local.h
.PHONY: test
# ! run after: make download.tests download.regress.multitest.out
# ! hint: make test | tee multi_test.log | grep '^not ok' | tee multi_test.err.log
# - set PATH so scripts invoke the new sed executable
# - run multi_test.sh (requires /usr/share/dict/words regress.multitest.out/*)
# - optionally run inplace_race_test.sh
# - ignore legacy_test.sh regress.* (require m4 regress.m4 hanoi.sed math.sed)
# - ignore sed2_test.sh (requires ATF, cf. https://github.com/freebsd/freebsd-src/tree/main/contrib/atf)
test : | /usr/share/dict/words
> cd $(patsubst %/,%,$(tests.ldir)); \
PATH="..:$$PATH"; \
$(SHELL) multi_test.sh \
$(if $(racetest),; $(SHELL) inplace_race_test.sh && rm -f file[0-9] file[0-9].prev)
./adapt/local.c
/* Support for non-GNU functions, cf. man.freebsd.org */
#ifdef notonbsd
#include <err.h>
#include <limits.h>
#include <regex.h>
#include <stdio.h>
#include <stddef.h>
#include <string.h>
#include <stdarg.h>
#include <errno.h>
#include "defs.h"
#include "extern.h"
#include "wcwidth9.h" /* https://github.com/joshuarubin/wcwidth9 */
void
errc(int eval, int code, const char *fmt, ...)
{
va_list args;
va_start(args, fmt);
errno = code;
err(eval, fmt, args);
va_end(args);
}
char*
getprogname()
{
return (program_invocation_short_name);
}
int
wcwidth(wchar_t wc)
{
return wcwidth9(wc);
}
size_t
strlcpy(char *dst, const char *src, size_t dstsize)
{
return snprintf(dst, dstsize, "%s", src);
}
size_t
strlcat(char *dst, const char *src, size_t dstsize)
{
int dlen, slen, dslen, addlim;
dlen = strlen(dst);
slen = strlen(src);
dslen = dlen + slen;
addlim = (dstsize > dslen ? slen : dstsize - dlen - 1);
if ( addlim > 0 )
strncat(dst, src, addlim);
return dslen;
}
#endif /* notonbsd */
./adapt/local.h
/* To be appended to extern.h */
#ifdef notonbsd
extern char *program_invocation_short_name;
char *getprogname();
void errc(int eval, int code, const char *fmt, ...);
int wcwidth(wchar_t wc);
size_t strlcpy(char *dst, const char *src, size_t dstsize);
size_t strlcat(char *dst, const char *src, size_t dstsize);
#endif /* notonbsd */
UPDATE on 2022-01-07
Following Makefile to build macOS sed on a Debian system uses the
same ./adapt/local.c and ./adapt/local.h as listed above. At my end
make download all builds the executable without errors but warns about
an ignored return value of fchown if $(CFLAGS) contains -O3.
make download.TEST test downloads test files, generates and runs
test-edited.sh (a modified TEST/sed.test) which adjusts test
parameters and fixes a redirection issue. Test logs show that 113
(30+83) out of 115 (30+85) test cases succeed (30 in test_error(),
the rest numbered):
test 2.8 fails: sed -n -e '0p' lines1 is run by macOS sed (and
FreeBSD sed) but GNU sed says "invalid usage of line address 0"
test 7.1 fails, even when line continuation char.s are eliminated:
l command of macOS sed (and FreeBSD sed) outputs backslash as \
but POSIX.1-2017
(and GNU sed) says \\
Test 7.7 (involving /usr/share/dict/words) is an indication that
the equivalent FreeBSD sed testcase #75 (see above) should be run
against the local file, not one produced by the test author.
./Makefile
Note: recipe prefix is > (at beginning of line), not tab.
# desc:
# Download, build, test macOS sed (dated 2017-03-27) on GNU/Linux.
# compat:
# dash 0.5.10 GNU make 4.2.1 GNU wget 1.20.3 GNU gcc 9.3.0
# GNU sed 4.7 GNU coreutils 8.30 man 2.9.1
# ref:
# https://opensource.apple.com/source/text_cmds/text_cmds-106/sed/
# https://github.com/joshuarubin/wcwidth9
# see:
# Last sed manpage (dated May 10, 2005) captured by archive.org on Aug 8, 2017
# https://web.archive.org/web/20170808213955/https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/sed.1.html
# https://leancrew.com/all-this/2021/03/apple-and-links/
# https://opensource.apple.com/source/text_cmds/text_cmds-106/text_cmds.plist
# https://unix.stackexchange.com/questions/13711/differences-between-sed-on-mac-osx-and-other-standard-sed
# files:
# Makefile adapt/local.c adapt/local.h
# howto:
# make download
# make all
# (optional) make download.TEST test
# note:
# Mind the $(wgetFlags) and $(CFLAGS)
SHELL := /bin/sh
refsed ?= /usr/bin/sed --posix
wgetFlags ?= --no-verbose --wait=1
# $(call wgetCmd,subtarget-name)
define wgetCmd =
wget $(wgetFlags) --no-host-directories --directory-prefix=$($1.ldir) \
-- $(addprefix $($1.url),$($1.names))
endef
#
hdrs := defs.h extern.h
srcs := misc.c compile.c process.c main.c
hdrx := wcwidth9.h
srcx := local.c
objs := $(patsubst %.c,%.o,$(srcs) $(srcx))
exe := sed
#
sed.genfiles := $(exe) $(exe).1.gz $(exe).pdf
test.gendirs := ./sed.out/ ./nsed.out/
test.genfiles := test-edited.sh test.log test-diff.log $(addsuffix *,$(test.gendirs))
#
subtargets := wcwidth9 sed TEST
dnldtargets := $(addprefix download.,$(subtargets))
#
wcwidth9.url := https://github.com/joshuarubin/wcwidth9/raw/master/
wcwidth9.names := $(hdrx)
wcwidth9.ldir := ./
sed.url := https://opensource.apple.com/source/text_cmds/text_cmds-106/sed/
sed.names := POSIX sed.1 $(srcs) $(hdrs)
sed.ldir := ./
TEST.url := $(sed.url)TEST/
TEST.names := hanoi.sed math.sed sed.test
TEST.ldir := ./TEST/
.RECIPEPREFIX = >
.DELETE_ON_ERROR:
.PHONY: all testclean clean realclean
all : $(sed.genfiles)
testclean :
> rm -f -d -- $(test.genfiles) $(patsubst %/,%,$(test.gendirs))
clean : testclean
> rm -f -- $(objs) $(sed.genfiles)
realclean : clean
> rm -f -- local.c $(foreach T,$(subtargets),$(addprefix $($(T).ldir),$($(T).names)))
> rmdir --ignore-fail-on-non-empty -- $(patsubst %/,%, \
$(foreach T,$(subtargets),$(filter-out %. %./,$($(T).ldir))))
# notonbsd: enable modifications in extern.h and $(srcx)
# __FBSDID: don't embed RCS ID
$(objs) : CFLAGS += $(if $(DEBUG),,-s -O3) -Dnotonbsd -D__FBSDID\(s\)=
$(objs) : $(srcs) $(hdrs)
extern.h : adapt/local.h
> grep -q '^.ifdef\s*notonbsd' $# || { cat $# $< > $#.tmp && mv -f -- $#.tmp $# ; }
local.c : adapt/local.c
> cp $< $#
local.o : $(hdrx)
main.o : CFLAGS += -D__unreachable=__builtin_unreachable
$(exe) : $(objs)
> $(LINK.c) $^ -o $#
%.1.gz : %.1
> gzip -fk $<
%.ps : %.1
> man -l -t $< > $#
%.pdf : %.ps
> ps2pdf $< $#
.PHONY : download.all $(dnldtargets) download
download.all : $(dnldtargets)
$(dnldtargets) :
> $(call wgetCmd,$(patsubst download.%,%,$#))
download : download.wcwidth9 download.sed
> touch adapt/local.h
.PHONY: test
# run after: make download.TEST
# notes:
# test 2.8: GNU sed says "invalid usage of line address 0"
# test 7.1 fails, even when line continuation char.s are eliminated:
# `l` command of macOS sed (and BSD sed) outputs backslash as '\'
# but POSIX.1-2017 (and GNU sed) says `\\`
test : test-diff.log
> printf '## test results in "%s"\n' '$<' 1>&2
test-diff.log : test.log
> - diff -c $(test.gendirs) > $#
# notes:
# `LANG=en`: cause test 7.1 not to output unicode/widechar
# creates $(test.gendirs)
# all tests in test_error() are supposed to produce error output
test.log : test-edited.sh
> LANG=en $(SHELL) $< 1>$# 2>&1
> rm -f -- lines[1-4] script[1-2]
test-edited.sh : $(TEST.ldir)sed.test $(exe)
> $(refsed) \
-e '# main() : adjust test parameters' \
-e '/^.BASE=/ s,.*,BASE="$(refsed)",' \
-e '/^.TEST=/ s,.*,TEST="./$(exe)",' \
-e '# be silent' \
-e '/^.BSD=/ s,.*,BSD=0,' \
-e '/^.GNU=/ s,.*,GNU=0,' \
-e '/^.SUN=/ s,.*,SUN=0,' \
-e '# allow "sed --posix" as exename' \
-e '/^.tests / s,\(\$$[A-Z][A-Z]*\),"\1",g' \
-e '# comment out "diff -c"' \
-e '/^.diff -c/ s,^,#,' \
-e '# test_error(): fix stdin redirection' \
-e '/^.exec 0>&3 4>&1 5>&2/ s,0>&3,3<\&0,' \
-e '/^.exec 0>&3 1>&4 2>&5/ s,0>&3,0<\&3,' \
$< > $#
.PHONY : test-7.1-compare
# special case
test-7.1-compare : $(addsuffix .nolinecontinuation,$(wildcard $(addsuffix *_7.1,$(test.gendirs))))
> - diff -c $^
%.nolinecontinuation : %
> $(refsed) -e ':a' -e '/[^\\]\\$$/N; s/\\\n//; ta' < $< > $#
Possibly of interest:
$ apt-cache search '^freebsd'
ctfutils - FreeBSD CTF utilities
freebsd-buildutils - Utilities for building FreeBSD sources
freebsd-glue - Emulate a FreeBSD build environment
freebsd-manpages - Manual pages for a GNU/kFreeBSD system
freebsd-mk - FreeBSD makefile templates for bmake
libarchive-tools - FreeBSD implementations of 'tar' and 'cpio' and other archive tools
libfreebsd-glue-0 - FreeBSD glue environment (shared objects)
libipx2 - FreeBSD IPX address conversion support library
libsbuf6 - FreeBSD string buffer library
libutil-freebsd-9 - FreeBSD utility library
Story
Case 1
I accidentally wrote my Assembly code in the .data section. I compiled it and executed it. The program ran normally under Linux 5.4.0-53-generic even though I didn't specify a flag like execstack.
Case 2:
After that, I executed the program under Linux 5.9.0-050900rc5-generic. The program got SIGSEGV. I inspected the virtual memory permission by reading /proc/$pid/maps. It turned out that the section is not executable.
I think there is a configuration on Linux that manages that permission. But I don't know where to find.
Code
[Linux 5.4.0-53-generic]
Run (normal)
ammarfaizi2#integral:/tmp$ uname -r
5.4.0-53-generic
ammarfaizi2#integral:/tmp$ cat test.asm
[section .data]
global _start
_start:
mov eax, 60
xor edi, edi
syscall
ammarfaizi2#integral:/tmp$ nasm --version
NASM version 2.14.02
ammarfaizi2#integral:/tmp$ nasm -felf64 test.asm -o test.o
ammarfaizi2#integral:/tmp$ ld test.o -o test
ammarfaizi2#integral:/tmp$ ./test
ammarfaizi2#integral:/tmp$ echo $?
0
ammarfaizi2#integral:/tmp$ md5sum test
7ffff5fd44e6ff0a278e881732fba525 test
ammarfaizi2#integral:/tmp$
Check Permission (00400000-00402000 rwxp), so it is executable.
## Debug
gef➤ shell cat /proc/`pgrep test`/maps
00400000-00402000 rwxp 00000000 08:03 7471589 /tmp/test
7ffff7ffb000-7ffff7ffe000 r--p 00000000 00:00 0 [vvar]
7ffff7ffe000-7ffff7fff000 r-xp 00000000 00:00 0 [vdso]
7ffffffde000-7ffffffff000 rwxp 00000000 00:00 0 [stack]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall]
gef➤
[Linux 5.9.0-050900rc5-generic]
Run (Segfault)
root#esteh:/tmp# uname -r
5.9.0-050900rc5-generic
root#esteh:/tmp# cat test.asm
[section .data]
global _start
_start:
mov eax, 60
xor edi, edi
syscall
root#esteh:/tmp# nasm --version
NASM version 2.14.02
root#esteh:/tmp# nasm -felf64 test.asm -o test.o
root#esteh:/tmp# ld test.o -o test
root#esteh:/tmp# ./test
Segmentation fault (core dumped)
root#esteh:/tmp# echo $?
139
root#esteh:/tmp# md5sum test
7ffff5fd44e6ff0a278e881732fba525 test
root#esteh:/tmp#
Check Permission (00400000-00402000 rw-p), so it is NOT executable.
## Debug
gef➤ shell cat /proc/`pgrep test`/maps
00400000-00402000 rw-p 00000000 fc:01 2412 /tmp/test
7ffff7ff9000-7ffff7ffd000 r--p 00000000 00:00 0 [vvar]
7ffff7ffd000-7ffff7fff000 r-xp 00000000 00:00 0 [vdso]
7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0 [stack]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall]
gef➤
objdump -p
root#esteh:/tmp# objdump -p test
test: file format elf64-x86-64
Program Header:
LOAD off 0x0000000000000000 vaddr 0x0000000000400000 paddr 0x0000000000400000 align 2**12
filesz 0x0000000000001009 memsz 0x0000000000001009 flags rw-
Questions
Where is the configuration on Linux that manages default ELF sections permission?
Are my observations on permissions correct?
Summary
Default permission for .data section on Linux 5.4.0-53-generic is executable.
Default permission for .data section on Linux 5.9.0-050900rc5-generic is NOT executable.
Your binary is missing PT_GNU_STACK. As such, this change appears to have been caused by commit 9fccc5c0c99f238aa1b0460fccbdb30a887e7036:
From 9fccc5c0c99f238aa1b0460fccbdb30a887e7036 Mon Sep 17 00:00:00 2001
From: Kees Cook <keescook#chromium.org>
Date: Thu, 26 Mar 2020 23:48:17 -0700
Subject: x86/elf: Disable automatic READ_IMPLIES_EXEC on 64-bit
With modern x86 64-bit environments, there should never be a need for
automatic READ_IMPLIES_EXEC, as the architecture is intended to always
be execute-bit aware (as in, the default memory protection should be NX
unless a region explicitly requests to be executable).
There were very old x86_64 systems that lacked the NX bit, but for those,
the NX bit is, obviously, unenforceable, so these changes should have
no impact on them.
Suggested-by: Hector Marco-Gisbert <hecmargi#upv.es>
Signed-off-by: Kees Cook <keescook#chromium.org>
Signed-off-by: Borislav Petkov <bp#suse.de>
Reviewed-by: Jason Gunthorpe <jgg#mellanox.com>
Link: https://lkml.kernel.org/r/20200327064820.12602-4-keescook#chromium.org
---
arch/x86/include/asm/elf.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/elf.h b/arch/x86/include/asm/elf.h
index 397a1c74433ec..452beed7892bb 100644
--- a/arch/x86/include/asm/elf.h
+++ b/arch/x86/include/asm/elf.h
## -287,7 +287,7 ## extern u32 elf_hwcap2;
* CPU: | lacks NX* | has NX, ia32 | has NX, x86_64 |
* ELF: | | | |
* ---------------------|------------|------------------|----------------|
- * missing PT_GNU_STACK | exec-all | exec-all | exec-all |
+ * missing PT_GNU_STACK | exec-all | exec-all | exec-none |
* PT_GNU_STACK == RWX | exec-stack | exec-stack | exec-stack |
* PT_GNU_STACK == RW | exec-none | exec-none | exec-none |
*
## -303,7 +303,7 ## extern u32 elf_hwcap2;
*
*/
#define elf_read_implies_exec(ex, executable_stack) \
- (executable_stack == EXSTACK_DEFAULT)
+ (mmap_is_ia32() && executable_stack == EXSTACK_DEFAULT)
struct task_struct;
--
cgit 1.2.3-1.el7
This was first present in the 5.8 series. See also Unexpected exec permission from mmap when assembly files included in the project.
This is only a guess: I think the culprit is the READ_IMPLIES_EXEC personality that was being set automatically in the absence of a PT_GNU_STACK segment.
In the 5.4 kernel source we can find this piece of code:
SET_PERSONALITY2(loc->elf_ex, &arch_state);
if (elf_read_implies_exec(loc->elf_ex, executable_stack))
current->personality |= READ_IMPLIES_EXEC;
That's the only thing that can transform an RW section into an RWX one. Any other use of PROC_EXEC didn't seem to be changed or relevant to this question, to me.
The executable_stack is set here:
for (i = 0; i < loc->elf_ex.e_phnum; i++, elf_ppnt++)
switch (elf_ppnt->p_type) {
case PT_GNU_STACK:
if (elf_ppnt->p_flags & PF_X)
executable_stack = EXSTACK_ENABLE_X;
else
executable_stack = EXSTACK_DISABLE_X;
break;
But if the PT_GNU_STACK segment is not present, that variable retains its default value:
int executable_stack = EXSTACK_DEFAULT;
Now this workflow is identical in both 5.4 and the latest kernel source, what changed is the definition of elf_read_implies_exec:
Linux 5.4:
/*
* An executable for which elf_read_implies_exec() returns TRUE will
* have the READ_IMPLIES_EXEC personality flag set automatically.
*/
#define elf_read_implies_exec(ex, executable_stack) \
(executable_stack != EXSTACK_DISABLE_X)
Latest Linux:
/*
* An executable for which elf_read_implies_exec() returns TRUE will
* have the READ_IMPLIES_EXEC personality flag set automatically.
*
* The decision process for determining the results are:
*
* CPU: | lacks NX* | has NX, ia32 | has NX, x86_64 |
* ELF: | | | |
* ---------------------|------------|------------------|----------------|
* missing PT_GNU_STACK | exec-all | exec-all | exec-none |
* PT_GNU_STACK == RWX | exec-stack | exec-stack | exec-stack |
* PT_GNU_STACK == RW | exec-none | exec-none | exec-none |
*
* exec-all : all PROT_READ user mappings are executable, except when
* backed by files on a noexec-filesystem.
* exec-none : only PROT_EXEC user mappings are executable.
* exec-stack: only the stack and PROT_EXEC user mappings are executable.
*
* *this column has no architectural effect: NX markings are ignored by
* hardware, but may have behavioral effects when "wants X" collides with
* "cannot be X" constraints in memory permission flags, as in
* https://lkml.kernel.org/r/20190418055759.GA3155#mellanox.com
*
*/
#define elf_read_implies_exec(ex, executable_stack) \
(mmap_is_ia32() && executable_stack == EXSTACK_DEFAULT)
Note how in the 5.4 version the elf_read_implies_exec returned a true value if the stack was not explicitly marked as not executable (via the PT_GNU_STACK segment).
In the latest source, the check is now more defensive: the elf_read_implies_exec is true only on 32-bit executable, in the case where no PT_GNU_STACK segment was found in the ELF binary.
I assembled your program, linked it, and found no PT_GNU_STACK segment, so this may be the reason.
If this is indeed the issue and if I followed the code correctly, if you set the stack as not executable in the binary, its data section should not be mapped executable anymore (not even on Linux 5.4).
Im passionate about assembly and wanted to start coding from home on linux instead of mac I usually use.
I really struggle for 4 days about this issue.
you can find my makefile and clone repository at the following url:
<code>NAME = libfts.a
ASM_FILES = ft_isascii \
OS := $(shell uname)
ifeq ($(OS), Darwin)
ASM_COMPILER = ~/.brew/bin/nasm -f macho64 -g
else
ASM_COMPILER = nasm -f elf64 -g
endif
ASM_SRC_DIR = srcs/
ASM_OBJ_DIR_NAME = obj
ASM_OBJ_DIR = $(ASM_OBJ_DIR_NAME)/
ASM_OBJ := $(addsuffix .o,$(ASM_FILES))
ASM_OBJ := $(addprefix $(ASM_OBJ_DIR),$(ASM_OBJ))
TEST = maintest.out
TEST_FILES = maintest
C_COMPILER = clang -Wall -Werror -Wextra -O3
TEST_DIR_NAME = test
TEST_DIR = $(TEST_DIR_NAME)/
TEST_OBJ := $(addsuffix .o,$(TEST_FILES))
TEST_OBJ := $(addprefix $(TEST_DIR),$(TEST_OBJ))
OBJ_PATHS := $(ASM_OBJ) $(TEST_OBJ)
all: $(NAME)
$(NAME): $(ASM_OBJ)
ar rc $(NAME) $(ASM_OBJ)
test: re $(TEST_OBJ)
$(C_COMPILER) -L. $(NAME) $(TEST_OBJ) -o $(TEST)
$(ASM_OBJ): $(ASM_OBJ_DIR)%.o: $(ASM_SRC_DIR)%.s
#/bin/mkdir -p $(ASM_OBJ_DIR)
$(ASM_COMPILER) $< -o $#
$(TEST_OBJ): $(TEST_DIR)%.o: $(TEST_DIR)%.c
$(C_COMPILER) -c -I. $< -o $#
clean:
-/bin/rm -f $(OBJ_PATHS)
/usr/bin/find . -name "$(ASM_OBJ_DIR_NAME)" -maxdepth 1 -type d -empty -delete
fclean: clean
-/bin/rm -f $(NAME)
-/bin/rm -f $(TEST)
re: fclean all
.PHONY: all clean fclean re
I have this error message when I try "make test" on the linux:
test/maintest.o: In function `main':
test/maintest.c:(.text+0x33): undefined reference to `ft_isascii'
undefined reference to etc.
.h content:
#ifndef _LIBFTS
# define _LIBFTS
#include <stddef.h>
int ft_isascii(int c);
#endif
ft_isascii.s content:
global _ft_isascii
section .text
_ft_isascii: ; int ft_isascii
and edi, 0xffffff80 ; mask with the 128 firsts bits left to 0 as ASCII range from 0 to 7f in hexa (just below 80)
sete al ; SETE sets AL to 1 if above condition code means "equal", otherwise it sets AL to 0.
movzx eax, al
ret
I would REALLY be thanksful for any tips to solve this issue...
Regards,
There are two problems to be fixed:
First, on ELF targets (most Unixes except macOS), C functions are not decorated with an underscore. To fix your code, remove the leading underscore from all symbols. Make sure to remove it everywhere.
Second, the linker when looking at an archive (.a file) only picks files it needs right now to satisfy dependencies. So when you pass the archive before maintest.o, the linker doesn't take anything from the archive at all as it doesn't need any ft_... symbols at that point. These symbols are only needed once the linker has seen maintest.o. To fix this issue, move the $(NAME) operand to after $(TEST_OBJ). As a general rule of thumb, always place libraries after object files on the linker command line.
On macOS, you won't observe this problem because they use lld, the LLVM linker, which is a bit unconventional in that it defers the choice which objects to take out of archives until it has looked at the symbol tables of all operands, making your actually broken invocation work. Don't depend on this behaviour, please.
I build httrack on Cygwin, but it is always "checking for iconv in -liconv... no" and compile error.
collect2: error: ld returned 1 exit status
./.libs/libhttrack.a(htscharset.o): In function `hts_convertStringCharset':
/home/myname/httrack/src/htscharset.c:441: undefined reference to `libiconv_open'
/home/myname/httrack/src/htscharset.c:460: undefined reference to `libiconv'
/home/myname/httrack/src/htscharset.c:491: undefined reference to `libiconv_close
I have install libiconv1.1.4 in /usr/local/libiconv, but it's still error.
nm /lib/libiconv.a | grep iconv
iconv.o:
00000000 D __libiconv_version
00015960 T _iconv_canonicalize
00015120 T _libiconv
00015180 T _libiconv_close
00014d00 T _libiconv_open
000151a0 T _libiconv_open_into
00015690 T _libiconvctl
00015810 T _libiconvlist
U _libiconv_relocate
000000d0 T _libiconv_relocate
00000000 T _libiconv_set_relocation_prefix
iconv-exports.o:
00000000 D __imp___libiconv_version
U __libiconv_version
libiconv.res.o:
The code in configure for iconv check:
### Check for iconv
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for iconv in -liconv" >&5
$as_echo_n "checking for iconv in -liconv... " >&6; }
if ${ac_cv_lib_iconv_iconv+:} false; then :
$as_echo_n "(cached) " >&6
else
ac_check_lib_save_LIBS=$LIBS
LIBS="-liconv $LIBS"
httrack requires several patches to build and operate correctly on Cygwin. The libiconv patch fixes the iconv configure test to work on Cygwin (and other platforms which use GNU libiconv).
First of all, I'm trying to get used to makefiles but yet I#m new with this. The following file is supposed to, first, compile all ./src/*.cpp files to ./src/*.o (where the filename survives) and afterwards complete compilation with simulation.cpp and linking the whole stuff together. Now, make returns the error message:
make: -c: Command not found
I have literally no clue how to proceed! Would the wildcard-construct even work in the way desired? Thanks a lot for your effort!
#basic stuff
TRUE = 1
FALSE = 0
SHELL := #!/bin/bash
# path names
SRCPATH = ./src/
CLEANPATH = ./res/ \
./crash/
# source files.
MAIN = simulation.cpp
OBJS = $(wildcard $(SRCPATH)*.o)
SRCS = $(wildcard $(SRCPATH)*.cpp)
INCLUDES = $(wildcard $(SRCPATH)*.h)
#GLOBAL MACROS PASSED TO PROGRAM!
MODEL_MRT = $(TRUE) #if true model used is MRT else SRT
PARALLEL = $(TRUE)
GRAVITY = $(TRUE)
# output file name
OUT = simulation
# C++ compiler flags (-g -O2 -Wall)
CXXFLAGS = -g -Wall -O -fopenmp
CXXDEFINES = -D MODEL=$(MODEL_MRT) -D PARALLEL=$(PARALLEL) -D GRAVITY=$(GRAVITY)
# compiler
CXX = g++
$(OUT) : $(OBJS)
$(CXX) $(CXXFLAGS) $(MAIN) $(OBJS) $(CXXDEFINES) -o $(OUT)
$(OBJS) : $(SRCS) $(INCLUDES)
$(CXX) $(CXXFLAGS) -c $(SRCS) -o $(OBJS)
clean : $(OUT)
rm $(OBJS)
rm $(CLEANPATH)/*.*
run : $(OUT) clean
./$(OUT)
.PHONY: clean run
You're tricking make with your SHELL variable, it sees is at empty as it is just a comment.
Change
SHELL := #!/bin/bash
to
SHELL := /bin/bash
This line:
SHELL := #!/bin/bash
is incorrect.
Your makefile should work perfectly well if you leave that line out altogether. If you do need something there, try
SHELL := /bin/bash