crontab with ed by commands on stream, results in "no modification made" - linux

I am trying to append a line to my crontab file. I know there are other ways to work around this problem, but still want to know what caused it. The command is run on raspberry pi 3 B+, raspbian lite is installed, with GNU ed 1.15, cron 3.0pl1-134+deb10u1.
The command that I'm stuck on is this:
$ echo -e 'a\n#asdf\n.\nwQ' | EDITOR=ed crontab -e
902
909
No modification made
I'm expecting it to add line #asdf at the end of my crontab file, but it doesn't.
Setting EDITOR='tee -a' as suggested on https://stackoverflow.com/a/30123606/8842387 does not solve the problem. So I guess it is the problem with cron.
Strangely enough, when I give ed commands from the keyboard directly, rather than streaming it, it just works. Maybe subshell creation caused the problem?
Here I'm attaching a few of the last lines from strace result.
$ echo -e 'a\n#asdf\n.\nwQ' | EDITOR=ed strace crontab -e
execve("/usr/bin/crontab", ["crontab", "-e"], 0x7ee54c14 /* 29 vars */) = 0
access("/etc/suid-debug", F_OK) = -1 ENOENT (No such file or directory)
...
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\7\0\0\0\7\0\0\0\0"..., 4096) = 659
_llseek(3, -393, [266], SEEK_CUR) = 0
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\7\0\0\0\7\0\0\0\0"..., 4096) = 393
close(3) = 0
getpid() = 18579
socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path="/dev/log"}, 110) = 0
send(3, "<78>Nov 20 15:31:25 crontab[1857"..., 56, MSG_NOSIGNAL) = 56
openat(AT_FDCWD, "crontabs/pi", O_RDONLY) = -1 EACCES (Permission denied)
openat(AT_FDCWD, "/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=2995, ...}) = 0
read(4, "# Locale name alias data base.\n#"..., 4096) = 2995
read(4, "", 4096) = 0
close(4) = 0
openat(AT_FDCWD, "/usr/share/locale/en_GB.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_GB.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_GB/LC_MESSAGES/libc.mo", O_RDONLY) = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=1433, ...}) = 0
mmap2(NULL, 1433, PROT_READ, MAP_PRIVATE, 4, 0) = 0x76f50000
close(4) = 0
openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "crontabs/pi/: fdopen: Permission"..., 39crontabs/pi/: fdopen: Permission denied) = 39
exit_group(1) = ?
+++ exited with 1 +++
openat(AT_FDCWD, "crontabs/pi", O_RDONLY) = -1 EACCES (Permission denied) looks a bit suspicious, but not sure why it opens the file read-only.
EDIT:
As suggested by #tink, I ran EDITOR=ed strace crontab -e to see what strace gives on an interactive session. The result was almost same (only varying on pid and fd numbers).
I noticed that running echo "..." | EDITOR=ed crontab -e exited with message No modification made but with strace the process halts without any messages. (EDITOR=ed strace crontab -e 2>&1 | grep "No mod" prints nothing). Guess the strace triggers different errors.

Following up on my VISUAL comment, these worked for me:
( unset VISUAL; printf '%s\n' a '#abcd' . wq | EDITOR=ed crontab -e )
printf '%s\n' a '#abcd' . wq | VISUAL=ed crontab -e
In my environment, both VISUAL and EDITOR are set to "vim"
Or, more roundabout, but don't need to monkey with env vars. This one also allows you to do it silently:
crontab <(printf '%s\n' a '#asdf' . '%p' | ed -s <(crontab -l))
I was doing the above on a Mac. On Linux, I can reproduce your observations, but can't explain them.
A small tweak to the last command works:
printf '%s\n' a '#asdf' . '%p' Q | ed -s <(crontab -l) | crontab -

TLDR; (sleep 1; echo -e 'a\n#asdf\n.\nwQ') | EDITOR=ed crontab -e works!
The problem was on crontab.
When I invoke crontab -e it creates a temporary copy of the user cron table in /tmp directory.
Then opens the temporary file with an editor specified by $EDITOR.
After the editing is done, crontab check if the file modification date have changed since its creation.
This is implemented in the patch that enables editing cron table via temporary file.
In my case, ed getting its command from stdin finished the editing too fast so that even a single digit of the modification timestamp of the temporary file had not been changed.
As crontab considered no human can make edition that fast, it assumes no modification made and discards it.
To bypass this behavior, I added sleep 1 before the release of the command.
This will hold ed to wait for its command from stdin after crontab created tempfile, which effectively lets the modification timestamp different.

Related

Why does this strace on a pipeline not finish

I have a directory with a single file, one.txt. If I run ls | cat, it works fine. However, if I try to strace both sides of this pipeline, I do see the output of the command as well as strace, but the process doesn't finish.
strace ls 2> >(stdbuf -o 0 sed 's/^/command1:/') | strace cat 2> >(stdbuf -o 0 sed 's/^/command2:/')
The output I get is:
command2:execve("/usr/bin/cat", ["cat"], [/* 50 vars */]) = 0
command2:brk(0) = 0x1938000
command2:mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f87e5a93000
command2:access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
<snip>
command2:open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
command2:fstat(3, {st_mode=S_IFREG|0644, st_size=106070960, ...}) = 0
command2:mmap(NULL, 106070960, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f87def8a000
command2:close(3) = 0
command2:fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0
command2:fstat(0, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
command2:fadvise64(0, 0, 0, POSIX_FADV_SEQUENTIAL) = -1 ESPIPE (Illegal seek)
command2:read(0, "command1:execve(\"/usr/bin/ls\", ["..., 65536) = 4985
command1:execve("/usr/bin/ls", ["ls"], [/* 50 vars */]) = 0
command1:brk(0) = 0x1190000
command1:mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fae869c3000
command1:access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
<snip>
command1:close(3) = 0
command1:fstat(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
command2:write(1, "command1:close(3) "..., 115) = 115
command2:read(0, "command1:mmap(NULL, 4096, PROT_R"..., 65536) = 160
command1:mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fae869c2000
one.txt
command1:write(1, "one.txt\n", 8) = 8
command2:write(1, "command1:mmap(NULL, 4096, PROT_R"..., 160) = 160
command2:read(0, "command1:close(1) "..., 65536) = 159
command1:close(1) = 0
command1:munmap(0x7fae869c2000, 4096) = 0
command1:close(2) = 0
command2:write(1, "command1:close(1) "..., 159) = 159
command2:read(0, "command1:exit_group(0) "..., 65536) = 53
command1:exit_group(0) = ?
command2:write(1, "command1:exit_group(0) "..., 53) = 53
command2:read(0, "command1:+++ exited with 0 +++\n", 65536) = 31
command1:+++ exited with 0 +++
command2:write(1, "command1:+++ exited with 0 +++\n", 31) = 31
and it hangs from then on. ps reveals that both commands in the pipeline (ls and cat here) are running.
I am on RHEL7 running Bash version 4.2.46.
I put a strace on your strace:
strace bash -c 'strace true 2> >(cat > /dev/null)'
It hangs on a wait4, indicating that it's stuck waiting on children. ps f confirms this:
24740 pts/19 Ss 0:00 /bin/bash
24752 pts/19 S+ 0:00 \_ strace true
24753 pts/19 S+ 0:00 \_ /bin/bash
24755 pts/19 S+ 0:00 \_ cat
Based on this, my working theory is that this effect is a deadlock because:
strace waits on all children, even the ones it didn't spawn directly
Bash spawns the process substitution as a child of the process. Since the process substitution is attached to stderr, it essentially waits for the parent to exit.
This suggests at least two workarounds, both of which appear to work:
strace -D ls 2> >(nl)
{ strace ls; true; } 2> >(nl)
-D, to quote the man page, "[runs the] tracer process as a detached grandchild, not as parent of the tracee". The second one forces bash to do another fork to run strace by adding another command to do after.
In both cases, the extra forks mean that the process substitution doesn't end up as strace's child, avoiding the issue.

What happens internally when `ls *.c` is executed?

I've got very interested in Linux internals recently, and currently trying to understand how things work.
I knew that when I type ls
opendir() - function is called;
readdir() - function called for each directory entry in directory data store;
stat() - function can be called to get additional information on files, if required.
Please correct me if I'm missing something or if it's wrong.
The part which is mystery to me is filename expansion(globbing).
I've compared the output of strace ls
open(".", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=270336, ...}) = 0
getdents(3, /* 14 entries */, 32768) = 440
getdents(3, /* 0 entries */, 32768) = 0
close(3) = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
write(1, "2q.c ds.c fglob fnoglob\n", 272q.c ds.c fglob fnoglob
and strace ls *.c,
stat("2q.c", {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
lstat("2q.c", {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
stat("ds.c", {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
lstat("ds.c", {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
write(1, "2q.c ds.c\n", 112q.c ds.c
) = 11
and from my limited knowledge can tell that in the first case, it indeed behaves as I expected open, stat followed by getdents.
But the later one, with shell globbing isn't clear to me, because it there's already a list of files, which match the pattern. Where does this list came from?
Thanks!
Shell globbing patterns on the command line are expanded by the shell before the utility is invoked.
You can see this by enabling tracing in the shell with set -x:
$ set -x
$ ls -l f*
+ ls -l file1 file2 file3
-rw-r--r-- 1 kk wheel 0 May 11 16:49 file1
-rw-r--r-- 1 kk wheel 0 May 11 16:49 file2
-rw-r--r-- 1 kk wheel 0 May 11 16:49 file3
As you can see, the shell tells you what command it invokes (at the + prompt), and at that point it has already expanded the pattern on the command line.
The ls command does not do filename globbing. In fact, if you single quote the globbing pattern to protect it from the shell, ls is bound to be confused:
$ ls -l 'f*'
+ ls -l f*
ls: f*: No such file or directory
(unless there's actually something in the current directory called f* of course).

chown does not set SGID

I am trying to create a a directory with permissions 02770, so that the resultant permissions would be drwxrws---
When I run the below commands I get the expected behavior
rsam.svtest2.serendipity> (/home/svtest2)
$ mkdir abc
rsam.svtest2.serendipity> (/home/svtest2)
$ ls -lrt
drwxrwxr-x 2 svtest2 users 6 Apr 18 10:57 abc
rsam.svtest2.serendipity> (/home/svtest2)
$ chmod 02770 abc
rsam.svtest2.serendipity> (/home/svtest2)
$ ls -lrt
drwxrws--- 2 svtest2 users 6 Apr 18 10:57 abc
UPDATE#1
Following from above, after running mkdir and chmod on a directory, when I run chown the SGID bit gets cleared off.
rsam.svtest2.serendipity> (/home/svtest2)
$ chown svtest2:users abc
rsam.svtest2.serendipity> (/home/svtest2)
$ ls -lrt
drwxrwx--- 2 svtest2 users 6 Apr 18 10:57 abc
From the chown documentation,
Only a privileged process (Linux: one with the CAP_CHOWN capability)
may change the owner of a file. The owner of a file may change the
group of the file to any group of which that owner is a member. A
privileged process (Linux: with CAP_CHOWN) may change the group
arbitrarily.
The problem is that my user svtest does not have CAP_CHOWN capability.
Now the question boils down to - How to I get the user to have CAP_CHOWN capability?
It looks like there is some instruction here - SO - setting CAP_CHOWN
but I am yet to try it out.
However, when I run below C++ code(part of tuxedo server)
// Check if the directory exists and if not creates the directory
// with the given permissions.
struct stat st;
int lreturn_code = stat(l_string, &st);
if (lreturn_code != 0 &&
(mkdir(l_string, lpermission) != 0 ||
chmod(l_string, lpermission) != 0)) {
....
....
}
....
....
// Convert group name to group id into lgroup
if (chown(l_string, -1, lgroup) != 0) {
// System error.
}
The directory is created as below:
$ ls -l|grep DirLevel1
drwxrwx--- 2 svtest2 users 6 Apr 18 11:14 DirLevel1
Notice that the SGUID bit is not set as against when the commands were run directly as mentioned above.
Excerpt from strace for the operation:
5864 stat("/home/svtest2/data/server/log/DirLevel1/", 0x7ffd235f29f0) = -1 ENOENT (No such file or directory)
5864 mkdir("/home/svtest2/data/server/log/DirLevel1/", 02770) = 0
5864 chmod("/home/svtest2/data/server/log/DirLevel1/", 02770) = 0
5864 socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 15
5864 connect(15, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
5864 close(15) = 0
5864 socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 15
5864 connect(15, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
5864 close(15) = 0
5864 open("/etc/group", O_RDONLY|O_CLOEXEC) = 15
5864 fstat(15, {st_mode=S_IFREG|0644, st_size=652, ...}) = 0
5864 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f414d7c4000
5864 read(15, "root:x:0:\nbin:x:1:\ndaemon:x:2:\ns"..., 4096) = 652
5864 close(15) = 0
5864 munmap(0x7f414d7c4000, 4096) = 0
5864 chown("/home/svtest2/data/server/log/DirLevel1/", 4294967295, 100) = 0
5864 write(7, "\0\0\2~\6\0\0\0\0\0\21i\216\376\377\377\377\377\377\377\377\1\0\0\0\0\0\0\0\1\0\0"..., 638) = 638
5864 read(7, "\0\0\0\300\6\0\0\0\0\0\10\0\0\0\0\250\0\0\0\0\0\0\0\0\0(\0\0\0\0\0\0"..., 8208) = 192
5864 write(7, "\0\0\1}\6\0\0\0\0\0\3h\221\1\0\0\0\0\0\0\0\376\377\377\377\377\377\377\377\250\0\0"..., 381) = 381
5864 read(7, "\0\0\0\26\6\0\0\0\0\0\10\4\0\0\0\t\1\0\0\0\215\f", 8208) = 22
5864 msgsnd(43679799, {805306373, "y\0\0\0007\200\232\2\0\0\0\0\f\2\0\0\0\0\0\200\0\0\0\0\0\0\0\0\0\0\0\0"...}, 516, IPC_NOWAIT) = 0
5864 msgrcv(43614264,
From http://man.sourcentral.org/RHEL7/2+chown,
When the owner or group of an executable file are changed by an
unprivileged user the S_ISUID and S_ISGID mode bits are cleared. POSIX
does not specify whether this also should happen when root does the
chown(); the Linux behavior depends on the kernel version. In case of
a non-group-executable file (i.e., one for which the S_IXGRP bit is
not set) the S_ISGID bit indicates mandatory locking, and is not
cleared by a chown().
The above highlights a possible scenario, but I'm not sure how that is applicable to my case because it is not executable file but a directory.
Since *nix system consider a file to be an executable by seeing the 'x' permission bit, I believe a searchable directory might be considered as an executable, too.

what functions are called when i do vi

When I do a vi filename from the command prompt, what fuse functions are called if I am using the fusexmp example ? I could guess mknod, open are called.
When I do a write ie when i do :wq write is called. is that right.
There's no fantastically easy way to see which FUSE functions are called for any given file operation, but running strace(1) will record the system calls, which is quite close to the FUSE functions:
$ strace -o /tmp/vim.all vim /etc/motd
A lot of those system calls aren't related to the one file specifically, but to the process of loading vim, its dynamically linked libraries, your local configuration, and all its supporting files.
Here's some selected lines that refer to the /etc/motd that I opened:
stat("/etc/motd", {st_mode=S_IFREG|0644, st_size=183, ...}) = 0
stat("/etc/motd", {st_mode=S_IFREG|0644, st_size=183, ...}) = 0
stat("/etc/motd", {st_mode=S_IFREG|0644, st_size=183, ...}) = 0
stat("/etc/motd", {st_mode=S_IFREG|0644, st_size=183, ...}) = 0
access("/etc/motd", W_OK) = -1 EACCES (Permission denied)
open("/etc/motd", O_RDONLY) = 7
close(7) = 0
open("/etc/motd", O_RDONLY) = 7
read(7, "Welcome to Ubuntu 11.04 (GNU/Lin"..., 8192) = 183
read(7, "", 65536) = 0
close(7) = 0
stat("/etc/motd", {st_mode=S_IFREG|0644, st_size=183, ...}) = 0
The intervening lines make the repeated stat(2) calls a little less silly looking.

Linux proc/pid/fd for stdout is 11?

Executing a script with stdout redirected to a file. So /proc/$$/fd/1 should point to that file (since stdout fileno is 1). However, actual fd of the file is 11. Please, explain, why.
Here is session:
$ cat hello.sh
#!/bin/sh -e
ls -l /proc/$$/fd >&2
$ ./hello.sh > /tmp/1
total 0
lrwx------ 1 nga users 64 May 28 22:05 0 -> /dev/pts/0
lrwx------ 1 nga users 64 May 28 22:05 1 -> /dev/pts/0
lr-x------ 1 nga users 64 May 28 22:05 10 -> /home/me/hello.sh
l-wx------ 1 nga users 64 May 28 22:05 11 -> /tmp/1
lrwx------ 1 nga users 64 May 28 22:05 2 -> /dev/pts/0
I have a suspicion, but this is highly dependent on how your shell behaves. The file descriptors you see are:
0: standard input
1: standard output
2: standard error
10: the running script
11: a backup copy of the script's normal standard out
Descriptors 10 and 11 are close on exec, so won't be present in the ls process. 0-2 are, however, prepared for ls before forking. I see this behaviour in dash (Debian Almquist shell), but not in bash (Bourne again shell). Bash instead does the file descriptor manipulations after forking, and incidentally uses 255 rather than 10 for the script. Doing the change after forking means it won't have to restore the descriptors in the parent, so it doesn't have the spare copy to dup2 from.
The output of strace can be helpful here.
The relevant section is
fcntl64(1, F_DUPFD, 10) = 11
close(1) = 0
fcntl64(11, F_SETFD, FD_CLOEXEC) = 0
dup2(2, 1) = 1
stat64("/home/random/bin/ls", 0xbf94d5e0) = -1 ENOENT (No such file or
+++++++>directory)
stat64("/usr/local/bin/ls", 0xbf94d5e0) = -1 ENOENT (No such file or directory)
stat64("/usr/bin/ls", 0xbf94d5e0) = -1 ENOENT (No such file or directory)
stat64("/bin/ls", {st_mode=S_IFREG|0755, st_size=96400, ...}) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
+++++++>child_tidptr=0xb75a8938) = 22748
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22748
--- SIGCHLD (Child exited) # 0 (0) ---
dup2(11, 1) = 1
So, the shell moves the existing stdout to an available file descriptor above 10 (namely, 11), then moves the existing stderr onto its own stdout (due to the >&2 redirect), then restores 11 to its own stdout when the ls command is finished.

Resources