find exec rm: No such file or directory - linux

Why "No such file or directory" in example below?
There are workarounds to not get this error, the question is why in this exact example the error is produced.
cd /tmp
mkdir -p TOP1/AB/AC/AD/AE/AF
mkdir -p TOP2/AB/AC/AD/AE/AF
find . -iname 'A*' -exec rm -rvf {} \;
removed directory: `./TOP1/AB/AC/AD/AE/AF'
removed directory: `./TOP1/AB/AC/AD/AE'
removed directory: `./TOP1/AB/AC/AD'
removed directory: `./TOP1/AB/AC'
removed directory: `./TOP1/AB'
find: `./TOP1/AB': No such file or directory
removed directory: `./TOP2/AB/AC/AD/AE/AF'
removed directory: `./TOP2/AB/AC/AD/AE'
removed directory: `./TOP2/AB/AC/AD'
removed directory: `./TOP2/AB/AC'
removed directory: `./TOP2/AB'
find: `./TOP2/AB': No such file or directory
Using -depth or \+ or -delete fix this issue:
find . -iname 'A*' -exec rm -rvf {} \+
find . -depth -iname 'A*' -exec rm -rvf {} \;
find . -iname 'A*' -delete

I think the answer is in the man page itself: https://linux.die.net/man/1/find
Also see \; vs +: Using semicolon (;) vs plus (+) with exec in find
To understand the whole process better, you can print the files instead of deleting them. Considering the TOP1 folder for example:
find . -iname 'A*' -exec echo "{}" \;
Output:
./TOP1/AB
./TOP1/AB/AC
./TOP1/AB/AC/AD
./TOP1/AB/AC/AD/AE
./TOP1/AB/AC/AD/AE/AF
What \; does is: Run file on every file in or below the current directory.
The order is important here, i.e. first the parent folder is treated, and then its child folders are recursively traversed.
On the other hand, this is how rm -frv works:
rm -frv ./TOP1
Output:
removed directory './TOP1/AB/AC/AD/AE/AF'
removed directory './TOP1/AB/AC/AD/AE'
removed directory './TOP1/AB/AC/AD'
removed directory './TOP1/AB/AC'
removed directory './TOP1/AB'
removed directory './TOP1'
So rm -fr works practically the other way round than find, which is the main cause of the problem here:
find together with \; first starts with the parent folder TOP1, where the folder TOP1 and its contents will be completely deleted thanks to the part -exec rm -rvf {} \;. On the next round find then will try to handle the next child folder below, where it apparently first checks (via stat ./TOP1), if the base folder still exists, and it only then goes one level deeper. But since ./TOP1 was already deleted in the first round, stat ./TOP1 will fail, hence the error message.
With the other parameters available for the find you can obviously change the default behavior:
-depth: Process each directory's contents before the (parent) directory itself. The -delete action also implies -depth. (i.e. treversing from very bottom to top - just like rm -fr does!)
-delete: Use of -delete automatically turns on the -depth option
Using +: the -exec rm -rvf {} + part is executed only once, e.g. -exec rm -rvf {} +:
rm -frv ./TOP1/AB ./TOP1/AB/AC ./TOP1/AB/AC/AD ./TOP1/AB/AC/AD/AE ./TOP1/AB/AC/AD/AE/AF
Last but not least:
When rm is used with the -fr parameter, it becomes very "tolerant", so, no error is thrown. It will try to forcibly and recursively delete everything that is passed to it; and, if it encounters files/directories that do not exist, it will not complain as well.

This is an implementation detail of find. You'll find the ultimate answer to your question in it's source code.
On Linux another way to understand what is happening behind the scenes is strace. I've prepared some interesting output from strace below. You may have a look at the order of the openat system calls issues by both find invocations (with and without -depth):
mkdir -pv TOP{1,2}/AB/AC/AD/AE/AF && strace -o foo find . -depth -iname 'A*' -exec rm -rvf {} \; ; grep openat foo
mkdir: created directory 'TOP1/AB'
mkdir: created directory 'TOP1/AB/AC'
mkdir: created directory 'TOP1/AB/AC/AD'
mkdir: created directory 'TOP1/AB/AC/AD/AE'
mkdir: created directory 'TOP1/AB/AC/AD/AE/AF'
mkdir: created directory 'TOP2/AB'
mkdir: created directory 'TOP2/AB/AC'
mkdir: created directory 'TOP2/AB/AC/AD'
mkdir: created directory 'TOP2/AB/AC/AD/AE'
mkdir: created directory 'TOP2/AB/AC/AD/AE/AF'
removed directory './TOP1/AB/AC/AD/AE/AF'
removed directory './TOP1/AB/AC/AD/AE'
removed directory './TOP1/AB/AC/AD'
removed directory './TOP1/AB/AC'
removed directory './TOP1/AB'
removed directory './TOP2/AB/AC/AD/AE/AF'
removed directory './TOP2/AB/AC/AD/AE'
removed directory './TOP2/AB/AC/AD'
removed directory './TOP2/AB/AC'
removed directory './TOP2/AB'
5:openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
9:openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
20:openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
28:openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
44:openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = 3
52:openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
60:openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
93:openat(AT_FDCWD, "/proc/filesystems", O_RDONLY|O_CLOEXEC) = 3
99:openat(AT_FDCWD, ".", O_RDONLY|O_CLOEXEC) = 3
102:openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 4
106:openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = 4
114:openat(AT_FDCWD, ".", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 4
125:openat(5, "TOP1", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 6
137:openat(7, "AB", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 6
149:openat(8, "AC", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 6
161:openat(9, "AD", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 6
173:openat(10, "AE", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 6
186:openat(11, "AF", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 5
223:openat(8, "..", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 5
231:openat(5, "..", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 6
237:openat(6, "TOP2", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 5
249:openat(7, "AB", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 5
261:openat(8, "AC", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 5
273:openat(9, "AD", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 5
285:openat(10, "AE", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 5
298:openat(11, "AF", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 5
332:openat(8, "..", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 5
340:openat(5, "..", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 6
mkdir -pv TOP{1,2}/AB/AC/AD/AE/AF && strace -o foo find . -iname 'A*' -exec rm -rvf {} \; ; grep openat foo
mkdir: created directory 'TOP1/AB'
mkdir: created directory 'TOP1/AB/AC'
mkdir: created directory 'TOP1/AB/AC/AD'
mkdir: created directory 'TOP1/AB/AC/AD/AE'
mkdir: created directory 'TOP1/AB/AC/AD/AE/AF'
mkdir: created directory 'TOP2/AB'
mkdir: created directory 'TOP2/AB/AC'
mkdir: created directory 'TOP2/AB/AC/AD'
mkdir: created directory 'TOP2/AB/AC/AD/AE'
mkdir: created directory 'TOP2/AB/AC/AD/AE/AF'
removed directory './TOP1/AB/AC/AD/AE/AF'
removed directory './TOP1/AB/AC/AD/AE'
removed directory './TOP1/AB/AC/AD'
removed directory './TOP1/AB/AC'
removed directory './TOP1/AB'
find: ‘./TOP1/AB’: No such file or directory
removed directory './TOP2/AB/AC/AD/AE/AF'
removed directory './TOP2/AB/AC/AD/AE'
removed directory './TOP2/AB/AC/AD'
removed directory './TOP2/AB/AC'
removed directory './TOP2/AB'
find: ‘./TOP2/AB’: No such file or directory
5:openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
9:openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
20:openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
28:openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
44:openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = 3
52:openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
60:openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
93:openat(AT_FDCWD, "/proc/filesystems", O_RDONLY|O_CLOEXEC) = 3
99:openat(AT_FDCWD, ".", O_RDONLY|O_CLOEXEC) = 3
102:openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 4
106:openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = 4
114:openat(AT_FDCWD, ".", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 4
125:openat(5, "TOP1", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 6
142:openat(7, "AB", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)
143:openat(AT_FDCWD, "/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 6
148:openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/findutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
149:openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/findutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
150:openat(AT_FDCWD, "/usr/share/locale-langpack/en_US/LC_MESSAGES/findutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
151:openat(AT_FDCWD, "/usr/share/locale-langpack/en/LC_MESSAGES/findutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
154:openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
155:openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
156:openat(AT_FDCWD, "/usr/share/locale-langpack/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
157:openat(AT_FDCWD, "/usr/share/locale-langpack/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
164:openat(5, "TOP2", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = 6
179:openat(7, "AB", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)

Related

crontab with ed by commands on stream, results in "no modification made"

I am trying to append a line to my crontab file. I know there are other ways to work around this problem, but still want to know what caused it. The command is run on raspberry pi 3 B+, raspbian lite is installed, with GNU ed 1.15, cron 3.0pl1-134+deb10u1.
The command that I'm stuck on is this:
$ echo -e 'a\n#asdf\n.\nwQ' | EDITOR=ed crontab -e
902
909
No modification made
I'm expecting it to add line #asdf at the end of my crontab file, but it doesn't.
Setting EDITOR='tee -a' as suggested on https://stackoverflow.com/a/30123606/8842387 does not solve the problem. So I guess it is the problem with cron.
Strangely enough, when I give ed commands from the keyboard directly, rather than streaming it, it just works. Maybe subshell creation caused the problem?
Here I'm attaching a few of the last lines from strace result.
$ echo -e 'a\n#asdf\n.\nwQ' | EDITOR=ed strace crontab -e
execve("/usr/bin/crontab", ["crontab", "-e"], 0x7ee54c14 /* 29 vars */) = 0
access("/etc/suid-debug", F_OK) = -1 ENOENT (No such file or directory)
...
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\7\0\0\0\7\0\0\0\0"..., 4096) = 659
_llseek(3, -393, [266], SEEK_CUR) = 0
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\7\0\0\0\7\0\0\0\0"..., 4096) = 393
close(3) = 0
getpid() = 18579
socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path="/dev/log"}, 110) = 0
send(3, "<78>Nov 20 15:31:25 crontab[1857"..., 56, MSG_NOSIGNAL) = 56
openat(AT_FDCWD, "crontabs/pi", O_RDONLY) = -1 EACCES (Permission denied)
openat(AT_FDCWD, "/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=2995, ...}) = 0
read(4, "# Locale name alias data base.\n#"..., 4096) = 2995
read(4, "", 4096) = 0
close(4) = 0
openat(AT_FDCWD, "/usr/share/locale/en_GB.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_GB.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_GB/LC_MESSAGES/libc.mo", O_RDONLY) = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=1433, ...}) = 0
mmap2(NULL, 1433, PROT_READ, MAP_PRIVATE, 4, 0) = 0x76f50000
close(4) = 0
openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "crontabs/pi/: fdopen: Permission"..., 39crontabs/pi/: fdopen: Permission denied) = 39
exit_group(1) = ?
+++ exited with 1 +++
openat(AT_FDCWD, "crontabs/pi", O_RDONLY) = -1 EACCES (Permission denied) looks a bit suspicious, but not sure why it opens the file read-only.
EDIT:
As suggested by #tink, I ran EDITOR=ed strace crontab -e to see what strace gives on an interactive session. The result was almost same (only varying on pid and fd numbers).
I noticed that running echo "..." | EDITOR=ed crontab -e exited with message No modification made but with strace the process halts without any messages. (EDITOR=ed strace crontab -e 2>&1 | grep "No mod" prints nothing). Guess the strace triggers different errors.
Following up on my VISUAL comment, these worked for me:
( unset VISUAL; printf '%s\n' a '#abcd' . wq | EDITOR=ed crontab -e )
printf '%s\n' a '#abcd' . wq | VISUAL=ed crontab -e
In my environment, both VISUAL and EDITOR are set to "vim"
Or, more roundabout, but don't need to monkey with env vars. This one also allows you to do it silently:
crontab <(printf '%s\n' a '#asdf' . '%p' | ed -s <(crontab -l))
I was doing the above on a Mac. On Linux, I can reproduce your observations, but can't explain them.
A small tweak to the last command works:
printf '%s\n' a '#asdf' . '%p' Q | ed -s <(crontab -l) | crontab -
TLDR; (sleep 1; echo -e 'a\n#asdf\n.\nwQ') | EDITOR=ed crontab -e works!
The problem was on crontab.
When I invoke crontab -e it creates a temporary copy of the user cron table in /tmp directory.
Then opens the temporary file with an editor specified by $EDITOR.
After the editing is done, crontab check if the file modification date have changed since its creation.
This is implemented in the patch that enables editing cron table via temporary file.
In my case, ed getting its command from stdin finished the editing too fast so that even a single digit of the modification timestamp of the temporary file had not been changed.
As crontab considered no human can make edition that fast, it assumes no modification made and discards it.
To bypass this behavior, I added sleep 1 before the release of the command.
This will hold ed to wait for its command from stdin after crontab created tempfile, which effectively lets the modification timestamp different.

Copy and move's command effect on inode

I interpret inode as a pointer to the actual place where the file is stored.
But I have problem understanding:
If I use cp file1 file2 in a place where file2 already exists, the inode doesn't change. And If there is originally a hard-link to file2, they now both point to the new file just copied here.
The only reason I can think of is that Linux interprets this as modifying
the file instead of deleting and creating a new file. I don't understand why it's designed this way?
But when I use mv file1 file2, the inode changes to the inode of file1.
You are correct in stating that cp will modify the file instead of deleting and recreating.
Here is a view of the underlying system calls as seen by strace (part of the output of strace cp file1 file2):
open("file2", O_WRONLY|O_TRUNC) = 4
stat("file2", {st_mode=S_IFREG|0664, st_size=6, ...}) = 0
stat("file1", {st_mode=S_IFREG|0664, st_size=3, ...}) = 0
stat("file2", {st_mode=S_IFREG|0664, st_size=6, ...}) = 0
open("file1", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0664, st_size=3, ...}) = 0
open("file2", O_WRONLY|O_TRUNC) = 4
fstat(4, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
read(3, "hi\n", 65536) = 3
write(4, "hi\n", 3) = 3
read(3, "", 65536) = 0
close(4) = 0
close(3) = 0
As you can see, it detects that file2 is present (stat returns 0), but then opens it for writing (O_WRONLY|O_TRUNC) without first doing an unlink.
See for example POSIX.1-2017, which specifies that the destination file shall only be unlink-ed where it could not be opened for writing and -f is used:
A file descriptor for dest_file shall be obtained by performing
actions equivalent to the open() function defined in the System
Interfaces volume of POSIX.1-2017 called using dest_file as the path
argument, and the bitwise-inclusive OR of O_WRONLY and O_TRUNC as the
oflag argument.
If the attempt to obtain a file descriptor fails and the -f option is
in effect, cp shall attempt to remove the file by performing actions
equivalent to the unlink() function defined in the System Interfaces
volume of POSIX.1-2017 called using dest_file as the path argument. If
this attempt succeeds, cp shall continue with step 3b.
This implies that if the destination file exists, the copy will succeed (without resorting to -f behaviour) if the cp process has write permission on it (not necessarily run as the user that owns the file), even if it does not have write permission on the containing directory. By contrast, unlinking and recreating would require write permission on the directory. I would speculate that this is behind the reason why the standard is as it is.
The --remove-destination option on GNU cp will make it do instead what you thought ought to be the default.
Here is the relevant part of the output of strace cp --remove-destination file1 file2. Note the unlink this time.
stat("file2", {st_mode=S_IFREG|0664, st_size=6, ...}) = 0
stat("file1", {st_mode=S_IFREG|0664, st_size=3, ...}) = 0
lstat("file2", {st_mode=S_IFREG|0664, st_size=6, ...}) = 0
unlink("file2") = 0
open("file1", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0664, st_size=3, ...}) = 0
open("file2", O_WRONLY|O_CREAT|O_EXCL, 0664) = 4
fstat(4, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
read(3, "hi\n", 65536) = 3
write(4, "hi\n", 3) = 3
read(3, "", 65536) = 0
close(4) = 0
close(3) = 0
When you use mv and the source and destination paths are on the same file filesystem, it will do an rename, and this will have the effect of unlinking any existing file at the target path. Here is the relevant part of the output of strace mv file1 file2.
access("file2", W_OK) = 0
rename("file1", "file2") = 0
In either case where an destination path is unlinked (whether explicitly by unlink() as called from cp --remove-destination, or as part of the effect of rename() as called from mv), the link count of the inode to which it was pointing will be decremented, but it will remain on the filesystem if either the link count is still >0 or if any processes have open filehandles on it. Any other (hard) links to this inode (i.e. other directory entries for it) will remain.
Investigating using ls -i
ls -i will show the inode numbers (as the first column when combined with -l), which helps demonstrate what is happening.
Example with default cp action
$ rm file1 file2 file3
$ echo hi > file1
$ echo world > file2
$ ln file2 file3
$ ls -li file*
49 -rw-rw-r-- 1 myuser mygroup 3 Jun 13 10:43 file1
50 -rw-rw-r-- 2 myuser mygroup 6 Jun 13 10:43 file2
50 -rw-rw-r-- 2 myuser mygroup 6 Jun 13 10:43 file3
$ cp file1 file2
$ ls -li file*
49 -rw-rw-r-- 1 myuser mygroup 3 Jun 13 10:43 file1
50 -rw-rw-r-- 2 myuser mygroup 3 Jun 13 10:43 file2 <=== exsting inode
50 -rw-rw-r-- 2 myuser mygroup 3 Jun 13 10:43 file3 <=== exsting inode
(Note existing inode 50 now has size 3).
Example with --remove-destination
$ rm file1 file2 file3
$ echo hi > file1
$ echo world > file2
$ ln file2 file3
$ ls -li file*
49 -rw-rw-r-- 1 myuser mygroup 3 Jun 13 10:46 file1
50 -rw-rw-r-- 2 myuser mygroup 6 Jun 13 10:46 file2
50 -rw-rw-r-- 2 myuser mygroup 6 Jun 13 10:46 file3
$ cp --remove-destination file1 file2
$ ls -li file*
49 -rw-rw-r-- 1 myuser mygroup 3 Jun 13 10:46 file1
55 -rw-rw-r-- 1 myuser mygroup 3 Jun 13 10:47 file2 <=== new inode
50 -rw-rw-r-- 1 myuser mygroup 6 Jun 13 10:46 file3 <=== existing inode
(Note new inode 55 has size 3. Unmodified inode 50 still has size 6.)
Example with mv
$ rm file1 file2 file3
$ echo hi > file1
$ echo world > file2
$ ln file2 file3
$ ls -li file*
49 -rw-rw-r-- 1 myuser mygroup 3 Jun 13 11:05 file1
50 -rw-rw-r-- 2 myuser mygroup 6 Jun 13 11:05 file2
50 -rw-rw-r-- 2 myuser mygroup 6 Jun 13 11:05 file3
$ mv file1 file2
$ ls -li file*
49 -rw-rw-r-- 1 myuser mygroup 3 Jun 13 11:05 file2 <== existing inode
50 -rw-rw-r-- 1 myuser mygroup 6 Jun 13 11:05 file3 <== existing inode
#alaniwi's answer covers what is is happening, but there's an implicit why here as well.
The reason cp works the way it does is to provide a way of replacing a file with mulitple names, having all of those names refer to the new file. When the destination of cp is a file that already exists, possibly with multiple names via hard or soft links, cp will make all of those names refer to the new file. There will be no 'orphan' references to the old file left over.
Given this command, it is pretty easy to get the 'just change the file for one name' behavior -- unlink the file first. Given just that as a primitive it would be very hard to implement the 'change all references to point to the new contents' behavior.
Of course, doing rm+cp has some race condition issues (it is two commands), which is why the install command got added in BSD unix -- it basically just does rm + cp, along with some checks to make it atomic in the rare case two people try to install to the same path simultaneously, as well as the more serious problems of someone reading from the file you're trying to install to (a problem with plain cp). Then the GNU version added options to backup the old version and various other useful bookkeeping.
An inode is a collection of metadata for a file, i.e. information about a file, in a Unix/ Unix-like filesystem. It includes permission data, last access/ modify time, file size, etc.
Notably, a file's name/ path is not part of the inode. A filename is just a human-readable identifier for an inode. A file can have one or more names, the number of which is represented in the inode by its number of "links" (hard links). The number associated with the inode, the inode number, which I believe you're interpreting as its physical location on disk, is rather simply a unique identifier for the inode. An inode does contain the location of the file on disk, but that is not the inode number.
So knowing this, the difference you're seeing is in how cp and mv function. When you cp a file you're creating a new inode with a new name and copying the contents of the old file to a new location on disk. When you mv a file all you're doing is changing one of its names. If the new name is already the name of another file, the name is disassociated with the old file (and the old file's link count is reduced by 1) and associated with the new file.
You can read more about inodes here.

Why is this vim plugin not getting loaded?

I am using pathogen to build a plugin bundle to manage my blog. For starters I wanted to factor out some commands I defined directly in my vimrc and move them to a plugin. Here's how I set up the bundle:
call pathogen#infect('/home/frew/code/blog/etc/vim-bundle/{}')
Here is a brief snippet of the beginning of the plugin:
echom "loaded hugo plugin"
if exists('g:loaded_hugo') || &cp || v:version < 700
finish
endif
let g:loaded_hugo = 1
function! Tagged(tag)
exe 'args `bin/tag-files ' . a:tag . '`'
endfunction
command! -nargs=1 Tagged call Tagged('<args>')
And here is the directory structure of the bundle:
/home/frew/code/blog/etc/vim-bundle/
└── hugo
└── plugin
└── hugo.vim
2 directories, 1 file
The echom is never getting run, and furthermore I straced a gvim and here is what I found matching /home/frew/code/blog/etc/vim-bundle/: (I filtered out the duplicates with sort -u)
open("/home/frew/code/blog/etc/vim-bundle/hugo/ftdetect/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/frew/code/blog/etc/vim-bundle/hugo/ftplugin/html/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/frew/code/blog/etc/vim-bundle/hugo/ftplugin/markdown/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/frew/code/blog/etc/vim-bundle/hugo/ftplugin/markdown/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC <unfinished ...>
open("/home/frew/code/blog/etc/vim-bundle/hugo/ftplugin/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/frew/code/blog/etc/vim-bundle/hugo/ftplugin/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC <unfinished ...>
open("/home/frew/code/blog/etc/vim-bundle/hugo/ftplugin/vim/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/markdown/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/vim/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/vim/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC <unfinished ...>
open("/home/frew/code/blog/etc/vim-bundle/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 18
stat("/home/frew/code/blog/etc/vim-bundle/hugo/after", 0x7fffd759c6d0) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/autoload/ctrlp/utils.vim", 0x7fffd759a620) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/autoload/editorconfig.vim", 0x7fffd759c650) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/autoload/gitgutter/async.vim", 0x7fffd759d2f0) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/autoload/gitgutter/debug.vim", <unfinished ...>
stat("/home/frew/code/blog/etc/vim-bundle/hugo/autoload/gitgutter/diff.vim", 0x7fffd759e6d0) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/autoload/gitgutter/sign.vim", 0x7fffd75a0040) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/autoload/htmlcomplete.vim", 0x7fffd759c4c0) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/filetype.vim", 0x7fffd759f830) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/ftoff.vim", 0x7fffd759f830) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/ftplugin/html.vim", 0x7fffd759cf40) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/ftplugin/html.vim", 0x7fffd759d050) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/ftplugin/html.vim", 0x7fffd759d270) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/ftplugin/markdown.vim", 0x7fffd759da70) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/ftplugin/markdown.vim", 0x7fffd759dda0) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/ftplugin/markdown.vim", <unfinished ...>
stat("/home/frew/code/blog/etc/vim-bundle/hugo/ftplugin/vim.vim", 0x7fffd759e6e0) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/ftplugin/vim.vim", 0x7fffd759e830) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/ftplugin/vim.vim", <unfinished ...>
stat("/home/frew/code/blog/etc/vim-bundle/hugo/indent/markdown.vim", 0x7fffd759da70) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/indent/markdown.vim", 0x7fffd759db80) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/indent/markdown.vim", 0x7fffd759dda0) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/indent/vim.vim", 0x7fffd759e830) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/indent/vim.vim", <unfinished ...>
stat("/home/frew/code/blog/etc/vim-bundle/hugo/scripts.vim", 0x7fffd75a0060) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
stat("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/css.vim", 0x7fffd759ae20) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/css.vim", 0x7fffd759af30) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/css.vim", 0x7fffd759b150) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/html.vim", 0x7fffd759b9e0) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/html.vim", 0x7fffd759baf0) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/html.vim", 0x7fffd759bd10) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/javascript.vim", 0x7fffd759ae20) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/javascript.vim", 0x7fffd759af30) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/javascript.vim", 0x7fffd759b150) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/markdown.vim", 0x7fffd759c510) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/markdown.vim", 0x7fffd759c620) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/markdown.vim", 0x7fffd759c840) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/pod.vim", 0x7fffd759b230) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/pod.vim", 0x7fffd759b380) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/vb.vim", 0x7fffd759ae20) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/vb.vim", 0x7fffd759af30) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/vb.vim", 0x7fffd759b150) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/vim.vim", 0x7fffd759d2d0) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/syntax/vim.vim", <unfinished ...>
stat("/home/frew/code/blog/etc/vim-bundle/hugo/vimfiles", 0x7fffd759cce0) = -1 ENOENT (No such file or directory)
stat("/home/frew/code/blog/etc/vim-bundle/hugo/vimfiles/after", 0x7fffd759c750) = -1 ENOENT (No such file or directory)
Clearly it found the hugo plugin directory, it just never even tried looking in the related plugin subdirectory.
What am I missing here?
Update: Here's the current value of &rtp:
/home/frew/code/blog/etc/vim-bundle/hugo,/home/frew/.vim,/home/frew/.vim/bundle-lua/neocomplete,/home/frew/.vim/
bundle-python/editorconfig,/home/frew/.vim/bundle-python/ultisnips,/home/frew/.vim/bundle/FastFold,/home/frew/.v
im/bundle/IndentAnything,/home/frew/.vim/bundle/airline,/home/frew/.vim/bundle/better-whitespace,/home/frew/.vim
/bundle/colors-solarized,/home/frew/.vim/bundle/commentary,/home/frew/.vim/bundle/csv,/home/frew/.vim/bundle/ctr
lp,/home/frew/.vim/bundle/denite,/home/frew/.vim/bundle/eunuch,/home/frew/.vim/bundle/exchange,/home/frew/.vim/b
undle/fugitive,/home/frew/.vim/bundle/fugitive-gitlab,/home/frew/.vim/bundle/gitgutter,/home/frew/.vim/bundle/go
,/home/frew/.vim/bundle/goyo,/home/frew/.vim/bundle/inkpot,/home/frew/.vim/bundle/l9,/home/frew/.vim/bundle/last
place,/home/frew/.vim/bundle/lost,/home/frew/.vim/bundle/matchit,/home/frew/.vim/bundle/matchmaker,/home/frew/.v
im/bundle/obsession,/home/frew/.vim/bundle/pathogen,/home/frew/.vim/bundle/perl,/home/frew/.vim/bundle/projectio
nist,/home/frew/.vim/bundle/python,/home/frew/.vim/bundle/quick-scope,/home/frew/.vim/bundle/repeat,/home/frew/.
vim/bundle/sleuth,/home/frew/.vim/bundle/splitjoin,/home/frew/.vim/bundle/surround,/home/frew/.vim/bundle/tabula
r,/home/frew/.vim/bundle/terminus,/home/frew/.vim/bundle/textobj-between,/home/frew/.vim/bundle/textobj-entire,/
home/frew/.vim/bundle/textobj-underscore,/home/frew/.vim/bundle/textobj-user,/home/frew/.vim/bundle/unimpaired,/
home/frew/.vim/bundle/vinegar,/home/frew/.vim/bundle/visual-star-search,/home/frew/.vim/bundle/wipeout,/var/lib/
vim/addons,/usr/share/vim/vimfiles,/usr/share/vim/vim80,/usr/share/vim/vimfiles/after,/var/lib/vim/addons/after,
/home/frew/.vim/bundle/tabular/after,/home/frew/.vim/bundle-python/ultisnips/after,/home/frew/.vim/after
Tim Pope answered this question on github. Plugins get loaded before sessions, so modifying &rtp at session time is too late if you want to load plugins.

How to change umask so all files start with different modal bits than new directories

Specifiacally, I need to give files rw----r--
and dirs rwx--xr-x
Use umask 062.
This works because umask only unsets bits, and files aren't normally created with executable bits set in the first place:
$ umask 062
$ touch myfile; mkdir mydir
$ ls -ld myfile mydir
drwx--xr-x 1 user user 0 Dec 5 15:21 mydir
-rw----r-- 1 user user 0 Dec 5 15:21 myfile

chown does not set SGID

I am trying to create a a directory with permissions 02770, so that the resultant permissions would be drwxrws---
When I run the below commands I get the expected behavior
rsam.svtest2.serendipity> (/home/svtest2)
$ mkdir abc
rsam.svtest2.serendipity> (/home/svtest2)
$ ls -lrt
drwxrwxr-x 2 svtest2 users 6 Apr 18 10:57 abc
rsam.svtest2.serendipity> (/home/svtest2)
$ chmod 02770 abc
rsam.svtest2.serendipity> (/home/svtest2)
$ ls -lrt
drwxrws--- 2 svtest2 users 6 Apr 18 10:57 abc
UPDATE#1
Following from above, after running mkdir and chmod on a directory, when I run chown the SGID bit gets cleared off.
rsam.svtest2.serendipity> (/home/svtest2)
$ chown svtest2:users abc
rsam.svtest2.serendipity> (/home/svtest2)
$ ls -lrt
drwxrwx--- 2 svtest2 users 6 Apr 18 10:57 abc
From the chown documentation,
Only a privileged process (Linux: one with the CAP_CHOWN capability)
may change the owner of a file. The owner of a file may change the
group of the file to any group of which that owner is a member. A
privileged process (Linux: with CAP_CHOWN) may change the group
arbitrarily.
The problem is that my user svtest does not have CAP_CHOWN capability.
Now the question boils down to - How to I get the user to have CAP_CHOWN capability?
It looks like there is some instruction here - SO - setting CAP_CHOWN
but I am yet to try it out.
However, when I run below C++ code(part of tuxedo server)
// Check if the directory exists and if not creates the directory
// with the given permissions.
struct stat st;
int lreturn_code = stat(l_string, &st);
if (lreturn_code != 0 &&
(mkdir(l_string, lpermission) != 0 ||
chmod(l_string, lpermission) != 0)) {
....
....
}
....
....
// Convert group name to group id into lgroup
if (chown(l_string, -1, lgroup) != 0) {
// System error.
}
The directory is created as below:
$ ls -l|grep DirLevel1
drwxrwx--- 2 svtest2 users 6 Apr 18 11:14 DirLevel1
Notice that the SGUID bit is not set as against when the commands were run directly as mentioned above.
Excerpt from strace for the operation:
5864 stat("/home/svtest2/data/server/log/DirLevel1/", 0x7ffd235f29f0) = -1 ENOENT (No such file or directory)
5864 mkdir("/home/svtest2/data/server/log/DirLevel1/", 02770) = 0
5864 chmod("/home/svtest2/data/server/log/DirLevel1/", 02770) = 0
5864 socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 15
5864 connect(15, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
5864 close(15) = 0
5864 socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 15
5864 connect(15, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
5864 close(15) = 0
5864 open("/etc/group", O_RDONLY|O_CLOEXEC) = 15
5864 fstat(15, {st_mode=S_IFREG|0644, st_size=652, ...}) = 0
5864 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f414d7c4000
5864 read(15, "root:x:0:\nbin:x:1:\ndaemon:x:2:\ns"..., 4096) = 652
5864 close(15) = 0
5864 munmap(0x7f414d7c4000, 4096) = 0
5864 chown("/home/svtest2/data/server/log/DirLevel1/", 4294967295, 100) = 0
5864 write(7, "\0\0\2~\6\0\0\0\0\0\21i\216\376\377\377\377\377\377\377\377\1\0\0\0\0\0\0\0\1\0\0"..., 638) = 638
5864 read(7, "\0\0\0\300\6\0\0\0\0\0\10\0\0\0\0\250\0\0\0\0\0\0\0\0\0(\0\0\0\0\0\0"..., 8208) = 192
5864 write(7, "\0\0\1}\6\0\0\0\0\0\3h\221\1\0\0\0\0\0\0\0\376\377\377\377\377\377\377\377\250\0\0"..., 381) = 381
5864 read(7, "\0\0\0\26\6\0\0\0\0\0\10\4\0\0\0\t\1\0\0\0\215\f", 8208) = 22
5864 msgsnd(43679799, {805306373, "y\0\0\0007\200\232\2\0\0\0\0\f\2\0\0\0\0\0\200\0\0\0\0\0\0\0\0\0\0\0\0"...}, 516, IPC_NOWAIT) = 0
5864 msgrcv(43614264,
From http://man.sourcentral.org/RHEL7/2+chown,
When the owner or group of an executable file are changed by an
unprivileged user the S_ISUID and S_ISGID mode bits are cleared. POSIX
does not specify whether this also should happen when root does the
chown(); the Linux behavior depends on the kernel version. In case of
a non-group-executable file (i.e., one for which the S_IXGRP bit is
not set) the S_ISGID bit indicates mandatory locking, and is not
cleared by a chown().
The above highlights a possible scenario, but I'm not sure how that is applicable to my case because it is not executable file but a directory.
Since *nix system consider a file to be an executable by seeing the 'x' permission bit, I believe a searchable directory might be considered as an executable, too.

Resources