Failure of rsync of multi-user directory with sshfs fuse mount - linux

Question

I use rsync for automatic periodic syncing of the home folder (root user) in a linux server that is used by several people. A service that users need is the possibility of mounting remote directories through sshfs. However, when there is an sshfs mount, rsync fails giving the following messages
rsync: readlink_stat("/home/???/???") failed: Permission denied (13)
IO error encountered -- skipping file deletion
...
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.1]
Because of this error, the automated sync does not work as expected, in particular due to skipping the file deletion and a non-zero exit code. The sync is only necessary for the file system where home is mounted, so the wanted behavior is that the sshfs mounts be ignored. The -x / --one-file-system rsync option does not resolve it.
This problem is clearly explained in https://www.agwa.name/blog/post/how_fuse_can_break_rsync_backups . The follow-up article (https://www.agwa.name/blog/post/easily_running_fuse_in_an_isolated_mount_namespace) proposes a solution, though not an acceptable one because fuse mounts are only visible to the process created the mount.
I am looking for a solution that does not affect sshfs usability and is transparent for the users.

Answer 1

The problem is that FUSE denies stat access to other users, including root. Rsync requires stat access on all source files and directories specified. But when an rsync process owned by another user stats a FUSE mount-point, FUSE denies that process access to the mount-point's attributes, causing rsync to throw the said "permission denied" error. Mauricio Villega's solution works by telling rsync to skip FUSE mount-points listed by the mount command. Here is another version of Villega's solution that specifies a white-list of filesystem types using the findmnt command. I chose ext3 and ext4 but you may add other types as needed.
#!/bin/sh
# Which paths to rsync (note the lack of trailing slash tells rsync to preserve source path name at destination).
SOURCES=(
/home
)
# Which filesystem types are supported.
FSTYPES=(
ext3
ext4
)
# Rsync each source.
for SOURCE in ${SOURCES[#]}; do
# Build exclusion list (array of "--exclude=PATH").
excludedPaths=$(findmnt --invert --list --noheadings --output TARGET --types $(IFS=',';echo "${FSTYPES[*]}"))
printf -v exclusionList -- "--exclude=%s " ${excludedPaths[#]}
# Rsync.
rsync --archive ${exclusionList[#]} --hard-links --delete --inplace --one-file-system ${SOURCE} /backup
done
Note that it builds the exclusion list inside the loop to address a fundamental problem with this solution. That problem is due to rsync'ing from a live system where a user could create new FUSE mount-points while rsync is running. The exclusion list needs to be updated frequently enough to include new FUSE mount-points. You may divide the home directory further by each username by modifying the SOURCES array as shown.
SOURCES=(
/home/user1
/home/user2
)
If you are using LVM, an alternative solution is rsync from an LVM snapshot. An LVM snapshot provides a simple (e.g., no FUSE mount-points) and frozen view of the logical volume it is linked to. The downside is that you must reserve space for the LVM snapshot's copy-on-write (COW) activity. It is crucial that you discard the LVM snapshot after you are done with it; otherwise the LVM snapshot will continue to grow in size as modifications are made. Here is a sample script that uses LVM snapshots. Note that it does not need to build an exclusion list for rsync.
# Create and mount LVM snapshot.
lvcreate --extents 100%FREE --snapshot --name snapRoot /dev/vgSystem/lvRoot
mount -o ro /dev/mapper/snapRoot /root/mnt # Note that only root has access to this mount-point.
# Rsync each source.
for SOURCE in ${SOURCES[#]}; do
rsync --archive --hard-links --delete --inplace --one-file-system /root/mnt/${SOURCE} /backup
done
# Discard LVM snapshot.
umount /root/mnt
lvremove vgSystem/snapRoot
References:
"How FUSE Can Break Rsync Backups"

Answer 2

This error does not appear if the fuse mount points are excluded in the rsync command. Since it is an automated sync, the mount command can be used to obtain all fuse mount points. The output of the mount command may differ depending on the system, but in a debian jessie sshfs mounts appear as USER#HOST:MOUNTED_DIR on /path/to/mount/point type fuse.sshfs (rw,...). A simple way to automate the exclusion of fuse mounts in bash+sed is the following
SOURCE="/home/"
FUSEEXCLUDE=( $( mount |
sed -rn "
/ type fuse/ {
s|^[^ ]+ on ([^ ]+) type fuse.+|\1|;
/^${SOURCE//\//\\\/}.+/ {
s|^${SOURCE//\//\\\/}| --exclude |;
p;
}
}" ) )
rsync $OPTIONS "${FUSEEXCLUDE[#]}" "$SOURCE" "$TARGET"