lsyncd - OVERFLOW in event queue - Solution is to tune fs.inotify.max_queued_events - linux

Question

lsyncd is a fantastic alternative to NFS or NAS for replicating files among your Linux hosts. I have found the daemon works well with large Linux filesystems (many files, small to large sizes, xfs, ext4, luks) but requires some sysctl tuning as your filesystem grows.
This "question" is a note to myself so I can always find the answer via searching on stack overflow. Hope it helps you!
Github Project: https://github.com/axkibe/lsyncd
Exception in /var/log/lsyncd.log:
Thu Jun 18 17:48:52 2020 Normal: --- OVERFLOW in event queue ---
Thu Jun 18 17:48:52 2020 Normal: --- HUP signal, resetting ---
Thu Jun 18 17:48:52 2020 Normal: waiting for 1 more child processes.

Answer 1

Solution when you see "OVERFLOW in event queue" in lsyncd.log
From other knowledge bases, I had learned to tune max_user_watches, but by also tuning the max_queued_events, I corrected an OVERFLOW in event queue exception.
The temporary solution worked without needing to restart my lsyncd process.
I picked the number 1000000 as an arbitrarily large number. The default Ubuntu 18 value is 16384.
Temporary Solution
Check your current tuning values:
$ sysctl fs.inotify.max_queued_events
fs.inotify.max_queued_events = 16384
$ sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 8192
Update both max_user_watches and max_queued_events via shell
sudo sysctl fs.inotify.max_user_watches=1000000
sudo sysctl fs.inotify.max_queued_events=1000000
Permanent Solution, Persists after reboot
Update both max_user_watches and max_queued_events in /etc/sysctl.conf
fs.inotify.max_user_watches=1000000
fs.inotify.max_queued_events=1000000
Lsyncd.conf Basic Configuration
/etc/lsyncd/lsyncd.conf
settings {
logfile = "/var/log/lsyncd.log",
pidfile = "/var/run/lsyncd/lsyncd.pid",
insist = true
}
sync {
default.rsyncssh,
source="/var/application/data",
host="node2",
excludeFrom="/etc/lsyncd/exclude",
targetdir="/var/application/data",
rsync = {
archive = true,
compress = false,
whole_file = true
},
ssh = {
port = 22
}
}
System Details
Linux service1staging 5.0.0-36-generic #39~18.04.1-Ubuntu SMP Tue Nov 12 11:09:50 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Ubuntu 18.04.4 LTS
lsyncd --version
Version: 2.1.6