Shell script file watcher concurrency - linux

Question

I have the following script (shell script running on OEL5.6) that is currently scheduled via cron to pick files up from given directories (specified in a database table) and to call a processing script on them on a directory & filemask basis. The script works fine at the moment but with this implementation if one folder has a large amount of files to process even if all other folders complete the script won't exit until that one has, which means files landing in the other folders won't be picked up until the next run. I'd like to use a similar approach to this but to have it constantly checking folders for new files instead of sequentially running through all folders once and then exiting so it would run as more of a daemon constantly in the background. Any ideas rather than wrapping this in a while true loop? I've filtered out a bit of code from this example to keep it short.
readonly HOME_DIR="$(cd $(dirname $0)/;echo $PWD)"
export LOCK_DIR="/tmp/lock_folder"
check_lock() {
# Try and create the $LOCK_DIR lock directory. Exit script if failure.
# Do some checks to make sure the script is actually running and hasn't just failed and left a lock dir.
}
main(){
# Check to see if there's already an instance of the watcher running.
check_lock
# when the watcher script exits remove the lock directory for the next run
trap 'rm -r $LOCK_DIR;' EXIT
# Pull folder and file details into a csv file from the database -> $FEEDS_FILE
# Loop through all the files in given folders
while IFS="," read feed_name feed_directory file_mask
do
# Count the number of files to process using the directory and file mask
num_files=$(find $feed_directory/$file_mask -mmin +5 -type f 2> /dev/null | wc -l 2> /dev/null)
if [[ $num_files < 1 ]]; then
# There's no files older than 5 mins to pickup here. Move on to next folder.
continue
fi
# Files found! Try and create a new feed_name lock dir. This should always pass first loop.
if mkdir $LOCK_DIR/$feed_name 2>/dev/null; then
$HOME_DIR/another_script.sh "$feed_name" "$feed_directory" "$file_mask" & # Call some script to do processing. This script removes it's child lock dir when done.
else
log.sh "Watcher still running" f
continue
fi
# If the amount of processes running as indicated by child lock dirs present in $LOCK_DIR is greater than or equal to the max allowed then wait before re-trying another.
while [ $(find $LOCK_DIR -maxdepth 1 -type d -not -path $LOCK_DIR | wc -l) -ge 5 ]; do
sleep 10
done
done < $FEEDS_FILE
# Now all folders have been processed make sure that this script doesn't exit until all child scripts have completed (and removed their lock dirs).
while [ $(find $LOCK_DIR -type d | wc -l) -gt 1 ]; do
sleep 10
done
exit 0
}
main "$#"

Answer 1

One idea is to use inotifywait from inotify-tools to monitor the directories for changes. This is more efficient than constantly scanning the directories for changes. Something like that
inotifywait -m -r -e create,modify,move,delete /dir1 /dir2 |
while IFS= read -r event; do
# parse $event, act accordingly
done