bash: how to keep some delay between multiple instances of a script - linux

Question

I am trying to download 100 files using a script
I dont want at any point of time not more than 4 downloads are happening.
So i have create a folder /home/user/file_limit. In the script it creates a file here before the download and after the download is complete it will delete it.
The script will check the number of files in the folder is less than 4 then only it will allow to create a file in the folder /home/user/file_limit
I am running a script like this
today=`date +%Y-%m-%d-%H_%M_%S_%N`;
while true
do
sleep 1
# The below command will find number of files in the folder /home/user/file_limit
lines=$(find /home/user/file_limit -iname 'download_*' -type f| wc -l)
if [ $lines -lt 5 ]; then
echo "Create file"
touch "/home/user/file_limit/download_${today}"
break;
else
echo "Number of files equals 4"
fi
done
#After this some downloading happens and once the downloading is complete
rm "/home/user/file_limit/download_${today}"
The problem i am facing is when 100 such scripts are running. Eg when the number of files in the folder are less than 4, then many touch "/home/user/file_limit/download_${today}" gets executed simultaneously and all of them creates files. So the total number of files become more than 4 which i dont want because more downloads cause my system get slower.
How to ensure there is a delay between each script for checking the lines=$(find /home/user/file_limit -iname 'download_*' -type f| wc -l) so that only one touch command get executed.
Or HOw to ensure the lines=$(find /home/user/file_limit -iname 'download_*' -type f| wc -l) command is checked by each script in a queue. No two scripts can check it at the same time.

Answer 1

How to ensure there is a delay between each script for checking the lines=$(find ... | wc -l) so that only one touch command get executed
Adding a delay won't solve the problem. You need a lock, mutex, or semaphore to ensure that the check and creation of files is executed atomically.
Locks limit the number of parallel processes to 1. Locks can be created with flock (usually pre-installed).
Semaphores are generalized locks limiting the number concurrent processes to any number N. Semaphores can be created with sem (part of GNU parallel, has to be installed).
The following script allows 4 downloads in parallel. If 4 downloads are running and you start the script a 5th time then that 5th download will pause until one of the 4 running downloads finish.
#! /usr/bin/env bash
main() {
# put your code for downloading here
}
export -f main
sem --id downloadlimit -j4 main

Answer 2

My solution starts maximum MAXPARALELLJOBS number of process and waits until all of those processes are done...
Hope it helps your problem.
MAXPARALELLJOBS=4
count=0
while <not done the job>
do
((count++))
( <download job> ) &
[ ${count} -ge ${MAXPARALELLJOBS} ] && count=0 && wait
done