auto detect addition and deletion of any file in any specificy directory and subdirectory - linux

I am trying to write a tool to auto-detect any file addition or deletion to a specific directory. I know inotify can do this task in most efficent manner and code would be also shorter. But I cannot use inotify here as I am working on NFS directory , where inotify fail to detect any directory changes. (Which I understand) .
Here is a some rough code I wrote to monitor any directory , which is bad in many way. Also, sometimes it skip some events.
while true
do
touch temp_fs_state.log
touch temp_1_fs_state.log
find . -type f -ls |awk '{print $NF}' >> temp_fs_state.log
sleep 1
find . -type f -ls |awk '{print $NF}' >> temp_1_fs_state.log
diffResult=$(diff temp_fs_state.log temp_1_fs_state.log |grep -E '<|>' )
echo $diffResult |grep -Fq '>'
if [ "$?" -eq 0 ];then
echo "`echo $diffResult | awk '{print $NF}'` ADDED to the target directory!!"
fi
echo $diffResult |grep -Fq '<'
if [ "$?" -eq 0 ];then
echo "`echo $diffResult | awk '{print $NF}'` DELETED to the target directory!!"
fi
rm -rf temp_fs_state.log temp_1_fs_state.log
done
Current result:
Some files are not detected by current solution.
sh monitor
./x9 DELETED to the target directory!!
./x11 ADDED to the target directory!!
./x13 ADDED to the target directory!!
./x14 ADDED to the target directory!!
./x15 ADDED to the target directory!!
./x17 ADDED to the target directory!!
./x17 DELETED to the target directory!!
Question :
Can similar result would be achieved with any other shorter or smarter approach ?
Can we ensure no events are skipped ?
PS: Please assume , CPU performance is not concerned here as it's for testing purpose only.

Related

How to extract only file name return from diff command?

I am trying to prepare a bash script for sync 2 directories. But I am not able to file name return from diff. everytime it converts to array.
Here is my code :
#!/bin/bash
DIRS1=`diff -r /opt/lampp/htdocs/scripts/dev/ /opt/lampp/htdocs/scripts/www/ `
for DIR in $DIRS1
do
echo $DIR
done
And if I run this script I get out put something like this :
Only
in
/opt/lampp/htdocs/scripts/www/:
file1
diff
-r
"/opt/lampp/htdocs/scripts/dev/File
1.txt"
"/opt/lampp/htdocs/scripts/www/File
1.txt"
0a1
>
sa
das
Only
in
/opt/lampp/htdocs/scripts/www/:
File
1.txt~
Only
in
/opt/lampp/htdocs/scripts/www/:
file
2
-
second
Actually I just want to file name where I find the diffrence so I can take perticular action either copy/delete.
Thanks
I don't think diff produces output which can be parsed easily for your purposes. It's possible to solve your problem by iterating over the files in the two directories and running diff on them, using the return value from diff instead (and throwing the diff output away).
The code to do this is a bit long, but here it is:
DIR1=./one # set as required
DIR2=./two # set as required
# Process any files in $DIR1 only, or in both $DIR1 and $DIR2
find $DIR1 -type f -print0 | while read -d $'\0' -r file1; do
relative_path=${file1#${DIR1}/};
file2="$DIR2/$relative_path"
if [[ ! -f "$file2" ]]; then
echo "'$relative_path' in '$DIR1' only"
# Do more stuff here
elif diff -q "$file1" "$file2" >/dev/null; then
echo "'$relative_path' same in '$DIR1' and '$DIR2'"
# Do more stuff here
else
echo "'$relative_path' different between '$DIR1' and '$DIR2'"
# Do more stuff here
fi
done
# Process files in $DIR2 only
find $DIR2 -type f -print0 | while read -d $'\0' -r file2; do
relative_path=${file2#${DIR2}/};
file1="$DIR1/$relative_path"
if [[ ! -f "$file2" ]]; then
echo "'$relative_path' in '$DIR2 only'"
# Do more stuff here
fi
done
This code leverages some tricks to safely handle files which contain spaces, which would be very difficult to get working by parsing diff output. You can find more details on that topic here.
Of course this doesn't do anything regarding files which have the same contents but different names or are located in different directories.
I tested by populating two test directories as follows:
echo "dir one only" > "$DIR1/dir one only.txt"
echo "dir two only" > "$DIR2/dir two only.txt"
echo "in both, same" > $DIR1/"in both, same.txt"
echo "in both, same" > $DIR2/"in both, same.txt"
echo "in both, and different" > $DIR1/"in both, different.txt"
echo "in both, but different" > $DIR2/"in both, different.txt"
My output was:
'dir one only.txt' in './one' only
'in both, different.txt' different between './one' and './two'
'in both, same.txt' same in './one' and './two'
Use -q flag and avoid the for loop:
diff -rq /opt/lampp/htdocs/scripts/dev/ /opt/lampp/htdocs/scripts/www/
If you only want the files that differs:
diff -rq /opt/lampp/htdocs/scripts/dev/ /opt/lampp/htdocs/scripts/www/ |grep -Po '(?<=Files )\w+'|while read file; do
echo $file
done
-q --brief
Output only whether files differ.
But defitnitely you should check rsync: http://linux.die.net/man/1/rsync

Directory checking in Linux

I am trying to write a script shell that takes two arguments as directory names, and determines if Directory 1 contains Directory 2 or vice versa. And also if there is no relationship between them.
I know the command to check if a directory exists is find -type d, however i was a bit confused as how to check and parse then names. I know i would need if-else loops, just not sure how to check for the conditions?
find won't be needed.
Something similar to this (but not guaranteeing directory name with spaces or some special characters.):
if [ "$dir1" == "$dir2" ]; then
echo "$dir1 == $dir2";
exit;
fi
if grep -E -q "^$dir2" <<< $dir1; then
echo "$dir1 is contained by $dir2."
exit
fi
if grep -E -q "^$dir1" <<< $dir2; then
echo "$dir2 is contained by $dir1.";
fi
However, this does not deal with symbolic links. For example, sym1 -> /usr/local/bin and sym2 -> /usr/local, apparently, sym2 contains sym1.
In addition, this does not deal with strange looking directory names, like /usr/local/./bin, which is the same as /usr/local/bin, or even /usr/local/../bin, which is the same as /usr/bin
--- Update ---
DevSolar metioned that readlink -e can be used to resolve the symbolic link. In my test, it also resolves the strange looking directory names like those with . and ... Thanks to DevSolar.
Do you want this?
if [ -d $1 ];then
a=`find $1 -type d -name $2`
if [ $a ];then
echo "$1 has $2"
else
echo "$1 does NOT has $2"
fi
fi
if [ -d $2 ];then
b=`find $2 -type d -name $1`
if [ $b ];then
echo "$2 has $1"
else
echo "$2 does NOT has $1"
fi
fi
This will do the trick I think,
find -name directory1 |grep directory2
or vice-versa, then use
echo $?
it will give 0 for success and 1 for failure.

Changing directory and to download file using bash script and also extract it

I created a script to download file from URL and I want to download it in the specific directory but the problem is when its time in downloading it will not put to the directory given and also when extracting the file is in the given directory.
diskspace=$(df -h /var/ | sed '1d' | awk '{print $5}' | cut -d'%' -f1)
bundle=$(awk -F = '{print $2}' config.txt)
allowed=10
if [ "${diskspace}" -gt "${allowed}" ]; then
cd `/var/`
wget $bundle
else
echo "Not enough space to download the bundle"
echo $output
exit
fi
while true; do
for f in *.tar.gz; do
case $f in '*.tar.gz') exit 0;; esac
tar zxf "$f"
rm -v "$f"
done
done
Can Someone help me to this problem ? The thing that I want to happen is to download the file in the given directory and also extract it there. Help is greatly appreciated.

Shell script to delete files when disk is full

I am writing a small little script to clear space on my linux everyday via CRON if the cache directory grows too large.
Since I am really green at bash scripting, I will need a little bit of help from you linux gurus out there.
Here is basically the logic (pseudo-code)
if ( Drive Space Left < 5GB )
{
change directory to '/home/user/lotsa_cache_files/'
if ( current working directory = '/home/user/lotsa_cache_files/')
{
delete files in /home/user/lotsa_cache_files/
}
}
Getting drive space left
I plan to get the drive space left from the '/dev/sda5' command.
If returns the following value to me for your info :
Filesystem 1K-blocks Used Available Use% Mounted on<br>
/dev/sda5 225981844 202987200 11330252 95% /
So a little regex might be necessary to get the '11330252' out of the returned value
A little paranoia
The 'if ( current working directory = /home/user/lotsa_cache_files/)' part is just a defensive mechanism for the paranoia within me. I wanna make sure that I am indeed in '/home/user/lotsa_cache_files/' before I proceed with the delete command which is potentially destructive if the current working directory is not present for some reason.
Deleting files
The deletion of files will be done with the command below instead of the usual rm -f:
find . -name "*" -print | xargs rm
This is due to the inherent inability of linux systems to 'rm' a directory if it contains too many files, as I have learnt in the past.
Just another proposal (comments within code):
FILESYSTEM=/dev/sda1 # or whatever filesystem to monitor
CAPACITY=95 # delete if FS is over 95% of usage
CACHEDIR=/home/user/lotsa_cache_files/
# Proceed if filesystem capacity is over than the value of CAPACITY (using df POSIX syntax)
# using [ instead of [[ for better error handling.
if [ $(df -P $FILESYSTEM | awk '{ gsub("%",""); capacity = $5 }; END { print capacity }') -gt $CAPACITY ]
then
# lets do some secure removal (if $CACHEDIR is empty or is not a directory find will exit
# with error which is quite safe for missruns.):
find "$CACHEDIR" --maxdepth 1 --type f -exec rm -f {} \;
# remove "maxdepth and type" if you want to do a recursive removal of files and dirs
find "$CACHEDIR" -exec rm -f {} \;
fi
Call the script from crontab to do scheduled cleanings
I would do it this way:
# get the available space left on the device
size=$(df -k /dev/sda5 | tail -1 | awk '{print $4}')
# check if the available space is smaller than 5GB (5000000kB)
if (($size<5000000)); then
# find all files under /home/user/lotsa_cache_files and delete them
find /home/user/lotsa_cache_files -name "*" -delete
fi
Here's the script I use to delete old files in a directory to free up space...
#!/bin/bash
#
# prune_dir - prune directory by deleting files if we are low on space
#
DIR=$1
CAPACITY_LIMIT=$2
if [ "$DIR" == "" ]
then
echo "ERROR: directory not specified"
exit 1
fi
if ! cd $DIR
then
echo "ERROR: unable to chdir to directory '$DIR'"
exit 2
fi
if [ "$CAPACITY_LIMIT" == "" ]
then
CAPACITY_LIMIT=95 # default limit
fi
CAPACITY=$(df -k . | awk '{gsub("%",""); capacity=$5}; END {print capacity}')
if [ $CAPACITY -gt $CAPACITY_LIMIT ]
then
#
# Get list of files, oldest first.
# Delete the oldest files until
# we are below the limit. Just
# delete regular files, ignore directories.
#
ls -rt | while read FILE
do
if [ -f $FILE ]
then
if rm -f $FILE
then
echo "Deleted $FILE"
CAPACITY=$(df -k . | awk '{gsub("%",""); capacity=$5}; END {print capacity}')
if [ $CAPACITY -le $CAPACITY_LIMIT ]
then
# we're below the limit, so stop deleting
exit
fi
fi
fi
done
fi
To detect the occupation of a filesystem, I use this :
df -k $FILESYSTEM | tail -1 | awk '{print $5}'
that gives me the occupation percentage of the filesystem, this way, I don't need to compute it :)
If you use bash, you can use the pushd/popd operation to change directory and be sure to be in.
pushd '/home/user/lotsa_cache_files/'
do the stuff
popd
Here's what I do:
while read f; do rm -rf ${f}; done < movies-to-delete.txt

how to loop files in linux from svn status

As being quite a newbie in linux, I have the follwing question.
I have list of files (this time resulting from svn status) and i want to create a script to loop them all and replace tabs with 4 spaces.
So I want from
....
D HTML/templates/t_bla.tpl
M HTML/templates/t_list_markt.tpl
M HTML/templates/t_vip.tpl
M HTML/templates/upsell.tpl
M HTML/templates/t_warranty.tpl
M HTML/templates/top.tpl
A + HTML/templates/t_r1.tpl
....
to something like
for i in <files>; expand -t4;do cp $i /tmp/x;expand -t4 /tmp/x > $i;done;
but I dont know how to do that...
You can use this command:
svn st | cut -c8- | xargs ls
This will cut the first 8 characters leaving only a list of file names, without Subversion flags. You can also add grep before cut to filter only some type of changes, like /^M/. xargs will pass the list of files as arguments to a given command (ls in this case).
I would use sed, like so:
for i in files
do
sed -i 's/\t/ /' "$i"
done
That big block in there is four spaces. ;-)
I haven't tested that, but it should work. And I'd back up your files just in case. The -i flag means that it will do the replacements on the files in-place, but if it messes up, you'll want to be able to restore them.
This assumes that $files contains the filenames. However, you can also use Adam's approach at grabbing the filenames, just use the sed command above without the "$i".
Not asking for any votes, but for the record I'll post the combined answer from #Adam Byrtek and #Dan Fego:
svn st | cut -c8- | xargs sed -i 's/\t/ /'
I could not test it with real subversion output, but this should do the job:
svn st | cut -c8- | while read file; do expand -t4 $file > "$file-temp"; mv "$file-temp" "$file"; done
svn st | cut -c8- will generate a list of files without subversion flags. read will then save each entry in the variable $file and expand is used to replace the tabs with four spaces in each file.
Not quite what you're asking, but perhaps you should be looking into commit hooks in subversion?
You could create a hook to block check-ins of any code that contains tabs at the start of a line, or contains tabs at all.
In the repo directory on your subversion server there'll be a directory called hooks. Put something in there which is executable called 'pre-commit' and it'll be run before anything is allowed to be committed. It can return a status to block the commit if you wish.
Here's what I have to stop php files with syntax errors being checked in:
#!/bin/bash
REPOS="$1"
TXN="$2"
PHP="/usr/bin/php"
SVNLOOK=/usr/bin/svnlook
$SVNLOOK log -t "$TXN" "$REPOS" | grep "[a-zA-Z0-9]" > /dev/null
if [ $? -ne 0 ]
then
echo 1>&2
echo "You must enter a comment" 1>&2
exit 1
fi
CHANGED=`$SVNLOOK changed -t "$TXN" "$REPOS" | awk '{print $2}'`
for LINE in $CHANGED
do
FILE=`echo $LINE | egrep \\.php$`
if [ $? == 0 ]
then
MESSAGE=`$SVNLOOK cat -t "$TXN" "$REPOS" "${FILE}" | $PHP -l`
if [ $? -ne 0 ]
then
echo 1>&2
echo "***********************************" 1>&2
echo "PHP error in: ${FILE}:" 1>&2
echo "$MESSAGE" | sed "s| -| $FILE|g" 1>&2
echo "***********************************" 1>&2
exit 1
fi
fi
done

Resources