I have several directories like below. I want to list the directories which dont have the SQL files. For example "dev.mysite.com" in below example. Im using ubuntu 14.04 LTS
orange.com/
orange.com.sql
10.10.10.1/public_html/...
apple.edu.us/
apple.edu.sql
10.10.10.2/public_html/...
dev.mysite.com/
10.10.10.3/public_html/
example.com/
mysql_dbdump20150911.sql
10.10.11.11/public_html/...
Ive tried to achieve this using "find" with "cut" and "xargs" and moving those directories to the "dirwithsql" directory and take remaining ones as directory without sql file manually.
find . -maxdepth 2 -iname "*.sql" | cut -d'/' -f 2 | xargs -n 1 -I {} mv {} /backup/dirwithsql/{}
Ive tried with
find -maxdepth 2 ! -iname "*.sql" -exec dirname {} \;
But above command shows all directories
Is there any better method ?
Thank you
I had a similar problem recently, this worked for me
for f in $(find . -type d -maxdepth 2); do if [[ $(ls -1 $f | grep '.sql$'|wc -l) == 0 ]] ; then echo $f; fi; done
find . -type d -print0 | while read -d $'\0' n; do [ -f "$n"/*.sql ] || echo $n; done
Shall work even with exotic directory names (spaces, newlines...)
Related
I would like to find the newest sub directory in a directory and save the result to variable in bash.
Something like this:
ls -t /backups | head -1 > $BACKUPDIR
Can anyone help?
BACKUPDIR=$(ls -td /backups/*/ | head -1)
$(...) evaluates the statement in a subshell and returns the output.
There is a simple solution to this using only ls:
BACKUPDIR=$(ls -td /backups/*/ | head -1)
-t orders by time (latest first)
-d only lists items from this folder
*/ only lists directories
head -1 returns the first item
I didn't know about */ until I found Listing only directories using ls in bash: An examination.
This ia a pure Bash solution:
topdir=/backups
BACKUPDIR=
# Handle subdirectories beginning with '.', and empty $topdir
shopt -s dotglob nullglob
for file in "$topdir"/* ; do
[[ -L $file || ! -d $file ]] && continue
[[ -z $BACKUPDIR || $file -nt $BACKUPDIR ]] && BACKUPDIR=$file
done
printf 'BACKUPDIR=%q\n' "$BACKUPDIR"
It skips symlinks, including symlinks to directories, which may or may not be the right thing to do. It skips other non-directories. It handles directories whose names contain any characters, including newlines and leading dots.
Well, I think this solution is the most efficient:
path="/my/dir/structure/*"
backupdir=$(find $path -type d -prune | tail -n 1)
Explanation why this is a little better:
We do not need sub-shells (aside from the one for getting the result into the bash variable).
We do not need a useless -exec ls -d at the end of the find command, it already prints the directory listing.
We can easily alter this, e.g. to exclude certain patterns. For example, if you want the second newest directory, because backup files are first written to a tmp dir in the same path:
backupdir=$(find $path -type -d -prune -not -name "*temp_dir" | tail -n 1)
The above solution doesn't take into account things like files being written and removed from the directory resulting in the upper directory being returned instead of the newest subdirectory.
The other issue is that this solution assumes that the directory only contains other directories and not files being written.
Let's say I create a file called "test.txt" and then run this command again:
echo "test" > test.txt
ls -t /backups | head -1
test.txt
The result is test.txt showing up instead of the last modified directory.
The proposed solution "works" but only in the best case scenario.
Assuming you have a maximum of 1 directory depth, a better solution is to use:
find /backups/* -type d -prune -exec ls -d {} \; |tail -1
Just swap the "/backups/" portion for your actual path.
If you want to avoid showing an absolute path in a bash script, you could always use something like this:
LOCALPATH=/backups
DIRECTORY=$(cd $LOCALPATH; find * -type d -prune -exec ls -d {} \; |tail -1)
With GNU find you can get list of directories with modification timestamps, sort that list and output the newest:
find . -mindepth 1 -maxdepth 1 -type d -printf "%T#\t%p\0" | sort -z -n | cut -z -f2- | tail -z -n1
or newline separated
find . -mindepth 1 -maxdepth 1 -type d -printf "%T#\t%p\n" | sort -n | cut -f2- | tail -n1
With POSIX find (that does not have -printf) you may, if you have it, run stat to get file modification timestamp:
find . -mindepth 1 -maxdepth 1 -type d -exec stat -c '%Y %n' {} \; | sort -n | cut -d' ' -f2- | tail -n1
Without stat a pure shell solution may be used by replacing [[ bash extension with [ as in this answer.
Your "something like this" was almost a hit:
BACKUPDIR=$(ls -t ./backups | head -1)
Combining what you wrote with what I have learned solved my problem too. Thank you for rising this question.
Note: I run the line above from GitBash within Windows environment in file called ./something.bash.
I am trying to Create a shell script to move non empty files from Source directory to Target directory and add timestamp to them.
I am using
find . -type f -size +0 -print0 | xargs -I {} -r0 mv {} $Tgt_dir/{}_`date +%m%d%Y`
but its not working. Could you please help.
Thanks
You can use -printf in find to print the mv command with the full path of the source and just the basename in the destination, and pipe that to the shell:
date=$(date +%m%d%Y)
find . -type f -size +0 -printf "mv '%p' '$Tgt_dir/%f_$date'" | bash
%p is the full pathname, %f is the basename.
To move files with at least one line, write a command that counts the number of lines:
date=$(date +%m%d%Y)
find "$Src_dir" -type f -size +0 -printf "if [ $(wc -l '$p') -gt 1 ]; then mv '%p' '$Tgt_dir/%f_$date'; fi" | bash
Can you please suggest efficient way to move files from one location to their sub directory in a loop.
Ex:
/MY_PATH/User1/1234/Daily/abc.txt to /MY_PATH/User1/1234/Daily/Archive/abc.txt
/MY_PATH/User2/3456/Daily/def.txt to /MY_PATH/User2/3456/Daily/Archive/def.txt
/MY_PATH/User1/1111/Daily/hij.txt to /MY_PATH/User1/1111/Daily/Archive/hij.txt
/MY_PATH/User2/2222/Daily/def.txt to /MY_PATH/User2/2222/Daily/Archive/def.txt
I started in this way, but need your suggestions and best way to write it:
#!/bin/bash
dir1="/MyPath/"
subs= `ls $dir1`
for i in $subs; do
mv $dir1/$i/*/Daily $dir1/$i/*/Daily/Archive
done
My one line bash
for dir in $(
find MY_PATH -mindepth 3 -maxdepth 3 -type d -name Daily
);do
mkdir -p $dir/Archives
find $dir -maxdepth 1 -mindepth 1 ! -name Archives \
-exec mv -t $dir/Archives {} +
done
To quickly test:
mkdir -p MY_PATH/User{1,2,3,4}/{1234,2346,3333,2323}/Daily
touch MY_PATH/User{1,2,3,4}/{1234,2346,3333,2323}/Daily/{abc,bcd,def,feg,fds}.txt
for dir in $( find MY_PATH -mindepth 3 -maxdepth 3 -type d -name Daily );do
mkdir -p $dir/Archives; find $dir -maxdepth 1 -mindepth 1 ! -name Archives \
-exec mv -t $dir/Archives {} + ; done
ls -lR MY_PATH
This seem match OP's request
For more robust solution
There is a solution wich work with spaces somewhere in path...
Edited to include #mklement0's well pointed suggestion.
while IFS= read dir;do
mkdir -p "$dir"/Archives
find "$dir" -maxdepth 1 -mindepth 1 ! -name Archives \
-exec mv -t "$dir/Archives" {} +
done < <(
find MY_PATH -mindepth 3 -maxdepth 3 -type d -name Daily
)
Same demo;
mkdir -p MY_PATH/User{1,2,3,"4 3"}/{1234,"23 6",3333,2323}/Daily
touch MY_PATH/User{1,2,3,"4 3"}/{1234,"23 6",3333,2323}/Daily/{abc,"b c",def,hgz0}.txt
while read dir;do mkdir -p "$dir"/Archives;find "$dir" -maxdepth 1 -mindepth 1 \
! -name Archives -exec mv -t "$dir/Archives" {} +; done < <(
find MY_PATH -mindepth 3 -maxdepth 3 -type d -name Daily )
ls -lR MY_PATH
Assuming the directory structure is as you have shown in your examples, i.e.
MY_PATH/
subdir-level-1/
subdir-level-2/
Daily/
files
Archive/
Here's what you can do:
shopt -s nullglob # defend against globbing failure -- inspired by mklement0's answer
root="/MyPath"
for dir in "${root}"/*/*/Daily/; do
mkdir -p "${dir}/Archive" # if Archive might not exist; to be pedantic you should look at David C. Rankin's answer for error handling, but usually we know what we're doing so that's not necessary
find "${dir}" -maxdepth 1 -type f -print0 | xargs -0 mv -t "${dir}/Archive"
done
The reason I use find and xargs is to save a few processes; you can as well move files in each ${dir} one by one.
Update: #mklement0 suggested that find "${dir}" -maxdepth 1 -type f -print0 | xargs -0 mv -t "${dir}/Archive" can be further improved to
find "${dir}" -maxdepth 1 -type f -exec mv -t "${dir}/Archive" +
which is a very good point.
Try the following:
dir1="/MyPath"
for d in "$dir1"/*/*/Daily/; do
[[ -d $d ]] || break # break, if no subdirectories match
for f in "$d"/*; do # loop over files in */*/Daily/
[[ -f "$f" ]] || continue # skip non-files or if nothing matches
mv "$f" "$d"/Archive/
done
done
"$dir1"*/*/Daily/ matches all grandchild subdirectories of $dir1; thanks to the terminating /, only directories match; note that, as a result, $d ends in /.
Note that $d therefore ends in /, and, strictly speaking, needs no / later on when synthesizing paths with it (e.g., "$d"/*), but doing so does no harm and helps readability, as #4ae1e1 points out in a comment.
[[ -d $d ]] || break ensures that the loop is exited if no grandchild directories match (by default, a glob (pattern) that has no matches is passed as is to the loop).
for f in "$d"* loops over all entries (files and/or subdirs.) in $d:
[[ -f "$f" ]] || continue ensures that only files are processed or, in the event that nothing matches, the loop is exited.
mv "$f" "$d"/Archive/ then moves each file to subdir. Archive.
You need to check for, and if not present, create the destination directory before moving the file to Archive. If you cannot create the directory (due to permissions or otherwise), you skip the move. The following does not assume any limitation on depth, but will omit any directory containing Archive as an intermediate subdirectory:
oldifs="$IFS"
IFS=$'\n'
for i in $(find /MY_PATH -type f); do
[[ "$i" =~ Archive ]] && continue
[ -d "${i%/*}/Archive" ] || mkdir -p "${i%/*}/Archive"
[ -d "${i%/*}/Archive" ] || {
printf "error: unable to create '%s'\n" "${i%/*}/Archive"
continue
}
mv -fv "$i" "${i/Daily/Daily\/Archive}"
done
IFS="$oldifs"
Output when run
$ bash archive_daily.sh
mv -fv /MY_PATH/User1/1111/Daily/hij.txt /MY_PATH/User1/1111/Daily/Archive/hij.txt
mv -fv /MY_PATH/User1/1234/Daily/abc.txt /MY_PATH/User1/1234/Daily/Archive/abc.txt
mv -fv /MY_PATH/User2/3456/Daily/def.txt /MY_PATH/User2/3456/Daily/Archive/def.txt
mv -fv /MY_PATH/User2/2222/Daily/def.txt /MY_PATH/User2/2222/Daily/Archive/def.txt
Note: you can limit/tighten the file selection by adjusting the call to find populating the for loop (e.g. -name or -iname). This simply checks/moves every file to its Archive folder. To limit to only files with the .txt extension, you can specify find /MY_PATH -type f -name "*.txt". To limit to only files in the /MY_PATH/User1 and /MY_PATH/User2directories with a .txt extension, use find /MY_PATH/User[12] -type f -name "*.txt".
Note2: when looping on filenames, the paths & filenames should not contain non-standard characters for the current locale. Certainly you should not have the '\n' as a character in your filename. Setting IFS is required to protect against word splitting on spaces in either the path or filename.
Since you said efficient, anything with a subshell will fail in funny ways with lots of entries. You're better off using xargs:
#!/bin/bash
dir1="/MyPath/"
find $dir1 -name Daily -type d -depth 3 | while read i
do
pushd .
cd $i
mkdir Archive
find . -type f -depth 1 | xargs -J {} mv {} Archive
popd
done
The outer find will look for you Daily directories. It's very specific in that they have to be at a certain depth and directories, not regular files. The results gets piped into read, where each directory is entered, Archive is created, and files batch-copied with xargs ... mv. Complete file lists and directory lists are never stored in memory, so it scales very well.
I would like to build on the answer I found here: Bash script to find specific files in a hierarchy of files
find $dir -name $name -exec scp {} $destination \;
I have a file with a list of file names and I need to find those files on a backup disk, then copy those files found to a destination folder, and lastly print the files that could not be found to a new file.
the last step would be helpful so that I wouldn't need to make another list of files copied and then do a compare with original list.
If the script can then make a list of the copied files, and do a compare, then print the differences, then that's exactly what's required. Unless the shell process find can print to file each time it "Can't find" a file.
Assuming that your list is separated by newlines; something like this should work
#!/bin/bash
dir=someWhere
dest=someWhereElse
toCopyList=filesomewhere
notCopied=filesomewhereElse
while read line; do
find "$dir" -name "$line" -exec cp '{}' $dest \; -printf "%f\n"
done < "$toCopyList" > cpList
#sed -i 's#'$dir'/##' cpList
# I used # instead of / in sed to not confuse sed with / in $dir
# Also, I assumed the string in $dir doesnot end with a /
cat cpList "$toCopyList" | sort | uniq -c | sed -nr '/^ +1/s/^ +1 +(.*)/\1/p' > "$notCopied"
# Will not work if you give wild cards in your "toCopyList"
Hope it helps
while read fname ; do
find /FROM/WHERE/TO/COPY/ \
-type f \
-name "$fname" \
-exec cp \{\} /DESTINATION/DIR/ \; 2>/dev/null
find /DESTINATION/DIR/ \
-type f \
-name "$fname" &>/dev/null || \
echo $fname
done < FILESTOCOPY > MISSEDFILES
Will do.
i'm trying to write a shell script, for linux and solaris, that finds some specific files older than X days and then deletes them. the trick is that during this process there are a couple of files that must not be deleted.
for example from the following list of files i need to delete *.zip and keep *.log and *.something.*
1.zip
2.zip
3.log
prefix.something.suffix
finding the files and feeding them to rm was easy, but i'm having difficulties in excluding the files from the deletion list.
experimenting around i discovered one can benefit from multiple complex expressions grouped with logical operators like this:
find -L path -type f \( -name '*.log' \) -a ! \( -name '*.zip' -o -name '*something*' \) -mtime +3
cheers,
G
or you could do this:
find /appl/ftp -type f -mtime +30 |grep -vf [exclude_file] | xargs rm -rf;
I needed to find a way to provide a hard coded list of exclude files to not remove, but remove everything else that was older than 30 days. Here is a little script to perform a remove of all files older that 30 days, except files that are listed in the [exclude_file].
EXCL_FILES=`/bin/cat [exclude_file]`;
RM_FILE=`/usr/bin/find [path] -type f -mtime +30`;
for I in $RM_FILES;
do
for J in $EXCL_FILES;
do
grep $J $I;
if [[ $? == 0 ]]; then
/bin/rm $I;
if [[ $? != 0 ]]; then echo "PROBLEM: Could not remove $I"; exit 1; fi;
fi;
done;
done;