I want to move all my files older than 1000 days, which are distributed over various subfolders, from /home/user/documents into /home/user/archive. The command I tried was
find /home/user/documents -type f -mtime +1000 -exec rsync -a --progress --remove-source-files {} /home/user/archive \;
The problem is, that (understandably) all files end up being moved into the single folder /home/user/archive. However, what I want is to re-construct the file tree below /home/user/documents inside /home/user/archive. I figure this should be possible by simply replacing a string with another somehow, but how? What is the command that serves this purpose?
Thank you!
I would take this route instead of rsync:
Change directories so we can deal with relative path names instead of absolute ones:
cd /home/user/documents
Run your find command and feed the output to cpio, requesting it to make hard-links (-l) to the files, creating the leading directories (-d) and preserve attributes (-m). The -print0 and -0 options use nulls as record terminators to correctly handle file names with whitespace in them. The -l option to cpio uses links instead of actually copying the files, so very little additional space is used (just what is needed for the new directories).
find . -type f -mtime +1000 -print0 | cpio -dumpl0 /home/user/archives
Re-run your find command and feed the output to xargs rm to remove the originals:
find . -type f -mtime +1000 -print0 | xargs -0 rm
Here's a script too.
#!/bin/bash
[ -n "$BASH_VERSION" ] && [[ BASH_VERSINFO -ge 4 ]] || {
echo "You need Bash version 4.0 to run this script."
exit 1
}
# SOURCE=/home/user/documents/
# DEST=/home/user/archive/
SOURCE=$1
DEST=$2
declare -i DAYSOLD=10
declare -a DIRS=()
declare -A DIRS_HASH=()
declare -a FILES=()
declare -i E=0
# Check directories.
[[ -n $SOURCE && -d $SOURCE && -n $DEST && -d $DEST ]] || {
echo "Source or destination directory may be invalid."
exit 1
}
# Format source and dest variables properly:
SOURCE=${SOURCE%/}
DEST=${DEST%/}
SOURCE_LENGTH=${#SOURCE}
# Copy directories first.
echo "Creating directories."
while read -r FILE; do
DIR=${FILE%/*}
if [[ -z ${DIRS_HASH[$DIR]} ]]; then
PARTIAL=${DIR:SOURCE_LENGTH}
if [[ -n $PARTIAL ]]; then
TARGET=${DEST}${PARTIAL}
echo "'$TARGET'"
mkdir -p "$TARGET" || (( E += $? ))
chmod --reference="$DIR" "$TARGET" || (( E += $? ))
chown --reference="$DIR" "$TARGET" || (( E += $? ))
touch --reference="$DIR" "$TARGET" || (( E += $? ))
DIRS+=("$DIR")
fi
DIRS_HASH[$DIR]=.
fi
done < <(exec find "$SOURCE" -mindepth 1 -type f -mtime +"$DAYSOLD")
# Copy files.
echo "Copying files."
while read -r FILE; do
PARTIAL=${FILE:SOURCE_LENGTH}
cp -av "$FILE" "${DEST}${PARTIAL}" || (( E += $? ))
FILES+=("$FILE")
done < <(exec find "$SOURCE" -mindepth 1 -type f -mtime +"$DAYSOLD")
# Remove old files.
if [[ E -eq 0 ]]; then
echo "Removing old files."
rm -fr "${DIRS[#]}" "${FILES[#]}"
else
echo "An error occurred during copy. Not removing old files."
exit 1
fi
Related
I need to write script in loop which will count the number of files and directories and indicates which grater and by how much. Like etc: there are 10 more files than directories.
I was trying something like that but it just show files and directories and I don't have idea how to indicates which is greater etc. Thanks for any help
shopt -s dotglob
count=0
for dir in *; do
test -d "$dir" || continue
test . = "$dir" && continue
test .. = "$dir" && continue
((count++))
done
echo $count
for -f in *; do
"$fname"
done
Here is a recursive dir walk I used for something a while back. Added counting of dirs and files:
#!/bin/sh
# recursive directory walk
loop() {
for i in *
do
if [ -d "$i" ]
then
dir=$((dir+1))
cd "$i"
loop
else
file=$((file+1))
fi
done
cd ..
}
loop
echo dirs: $dir, files: $file
Paste it to a script.sh and run with:
$ sh script.sh
dirs: 1, files: 11
You can use the find command to make things simplier.
The following command will list all the files in the given path:
find "path" -mindepth 1 -maxdepth 1 -type f
And also using the -type d you will get the directories.
Piping find into the wc -l will give you the number instead of the actual file and directory names, so:
root="${1:-.}"
files=$( find "$root" -mindepth 1 -maxdepth 1 -type f | wc -l)
dirs=$( find "$root" -mindepth 1 -maxdepth 1 -type d | wc -l)
if [ $files -gt $dirs ]; then
echo "there are $((files - dirs)) more files"
elif [ $files -lt $dirs ]; then
echo "there are $((dirs - files)) more dirs"
else
echo "there are the same"
fi
Use could use find to get the number of files/folders in a directory. Use wc -l to count the number of found paths, which you could use to calculate/show the result;
#!/bin/bash
# Path to search
search="/Users/me/Desktop"
# Get number of files
no_files=$(find "$search" -type f | wc -l )
# Number of folders
no_folders=$(find "$search" -type d | wc -l )
echo "Files: ${no_files}"
echo "Folders: ${no_folders}"
# Caculate dif
diff=$((no_files - $no_folders))
# Check if there are more folders or files
if [ "$diff" -gt 0 ]; then
echo "There are $diff more files then folders!"
else
diff=$((diff * -1 ) # Invert negative number to positive (-10 -> 10)
echo "There are $diff more folders then files!"
fi;
Files: 13
Folders: 2
There are 11 more files then folders!
I'm trying to work on a script that will crawl my Plex media folder, find any header ".r00" files, extract them in their own directory, and trash the archive zips after it's done. I have two options I've been playing around with. Combined they do what I want, but I would like to have it all in one nice little script.
Option 1:
This script opens the "LinRAR" GUI, makes me navigate to a specific directory, finds and extracts any .r00 file in that directory, and successfully deleted all archive zips.
while true; do
if dir=$(zenity --title="LinRAR by dExIT" --file-selection --directory); then
if [[ ! -d $dir ]]; then
echo "$dir: Wrong Directory" >&2
else
( cd "$dir" && for f in *.r00; do [[ -f $f ]] || continue; rar e "$f" && rm "${f%00}"[0-9][0-9]; done )
fi
else
echo "$bold Selection cancelled $bold_off" >&2
exit 1
fi
zenity --title="What else...?" --question --text="More work to be done?" || break
done
Option 2:
This script cd's to my Plex folder, recursively finds any .r00 files, extracts to my /home/user folder, and does not remove the archive zips.
(cd '/home/user/Plex');
while [ "`find . -type f -name '*.r00' | wc -l`" -gt 0 ];
do find -type f -name "*.r00" -exec rar e -- '{}' \; -exec rm -- '{}' \;;
done
I would like to have something that takes the first working script, and applies the recursive find to all folders inside of /Plex instead of only letting me navigate to one folder at a time through the "LinRAR" GUI.
No need to use cd. find takes a starting directory.
It's that dot (.) you passed to it.
Also added another (more sane) alternative for the find & loop:
#!/bin/bash
while true; do
if dir=$(zenity --title="LinRAR by dExIT" --file-selection --directory); then
if [[ ! -d $dir ]]; then
echo "$dir: Wrong Directory" >&2
else
# Alternative 1 - a little more comfortable
files="$(find "${dir}" -type f -name '*.r00')"
for file in ${files}; do
rar e "${file}" && rm "${file}"
done
# Alternative 2 - based on your original code
while [ "`find "${dir}" -type f -name '*.r00' | wc -l`" -gt 0 ]; do
find "${dir}" -type f -name "*.r00" -exec rar e -- '{}' \; -exec rm -- '{}' \;;
done
fi
else
echo "$bold Selection cancelled $bold_off" >&2
exit 1
fi
zenity --title="What else...?" --question --text="More work to be done?" || break
done
According to the comments, I ran a small example of this code and it works perfectly fine:
#!/bin/bash
if dir=$(zenity --title="LinRAR by dExIT" --file-selection --directory); then
if [[ ! -d $dir ]]; then
echo "$dir: Wrong directory" >&2
else
find $dir -type f
fi
else
echo "cancelled"
fi
A directory is successfully picked and all its files are printed. If I chose to cancel in zenity, then it prints 'cancelled'.
As I was deleting many obsolete file trees on a Linux machine I was wondering if there is an easy way to remove files recursively while prompting only on directories.
I could use rm -ri but there some much files that it would be really annoying to answer for every one of them. What really matter to me is being prompted on folders to have more control on what happens.
I am not a bash expert so I am asking if there is a simple way to do this.
Here is my attempt with a long bash script:
#!/bin/bash
promptRemoveDir()
{
fileCount=$(ls -1 $1 | wc -l)
prompt=1
while [ $prompt == 1 ]
do
read -p "remove directory: $1($fileCount files) ? [yl]: " answer
case $answer in
[yY])
rm -r $1
prompt=0
;;
l)
echo $(ls -A $1)
;;
*)
prompt=0
;;
esac
done
}
removeDir()
{
if [ "$(ls -A $1)" ]
then dirs=$(find $1/* -maxdepth 0 -type d)
fi
if [[ -z $dirs ]]
then
promptRemoveDir $1
else
for dir in $dirs
do
removeDir $dir
done
promptRemoveDir $1
fi
}
for i in $*
do
if [ -d $i ]
then
removeDir $i
else
rm $i
fi
done
If i understand your question properly this should work
Dirs=$(find . -type d)
Removes just the files in the directories specified
for i in "$Dirs"; do read -p "Delete files in "$i": ";if [[ $REPLY == [yY] ]]; then find $i -maxdepth 1 -type f | xargs -0 rm ; fi ;done
If you want to delete the folders as well, this will read from lowest directory(none below it) upwards.
for i in $(echo "$Dirs" | sed '1!G;h;$!d' ); do read -p "Delete files in $i: ";if [[ $REPLY == [yY] ]]; then rm -r "$i"; fi ;done
Here's a simplified version from me. There's no need to use ls and find.
#!/bin/bash
shopt -s nullglob
shopt -s dotglob
function remove_dir_i {
local DIR=$1 ## Optional. We can just use $1.
local SUBFILES=("$DIR"/*) FILE
for (( ;; )); do
read -p "Remove directory: $DIR (${#SUBFILES[#]} files)? [YNLQ]: "
case "$REPLY" in
[yY])
echo rm -fr "$DIR"
return 0
;;
[nN])
for FILE in "${SUBFILES[#]}"; do
if [[ -d $FILE ]]; then
remove_dir_i "$FILE" || return 1
# else
# ## Apparently we skip deleting a file. If we do this
# ## we could actually simplify the function further
# ## since we also delete the file at first loop.
# # echo "Removing file \"$FILE.\""
# # rm -f "$FILE"
fi
done
return 0
;;
[lL])
printf '%s\n' "${SUBFILES[#]}"
;;
[qQ])
return 1
;;
# *)
# echo "Please answer Y(es), N(o), L(ist) or Q(uit)."
# ;;
esac
done
}
for FILE; do
if [[ -d $FILE ]]; then
remove_dir_i "$FILE"
else
# echo "Removing file \"$FILE.\""
echo rm -f "$FILE"
fi
done
Remove echo from rm commands when you're sure it's working already. Test:
rm -f /tmp/tar-1.27.1/ABOUT-NLS
rm -f /tmp/tar-1.27.1/acinclude.m4
rm -f /tmp/tar-1.27.1/aclocal.m4
rm -f /tmp/tar-1.27.1/AUTHORS
Remove directory: /tmp/tar-1.27.1/build-aux (12 files)? [YNLQ]: n
Remove directory: /tmp/tar-1.27.1/build-aux/snippet (5 files)? [YNLQ]: n
rm -f /tmp/tar-1.27.1/ChangeLog
rm -f /tmp/tar-1.27.1/ChangeLog.1
rm -f /tmp/tar-1.27.1/config.h.in
rm -f /tmp/tar-1.27.1/configure
rm -f /tmp/tar-1.27.1/configure.ac
rm -f /tmp/tar-1.27.1/COPYING
Remove directory: /tmp/tar-1.27.1/doc (25 files)? [YNLQ]: n
Remove directory: /tmp/tar-1.27.1/gnu (358 files)? [YNLQ]: n
Remove directory: /tmp/tar-1.27.1/gnu/uniwidth (2 files)? [YNLQ]: n
rm -f /tmp/tar-1.27.1/INSTALL
Remove directory: /tmp/tar-1.27.1/lib (19 files)? [YNLQ]:
...
Actually I just came upon the -depth option of the find command that is exactly what I was looking for. I can't believe I just missed that:
-depth Process each directory's contents before the directory itself. The -delete action also implies -depth.
So similar to #Jidder's code, I can write this:
dirs=$(find ./test_script -depth -type d); for i in $dirs; do read -p "Delete files in $i? " REPLY; if [[ $REPLY == [yY] ]]; then rm -r $i; fi; done;
And for more readability:
dirs=$(find ./test_script -depth -type d)
for i in $dirs
do
read -p "Delete files in $i? " REPLY
if [[ $REPLY == [yY] ]]
then rm -r $i
fi
done;
I am trying to create a script that will find all the files in a folder that contain, for example, the string 'J34567' and process them. Right now I can process all the files in the folder with my code, however, my script will not just process the contained string it will process all the files in the folder. In other words once I run the script even with the string name ./bashexample 'J37264' it will still process all the files even without that string name. Here is my code below:
#!/bin/bash
directory=$(cd `dirname .` && pwd)
tag=$1
echo find: $tag on $directory
find $directory . -type f -exec grep -sl "$tag" {} \;
for files in $directory/*$tag*
do
for i in *.std
do
/projects/OPSLIB/BCMTOOLS/sumfmt_linux < $i > $i.sum
done
for j in *.txt
do
egrep "device|Device|\(F\)" $i > $i.fail
done
echo $files
done
Kevin, you could try the following:
#!/bin/bash
directory='/home'
tag=$1
for files in $directory/*$tag*
do
if [ -f $files ]
then
#do your stuff
echo $files
fi
done
where directory is your directory name (you could pass it as a command-line argument too) and tag is the search term you are looking for in a filename.
Following script will give you the list of files that contain (inside the file, not in file name) the given pattern.
#!/bin/bash
directory=`pwd`
tag=$1
for file in $(find "$directory" -type f -exec grep -l "$tag" {} \;); do
echo $file
# use $file for further operations
done
What is the relevance of .std, .txt, .sum and .fail files to the files containing given pattern?
Its assumed there are no special characters, spaces, etc. in file names.
If that is the case following should help working around those.
How can I escape white space in a bash loop list?
Capturing output of find . -print0 into a bash array
There are multiple issues in your script.
Following is not required to set the operating directory to current directory.
directory=$(cd `dirname .` && pwd)
find is executed twice for the current directory due to $directory and ..
find $directory . -type f -exec grep -sl "$tag" {} \;
Also, result/output of above find is not used in for loop.
For loop is run for files in the $directory (sub directories not considered) with their file name having the given pattern.
for files in $directory/*$tag*
Following for loop will run for all .txt files in current directory, but will result in only one output file due to use of $i from previous loop.
for j in *.txt
do
egrep "device|Device|\(F\)" $i > $i.fail
done
This is my temporary solution. Please check if it follows your intention.
#!/bin/bash
directory=$(cd `dirname .` && pwd) ## Should this be just directory=$PWD ?
tag=$1
echo "find: $tag on $directory"
find "$directory" . -type f -exec grep -sl "$tag" {} \; ## Shouldn't you add -maxdepth 1 ? Are the files listed here the one that should be processed in the loop below instead?
for file in "$directory"/*"$tag"*; do
if [[ $file == *.std ]]; then
/projects/OPSLIB/BCMTOOLS/sumfmt_linux < "$file" > "${file}.sum"
fi
if [[ $file == *.txt ]]; then
egrep "device|Device|\(F\)" "$file" > "${file}.fail"
fi
echo "$file"
done
Update 1
#!/bin/bash
directory=$PWD ## Change this to another directory if needed.
tag=$1
echo "find: $tag on $directory"
while IFS= read -rd $'\0' file; do
echo "$file"
case "$file" in
*.std)
/projects/OPSLIB/BCMTOOLS/sumfmt_linux < "$file" > "${file}.sum"
;;
*.txt)
egrep "device|Device|\(F\)" "$file" > "${file}.fail"
;;
*)
echo "Unexpected match: $file"
;;
esac
done < <(exec find "$directory" -maxdepth 1 -type f -name "*${tag}*" \( -name '*.std' -or -name '*.txt' \) -print0) ## Change or remove the maxdepth option as wanted.
Update 2
#!/bin/bash
directory=$PWD
tag=$1
echo "find: $tag on $directory"
while IFS= read -rd $'\0' file; do
echo "$file"
/projects/OPSLIB/BCMTOOLS/sumfmt_linux < "$file" > "${file}.sum"
done < <(exec find "$directory" . -maxdepth 1 -type f -name "*${tag}*" -name '*.std' -print0)
while IFS= read -rd $'\0' file; do
echo "$file"
egrep "device|Device|\(F\)" "$file" > "${file}.fail"
done < <(exec find "$directory" -maxdepth 1 -type f -name "*${tag}*" -name '*.txt' -print0)
Below is a snippet from a larger script that exports a list of the subdirectories of a user-specified directory, and prompts the user before making directories with the same names in another user-specified directory.
COPY_DIR=${1:-/}
DEST_DIR=${2}
export DIRS="`ls --hide="*.*" -m ${COPY_DIR}`"
export DIRS="`echo $DIRS | sed "s/\,//g"`"
if [ \( -z "${DIRS}" -a "${1}" != "/" \) ]; then
echo -e "Error: Invalid Input: No Subdirectories To Output\n"&&exit
elif [ -z "${DEST_DIR}" ]; then
echo "${DIRS}"&&exit
else
echo "${DIRS}"
read -p "Create these subdirectories in ${DEST_DIR}?" ANS
if [ ${ANS} = "n|no|N|No|NO|nO" ]; then
exit
elif [ ${ANS} = "y|ye|yes|Y|Ye|Yes|YE|YES|yES|yeS|yEs|YeS" ]; then
if [ ${COPYDIR} = ${DEST_DIR} ]; then
echo "Error: Invalid Target: Source and Destination are the same"&&exit
fi
cd "${DEST_DIR}"
mkdir ${DIRS}
else
exit
fi
fi
However, the command ls --hide="*.*" -m ${COPY_DIR} also prints files in the list as well. Is there any way to reword this command so that it only prints out directories? I tried ls -d, but that doesn't work, either.
Any ideas?
You should never rely on the output of ls to provide filenames. See the following for reasons not to parse ls: http://mywiki.wooledge.org/ParsingLs
You can build a list of directories safely using GNU find's -print0 option and appending the results to an array.
dirs=() # create an empty array
while read -r -d $'\0' dir; do # read up to the next \0 and store the value in "dir"
dirs+=("$dir") # append the value in "dir" to the array
done < <(find "$COPY_DIR" -type d -maxdepth 1 -mindepth 1 ! -name '*.*') # find directories that do not match *.*
The -mindepth 1 prevents find from matching the $COPY_DIR itself.