Bash script to copy the directory structure from source directory into target directory - linux

I am very new to Bash scritping and to get some practice, I am attempting to write a script that takes in a source directory and a destination directory. The script will search the source directory and copy its structure of sub-directories into the target directory (any files will be ignored, just the directories themselves will be duplicated). The source directory can have any number of sub-directories at any depth. What would be the best way to achieve this? I have started by attempting to write a recursive function where, if a directory is found, the function is called recursively onto this directory. However, due to my lack of experience with scripting, I have been stumped.
Here is what I have so far:
#! /bin/bash
if [ ! $# -eq 2 ]; then
echo "ERROR: Script needs 2 arguments: $0 source/directory/ target/directory/"
exit
fi
SOURCE_DIR=$1
TARGET_DIR=$2
function searchForDirectory {
for FILE in ls $SOURCE_DIR/*; do #should be 'ls *current_directory*'
if [ -d $FILE ]; then
searchForDirectory #call function recursively onto $FILE
#Do stuff here to create copy of this directory in target dir
fi
done
}
if [ ! -d $SOURCE_DIR ]; then
echo "ERROR: Source directory does not exist"
exit
else
searchForDirectory
fi
I know that this is just a skeleton function, and a lot more work would need to be put into it, but I'm just looking for guidance as to whether this is the correct way to go about this before I go any further, and if so, what would be my next step? How do I pass a directory into my function?
EDIT: Here is my solution thanks to Ivan's help below
#! /bin/bash
if [ ! $# -eq 2 ]; then
echo -e "\nERROR: Script needs 2 arguments:\n$0 source/directory/ target/directory/\n"
exit
fi
function recursiveDuplication {
for file in `ls $1`; do
if [ -d $1/$file ] && [ ! -d $2/$file ]; then
mkdir $2/$file
recursiveDuplication $1/$file $2/$file
elif [[ $1/$file == *.png ]]; then
convert $1/$file $2/$file
fi
done
}
if [ ! -d $1 ]; then
echo -e "\nERROR: $1 does not exist\n"
exit
elif [ -d $2 ] && [ ! -w $2 ]; then
echo -e "\nERROR: You do not have permission to write to $2\n"
exit
elif [ ! -d $2 ]; then
echo -e "\nSUCCESS: $2 has been created"
mkdir $2
fi
recursiveDuplication $1 $2
There are two issues with this solution:
As Rany Albeg Wein explained below, using 'ls' is not a good solution - and I have seen why when there is a space in a directory/*.png name.
I am also attempting to copy any *.png file from the source to the target and convert it to a *.jpg image in the target. How can I do this? I am attempting to use ImageMagick's convert image.png image.jpg command, but do not know how to do so when the image.png is being referred to as $file?

you can simplify a lot
$ find a -type d | xargs -I d mkdir -p copy/d
copy the tree structure from directory a to new directory under copy
$ tree a
a
|-- b
| |-- c
| | `-- file4
| |-- d
| | `-- file4
| `-- file3
`-- file2
3 directories, 4 files
$ tree copy
copy
`-- a
`-- b
|-- c
`-- d
4 directories, 0 files

This solution has been tested with directory names containing special chars, a loop in the directory tree and a broken symbolic link.
There is bellow a stansard solution:
( cd source/ ; find . -depth -type d -printf "%p\0" | xargs -0 -I xxx mkdir -p ../destination/xxx )
And there is a second solution using gnu cpio:
( cd source/ ; find . -depth -type d -printf "%p\0" | cpio --null -pd ../destination )
The test:
$ mkdir -p source/a/ba/c/d/e
$ mkdir -p source/a/bb/c/d/e
$ mkdir -p source/a/b/ca/d/e
$ ln -s source/broken/link source/a/ba/c/d/e/broken
$ (cd source/a/ba && ln -s ../../a tree-loop)
$ mkdir -p source/z/"$(printf "\1\2\3\4\5\6\7\10\11\12\13\14\15\16\17\20\21\22\23\24\25\26\27\30\31\32\33\34\35\36\37\40\41\42\43\44\45\46\47\72\73\74\75\76\77\100\133\134\135\1336\137\140\173\174\175\176\177dir")"
$ touch source/a/b/"$(printf "\1\2\3\4\5\6\7\10\11\12\13\14\15\16\17\20\21\22\23\24\25\26\27\30\31\32\33\34\35\36\37\40\41\42\43\44\45\46\47\72\73\74\75\76\77\100\133\134\135\1336\137\140\173\174\175\176\177file")"
$ ls
source
$ find source -depth
source/z/??????????????????????????????? !"#$%&':;<=>?#[\][6_`{|}~?dir
source/z
source/a/ba/tree-loop
source/a/ba/c/d/e/broken
source/a/ba/c/d/e
source/a/ba/c/d
source/a/ba/c
source/a/ba
source/a/b/ca/d/e
source/a/b/ca/d
source/a/b/ca
source/a/b/??????????????????????????????? !"#$%&':;<=>?#[\][6_`{|}~?file
source/a/b
source/a/bb/c/d/e
source/a/bb/c/d
source/a/bb/c
source/a/bb
source/a
source
$ ( cd source/ ; find . -depth -type d -printf "%p\0" | xargs -0 -I xxx mkdir -p ../destination/xxx )
$ find destination/ -depth
destination/z/??????????????????????????????? !"#$%&':;<=>?#[\][6_`{|}~?dir
destination/z
destination/a/ba/c/d/e
destination/a/ba/c/d
destination/a/ba/c
destination/a/ba
destination/a/b/ca/d/e
destination/a/b/ca/d
destination/a/b/ca
destination/a/b
destination/a/bb/c/d/e
destination/a/bb/c/d
destination/a/bb/c
destination/a/bb
destination/a
destination/
The gnu cpio test:
$ rm -rf destination
$ ( cd source/ ; find . -depth -type d -printf "%p\0" | cpio --null -pd ../destination )
0 blocks
$ find destination -depth
destination/z/??????????????????????????????? !"#$%&':;<=>?#[\][6_`{|}~?dir
destination/z
destination/a/ba/c/d/e
destination/a/ba/c/d
destination/a/ba/c
destination/a/ba
destination/a/b/ca/d/e
destination/a/b/ca/d
destination/a/b/ca
destination/a/b
destination/a/bb/c/d/e
destination/a/bb/c/d
destination/a/bb/c
destination/a/bb
destination/a
destination

#!/bin/bash
# 1st argument - source dir, 2nd - destination
function rrr {
for i in `ls $1`
do
if [ -d $1/$i ]
then
mkdir $2/$i
rrr $1/$i $2/$i
fi
done
}
rrr $1 $2

Related

bash: cd: No such file or directory being returned but directory exists

I'm trying to run the following code
#!/bin/bash
source="/Users/9001128.ELOCAL/Downloads"
destination="/Users/Public/Documents"
days="+1"
destination=$(cd -- "$destination" && pwd) # make it an absolute path
cd -- "$source" &&
find . -type f -mtime +1 -atime +1 -exec sh -c '
mkdir -p "$0/${1%/*}"
mv "$1" "$0/$1"
' "$destination" {} \;
destination=$(cd -- "$destination" && pwd) # make it an absolute path
cd -- "$source" &&
find . -type f -mtime +1 -atime +1 -exec sh -c '
for x do
mkdir -p "$0/${x%/*}"
mv "$x" "$0/$x"
done
' "$destination" {} +
autoload -U zmv
mkdir_mv () {
mkdir -p -- $3:h
mv -- $2 $3
}
zmv -Qw -p mkdir_mv $source/'**/*(.m-'$days')' '$destination/$1$2'
Its function is to move the files to a new directory and maintain the directory structure, as in the example:
/Users/Public/Documents/file.txt
after moving to backup it would look like this:
backup/Users/Public/Documents/file.txt
Keeping the whole structure.
But when I run the script, I get the following message in the shell:
bash: cd: /Users/Public/Documents: No such file or directory
Does anyone know what I'm doing wrong so that the script doesn't work?

Prefixing sub-folder names to a filename, and moving file to current directory

I have a car stereo that reads USB sticks, but the stereo's "random" function only works on one folder, so I want to move all files to a parent folder, "STICK/music", but to have the Artist, and Album if any, added to the filename.
But what I've done is sort of hard-coded, and I'd like to be able to use different sticks.
I would like to have these ("RHYTHMBLUES" is the name of the stick):
/media/homer/RHYTHMBLUES/music/Artist/Album/name1.mp3
/media/homer/RHYTHMBLUES/music/Artist/Album/name2.mp3
/media/homer/RHYTHMBLUES/music/Artist/name3.mp3
/media/homer/RHYTHMBLUES/music/Artist/name4.mp3
renamed with folder names and moved to "music":
/media/homer/RHYTHMBLUES/music/Artist-Album-name1.mp3
/media/homer/RHYTHMBLUES/music/Artist-Album-name2.mp3
/media/homer/RHYTHMBLUES/music/Artist-name3.mp3
/media/homer/RHYTHMBLUES/music/Artist-name4.mp3
I thought of having a bash command to call "doit":
find . -type d -exec sh -c "cd \"{}\" ; /home/homer/doit \"{}\" " \;
and "doit" would contain:
#!/bin/sh -x
echo --------------------------------
pwd
for i in *.mp3
do new=$(printf "${1}/${i}")
# remove the ./ at the beginning of the name
new=`echo "${new}" | cut -c 3-`
# replace folder separators
new=`echo "${new}" | tr '/' '-'`
echo "$i" "/media/homer/RHYTHMBLUES/music/$new"
mv -T "$i" "/media/homer/RHYTHMBLUES/music/$new"
done
doitbaby.sh
#!/bin/sh
moveTo () { # $1=directory, $2=path, $3=targetName
local path="$(realpath "$1")"
for file in "$path"/*; do
if [ -d "$file" -a -n "$3" ]; then
moveTo "$file" "$2" "$3""${file##*/}"-
elif [ -d "$file" ]; then
moveTo "$file" "$path" "${file##*/}"-
elif [ -f "$file" -a "${file##*.}" = 'mp3' ]; then
echo 'FROM: '"$file"
echo 'TO : '"$2"/"$3""${file##*/}"
# mv "$file" "$2"/"$3""${file##*/}" # Uncomment this for real usage.
fi
done
}
removeEmptyDirs () {
local path="$(realpath "$1")"
for file in "$path"/*; do
if [ -d "$file" ]; then
rm -fR "$file"
fi
done
}
moveTo "$1"
# removeEmptyDirs "$1" # Uncomment this for real usage.
Note
I have commented 2 lines in the above script. Test it first before any further real actions.
Call the script like this: (Make sure the path is correct. This is the expected path by this script by the way.)
./doitbaby.sh '/media/homer/RHYTHMBLUES/music/'

How to rename file and folder with the same case as a reference one

I have to rename a complete folder tree ('target') recursively so that it has
the same files and folders name as on the file server ('server').
Example. On the 'target':
./target
./target/FILE
./target/dir2
./target/dir2/File2
./target/DIR1
./target/DIR1/file1
On the 'server':
./target
./target/file
./target/DIR2
./target/DIR2/File2
./target/dir1
./target/dir1/File1
I'm quite sure that if the filenames are the same (comparing in lower case)
the files are the same (maybe I can add a checksum comparing).
Final result should be that 'target' has the same filenames as the 'server'.
I've tried with bash (should be the best solution)... but bash hate me! Any clue? TY!
first, create 2 files, source.txt and target.txt containing the directories as printed in your question
You can achieve this for instance like this:
(cd path/to/the/server;find . -type d) > target.txt
(cd path/to/the/source;find . -type d) > source.txt
Then run this shell:
while read target_dir
do
fixed_name=$(grep -i ^$target_dir\$ source.txt)
if [ ! -z "$fixed_name" ] ; then
(cd path/to/the/server;mv $target_dir $fixed_name)
fi
done < target.txt
Basically, for each line of target.txt file, it looks the correct-case counterpart in the source.txt file. If found, then issues the rename command, located in the path/to/the/server directory.
Thanks Jean: u guide me to the solution! It's a little bit more complex:
#!/bin/bash
# 160912
# match filenames char-cases on target path as in server path
# usage : $0 targetDir serverDir
TARGET_PATH=$1
MAKE_EQUAL()
{
while read server_name
do
new_target_name=$(grep -i ^$server_name\$ targetList.txt)
if [ ! -z "$new_target_name" ] && [ "$server_name" != "$new_target_name" ] ; then
echo " $new_target_name --> $server_name"
(cd $TARGET_PATH;mv $new_target_name $server_name)
fi
done < serverList.txt
}
if [ ! -d "$1" -o ! -d "$2" ] ; then
echo "Paths not found!"
exit 0
fi
# Change directories changing depth level until is the last directory level
DIRLEVEL=1
LASTLEVEL=0
(cd $2;find . -maxdepth $DIRLEVEL -type d) > serverList.txt
while [ "$LASTLEVEL" == "0" ]
do
echo "Match directory level $DIRLEVEL:"
(cd $1;find . -maxdepth $DIRLEVEL -type d) > targetList.txt
MAKE_EQUAL
DIRLEVEL=`expr $DIRLEVEL + 1`
(cd $2;find . -maxdepth $DIRLEVEL -type d) > nextServerList.txt
diff nextServerList.txt serverList.txt > /dev/null
if [ $? -eq 0 ] ; then
LASTLEVEL=1
else
mv nextServerList.txt serverList.txt
fi
done
echo "Match files:"
(cd $1;find . -type f) > targetList.txt
(cd $2;find . -type f) > serverList.txt
MAKE_EQUAL
echo "Done!"

Archive old files only AND re-construct folder tree in archive

I want to move all my files older than 1000 days, which are distributed over various subfolders, from /home/user/documents into /home/user/archive. The command I tried was
find /home/user/documents -type f -mtime +1000 -exec rsync -a --progress --remove-source-files {} /home/user/archive \;
The problem is, that (understandably) all files end up being moved into the single folder /home/user/archive. However, what I want is to re-construct the file tree below /home/user/documents inside /home/user/archive. I figure this should be possible by simply replacing a string with another somehow, but how? What is the command that serves this purpose?
Thank you!
I would take this route instead of rsync:
Change directories so we can deal with relative path names instead of absolute ones:
cd /home/user/documents
Run your find command and feed the output to cpio, requesting it to make hard-links (-l) to the files, creating the leading directories (-d) and preserve attributes (-m). The -print0 and -0 options use nulls as record terminators to correctly handle file names with whitespace in them. The -l option to cpio uses links instead of actually copying the files, so very little additional space is used (just what is needed for the new directories).
find . -type f -mtime +1000 -print0 | cpio -dumpl0 /home/user/archives
Re-run your find command and feed the output to xargs rm to remove the originals:
find . -type f -mtime +1000 -print0 | xargs -0 rm
Here's a script too.
#!/bin/bash
[ -n "$BASH_VERSION" ] && [[ BASH_VERSINFO -ge 4 ]] || {
echo "You need Bash version 4.0 to run this script."
exit 1
}
# SOURCE=/home/user/documents/
# DEST=/home/user/archive/
SOURCE=$1
DEST=$2
declare -i DAYSOLD=10
declare -a DIRS=()
declare -A DIRS_HASH=()
declare -a FILES=()
declare -i E=0
# Check directories.
[[ -n $SOURCE && -d $SOURCE && -n $DEST && -d $DEST ]] || {
echo "Source or destination directory may be invalid."
exit 1
}
# Format source and dest variables properly:
SOURCE=${SOURCE%/}
DEST=${DEST%/}
SOURCE_LENGTH=${#SOURCE}
# Copy directories first.
echo "Creating directories."
while read -r FILE; do
DIR=${FILE%/*}
if [[ -z ${DIRS_HASH[$DIR]} ]]; then
PARTIAL=${DIR:SOURCE_LENGTH}
if [[ -n $PARTIAL ]]; then
TARGET=${DEST}${PARTIAL}
echo "'$TARGET'"
mkdir -p "$TARGET" || (( E += $? ))
chmod --reference="$DIR" "$TARGET" || (( E += $? ))
chown --reference="$DIR" "$TARGET" || (( E += $? ))
touch --reference="$DIR" "$TARGET" || (( E += $? ))
DIRS+=("$DIR")
fi
DIRS_HASH[$DIR]=.
fi
done < <(exec find "$SOURCE" -mindepth 1 -type f -mtime +"$DAYSOLD")
# Copy files.
echo "Copying files."
while read -r FILE; do
PARTIAL=${FILE:SOURCE_LENGTH}
cp -av "$FILE" "${DEST}${PARTIAL}" || (( E += $? ))
FILES+=("$FILE")
done < <(exec find "$SOURCE" -mindepth 1 -type f -mtime +"$DAYSOLD")
# Remove old files.
if [[ E -eq 0 ]]; then
echo "Removing old files."
rm -fr "${DIRS[#]}" "${FILES[#]}"
else
echo "An error occurred during copy. Not removing old files."
exit 1
fi

remove all non-directories from file list variable

Below is a snippet from a larger script that exports a list of the subdirectories of a user-specified directory, and prompts the user before making directories with the same names in another user-specified directory.
COPY_DIR=${1:-/}
DEST_DIR=${2}
export DIRS="`ls --hide="*.*" -m ${COPY_DIR}`"
export DIRS="`echo $DIRS | sed "s/\,//g"`"
if [ \( -z "${DIRS}" -a "${1}" != "/" \) ]; then
echo -e "Error: Invalid Input: No Subdirectories To Output\n"&&exit
elif [ -z "${DEST_DIR}" ]; then
echo "${DIRS}"&&exit
else
echo "${DIRS}"
read -p "Create these subdirectories in ${DEST_DIR}?" ANS
if [ ${ANS} = "n|no|N|No|NO|nO" ]; then
exit
elif [ ${ANS} = "y|ye|yes|Y|Ye|Yes|YE|YES|yES|yeS|yEs|YeS" ]; then
if [ ${COPYDIR} = ${DEST_DIR} ]; then
echo "Error: Invalid Target: Source and Destination are the same"&&exit
fi
cd "${DEST_DIR}"
mkdir ${DIRS}
else
exit
fi
fi
However, the command ls --hide="*.*" -m ${COPY_DIR} also prints files in the list as well. Is there any way to reword this command so that it only prints out directories? I tried ls -d, but that doesn't work, either.
Any ideas?
You should never rely on the output of ls to provide filenames. See the following for reasons not to parse ls: http://mywiki.wooledge.org/ParsingLs
You can build a list of directories safely using GNU find's -print0 option and appending the results to an array.
dirs=() # create an empty array
while read -r -d $'\0' dir; do # read up to the next \0 and store the value in "dir"
dirs+=("$dir") # append the value in "dir" to the array
done < <(find "$COPY_DIR" -type d -maxdepth 1 -mindepth 1 ! -name '*.*') # find directories that do not match *.*
The -mindepth 1 prevents find from matching the $COPY_DIR itself.

Resources