copy a directory structure with file names without content - linux

I have a huge directory structure of movie files. For analysis of that structure I want to copy the entire directory structure, i.e. folders and files however I don't want to copy all the movie files while I want to keep there file names. Ideally I get zero-byte files with the original movie file name.
I tried to and then rsync to my remote machine which didn't fetch the link files.
Any ideas how to do that w/o writing scripts?

You can use find:
find src/ -type d -exec mkdir -p dest/{} \; \
-o -type f -exec touch dest/{} \;
Find directory (-d) under (src/) and create (mkdir -p) them under dest/ or (-o) find files (-f) and touch them under dest/.
This will result in:
dest/src/<file-structre>
You can user mv creatively to resolve this issue.
Other (partial) solution can be achieved with rsync:
rsync -a --filter="-! */" sorce_dir/ target_dir/
The trick here is the --filter=RULE option that excludes (-) everything that is not (!) a directory (*/)

On ubuntu you can try:
cp -r --attributes-only <source_dir> <target_dir>
It doesn't copy file data.
From manpage of cp
--attributes-only
don't copy the file data, just the attributes
Note: I'm not sure this option available for other distributions, if anybody can confirm please update the answer.

I needed an alternative to this to sync only the file structure:
rsync --recursive --times --delete --omit-dir-times --itemize-changes "$src_path/" "$dst_path"
This is how I realized it:
# sync source to destination
while IFS= read -r -d '' src_file; do
dst_file="$dst_path${src_file/$src_path/}"
# new files
if [[ ! -e "$dst_file" ]]; then
if [[ -d "$src_file" ]]; then
mkdir -p "$dst_file"
elif [[ -f $src_file ]]; then
touch -r "$src_file" "$dst_file"
else
echo "Error: $src_file is not a dir or file"
fi
echo -n "+ "
ls -ld "$src_file"
# modification time changed (files only)
elif [[ -f $dst_file ]] && [[ $(date -r "$src_file") != $(date -r "$dst_file") ]]; then
touch -r "$src_file" "$dst_file"
echo -n "+ "
ls -ld "$src_file"
fi
done < <(find "$src_path" -print0)
# delete files in destination if they disappeared in source
while IFS= read -r -d '' dst_file; do
src_file="$src_path${dst_file/$dst_path/}"
# file disappeard on source
if [[ ! -e "$src_file" ]]; then
delinfo=$(ls -ld "$dst_file")
if [[ -d "$dst_file" ]] && rmdir "$dst_file" 2>/dev/null; then
echo -n "- $delinfo"
elif [[ -f $dst_file ]] && rm "$dst_file"; then
echo -n "- $delinfo"
fi
fi
done < <(find "$dst_path" -print0)
As you can see I use echo and ls to display changes.

ls > listOfMovie.txt; You will have the list of your films in a .txt file
.For multiple directories see the man page.

Related

Renaming files in subdirectories deletes the files

I am writing a script that renames *.MP4 files on an inserted SD card and then rsyncs them.
The directory with the *.MP4 files does not always have the same name:
eg: it could be /DCIM/123_PANA/ or /DCIM/141_PANA/ etc
So I'm trying to write a script that will see what folders are in the /DCIM path, and rename all the *.MP4 files, (there is also a MISC folder in this path which I suspect is causing the issue)
I am using a couple of variables to rename the files also
What I have is:
for f in /media/pi/LUMIX/DCIM/*; do
if [ -d "$f" ]; then
echo $f
for file in $(find $f -name 'P*.MP4')
do
echo $file ">" $(dirname "${file}")/$(date +"%d")$cardname$(basename $file)
mv $file $(dirname "${file}")/$(date +"%d")$cardname$(basename $file)
done
fi
done
But what seems to happen is I end up with a single file with the prefix only (say 08_nb1_) in the _PANA folder, all the others have been deleted. Obviously this is not my desired result!
UPDATE:
$cardname is of the format _nb2_
When I do as asked replace mv with echo here is the output:
/media/pi/LUMIX/DCIM/141_PANA
mv /media/pi/LUMIX/DCIM/141_PANA/P1410192.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410192.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410193.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410193.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410194.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410194.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410195.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410195.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410196.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410196.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410197.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410197.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410198.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410198.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410199.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410199.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410200.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410200.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410201.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410201.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410202.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410202.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410203.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410203.MP4
mv /media/pi/LUMIX/DCIM/141_PANA/P1410204.MP4 /media/pi/LUMIX/DCIM/141_PANA/09_nb2_P1410204.MP4
/media/pi/LUMIX/DCIM/MISC
OK I fixed it by filtering the additional directory names and limiting to only the ones with *_PANA - which solved the issue. I also added a the rsync part and demounted the SD card (if required using Zenity)
A text file placed on the SD card identifies it as a unique card giving each file a unique name when rsyncing it to the backup folder. Renaming on the SD card means
that it can still be used an written to if not full, but we then know which files have been backed up.
Very useful in the field when filming with multiple cards, crews. All running on a Rpi4
for f in /media/pi/LUMIX/DCIM/*_PANA/; do
if [ -d "$f" ]; then
echo "$f"
for file in $(find $f -wholename '*_PANA/P*.MP4')
do
mv "$file" $(dirname "${file}")/$(date +"%d")"$cardname"$(basename "${file}")
done
rsync --stats -u --progress "$f"/*.MP4 /media/pi/VDRIVE/ | tee /home/pi/Documents/ytu/rsync.txt | zenity --icon-name="dialog-warning" \
--width=300 --progress --pulsate --auto-close --auto-kill \
--title="Copying $sdn"
zenity --question --text="Unmount Card?"
if [ $? = 0 ]; then
umount /media/pi/LUMIX
else
exit
fi
fi
done

Avoid collision, if copying files

I was trying to copy all files of a certain filetype from all subfolders to one place. Unfortunately, this might cause collisions, if two files have the same name from two different subfolders.
I was using
find ./ -name '*.jpg' -exec mv -u '{}' . \;
How can I adjust this to automatically rename files (e.g. append "_1") to avoid collisions.
Or better: check if the files are the same (e.g. same size) beforehand. If yes, ignore (overwrite would be fine, too). If No, rename to avoid collision.
Suggestion would be appreciated. Thanks!
You could check before moving each individual file. Here I've used cksum to compare, which returns both the filesize and a simple checksum.
find ./ -name '*.jpg' -print0 |
while read -d '' -r path; do
file=$(basename "$path")
if [[ -e $file ]]; then
if [[ $(cksum "$file" | awk '{print $1 $2}') = $(cksum "$path" | awk '{print $1 $2}') ]]; then
continue
fi
read -n 1 -p "File '$file' would be overwritten by '$path', continue? (y/N) " -r prompt </dev/tty
if [[ $prompt != [Yy] ]]; then
continue
fi
fi
mv -f -v "$path" "$file"
done

Rewrite a script so it takes option arguments to control its behaviour

I created a script and it moves files with different extensions to their specified directories.
If the directory is not there, it creates another one (where the files will go), and it creates another directory where the remaining files with different extensions will go.
My first problem is that I want when I put -d and full path on the terminal it should move only media files, -l and full path to move all text files, then -x to change the extension to uppercase, then -u to lowercase.
Can somebody modify it for me and show me how to overcome this problem?
#!/bin/bash
From="/home/elg19/lone/doc"
To="/home/elg19/mu"
WA="/home/elg19/du"
MA="/home/elg19/dq"
WQ="/home/elg19/d2"
# this function checks if the directory exits and creates one if it does not then moves all doc files
function mama(){
if [[ ! -d "$WA" ]]; then
mkdir -p "$WA"
fi
cd "$From"
for i in pdf txt doc; do
find . -type f -name "*.${i}" -exec mv "{}" "$WA" \;
done
}
# this function checks if the directory exits and creates one if it does not then moves all media files
function so(){
if [[ ! -d "$To" ]]; then
mkdir -p "$To"
fi
cd "$From"
for i in mp3 mp4 swf; do
find . -type f -name "*.${i}" -exec mv "{}" "$To" \;
done
}
# this function checks if the directory exits and creates one if it does not then moves all image files
function soa(){
if [[ ! -d "$MA" ]]; then
mkdir -p "$MA"
fi
cd "$From"
for i in jpg gif png; do
find . -type f -name "*.${i}" -exec mv "{}" "$MA" \;
done
}
# this function checks if the directory exits and creates one if it does not then moves all the remaining files
function soaq(){
if [[ ! -d "$WQ" ]]; then
mkdir -p "$WQ"
fi
cd "$From"
for i in *; do
find . -type f -name "*.${i}" -exec mv "{}" "$WQ" \;
done
}
mama
so
soa
soaq
I don't know if the options suggested are mnemonic in your native language, but they are counter-mnemonic in English. I would suggest something more like:
-m path Move media files
-t path Move text files
-u Change extensions to upper-case
-l Change extensions to lower-case
The command to use for regular argument parsing like this is getopts (plural - many systems also have a command getopt, singular, which has different characteristics altogether).
The referenced page gives an example of how to use it:
The following example script parses and displays its arguments:
aflag=
bflag=
while getopts ab: name
do
case $name in
a) aflag=1;;
b) bflag=1
bval="$OPTARG";;
?) printf "Usage: %s: [-a] [-b value] args\n" $0
exit 2;;
esac
done
if [ ! -z "$aflag" ]; then
printf "Option -a specified\n"
fi
if [ ! -z "$bflag" ]; then
printf 'Option -b "%s" specified\n' "$bval"
fi
shift $(($OPTIND - 1))
printf "Remaining arguments are: %s\n" "$*"
The option -a doesn't take an argument; the option -b requires an argument.

Compare 2 directories and copy differences to directory 3

I have three directories. I would like to compare directory1 with directory2, then take those changes/new files and copy them over to directory3. Is there an easy way to do this, maybe by using linux diff and cp commands? I'm open to ideas.
Thanks!
Andrew
I believe this is what you want from your description.
for file in dir2/*; do
file_in_dir1=dir1/$(basename ${file})
if [ ! -e ${file_in_dir1} ]; then
# If the file in dir2 does not exist in dir1, copy
cp ${file} dir3
elif ! diff ${file} ${file_in_dir1}; then
# if the file in dir2 is different then the one in dir1, copy
cp ${file} dir3
fi
done
One thing I wasn't sure about is what you wanted if a file exists in dir1 but not dir2.
The thread yonder solves your problem quite nicely, I should think!
Copied from there:
#!/bin/bash
# setup folders for our different stages
DIST=/var/www/localhost/htdocs/dist/
DIST_OLD=/var/www/localhost/htdocs/dist_old/
DIST_UPGRADE=/var/www/localhost/htdocs/dist_upgrade/
cd $DIST
list=`find . -type f`
for a in $list; do
if [ ! -f "$DIST_OLD$a" ]; then
cp --parents $a $DIST_UPGRADE
continue
fi
diff $a $DIST_OLD$a > /dev/null
if [[ "$?" == "1" ]]; then
# File exists but is different so copy changed file
cp --parents $a $DIST_UPGRADE
fi
done
You can also do it without a bash script:
diff -qr ./dir1 ./dir2 | sed -e 's/^Only in\(.*\): \(.*\)/\1\/\2/g' -e 's/ and \..*differ$//g' -e 's/^Files //g' | xargs -I '{}' cp -Rf --parents '{}' ./dir3/
This solution removes all additional text from the diff command using sed, and then copies the files preserving the directory structure.
The two previously posted answers helped me get started but didn't get me all the way there. The solution posted by thomax was really close but I ran into an issue where the cp command on osx doesn't support the --parents parameter so I had to add some logic around the creation of subfolders which made things a bit messy and I had to restructure a bit. Here's what I wound up with:
#!/bin/bash
# setup folders for our different stages
DIST=/var/www/localhost/htdocs/dist/
DIST_OLD=/var/www/localhost/htdocs/dist_old/
DIST_UPGRADE=/var/www/localhost/htdocs/dist_upgrade/
cd $DIST
find . -type f | while read filename
do
newfile=false
modified=false
if [ ! -e "$DIST_OLD$filename" ]; then
newfile=true
echo "ADD $filename"
elif ! cmp $filename $DIST_OLD$filename &>/dev/null; then
modified=true
echo "MOD $filename"
fi
if $newfile || $modified; then
#massage the filepath to not include leading ./
filepath=$DIST_UPGRADE$(echo $filename | cut -c3-)
#create folder for it if it doesnt exist
destfolder=$(echo $filepath | sed -e 's/\/[^\/]*$/\//')
mkdir -p $destfolder
#copy new/modified file to the upgrade folder
cp $filename $filepath
fi
done
Consider you have dir1, dir2 and dir3 on the same level
with the content setup as below:
mkdir dir1
mkdir dir2
echo 1 > dir1/a
echo 1 > dir2/a
echo 2 > dir1/b
echo 3 > dir2/b
echo 4 > dir2/c
cp -r dir1 dir3
When you create and apply patch like this:
diff -ruN dir1 dir2 | patch -p1 -d dir3
Then you have content of dir2 and dir3 equivalent.
If your dir2 is not at the same level as dir1
then you have to edit filenames in the patch
so that you have equal amount of path components
in both dir1 and dir2 filenames.
You should better put your dir2 to the same level as dir1,
because there is no elegant way to do this (at least known to me).
Here follow an "ugly" way.
Consider your dir2 is located in some $BASEDIR
then you should update your diff to trim of the $BASEDIR from dir2's path
like this
diff -ruN dir1 $BASEDIR/dir2 | \
perl -slne 'BEGIN {$base =~ s/\//\\\//g; print $base}
s/\+\+\+ $base\//\+\+\+ /g; print' \
-- -base=$BASEDIR
And then you could apply the resulting path as above.

Appending rather than overwriting files when moving

I have the following directory structure:
+-archive
+-a
+-data.txt
+-b
+-data.txt
+-incoming
+-a
+-data.txt
+-c
+-data.txt
How do I do the equivalent of mv incoming/* archive/ but have the contents of the files in incoming appended to those in archive rather than overwrite them?
# move to incoming/ so that we don't
# need to strip a path prefix
cd incoming
# create directories that are missing in archive
for d in `find . -type d`; do
if [ ! -d "../archive/$d" ]; then
mkdir -p "../archive/$d"
fi
done
# concatenate all files to already existing
# ones (or automatically create them)
for f in `find . -type f`; do
cat "$f" >> "../archive/$f"
done
This should find any file in incoming and concatenate it to an existing file in archive.
The important part is to be inside incoming, because else we'd had to strip the path prefix (which is possible, but in the above case unnecessary). In the above case, a value of $f typically looks like ./a/data.txt, and hence the redirection goes to ../archive/./a/data.txt.
run it on the current directory.
find ./incoming -type f | while read -r FILE
do
dest=${FILE/incoming/archive}
cat "$FILE" >> "$dest"
done
the one in incoming/c would not be appended though
Here's a version with proper quoting:
#!/bin/sh
if [ -z "$1" ]; then
# acting as parent script
find incoming -type f -exec "$0" {} \;
else
# acting as child script
for in_file; do
if [ -f "$in_file" ]; then
destfile="${in_file/incoming/archive}"
test -d "$(dirname "$destfile")" || mkdir -p "$_"
cat "$in_file" >> "$destfile" &&
rm -f "$in_file"
fi
done
fi

Resources