Backup files with dir structure bash script - linux

I'm making a bash script that should backup all files and dir structure to another dir.
I made the following code to do that:
find . -type f -exec cp {} $HOME/$bdir \; -o -type d -exec mkdir -p {} $HOME/$bdir \; ;
The problem is, is that this only copies the files and not the dir structure.
NOTE: I may not use cp -r, cp -R or something like it because this code is part of an assignment.
I hope somebody can put me in the right direction. ;)
Joeri
EDIT:
I changed it to:
find . -type d -exec mkdir -p $HOME/$bdir/{} \; ;
find . -type f -exec cp {} $HOME/$bdir/{} \; ;
And it works! Ty guys ;)


This sounds like a job for rsync.
You mention that this is an assignment. What are your restrictions? Are you limited to only using find? Does it have to be a single command?
One way to do this is to do it in two find calls. The first call only looks for directories. When a directory is found, mkdir the corresponding directory in the destination hierarchy. The second find call would look for files, and would use a cp command like you currently have.
You can also take each filename, transform the path manually, and use that with the cp command. Here's an example of how to generate the destination filename:
> find . -type f | sed -e "s|^\./|/new/dir/|"
/new/dir/file1.txt
/new/dir/file2.txt
/new/dir/dir1/file1_1.txt
/new/dir/dir1/file1_2.txt
For your purposes, you could write a short bash script that take the source file as input, uses sed to generate the destination filename, and then passes those two paths to cp. The dirname command will return the directory portion of a filename, so mkdir -p $(dirname $destination_path) will ensure that the destination directory exists before you call cp. Armed with a script like that, you can simply have find execute the script for every file it finds.


cd olddir; tar c . | (cd newdir; tar xp)


Can you do your find with "-type d" and exec a "mkdir -p" first, followed by your find that copies the files rather than having it all in one command? It should probably also be mkdir -p $HOME/$bdir/{}.

Related

Bash command to flatten nested dicrectory

I have a directory structure like this
a/1/01.jpg
b/2/01.jpg
c/3/01.jpg
I want to make it into a structure like this
a1/01.jpg
b2/01.jpg
c3/01.jpg
So far I have a bash command that looks like this
find . -mindepth 2 -type d -execdir bash -c 'mv -i \"$1\" ./\"${1//\/\[/_[}\"' bash {} \;
However the command failed with these statements
mv: cannot stat '"./3"': No such file or directory
mv: cannot stat '"./2"': No such file or directory
mv: cannot stat '"./1"': No such file or directory
What am I doing wrong here? is there a better way to do this?

Doing that by using plain bash would be easier:
#!/bin/bash
for src in */*/; do
dst=${src/\/}
echo mkdir "$dst"
echo mv "$src"* "$dst"
done
Remove both echos if the output looks fine.
Or, a more efficient version:
for src in */*/; do
mv "$src" "${src/\/}"
done
but this version won't work properly when the destination directory (a1, b2, c3, etc.) already exists.

All operations need to be performed on the leaf directories. These are identified by having two links which you can find with:
$ find -type d -links 2
Once you have these directories, you only have to rename the directory
$ find -type d -links 2 -exec sh -c 'd1={};d2="${d1#./}"; [ "$d2" = "${d2//\//}" ] || mv -v "$d1" "${d2//\//}"' \;
Find will complain a bit as you moved a directory away that it was traversing, but this should do it.
If you have a pure structure, i.e. files only appear in leaf-directories, this should do it. All you need to do now is delete the empty directories:
$ find -type d -empty -delete
Be aware, however, that if a leaf directory was already empty, the latter will remove it.

Rename "*.sample.js" files to "*.js" in linux

Currently, I'm working on a project where I need to git ignore my local config files. This means that if the user clones the repository, he would see *.sample.js files inside the config dir.
If the user executes make config the following script is executed.
config: ###Misc Create config files
#cd src/backend/config
#cp cache.sample.js cache.js
#cp database.sample.js database.js
#cp steam.sample.js steam.js
#cp teamspeak.sample.js teamspeak.js
#cp website.sample.js website.js
#echo
This script basically removes the "sample" part from the file name. The above code needs to be modified when ever I add a new config file, which is not a practical thing.
I would like to convert this into a simple regex command like:
find -iname \*.sample.js -type f -exec rename -n 's/(.*sample\.js)/$1/' {} \;
This command doesn't work, at least on windows MINGW64 bash. I need a vanilla solution.

Thanks to #Jetchisel for a working solution:
find -iname '*.sample.js' -type f -exec bash -c 'for f; do cp "$f" "${f/.sample}"; done' sh {} +
Bonus: why that last + sign?
That will process as many files as possible while avoiding arg_max
as opposed to \; which will process one file per -exec call

Linux: copy ".svn" directories recursively

I know there are dozen of questions about similar topcis but I still can't beat this up.
I need to copy all .svn directories recursively from /var/foo to /var/foo2 on a Debian machine:
/var/www/foo/.svn
/var/www/foo/bar/.svn
...
I tried these two commands without success:
find /var/foo -name ".svn" -type f -exec cp {} ./var/foo2 \;
find /var/foo -name ".svn" -type d -exec cp {} /var/foo2 \;
Once only the svn directory right inside foo is copied, while another time nothing is copied.

Given following file structure:
./
./a/
./a/test/
./a/test/2
./b/
./b/3
./test/
./test/1
Running following script in the directory to be copied:
find -type d -iname test -exec sh -c 'mkdir -p "$(dirname ~/tmp2/{})"; cp -r {}/ ~/tmp2/{}' \;
Should copy all test directories to ~/tmp2/.
Points of interest:
Directories are copied to the destination on a one-by-one basis
Parent directories are created in advance so that cp doesn't complain about target not existing
Rather than just cp, cp -r is used
The whole command is wrapped with sh -c so that operations on {} such as dirname can be performed (so that the shell expands it for each directory separately, rather than expanding it once during calling the find)
Resulting structure in ~/tmp2:
./
./a/
./a/test/
./a/test/2
./test/
./test/1
So all you should need to do is to replace test with .svn and ~/tmp2 with directory of choice. Just remember about running it in the source directory, instead of using absolute paths.

I find that using tar for such operations makes the code often much more readable:
$ mkdir /var/www/foo2
$ cd /var/www/foo2
$ find ../foo/ -type d -name .svn -exec tar c \{\} \+ | \
tar x --strip-components=1
find will list all directories named .svn, and call tar to create (c) an archive file (that is sent to stdout) with all these directories. the archive on stdout is then extracted (x) by another tar instance in the target directory. the relative path portion (../) is automatically removed by the archiving tar, but since we also want to remove the first path component (foo/) we need to add --strip-components.
Note: This will only work if you do not have very many .svn directories you want to copy (more than $(getconf ARG_MAX)-2, which on my system is more than 200000).

Copying modified files and their file structure

I'm still new to linux scripting, so this might be a bit trivial.
As part of my script, I am trying to copy all the modified files (relative to the original backup I have made) to another folder while keeping the file structure.
I have found this method:
find /SourceFolder/ -newer /BackupFOlder/ -exec cp --parents \{\} /Destination
However the above command does not work, giving me the error:
find: missing argument to 'exec'
Why doesn't this work?
I also found this:
Copy files preserving folder structure
But I want to use cp command only.
Would really appreciate some help.

The -exec option requires a ; argument to tell it where the command ends, because you could have additional find options after it.
find /SourceFolder/ -newer /BackupFOlder/ -exec cp --parents {} /Destination \;
However, a better solution would be to use rsync:
rsync -a /SourceFolder /BackupFolder

Try
tar cf - . | (cd <some other dir>; tar xfv -)

Copy folder structure (without files) from one location to another

I want to create a clone of the structure of our multi-terabyte file server. I know that cp --parents can move a file and it's parent structure, but is there any way to copy the directory structure intact?
I want to copy to a linux system and our file server is CIFS mounted there.

You could do something like:
find . -type d > dirs.txt
to create the list of directories, then
xargs mkdir -p < dirs.txt
to create the directories on the destination.

cd /path/to/directories &&
find . -type d -exec mkdir -p -- /path/to/backup/{} \;

Here is a simple solution using rsync:
rsync -av -f"+ */" -f"- *" "$source" "$target"
one line
no problems with spaces
preserve permissions
I found this solution there

1 line solution:
find . -type d -exec mkdir -p /path/to/copy/directory/tree/{} \;

I dunno if you are looking for a solution on Linux. If so, you can try this:
$ mkdir destdir
$ cd sourcedir
$ find . -type d | cpio -pdvm destdir

This copy the directories and files attributes, but not the files data:
cp -R --attributes-only SOURCE DEST
Then you can delete the files attributes if you are not interested in them:
find DEST -type f -exec rm {} \;

This works:
find ./<SOURCE_DIR>/ -type d | sed 's/\.\/<SOURCE_DIR>//g' | xargs -I {} mkdir -p <DEST_DIR>"/{}"
Just replace SOURCE_DIR and DEST_DIR.

The following solution worked well for me in various environments:
sourceDir="some/directory"
targetDir="any/other/directory"
find "$sourceDir" -type d | sed -e "s?$sourceDir?$targetDir?" | xargs mkdir -p

This solves even the problem with whitespaces:
In the original/source dir:
find . -type d -exec echo "'{}'" \; > dirs2.txt
then recreate it in the newly created dir:
mkdir -p <../<SOURCEDIR>/dirs2.txt

Substitute target_dir and source_dir with the appropriate values:
cd target_dir && (cd source_dir; find . -type d ! -name .) | xargs -i mkdir -p "{}"
Tested on OSX+Ubuntu.

If you can get access from a Windows machine, you can use xcopy with /T and /E to copy just the folder structure (the /E includes empty folders)
http://ss64.com/nt/xcopy.html
[EDIT!]
This one uses rsync to recreate the directory structure but without the files.
http://psung.blogspot.com/2008/05/copying-directory-trees-with-rsync.html
Might actually be better :)

A python script from Sergiy Kolodyazhnyy
posted on Copy only folders not files?:
#!/usr/bin/env python
import os,sys
dirs=[ r for r,s,f in os.walk(".") if r != "."]
for i in dirs:
os.makedirs(os.path.join(sys.argv[1],i))
or from the shell:
python -c 'import os,sys;dirs=[ r for r,s,f in os.walk(".") if r != "."];[os.makedirs(os.path.join(sys.argv[1],i)) for i in dirs]' ~/new_destination
FYI:
Copy top level folder structure without copying files in linux
How do I copy a directory tree but not the files in Linux?

Another approach is use the tree which is pretty handy and navigating directory trees based on its strong options. There are options for directory only, exclude empty directories, exclude names with pattern, include only names with pattern, etc. Check out man tree
Advantage: you can edit or review the list, or if you do a lot of scripting and create a batch of empty directories frequently
Approach: create a list of directories using tree, use that list as an arguments input to mkdir
tree -dfi --noreport > some_dir_file.txt
-dfi lists only directories, prints full path for each name, makes tree not print the indentation lines,
--noreport Omits printing of the file and directory report at the end of the tree listing, just to make the output file not contain any fluff
Then go to the destination where you want the empty directories and execute
xargs mkdir < some_dir_file.txt

find source/ -type f | rsync -a --exclude-from - source/ target/
Copy dir only with associated permission and ownership

Simple way:
for i in `find . -type d`; do mkdir /home/exemplo/$i; done

cd oldlocation
find . -type d -print0 | xargs -0 -I{} mkdir -p newlocation/{}
You can also create top directories only:
cd oldlocation
find . -maxdepth 1 -type d -print0 | xargs -0 -I{} mkdir -p newlocation/{}

Here is a solution in php that:
copies the directories (not recursively, only one level)
preserves permissions
unlike the rsync solution, is fast even with directories containing thousands of files as it does not even go into the folders
has no problems with spaces
should be easy to read and adjust
Create a file like syncDirs.php with this content:
<?php
foreach (new DirectoryIterator($argv[1]) as $f) {
if($f->isDot() || !$f->isDir()) continue;
mkdir($argv[2].'/'.$f->getFilename(), $f->getPerms());
chown($argv[2].'/'.$f->getFilename(), $f->getOwner());
chgrp($argv[2].'/'.$f->getFilename(), $f->getGroup());
}
Run it as user that has enough rights:
sudo php syncDirs.php /var/source /var/destination

Resources