I want to rsync only certain file types (e.g. .py) and I want to exclude some directories (e.g. venv).
This is what I have tried:
rsync -avz --include='*/' --exclude='venv/' --include='*.py' --exclude='*' /tmp/src/ /tmp/dest/
But it doesn't work.
What am I missing?
With rsync you do not need to use --include="*.py" to include '*.py' files in the copy. The --include option will only include files that have been excluded by --exclude= before. rsync specifies ** as the wildcard specifier. For example, if you want to copy all .py files in the current directory (and subdirectories), but not copy anything from the venc directory, you can do something similar to:
rsync -uav --exclude="venc" **.py destination
(note -a implies -rlptgoD)
which would recursively copy all .py files in present working directory to destination excluding the venc directory.
To recursively copy only *.py files from all directories below the current path excluding any venc directories, you can build a temporary file with the results of find containing the *.py files and exclude files containing venc/ as part of the path, and then transfer all filenames in the temporary file using the --files-from and --no-R (no relative) options to rsync as:
$ find /path/to -type f -name "*.py" | grep -v 'venc/' > tmpfile \
rsync -uav --no-R --files-from=tmpfile / host:/dest/dir \
rm tmpfile
This will capture all *.py files in any subdirectories excluding all directories including the name venc/ and anything below them. The --no-R option is needed to prevent the absolute filenames in tmpfile from be taken as relative to the current working directory.
Related
Lets say i have a folder named Pictures and I want to move or copy all files out of this folder.
However I also want to move and harvest all of the files who are in sub folders so:
Pictures/1.png
Pictures/yolo/2.png
Pictures/yolo/swag/sand/3.png
Pictures/extra/fire/4.png
I want to move or copy all these files to another folder like results so I get:
results/1.png
results/2.png
results/3.png
results/4.png
Only I have no idea in advance what sub folders will be in the Pictures folder.
How can I accomplish this in bash/shell scripts ?
I also appreciate making it file type neutral so any files are harvested from their directories (not only .png like in my example) and I have no idea what the file name will be (I only used 1...4 because i did not have any idea how to name them).
You can do it like this:
find /absolute/path/to/Pictures -type f -name '*.png' -exec mv -i {} /absolute/path/to/results \;
Another option is to use xargs
find /absolute/path/to/Pictures -name '*.png' | xargs -I files mv files /absolute/path/to/results
You can simply copy all files and subdirectories along with their contents using cp's recursive option:
cp -pr <source_path>/* <destination_path>/
But, moving them recursively is a bit tricky, you will need to create tar files of the subdirectories and move them and then untar the tar files in destination path. As this is a complex process, as a workaround, you can copy the files/directories recursively and then delete the files from original path.
cp -pr <source_path>/* <destination_path>/ && rm -rf <source_path>/*
What I am trying to basically do with this shell script is to have the user input a directory via console and have the output be all of the .c files that are inside that directory and the sub-directories within it (the files are copied to the current location of the script).
rsync -va --include "*/" --include '*.c' --exclude '*' "$1/" .
However when I run the script (sh test.sh DirA) it copies all of the sub-directories within the $1 directory, with nothing inside them (not even the .c files). I want the output to not be the directories, but instead ONLY the .c files.
So if I were to ls in the current directory it should come out with a bunch of files like: file.c file1.c file2.c NOT dir1 dir2 dir3.
You should simply leave out --exclude '*'. As it occurs later on the command line, it is applied after a candidate file succeded the --include '*.c' test and causes any file to be rejected.
So, for simply copying any .c file within the given subtree to a hierarchy at current directory use a plain:
rsync -va --include '*.c' "$1" .
EDIT:
As rsync is all about copying and synchronizing hierarchies, it is the wrong tool for flattening hierarchies into a single directory. For such an operation you will need a command that is explicitly ignoring the source path for the target side (like plain cp).
E.g. in your case you could use:
find "$1" -name '*.c' -print0 | xargs -0 -I'{}' cp {} .
I'm trying to backup my www-folder but hidden folders like .config inside www are added to the backup. I want to exclude the folder "backups" and all folders (and files) starting with a dot.
The problem is that it copies all the hidden folders like .config to the zip-file.
Current code:
zip -r /var/www/backups/site/$(date +\%Y-\%m-\%d-\%H-\%M).zip /var/www -x "*backups*" "*.*" "*/.*"
This should work for you.
zip -r --exclude=*backups* --exclude=*/.* /var/www/backups/site/$(date +\%Y-\%m-\%d-\%H-\%M).zip /var/www
Use a linux find command with an exclude flag, then pipe it into zip.
The following command will exclude all paths under the current directory containing the keywords "backups" or files with "/." in the path and then pipe the files into zip.
find . | grep -v "\(backups\|/\.\)" | xargs zip archive.zip
I'm trying to write a script that copy all the files of one dir (with subdirs) to the root of another dir.
So Imagine I have this file structure:
/
pic.JPG
PIC5.JPG
FOLDER
pic2.JPG
pic3.JPG
FOLDER2
pic4.JPG
I want all the .JPG files from that directory and copy them over to another destination. But I don't want the directory structure, just the files.
This is what I've got:
"sudo rsync -aq --include '*/' --include '*.JPG' --exclude '*\' /source/picturesRoot/ /destination/flatView/
But it also copies the directories :(
I found this link on stackoverflow:
rsync : Recursively sync all files while ignoring the directory structure
I looked at the solution and didn't see much difference with my command, apart from the * and . in the path. I tried it but it didn't work.
I hope somebody can help me, thanks.
This answer cannot work for you because your pictures are not at the same level in directories. There is no option in rsync to skip the creation of directory structure. In the link you gave, it's working because the user explicitly select source files with *.
You can try something with find and rsync. Find will find files and rsync copy them.
Here is a solution :
find /source/picturesRoot -type f -name "*.JPG" -exec rsync -a {} /destination/flatView/ \;
Be careful, if two files have the same name just one will be in destination directory.
I have two directories and one is empty.
The first directory has many sub directories with hidden files. When I cp -r content from first directory to the second one, the hidden files gets copied too. Any solutions to escape them?
You can use rsync instead of cp:
rsync -av --exclude=".*" src dest
This excludes hidden files and directories. If you only want to exclude hidden directories, add a slash to the pattern:
rsync -av --exclude=".*/" src dest
You can do
cp -r SRC_DIR/* DEST_DIR
to exclude all .files and .dirs in the SRC_DIR level, but still it would copy any hidden files in the next level of sub-directories.
rsync has "-C" option
http://rsync.samba.org/ftp/rsync/rsync.html
Example:
rsync -vazC dir1 dir2
I came across the same need when I wanted to copy the files contained in a git repo, but excluding the .git folder, while using git bash.
If you don't have access to rsync, you can replicate the behavior of --exclude=".*" by using the find command along with xargs:
find ./src_dir -type f -not -path '*/.*' | xargs cp --parents -t ./dest_dir
To give more details:
find ./src_dir -type f -not -path '*/.*' will find all files in src_dir excluding the ones where the path contain a . at the beginning of a file or folder.
xargs cp --parents -t ./dest_dir will copy the files found to dest_dir, recreating the folder hierarchy thanks to the --parents argument.
Note: This will not copy empty folders. And will effectively exclude all hidden files and folders from being copied.
Link to relevant doc:
https://linux.die.net/man/1/cp
https://linux.die.net/man/1/find