I'm having problems getting my rsync syntax right and I'm wondering if my scenario can actually be handled with rsync. First, I've confirmed that rsync is working just fine between my local host and my remote host. Doing a straight sync on a directory is successful.
Here's what my filesystem looks like:
uploads/
1260000000/
file_11_00.jpg
file_11_01.jpg
file_12_00.jpg
1270000000/
file_11_00.jpg
file_11_01.jpg
file_12_00.jpg
1280000000/
file_11_00.jpg
file_11_01.jpg
file_12_00.jpg
What I want to do is run rsync only on files that begin with "file_11_" in the subdirectories and I want to be able to run just one rsync job to sync all of these files in the subdirectories.
Here's the command that I'm trying:
rsync -nrv --include="**/file_11*.jpg" --exclude="*" /Storage/uploads/ /website/uploads/
This results in 0 files being marked for transfer in my dry run. I've tried various other combinations of --include and --exclude statements, but either continued to get no results or got everything as if no include or exclude options were set.
Anyone have any idea how to do this?
The problem is that --exclude="*" says to exclude (for example) the 1260000000/ directory, so rsync never examines the contents of that directory, so never notices that the directory contains files that would have been matched by your --include.
I think the closest thing to what you want is this:
rsync -nrv --include="*/" --include="file_11*.jpg" --exclude="*" /Storage/uploads/ /website/uploads/
(which will include all directories, and all files matching file_11*.jpg, but no other files), or maybe this:
rsync -nrv --include="/[0-9][0-9][0-9]0000000/" --include="file_11*.jpg" --exclude="*" /Storage/uploads/ /website/uploads/
(same concept, but much pickier about the directories it will include).
rsync include exclude pattern examples:
"*" means everything
"dir1" transfers empty directory [dir1]
"dir*" transfers empty directories like: "dir1", "dir2", "dir3", etc...
"file*" transfers files whose names start with [file]
"dir**" transfers every path that starts with [dir] like "dir1/file.txt", "dir2/bar/ffaa.html", etc...
"dir***" same as above
"dir1/*" does nothing
"dir1/**" does nothing
"dir1/***" transfers [dir1] directory and all its contents like "dir1/file.txt", "dir1/fooo.sh", "dir1/fold/baar.py", etc...
And final note is that simply dont rely on asterisks that are used in the beginning for evaluating paths; like "**dir" (its ok to use them for single folders or files but not paths) and note that more than two asterisks dont work for file names.
Here's my "teach a person to fish" answer:
Rsync's syntax is definitely non-intuitive, but it is worth understanding.
First, use -vvv to see the debug info for rsync.
$ rsync -nr -vvv --include="**/file_11*.jpg" --exclude="*" /Storage/uploads/ /website/uploads/
[sender] hiding directory 1280000000 because of pattern *
[sender] hiding directory 1260000000 because of pattern *
[sender] hiding directory 1270000000 because of pattern *
The key concept here is that rsync applies the include/exclude patterns for each directory recursively. As soon as the first include/exclude is matched, the processing stops.
The first directory it evaluates is /Storage/uploads. Storage/uploads has 1280000000/, 1260000000/, 1270000000/ dirs/files. None of them match file_11*.jpg to include. All of them match * to exclude. So they are excluded, and rsync ends.
The solution is to include all dirs (*/) first. Then the first dir component will be 1260000000/, 1270000000/, 1280000000/ since they match */. The next dir component will be 1260000000/. In 1260000000/, file_11_00.jpg matches --include="file_11*.jpg", so it is included. And so forth.
$ rsync -nrv --include='*/' --include="file_11*.jpg" --exclude="*" /Storage/uploads/ /website/uploads/
./
1260000000/
1260000000/file_11_00.jpg
1260000000/file_11_01.jpg
1270000000/
1270000000/file_11_00.jpg
1270000000/file_11_01.jpg
1280000000/
1280000000/file_11_00.jpg
1280000000/file_11_01.jpg
https://download.samba.org/pub/rsync/rsync.1
Related
I'm relatively new to bash and I have tried multiples solutions that I could find here but none of them seem to be working in my case. It's pretty simple, I have a folder that looks like this:
- images/
- 0_image_1.jpg
- 0_image_2.jpg
- 0_image_3.jpg
- 1_image_1.jpg
- 1_image_2.jpg
- 1_image_3.jpg
and I would like to move these jpg files into subfolders based on the prefix number like so:
- images_0/
- 0_image_1.jpg
- 0_image_2.jpg
- 0_image_3.jpg
- images_1/
- 1_image_1.jpg
- 1_image_2.jpg
- 1_image_3.jpg
Is there a bash command that could do that in a simple way ?
Thank you
for src in *_*.jpg; do
dest=images_${src%%_*}/
echo mkdir -p "$dest"
echo mv -- "$src" "$dest"
done
Remove both echos if the output looks good.
I would do this with rename a.k.a. Perl rename. It is extremely powerful and performant. Here's a command for your use case:
rename --dry-run -p '$_="images_" . substr($_,0,1) . "/" . $_' ?_*jpg
Let's dissect that. At the right end, we specify we only want to work on files that start with a single character/digit before an underscore so we don't do damage trying to apply the command to files it wasn't meant for. Then --dry-run means it doesn't actually do anything, it just shows you what it would do - this is a very useful feature. Then -p which handily means "create any necessary directories for me as you go". Then the meat of the command. It passes you the current filename in a variable called $_ and we then need to create a new variable called $_ to say what we want the file to be called. In this case we just want the word images_ followed by the first digit of the existing filename and then a slash and the original name. Simples!
Sample Output
'0_image_1.jpg' would be renamed to 'images_0/0_image_1.jpg'
'0_image_2.jpg' would be renamed to 'images_0/0_image_2.jpg'
'1_image_3.jpg' would be renamed to 'images_1/1_image_3.jpg'
Remove the --dry-run and run again for real, if the output looks good.
Using rename has several benefits:
that it will warn and avoid any conflicts if two files rename to the same thing,
that it can rename across directories, creating any necessary intermediate directories on the way,
that you can do a dry run first to test it,
that you can use arbitrarily complex Perl code to specify the new name.
Note: On macOS, you can install rename using homebrew:
brew install rename
Note: On some Ones, rename is referred to as prename for Perl rename.
Pulling my hair out here trying to get this to work. Heres an example of the details and command.
I have a file with a list of directories named list.txt The contents look like this:
HYTTCCCXX
HYTVNCCXX
HYV5TCCXX
My rsync command looks like:
rsync -av --recursive --files-from='/tmp/list.txt' /test/apple/ /destination/files/
The issue is that when I run the command, it includes both
/test/ (which is an autofs top level, so contains nothing really) and /test/apple/ in the files to be transferred. Causing the files to be written twice into the destination as if I left the trailing slash off my source.
So the destination ends up with both the directories in the list, and another copy of the source like:
/destination/files/HYW22CCXX
/destination/files/HYTVNCCXX
/destination/files/HYV5TCCXX
/destination/files/test/apple/HYW22CCXX
/destination/files/test/apple/HYW22CCXX
/destination/files/test/apple/HYTVNCCXX
So I end up with 2 copies of everything.
Ive tried every combination of exclude like --exclude='/test/apple/' or --exclude='/test/* or --exclude='apple/* to try and keep it from being included. But nothing works.
Any ideas? Im going bananas trying to figure this out.
Thank you!
This is due to the fact that the --files-from option implies --relative.
Quote from the rsync man page, the section on --files-from:
The --relative (-R) option is implied, which preserves the path information that is specified for each item in the file (use --no-relative or --no-R if you want to turn that off).
Try the following options and see if it helps:
rsync -av --recursive --no-relative --files-from='/tmp/list.txt' /test/apple/ /destination/files/
I'm trying to create a GNU Makefile rule that copies files (found via VPATH) from one directory to another, preserving their directory structure.
There are zillions of ways to do this (starting with cp -r) but it seems that none of them work in the context of make, where the copying is initiated in the target directory.
E.g.
cp ../src/foo.c ../src/bar.c .
All the source files share a common directory (only known at runtime), and this common directory should be stripped away.
E.g.
$ srcdir=../../knurgl
$ cp ${srcdir}/src/foo.c ${srcdir}/src/bar.c .
$ find . -type f
./src/foo.c
./src/bar.c
even though the common directory is known at runtime, it can be arbitrary and even include the current directory . (in which case the operation should be a nop).
This is what i tried:
cp
cp --parent ${srcdir}/src/foo.c ${srcdir}/src/bar.c .
but rightfully this refuses to work when called from the target directory (as it would always copy the files onto themselves).
tar
tar c ${srcdir}/src/foo.c ${srcdir}/src/bar.c | tar x
this strips away any relative directories, but keeps the rest (so I end up with ./knurgl/src/foo.c instead of ./src/foo.c.
The --strip-components option doesn't help me much, as i don't know the depth of ${srcdir}.
Instead of
cp --parent ${srcdir}/src/foo.c ${srcdir}/src/bar.c .
(which doesn't work because it doesn't strip $srcdir) you can write
(wd=$PWD; cd $srcdir; cp --parent src/foo.c src/bar.c $wd)
make has built-in functions for handling strings. To replace old_base_dir with new_base_dir in the variable path, call:
$(path:old_base_dir/%=new_base_dir/%)
You can also let it perform the substitution on a list:
$(foreach path,$(path_list),$(path:old_base_dir/%=new_base_dir/%)
Here, the variable path_list contains multiple files. Note though that this will break if the file names contain spaces.
The manual of GNU make describes many more useful functions.
We have tomcat server located at /opt/tomcat7.0 i want to sync only logs directory to remote server, I am trying following rsync command with exclude * everything and include logs but it doesn't syncing anything.
following are tomcat directories (I only want to sync logs directory)
[rsync#server1]$ ls /opt/tomcat7.0
bin/ conf/ lib/ logs/ temp/ webapps/ work/
here is rsync command
[rsync#logserver]$ rsync -avz --delete --copy-links --include='logs' --exclude='*' server1:/opt/tomcat7.0 /path/to/destination/.
receiving incremental file list
sent 25 bytes received 10 bytes 6.36 bytes/sec
total size is 0 speedup is 0.00
what am i doing wrong?
Any reason not to do:
rsync -avz --delete --copy-links server1:/opt/tomcat7.0/logs /path/to/destination/.
The manpage explains why this does not work and what can be done to make it work:
Note that, when using the --recursive (-r) option (which is implied by -a), every subcomponent of every path
is visited from the top down, so include/exclude patterns get applied recursively to each subcomponent's full
name (e.g. to include "/foo/bar/baz" the subcomponents "/foo" and "/foo/bar" must not be excluded). The
exclude patterns actually short-circuit the directory traversal stage when rsync finds the files to send. If
a pattern excludes a particular parent directory, it can render a deeper include pattern ineffectual because
rsync did not descend through that excluded section of the hierarchy. This is particularly important when
using a trailing '*' rule. For instance, this won't work:
+ /some/path/this-file-will-not-be-found
+ /file-is-included
- *
This fails because the parent directory "some" is excluded by the '*' rule, so rsync never visits any of the
files in the "some" or "some/path" directories. One solution is to ask for all directories in the hierarchy
to be included by using a single rule: "+ */" (put it somewhere before the "- *" rule), and perhaps use the
--prune-empty-dirs option. Another solution is to add specific include rules for all the parent dirs that
need to be visited. For instance, this set of rules works fine:
+ /some/
+ /some/path/
+ /some/path/this-file-is-found
+ /file-also-included
- *
I have following architecture of files in a directory.
Directory
/A/abc.xyz
/B/abc.xyz
/C/abc.xyz
/D/abc.xyz
/E/abc.xyz
I want to execute a program on acb.xyz in each SubDirectory. Save Output files in different directory i.e. Directory/processed with the name of SubDirectory appended in the name of output files.
Can it be written in following way? Need corrections.
for i in `ls "Directory/"`
do
program.pl $i/abc.xyz > processed/$i-abc.xyz
done
for dir in Directory/*; do
program.pl "$dir/abc.xyz" > "processed/${dir##*/}-abc.xyz"
done
The ${dir##*/} part strips the leading directory names from $dir, so Directory/A becomes just A. I added quotes to ensure directory names with whitespace don't cause issue (a good habit, even if you know there are no spaces).
As an alternative to the string munging you could simplify this if you first change directory:
cd Directory
for dir in *; do
program.pl "$dir/abc.xyz" > "processed/$dir-abc.xyz"
done