I want a simple and working (multiple) exclude option inside my rsync command. Lets say i will exclude a file and a directory:
/var/www/html/test.txt
/var/www/html/images/
What i did is:
rsync -avz --exclude="/var/www/html/test.txt" --exclude="/var/www/html/images/" /var/www/html root#xx.xx.xx.xx:/var/www
or
rsync -avz --exclude=/var/www/html/test.txt --exclude=/var/www/html/images/ /var/www/html root#xx.xx.xx.xx:/var/www
or
rsync -avz --exclude /var/www/html/test.txt --exclude /var/www/html/images/ /var/www/html root#xx.xx.xx.xx:/var/www
..
But however, the --exclude is NOT WORKING!
Everything is going out!
How to do it in this simple format please?
Note: I also don't want to use external exclusion list file. Just want all in one simple command.
i got it solved by myself after i've learned and tested many times. The real problem was the understandable (for me) --exclude option usage format.
I don't know how others are doing but i just found out that:
"--exclude" path CAN NOT be the full absolute path!
Because i was using the path(s) like: --exclude /var/www/html/text.txt which caused the thing DOES NOT work. So i used like:
--exclude text.txt --exclude images/
.. and it WORKS!
I personnaly like the --exclude={text.txt,images/} format.
Remember that with rsync, all exclude (or include) paths beginning with / are are anchored to the root of transfer which in your example will be the /var/www/html directory!!, so if you specify /text.txt it will be only the file which is # the root of your transfer directory not above in the tree. You can find more infos and examples here
Related
Currently i only RSync-ing the Directories as like:
* * * * * rsync -avz /var/www/public_html/images root#<remote-ip>:/var/www/public_html
So how do i rsync one single file like, /var/www/public_html/.htaccess ?
You do it the same way as you would a directory, but you specify the full path to the filename as the source. In your example:
rsync -avz --status=progress /var/www/public_html/.htaccess root#<remote-ip>:/var/www/public_html/
As mentioned in the comments: since -a includes recurse, one little typo can make it kick off a full directory tree transfer, so a more fool-proof approach might to just use -vz, or replace it with -lptgoD.
Basic syntax
rsync options source destination
Example
rsync -az /var/www/public_html/filename root#<remote-ip>:/var/www/public_html
Read more
Michael Place's answer works great if, relative to the root directory for both the source and target, all of the directories in the file's path already exist.
But what if you want to sync a file with this source path:
/source-root/a/b/file
to a file with the following target path:
/target-root/a/b/file
and the directories a and b don't exist?
You need to run an rsync command like the following:
rsync -r --include="/a/" --include="/a/b/" --include="/a/b/file" --exclude="*" [source] [target]
To date, two of the answers aren't quite right, they'll get more than one file, and the other isn't as simple as it could be, here's a simpler answer IMO.
The following gets exactly one file, but you have to create the dest directory with mkdir. This is probably the fastest option:
mkdir -p ./local/path/to/file
rsync user#remote:/remote/path/to/file/ -zarv --include "filename" --exclude "*" ./local/path/to/file/
If there is only one instance of file in /remote/path, rsync can create directories for you if you do the following. This will probably take a little more time because it searches more directories. Plus it's will create empty directories for directories in /remote/path that are not in ./local
cd ./local
rsync user#remote:/remote/path -zarv --include "*/" --include "filename" --exclude "*" .
Keep in mind that the order of --include and --exclude matters.
Aside from the good above answers, rsync expects the destination to be a directory and not a filename. Suppose you are copying the word list file words to /tmp, don't do this:
rsync -az /user/share/dict/words /tmp/words # does not work
'cp' is tolerant of this form, but rsync isn't - it will fail because it doesn't see a directory at /tmp/words. Snip off the destination filename and it works:
rsync -az /user/share/dict/words /tmp
Note that rsync won't let you change the filename during the copy, and cp will.
I want to scp several files from remote to local, the files in remote is like this:
/data/1792348/a.stat
/data/1792348/b.stat
/data/187657/a.stat
/data/187657/b.stat
... ...
1792348 187657 etc, the middle directory name is random.
how can i scp all the files ends with .stat from remote to local?
if i tried scp -P36000 user#host:/data//*.stat .*, i can only get 2 files a.stat b.stat.
why i can's submit this question?
i really don't know how to solve this, and hadn't search a answer from google.
i would use rsync (which uses scp internally; but is way more elaborate, e.g. it will only transmit minimal changesets of data, so if you run it several times, you will get an impressive speedup)
rsync -avz /data/ \
--include "*/" --include "*.stat" --exclude "*" \
user#host:/path/to/dest/data/
I am trying to rsync directory A of server1 with directory B of server2.
Sitting in the directory A of server1, I ran the following commands.
rsync -av * server2::sharename/B
but the interesting thing is, it synchronizes all files and directories except .htaccess or any hidden file in the directory A. Any hidden files within subdirectories get synchronized.
I also tried the following command:
rsync -av --include=".htaccess" * server2::sharename/B
but the results are the same.
Any ideas why hidden files of A directory are not getting synchronized and how to fix it. I am running as root user.
thanks
This is due to the fact that * is by default expanded to all files in the current working directory except the files whose name starts with a dot. Thus, rsync never receives these files as arguments.
You can pass . denoting current working directory to rsync:
rsync -av . server2::sharename/B
This way rsync will look for files to transfer in the current working directory as opposed to looking for them in what * expands to.
Alternatively, you can use the following command to make * expand to all files including those which start with a dot:
shopt -s dotglob
See also shopt manpage.
For anyone who's just trying to sync directories between servers (including all hidden files) -- e.g., syncing somedirA on source-server to somedirB on a destination server -- try this:
rsync -avz -e ssh --progress user#source-server:/somedirA/ somedirB/
Note the slashes at the end of both paths. Any other syntax may lead to unexpected results!
Also, for me its easiest to perform rsync commands from the destination server, because it's easier to make sure I've got proper write access (i.e., I might need to add sudo to the command above).
Probably goes without saying, but obviously your remote user also needs read access to somedirA on your source server. :)
I had the same issue.
For me when I did the following command the hidden files did not get rsync'ed
rsync -av /home/user1 server02:/home/user1
But when I added the slashes at the end of the paths, the hidden files were rsync'ed.
rsync -av /home/user1/ server02:/home/user1/
Note the slashes at the end of the paths, as Brian Lacy said the slashes are the key. I don't have the reputation to comment on his post or I would have done that.
I think the problem is due to shell wildcard expansion. Use . instead of star.
Consider the following example directory content
$ ls -a .
. .. .htaccess a.html z.js
The shell's wildcard expansion translates the argument list that the rsync program gets from
-av * server2::sharename/B
into
-av a.html z.js server2::sharename/B
before the command starts getting executed.
The * tell to rsynch to not synch hidden files. You should not omit it.
On a related note, in case any are coming in from google etc trying to find while rsync is not copying hidden subfolders, I found one additional reason why this can happen and figured I'd pay it forward for the next guy running into the same thing: if you are using the -C option (obviously the --exclude would do it too but I figure that one's a bit easier to spot).
In my case, I had a script that was copying several folders across computers, including a directory with several git projects and I noticed that the I couldn't run any of the normal git commands in the copied repos (yes, normally one should use git clone but this was part of a larger backup that included other things). After looking at the script, I found that it was calling rsync with 7 or 8 options.
After googling didn't turn up any obvious answers, I started going through the switches one by one. After dropping the -C option, it worked correctly. In the case of the script, the -C flag appears to have been added as a mistake, likely because sftp was originally used and -C is a compression-related option under that tool.
per man rsync, the option is described as
--cvs-exclude, -C auto-ignore files in the same way CVS does
Since CVS is an older version control system, and given the man page description, it makes perfect sense that it would behave this way.
I want to create script that copy my project and make it zip archive. I want to exclude all folder named .svn in all sub directories. Any suggestion?
I'd use rsync's FILTER RULES for this:
Create an .rsync-filter file (in the origin directory) containing, e.g.
-.svn/
Now run rsync like an exalted copy:
rsync -aFF origin/ destination/
You can do this using rsync. Although this is designed to synchronise directories across servers, it can also be used to copy directories on a single machine.
rsync has a --exclude option to exclude files and directories by pattern. See http://www.samba.org/ftp/rsync/rsync.html for help and examples.
Just call the zip utility on your project’s folder and use the -r option for recursive plus the -x option to exclude files / folders by pattern.
zip -r target-filename.zip source-folder -x \*exclude-pattern\*
exclude-pattern in your case would be .svn
See also man zip
Having an issues with rsync. I'm using rsync as a glorified cp command. I have in a script the following code.
rsync -aL --exclude /path/to/exclude/ --exclude='.*' /source/ /destination
I can get the rsync to exclude any hidden files. Hence the '.*' I cannot get the exclude dir to exclude. I've tried using an '=' sign, surrounding the dir with double quotes, with single quotes. Any help would be greatly appreciated. Thanks in advance.
Actually, neither Erik's nor Antoni's answer is fully accurate.
Erik is halfway right in saying that
As test/a is the base directory synced from, the exclude pattern is specified by starting with a/
It is true that the exclude pattern's root is test/a (i.e. the pattern /some/path binds to test/a/some/path), but that's not the whole story.
From the man page:
if the pattern starts with a / then it is anchored to a particular spot in the hierarchy of files, otherwise it is matched against the end of the pathname. This is similar to a leading ^ in regular expressions. Thus "/foo" would match a file named "foo" at either the
"root of the transfer" (for a global rule) or in the merge-file's directory (for a per-directory rule).
We can ignore the per-directory bit as it doesn't apply to us here.
Therefore, rsync -nvraL test/a test/dest --exclude=a/b/c/d will most definitely exclude test/a/b/c/d (and children), but it'll also exclude test/a/other/place/a/b/c/d.
rsync -nvraL test/a test/dest --exclude=/b/c/d, on the other hand, will exclude only test/a/b/c/d (and children) (test/a being the point to which / is anchored).
This is why you still need the anchoring inital slash if you want to exclude that specific path from being backed up. This might seem like a minor detail, and it will be so the more specific your exclude pattern becomes (e.g. Pictures vs. home/daniel/Pictures) but it might just come around to bite you in the butt.
mkdir -p test/a/b/c/d/e
mkdir -p test/dest
rsync -nvraL test/a test/dest --exclude=a/b/c/d
This works. As test/a is the base directory synced from, the exclude pattern is specified by starting with a/
Show us the real paths/excludes if this doesn't help.
Running rsync with -vn will list dirs/files - the pattern is matched against the format that rsync prints.
Following Erik's example you want to do this:
rsync -nvraL test/a/ test/dest --exclude=/b/c/d
Either of these would work:
rsync -nvraL test/a test/dest --exclude=/a/b/c/d
rsync -nvraL test/a/ test/dest --exclude=/b/c/d
Note the ending / in source path makes a critical difference to how --exclude should be specified. This assumes we have:
mkdir -p test/a/b/c/d/e
mkdir -p test/a/other/place/a/b/c/d
mkdir -p test/dest
Original Post has difficulty when exclude path starts with a /. Daniel’s answer is correct that this initial / in exclude path might be desirable to exclude a specific path, and that this initial / should be understood like leading ^ in regular expressions. However, his answer has a critical typo about the ending / in source path.
If you want to specify absolute paths you can convert them to relative paths using realpath:
--exclude="$(realpath --relative-to=$PWD /home/file)"