I'm trying to copy all XML files whose name start with foo
rsync /source/dir/foo*.xml /dest/dir
If there aren't any files matching this pattern rsync throws error:
rsync: link_stat "/source/dir/foo*.xml" failed: No such file or directory (2)
Should I care about this error? Is there a way to suppress it? If there's at least one file matching the pattern then the command runs without errors.
There is a bash setting to avoid this:
shopt -s failglob
From the man page:
failglob
If set, patterns which fail to match filenames during pathname expansion result in an expansion error.
Otherwise, you can use an if to just not run the rsync when there's nothing to do.
If you really do want to sync nothing, so that it deletes files that don't exist, then the command might be like this:
rsync --include 'foo*.xml' \
--exclude '**' \
--delete \
/source/dir/ /dest/dir
The trailing slash is significant.
Related
I would like to ask some help regarding my script below:
#!/bin/bash
year=$(date +%Y)
lastyear=$((year-1))
month=$(date +%m)
log="$lastyear$month"
mkdir -p "/root/temp/$lastyear"
mkdir -p "/root/temp/$lastyear/$month"
cd /root
mv -f "*$log*" "/root/temp/$lastyear/$month"
Error Prompt is
mv: cannot stat `*201602*': No such file or directory
My target was to move all the files that has the specific "201602" string in file name to a specific location.
Sample logs files is OUTXXX-201602XXX, INXXX-201602XXX, 201602XXXX.
This will be implemented through crontab, because there are about 500k+ log files to be transferred and using find will receive a too much argument error T_T.
Any suggestions will help!
You have to put globbing characters outside of quotes for them to be interpreted.
Try instead:
mv -f *"$log"* …
The shell will still keep the resulting matches as one word, so this is OK even if the filenames have spaces.
I'm writing a very simple script, that will interact with my git repository, but I've reached a point, where I can't figure out why the following is happening.
Having:
destPath='~/Dropbox/Shared/Alex\&Stuff'
destFile='file.zip'
#git archive --format zip --output $destFile master
echo $destPath/$destFile
rm $destPath/$destFile
The echo outputs the correct path:
~/Dropbox/Shared/Alex\&Stuff/file.zip
But the rm fails with the following:
rm: cannot remove ‘~/Dropbox/Shared/Alex\\&Stuff/file.zip’: No such file or directory
So, why the extra backslash is added when rm is executed? Alex\\$Stuff instead of Alex\$Stuff ?
Tilde character needs to be outside quote to be expnded:
destPath=~/Dropbox/Shared/Alex\&Stuff
destFile='file.zip'
#git archive --format zip --output $destFile master
echo "$destPath/$destFile"
rm "$destPath/$destFile"
Try
destPath="$HOME/Dropbox/Shared/Alex\&Stuff"
Tildas do not always expand to $HOME. And the ampersand doesn't need the backslash, unless you want an actual backslash there.
As for the double backslash, I would guess that that's how rm quotes its internal strings (i.e., backslashes have special meaning and a single backslash needs to be written as '\'--C does it this way, for example)
I'm attempting to sync all files from within a large directory structure into a single root directory (ie not creating the sub directories but still including all recursive files).
Environment:
Ubuntu 12.04 x86
RSYNC version 3.0.9
GNU bash version 4.2.25(1)
So far I have this command called from a bash script which works fine and provides the basic core functionality required:
shopt -s globstar
rsync -adv /path/to/source/**/. /path/to/dest/. --exclude-from=/myexcludefile
The contents of myexcludefile are:
filename
*/
# the */ prevents all of the directories appearing in /path/to/dest/
# other failed attempts have included:
directory1
directory1/
directory1/*
I now need to exclude files that are located inside certain directories in the source tree. However due to the globstar approach of looking in all directories rsync is unable to match directories to exclude. In other words, with the exception of my /* and filename rules, everything else is completely ignored.
So I'm looking for some assistance on either the excludes syntax or if there's another way of achieving the rsync of many directories into a single destination directory that doesn't use my globstar approach.
Any help or advice would be very gratefully received.
If you want to exclude directories from a globstar match, you can save those to an array, then filter the contents of that array based on a file.
Example:
#!/bin/bash
shopt -s globstar
declare -A X
readarray -t XLIST < exclude_file.txt
for A in "${XLIST[#]}"; do
X[$A]=.
done
DIRS=(/path/to/source/**/.)
for I in "${!DIRS[#]}"; do
D=${DIRS[I]}
[[ -n ${X[$D]} ]] && unset 'DIRS[I]'
done
rsync -adv "${DIRS[#]}" /path/to/dest/.
Run with:
bash script.sh
Note that values in exclude_file.txt should really match expanded values in /path/to/source/**/..
I use the following bash script to copy only files of certain extension(in this case *.sh), however it still copies over all the files. what's wrong?
from=$1
to=$2
rsync -zarv --include="*.sh" $from $to
I think --include is used to include a subset of files that are otherwise excluded by --exclude, rather than including only those files.
In other words: you have to think about include meaning don't exclude.
Try instead:
rsync -zarv --include "*/" --exclude="*" --include="*.sh" "$from" "$to"
For rsync version 3.0.6 or higher, the order needs to be modified as follows (see comments):
rsync -zarv --include="*/" --include="*.sh" --exclude="*" "$from" "$to"
Adding the -m flag will avoid creating empty directory structures in the destination. Tested in version 3.1.2.
So if we only want *.sh files we have to exclude all files --exclude="*", include all directories --include="*/" and include all *.sh files --include="*.sh".
You can find some good examples in the section Include/Exclude Pattern Rules of the man page
The answer by #chepner will copy all the sub-directories whether it contains files or not. If you need to exclude the sub-directories that don't contain the file and still retain the directory structure, use
rsync -zarv --prune-empty-dirs --include "*/" --include="*.sh" --exclude="*" "$from" "$to"
Here's the important part from the man page:
As the list of files/directories to transfer is built, rsync checks each name to be transferred against the list of include/exclude patterns in turn, and the first matching pattern is acted on: if it is an exclude pattern, then that file is skipped; if it is an include pattern then that filename is not skipped; if no matching pattern is found, then the filename is not skipped.
To summarize:
Not matching any pattern means a file will be copied!
The algorithm quits once any pattern matches
Also, something ending with a slash is matching directories (like find -type d would).
Let's pull apart this answer from above.
rsync -zarv --prune-empty-dirs --include "*/" --include="*.sh" --exclude="*" "$from" "$to"
Don't skip any directories
Don't skip any .sh files
Skip everything
(Implicitly, don't skip anything, but the rule above prevents the default rule from ever happening.)
Finally, the --prune-empty-directories keeps the first rule from making empty directories all over the place.
One more addition: if you need to sync files by its extensions in one dir only (without of recursion) you should use a construction like this:
rsync -auzv --include './' --include '*.ext' --exclude '*' /source/dir/ /destination/dir/
Pay your attention to the dot in the first --include. --no-r does not work in this construction.
EDIT:
Thanks to gbyte.co for the valuable comment!
EDIT:
The -uzv flags are not related to this question directly, but I included them because I use them usually.
Wrote this handy function and put in my bash scripts or ~/.bash_aliases. Tested sync'ing locally on Linux with bash and awk installed. It works
selrsync(){
# selective rsync to sync only certain filetypes;
# based on: https://stackoverflow.com/a/11111793/588867
# Example: selrsync 'tsv,csv' ./source ./target --dry-run
types="$1"; shift; #accepts comma separated list of types. Must be the first argument.
includes=$(echo $types| awk -F',' \
'BEGIN{OFS=" ";}
{
for (i = 1; i <= NF; i++ ) { if (length($i) > 0) $i="--include=*."$i; } print
}')
restargs="$#"
echo Command: rsync -avz --prune-empty-dirs --include="*/" $includes --exclude="*" "$restargs"
eval rsync -avz --prune-empty-dirs --include="*/" "$includes" --exclude="*" $restargs
}
Advantages:
short handy and extensible when one wants to add more arguments (i.e. --dry-run).
Example:
selrsync 'tsv,csv' ./source ./target --dry-run
If someone looks for this…
I wanted to rsync only specific files and folders and managed to do it with this command: rsync --include-from=rsync-files
With rsync-files:
my-dir/
my-file.txt
- /*
Source directory is determined like so:
SHOW=${PWD##*/}
SRC=wells#server.com:"/mnt/bigfish/video/TV/${SHOW}/"
So it comes out something like:
wells#server.com:/mnt/bigfish/video/TV/The Name Of the Show With Spaces/
Then trying to run rsync like so:
rsync -avz -e ssh "${SRC}" .
But it tells me that ""/mnt/bigfish/video/TV/The" is not a directory, ""/mnt/bigfish/video/TV/Name" is not a directory, etc, for however many space-delimited words are in the name of the source directory.
How can I rectify this egregiously annoying issue?
UPDATE I'm running this on OS 10.6, and I ended up string-replacing spaces with escaped spaces like so:
SRC=wells#kittenfactory.com:"/mnt/bigfish/video/TV/${SHOW// /\ }/"
From the rsync manual:
-s, --protect-args
This option sends all filenames and most options to the
remote
rsync without allowing the remote shell to interpret them.
This
means that spaces are not split in names, and any
non-wildcard
special characters are not translated (such as ~, $, ;, &,
etc.). Wildcards are expanded on the remote host by
rsync
(instead of the shell doing it).
As your question is dedicated to OS X, according to the Apple rsync manual you can accomplish this using either simple quotes or the wildcard ?:
rsync -av host:'file\ name\ with\ spaces' /dest
rsync -av host:file?name?with?spaces /dest
Just had to do this and using the simple quotes works perfectly:
rsync -r --partial --progress --exclude=".cvs" --exclude=".svn" --exclude=".git" --rsh=ssh root#datakeep.local:'/volume1/devel/__To\ SORT/___XXXXX\ Saves\ 2011-04' ./Saves2011
This works:
rsync -avz -e ssh "wells#server.com:\"/mnt/bigfish/video/TV/${SHOW}/\""
So set:
SRC=wells#server.com:\"/mnt/bigfish/video/TV/${SHOW}/\"
At least, here on Debian it works like a charm, no OS 10 available to test with here.
You can do this on OSX if you're dealing with arguments in a script:
ESCAPED_SRC="$(echo "$SRC" | tr ' ' '\\ ')"
ESCAPED_DEST="$(echo "$DEST" | tr ' ' '\\ ')"
rsync -ravP "$ESCAPED_SRC" "$ESCAPED_DEST"