How do I copy symbolic links only (and not the file it points to) or other files using rsync?
I tried
rsync -uvrl input_dir output_dir
but I need to exclusively copy the symbolic links only ?
any trick using include exclude options?
Per this question+answer, you can script this as a pipe. Pipes are an integral part of shell programming and shell scripting.
find /path/to/files -type l -print | \
rsync -av --files-from=- /path/to/files user#targethost:/path
What's going on here?
The find command starts at /path/to/files and steps recursively through everything "under" that point. The options to find are conditions that limit what gets output by the -print option. In this case, only things of -type l (symbolic link, according to man find) will be printed to find's "standard output".
These files become the "standard input" of the rsync command's --file-from option.
Give it a shot. I haven't actually tested this, but it seems to me that it should work.
You can generate a list of files excluding links with find input_dir -not -type l, rsync has an option --exclude-from=exlude_file.txt
you can do it in two steps :
find input_dir -not -type l > /tmp/rsync-exclude.txt
rsync -uvrl --exclude-from=/tmp/rsync-exclude.txt input_dir output_dir
one line bash :
rsync -urvl --exclude-from=<(find input_dir -not -type l | sed 's/^..//') input_dir output_dir
You can do it more easily like:
find /path/to/dir/ -type l -exec rsync -avP {} ssh_server:/path/to/server/ \;
EDIT:
If you want to copy symbolic links of the current directory only without making it recursive. You can do:
find /path/to/dir/ -maxdepth 1 -type l -exec rsync -avP {} ssh_server:/path/to/server/ \;
I prefer this:
find ./ -type l -print > /tmp/list_of_links.txt
rsync -av --files-from=/tmp/list_of_links.txt /path/to/files user#targethost:/path
The reason is simple. In the previous suggested version, I had to enter my password with every file. This way I can send all symlinks at once, with just one password entered.
Related
I have some automated downloads in a proprietary linux distro.
They go to a temp scratch disk. I want to move them when they're finished to the main RAID array. The best way I can see to do this is to check the folders on the disk to see if the contents have changed in the last minute. If not then its probably finished downloading and then move it.
Assuming there could be hundreds of folders or just one in this location and its all going to the same place. Whats the best way to write this?
I can get a list of folder sizes with
du -h directory/name
The folders can contain multiple files anywhere from 1.5mb to 10GB
Temp Loc: /volume2/4TBScratch/Processing
Dest Loc when complete: /volume1/S/00 Landing
EDIT:
Using this:
find /volume2/4TBScratch/Processing -mindepth 1 -type d -not -mmin +10 -exec mv "{}" "/volume1/S/00 Landing" \;
find: `/volume2/4TBScratch/Processing/test': No such file or directory
4.3#
yet it DOES copy the relevant folders and all files. But the error worries me that something might go wrong in the future.... is it because there is multiple files and it's running the same move command for EACH file or folder in the root folder? But since it moves it all on the first iteration it cant find it on the next ones?
EDIT2:
Using Rsync
4.3# find /volume2/4TBScratch/Processing -mindepth 1 -type d -not -mmin +10 -exec rsync --remove-source-files "{}" "/volume1/S/00 Landing" \;
skipping directory newtest
skipping directory erw
RESOLVED: EDIT3
Resolved with the help in the comments below. Final script looks like this:
find /volume2/4TBScratch/Processing -mindepth 1 -type d -not -mmin +10 -exec rsync -a --remove-source-files "{}" "/volume1/S/00 Landing" \;
find /volume2/4TBScratch/Processing -depth -type d -empty -delete
rsync to move folders and files but leaves empty root dir
the next command finds empty folders and removes them.
Thanks all!
You can use GNU find with options -size for detecting files/folders of certain size and use mv with the -exec option to move to destination directory. The syntax is
find /volume2/4TBScratch/Processing -type d -maxdepth 1 -size -10G -exec mv "{}" "/volume1/S/00 Landing" \;
Using rsync
find /volume2/4TBScratch/Processing -type d -maxdepth 1 -size -10G -exec rsync --remove-source-files "{}" "/volume1/S/00 Landing" \;
The size with a - sign to indicate less than the mentioned size which in this case is 10GB. A note on each of the flags used
-type d -> For identifying only the folders from the source path.
-maxdepth 1 -> To look only on the current source directory and not
being recursive.
-exec -> Execute command following it.
Alternatively, if you want to find files that are last modified over a certain time(minutes), find has an option for -mmin which can be set to a value. E.g. -mmin -5 would return files modified five minutes ago.
So suggest adding it to your requirement, for x as you need and see if the directories are listed, then you can add the -exec option for moving the directories
find /volume2/4TBScratch/Processing -type d -maxdepth 1 -mmin -2 -size -10G
Refer to the GNU documentation for finding files according to size on how this works.
Note:- The double quotes("") are added to avoid Bash from splitting the names containing spaces.
At the moment I recursively remove all softlinks from my current working directory like this:
find . -type l -delete
But I don't want to remove symlinks pointing to a directory anymore.
Is there simple way to customize the find command or do I have to omit the -delete and script something to inspect every found softlink "myself" before removing?
As already suggested in the comments, you can use the test utility for this; but you don't need readlink because test -d always resolves symbolic links.
# replace -print with -exec rm {} +
find . -type l ! -exec test -d {} \; -print
It might be slow due to the overhead from spawning a new process for each symlink though. If that's a problem, you can incorporate a small shell script in your find command to process them in bulks.
find . -type l -exec sh -c '
for link; do
shift
if ! test -d "$link"; then
set "$#" "$link"
fi
done
# remove echo
echo rm "$#"' sh {} +
Or, if you have GNU find installed, you can utilize the -xtype primary.
# replace -print with -delete
find -type l ! -xtype d -print
What I want:
In a bash script: Find all files in current directory that contain a certain string "teststring" and cop them into a subfolder "./testfolder"
Found this to find the filenames which im looking for
find . -type f -print0 | xargs -0 grep -l "teststring"
..and this to copy found files to another folder (here selecting by strings in filename):
find . -type f -iname "stringinfilename" -exec cp {} ./testfolder/ \;
Whats the best way to combine both commands to achieve what I described at the top?
Just let find do both:
find . -name subdir -prune -o -type f -exec \
grep -q teststring "{}" \; -exec cp "{}" subdir \;
Note that things like this are much easier if you don't try to add to the directory you're working in. In other words, write to a sibling dir instead of writing to a subdirectory. If you want to wind up with the data in a subdir, mv it when you're done. That way, you don't have to worry about the prune (ie, you don't have to worry about find descending into the subdir and attempting to duplicate the work).
How to search (using find command) for directories and copy all the files and directory itself to another directory in linux?
Here is what I have so far:
find -type d -name "*.ABC" -exec {} /Desktop/NewFile \;
I get this as output:
find: './GAE/.ABC: PERMISSION DENIED
Please Help, Thanks!
Your error here above has nothing to do with file read permission. You're trying to execute the directories you find! Avoid running commands as root or sudo unless: (1) you really need it and (2) you really know what you're doing. Quite often people asking for root or sudo privileges are exactly the ones should not have it.
That said... there are several ways to copy a directory tree under *nix. This is just one possible approach:
$ find <start> -type d -name \*.ABC -exec cp -av {} <target> \;
Where:
<start> is a directory name. It's used to tell find where to start its search (for example /usr/local or $HOME)
<target> is another directory name to define the final destination of your copied directories
UPDATE
In case you want to search for multiple paths...
$ find <start> -type d \( -name \*.ABC -o -name \*.DEF \) -exec cp -av {} <target> \;
This should work:
find ./source_dir -name \*.png -print0 | xargs -0 cp -t path/to/destination
For more info, you can look up here.
I want to copy the recently updated multiple file into another directory.
I am having 1.xml,2.xml,3.xml.... in this directory recently someone updated file or added new file into the directory,So i want to copy those files into the destination directory ..Its like synchronization of 2 directories.
For that I have tried below commend
find home/deployment/server/services/ -type f -mtime 1 | xargs cp /home/application/
and below one also
find home/deployment/server/services/ -type f -mtime 1 -exec cp /home/application/
I am not getting any file into destination after updating 1.xml file,So I have added new file 4.xml even that also not updating in destination directory.
How to process recently updated or newly added multiple files.
Thanks in advance.
Short answer:
use xargs to mv the "find" directory into another directory
Long answer: As I recall (not tested) for exec syntax is
find . -type f --mtime 1 -exec cp {} /destination/path/ +
"{}" is an argument which came from command "find"
For xargs
find . -type f --mtime 1 | xargs -0 -I {} cp {} /destination/path/
I do this often but use \; instead of + and usually -cnewer rather than -mtime.
\; executes the cp command on files individually instead of as a group.
+ executes as a group with as many paths as xterm will take. It may do this multiple time if there are a lot of files.
the \ in front of the ; option is required or bash will think it is the end of the command.
find ./ -mtime -1 -exec cp {} /path/ \; -print
Use the -print at the end to get a list of the files that were copied.