So, I have a list of files in a text file. I believe it's about 100,000 files.
The files in said list are spread across many directories, have different sizes, filenames, extensions, ages, etc.
I am trying to find a way to move those files, and just those, to another drive.
Complicating factor: some of the files have the same name, but are not the same file. They can't just be moved into one folder with an overwriting or ignoring policy towards multiples.
Preferably, I would like them to retain their directory structure, but only have the files that I want inside the destination directory. (the destination drive isn't big enough to simply copy everything).
Below is an example of some lines in the file:
media/dave/xdd/cruzer/F#(NTFS 1)/Raw Files/Portable Network Graphic file/3601-3900/FILE3776.PNG/Windows/winsxs/amd64_microsoft-windows-o..disc-style-memories_31bf3856ad364e35_6.1.7600.16385_none_51190840a935f980/Title_mainImage-mask.png
media/dave/xdd/d1/other/hd1/Program Files/DVD Maker/Shared/DvdStyles/Memories/Title_content-background.png
I have tried to use
rsync -a --files-from=/sourcefile.txt / /media/destinationhdd
However, this just tries to copy my root directory to the destination. Please help, how to I just copy the accursed files that I want to?
cat list | xargs tar cf - | (cd dest; tar xvfp -)
Where list is the file which contains all the file paths.
dest is the target directory
Related
I know there has been a huge discussion about this but I have not found something this specific.
Im trying to copy all .key files in /home// directory
This does not work
/usr/bin/rsync -auPA --include="*/*.key" --exclude="*" /home/* /tmp/test
This works but it copies over unwanted empty directories like /home/uname/Documents
/usr/bin/rsync -auPA --include="*/" --include="*.key" --exclude="*" /home /tmp/test
Basically what i need for rsync to do is to copy only files with .key extension and only create necessarily folders that contain .key files
I think you are looking for the -m option. From the man page:
-m, --prune-empty-dirs
This option tells the receiving rsync to get rid of empty directories from the file-list, including nested directories that
have no non-directory children. This is useful for avoiding the creation of a bunch of useless directories when the sending
rsync is recursively scanning a hierarchy of files using include/exclude/filter rules.
Note that the use of transfer rules, such as the --min-size option, does not affect what goes into the file list, and thus
does not leave directories empty, even if none of the files in a directory match the transfer rule.
Because the file-list is actually being pruned, this option also affects what directories get deleted when a delete is active.
However, keep in mind that excluded files and directories can prevent existing items from being deleted due to an exclude both
hiding source files and protecting destination files. See the perishable filter-rule option for how to avoid this.
You can prevent the pruning of certain empty directories from the file-list by using a global "protect" filter. For instance,
this option would ensure that the directory "emptydir" was kept in the file-list:
--filter ’protect emptydir/’
Here’s an example that copies all .pdf files in a hierarchy, only creating the necessary destination directories to hold the
.pdf files, and ensures that any superfluous files and directories in the destination are removed (note the hide filter of
non-directories being used instead of an exclude):
rsync -avm --del --include=’*.pdf’ -f ’hide,! */’ src/ dest
If you didn’t want to remove superfluous destination files, the more time-honored options of "--include='*/' --exclude='*'"
would work fine in place of the hide-filter (if that is more natural to you).
If I copy files from one directory to another directory:
Will their inode numbers also change?
Changes in file of one directory, will it reflect in same file of another directory also?
When I use command like:
cp -r dir1/ dir2/
With a simple copy the file system handle the copied files as newly created ones, therefore assining new inodes to them.
Any change made in the origin wouldn't change the copys. This only happens when you create symbolic or hard links between files.
You can check the inodes of your files with "ls -i filename".
I used the command mv to move files from directory /a/b to directory /v/c. I wanted the whole 'b' directory to be moved to the path /v/c.
Now while running this command- mv /a/b /v/c I interrupted it in middle where the source had a large amount of data. Later I deleted directory 'c' since I thought it had partial files.
Now my question is will the directory 'b' contain all the original files along with the files that where moved to path /v/c? Or did I lose files by deleting the directory 'c'?
mv across filesystems will:
create the destination directory
for each file: copy and remove original
remove origin directory
Thus, if you interrupt it, some of the files will have been moved but not all. A mv of a directory within the same filesystem is atomic as it's just re-linking the directory's inode to a new location.
At one time, mv could only do the latter.
I believe it depends on if the source and destination directories were on the same file system or different file systems. If they were on the same file system then a "move" just changes the path information for each file. But if they're on different file systems the "move" command will copy one file at a time, and subsequently delete it on the source.
So, in your scenario if the source and destination were on separate file systems then yes, you just lost files if you interrupted mv and then deleted "c".
I have two directory structures on a USB drive that have various files that are the same, and each have files that the other one doesn't.
What I want to do is to move over directory structure B to A. It is ok that B's content is gone after the merge. Directories in A must not be erased, because otherwise I lose A's content. The mv command won't work I think, because it will complain that it can't move a directory because a destination directory in the same place is not empty. mv B/* A/ won't work either because some sub directory will also not be empty.
cp -a B/* A/ is bad (even with -u), because it will take way too long, because the files are on a USB drive, and there possibly too many of them, making the drive run out of capacity.
rsync has the same problem, because it doesn't appear to have a move/rename feature, and it can only move files by copying them.
So either, I'm going to have to write a script that will recursively run through B, and create missing directories and move missing files to A.
But I'm hoping that there is a command or option or utility that I don't know about.
I believe cpio has the capabilities you are wanting. This command:
cd B
find . -type f -print0 | cpio -0dumpl A/.
Will find all files in B, pass them to cpio with null termination to properly handle odd file names, create necessary directories (cpio -d), preserve ownership, permissions and timestamps (-m), and use linking to create the destination files where possible (-l) unconditionally (-u).
Using rsync, how can I rename files when copying with the --files-from argument? I have about 190,000 files, each of which need to be renamed when copying from source to destination. I plan to have the list of files in a text file to pass to the --files-from argument.
Not entirely true... you CAN rename files enroute with rsync, but only if you rsync one file at a time, and set the --no-R --no-implied-dirs options, then explicitly set the destination name in the destination path.
But at that point, you may just want to use some other tool.
This, for example, would work:
rsync --no-R --no-implied-dirs
1.2.3.4::module/$FILENAME
/$PATH/$TOFILE/$NEWFILENAME
There is no way to arbitrarily rename files with rsync. All rsync can do is move files to a different directory.
You must use a second tool either on the sending or receiving side to rename the files.