Pulling my hair out here trying to get this to work. Heres an example of the details and command.
I have a file with a list of directories named list.txt The contents look like this:
HYTTCCCXX
HYTVNCCXX
HYV5TCCXX
My rsync command looks like:
rsync -av --recursive --files-from='/tmp/list.txt' /test/apple/ /destination/files/
The issue is that when I run the command, it includes both
/test/ (which is an autofs top level, so contains nothing really) and /test/apple/ in the files to be transferred. Causing the files to be written twice into the destination as if I left the trailing slash off my source.
So the destination ends up with both the directories in the list, and another copy of the source like:
/destination/files/HYW22CCXX
/destination/files/HYTVNCCXX
/destination/files/HYV5TCCXX
/destination/files/test/apple/HYW22CCXX
/destination/files/test/apple/HYW22CCXX
/destination/files/test/apple/HYTVNCCXX
So I end up with 2 copies of everything.
Ive tried every combination of exclude like --exclude='/test/apple/' or --exclude='/test/* or --exclude='apple/* to try and keep it from being included. But nothing works.
Any ideas? Im going bananas trying to figure this out.
Thank you!
This is due to the fact that the --files-from option implies --relative.
Quote from the rsync man page, the section on --files-from:
The --relative (-R) option is implied, which preserves the path information that is specified for each item in the file (use --no-relative or --no-R if you want to turn that off).
Try the following options and see if it helps:
rsync -av --recursive --no-relative --files-from='/tmp/list.txt' /test/apple/ /destination/files/
Related
I'm relatively new to bash and I have tried multiples solutions that I could find here but none of them seem to be working in my case. It's pretty simple, I have a folder that looks like this:
- images/
- 0_image_1.jpg
- 0_image_2.jpg
- 0_image_3.jpg
- 1_image_1.jpg
- 1_image_2.jpg
- 1_image_3.jpg
and I would like to move these jpg files into subfolders based on the prefix number like so:
- images_0/
- 0_image_1.jpg
- 0_image_2.jpg
- 0_image_3.jpg
- images_1/
- 1_image_1.jpg
- 1_image_2.jpg
- 1_image_3.jpg
Is there a bash command that could do that in a simple way ?
Thank you
for src in *_*.jpg; do
dest=images_${src%%_*}/
echo mkdir -p "$dest"
echo mv -- "$src" "$dest"
done
Remove both echos if the output looks good.
I would do this with rename a.k.a. Perl rename. It is extremely powerful and performant. Here's a command for your use case:
rename --dry-run -p '$_="images_" . substr($_,0,1) . "/" . $_' ?_*jpg
Let's dissect that. At the right end, we specify we only want to work on files that start with a single character/digit before an underscore so we don't do damage trying to apply the command to files it wasn't meant for. Then --dry-run means it doesn't actually do anything, it just shows you what it would do - this is a very useful feature. Then -p which handily means "create any necessary directories for me as you go". Then the meat of the command. It passes you the current filename in a variable called $_ and we then need to create a new variable called $_ to say what we want the file to be called. In this case we just want the word images_ followed by the first digit of the existing filename and then a slash and the original name. Simples!
Sample Output
'0_image_1.jpg' would be renamed to 'images_0/0_image_1.jpg'
'0_image_2.jpg' would be renamed to 'images_0/0_image_2.jpg'
'1_image_3.jpg' would be renamed to 'images_1/1_image_3.jpg'
Remove the --dry-run and run again for real, if the output looks good.
Using rename has several benefits:
that it will warn and avoid any conflicts if two files rename to the same thing,
that it can rename across directories, creating any necessary intermediate directories on the way,
that you can do a dry run first to test it,
that you can use arbitrarily complex Perl code to specify the new name.
Note: On macOS, you can install rename using homebrew:
brew install rename
Note: On some Ones, rename is referred to as prename for Perl rename.
I'm looking for a way to flatten directories from a /year/month/day/directory format to just /directory via rsync
The source directories, containing data files, are formatted like this:
/year/month/day/round-number/files
Source:
/2018/06/01/round-1111/(files)
/2018/06/01/round-1112/(etc)
/2018/06/01/round-1113
/2018/06/02/round-1114
/2018/06/02/round-1115
/2018/06/02/round-1116
/2018/06/03/round-1117
/2018/06/03/round-1118
/2018/06/03/round-1119
I need them to come out like this at the Destination:
/round-1111/(files)
/round-1112/(etc)
/round-1113
/round-1114
/round-1115
/round-1116
/round-1117
/round-1118
/round-1119
The command I'm using right now is basically "rsync -a source destination"
I'd like to keep the processing load of this command low, as it needs to run frequently while also not disturbing the source server too much.
I have been trying to change the suffix on my backup files using the --suffix function but I'm not quite sure how to do it. Currently this line of code
find ./$1 -name "IMG_****.JPG" -exec cp --backup=t {} ./$2 \;
searches the first command line argument directory for images in the IMG_****.JPG format and copies them to the directory entered second, making copies of any files with duplicate names and adding the =t suffix to the end giving IMG_****.JPG.~1~ etc. Instead of .~1~ I would like to add something like .JPG, any ideas on how to use the --suffix to do this?
Read the man page:
The backup suffix is '~', unless set with --suffix or SIMPLE_BACKUP_SUFFIX.
It should be pretty obvious from this sentence that supplying --suffix is equivalent to setting SIMPLE_BACKUP_SUFFIX, which as its name suggests only applies to simple backups (i.e., --backup=simple or --backup=never). E.g.,
> touch src dst
> cp --backup=simple --suffix=.bak src dst
> ls src* dst*
dst dst.bak src
However, you are requesting numbered backups through --backup=t, so the suffixes you will get will always be .~1~, .~2~, etc., unaffected by --suffix.
I need to rename hundreds of files in Linux to change the unique identifier of each from the command line. For sake of examples, I have a file containing:
old_name1 new_name1
old_name2 new_name2
and need to change the names from new to old IDs. The file names contain the IDs, but have extra characters as well. My plan is therefore to end up with:
abcd_old_name1_1234.txt ==> abcd_new_name1_1234.txt
abcd_old_name2_1234.txt ==> abcd_new_name2_1234.txt
Use of rename is obviously fairly helpful here, but I am struggling to work out how to iterate through the file of the desired name changes and pass this as input into rename?
Edit: To clarify, I am looking to make hundreds of different rename commands, the different changes that need to be made are listed in a text file.
Apologies if this is already answered, I've has a good hunt, but can't find a similar case.
rename 's/^(abcd_)old_name(\d+_1234\.txt)$/$1new_name$2/' *.txt
Should work, depending on whether you have that package installed. Also have a look at qmv (rename-utils)
If you want more options, use e.g.
shopt -s globstart
rename 's/^(abcd_)old_name(\d+_1234\.txt)$/$1new_name$2/' folder/**/*.txt
(finds all txt files in subdirectories of folder), or
find folder -type f -iname '*.txt' -exec rename 's/^(abcd_)old_name(\d+_1234\.txt)$/$1new_name$2/' {} \+
To do then same using GNU find
while read -r old_name new_name; do
rename "s/$old_name/$new_name/" *$old_name*.txt
done < file_with_names
In this way, you read the IDs from file_with_names and rename the files replacing $old_name with $new_name leaving the rest of the filename untouched.
I was about to write a php function to do this for myself, but I came upon a faster method:
ls and copy & paste the directory contents into excel from the terminal window. Perhaps you may need to use on online line break removal or addition tool. Assume that your file names are in column A In excel, use the following formula in another column:
="mv "&A1&" prefix"&A1&"suffix"
or
="mv "&A1&" "&substitute(A1,"jpeg","jpg")&"suffix"
or
="mv olddirectory/"&A1&" newdirectory/"&A1
back in Linux, create a new file with
nano rename.txt and paste in the values from excel. They should look something like this:
mv oldname1.jpg newname1.jpg
mv oldname1.jpg newname2.jpg
then close out of nano and run the following command:
bash rename.txt. Bash just runs every line in the file as if you had typed it.
and you are done! This method gives verbose output on errors, which is handy.
I'm having problems getting my rsync syntax right and I'm wondering if my scenario can actually be handled with rsync. First, I've confirmed that rsync is working just fine between my local host and my remote host. Doing a straight sync on a directory is successful.
Here's what my filesystem looks like:
uploads/
1260000000/
file_11_00.jpg
file_11_01.jpg
file_12_00.jpg
1270000000/
file_11_00.jpg
file_11_01.jpg
file_12_00.jpg
1280000000/
file_11_00.jpg
file_11_01.jpg
file_12_00.jpg
What I want to do is run rsync only on files that begin with "file_11_" in the subdirectories and I want to be able to run just one rsync job to sync all of these files in the subdirectories.
Here's the command that I'm trying:
rsync -nrv --include="**/file_11*.jpg" --exclude="*" /Storage/uploads/ /website/uploads/
This results in 0 files being marked for transfer in my dry run. I've tried various other combinations of --include and --exclude statements, but either continued to get no results or got everything as if no include or exclude options were set.
Anyone have any idea how to do this?
The problem is that --exclude="*" says to exclude (for example) the 1260000000/ directory, so rsync never examines the contents of that directory, so never notices that the directory contains files that would have been matched by your --include.
I think the closest thing to what you want is this:
rsync -nrv --include="*/" --include="file_11*.jpg" --exclude="*" /Storage/uploads/ /website/uploads/
(which will include all directories, and all files matching file_11*.jpg, but no other files), or maybe this:
rsync -nrv --include="/[0-9][0-9][0-9]0000000/" --include="file_11*.jpg" --exclude="*" /Storage/uploads/ /website/uploads/
(same concept, but much pickier about the directories it will include).
rsync include exclude pattern examples:
"*" means everything
"dir1" transfers empty directory [dir1]
"dir*" transfers empty directories like: "dir1", "dir2", "dir3", etc...
"file*" transfers files whose names start with [file]
"dir**" transfers every path that starts with [dir] like "dir1/file.txt", "dir2/bar/ffaa.html", etc...
"dir***" same as above
"dir1/*" does nothing
"dir1/**" does nothing
"dir1/***" transfers [dir1] directory and all its contents like "dir1/file.txt", "dir1/fooo.sh", "dir1/fold/baar.py", etc...
And final note is that simply dont rely on asterisks that are used in the beginning for evaluating paths; like "**dir" (its ok to use them for single folders or files but not paths) and note that more than two asterisks dont work for file names.
Here's my "teach a person to fish" answer:
Rsync's syntax is definitely non-intuitive, but it is worth understanding.
First, use -vvv to see the debug info for rsync.
$ rsync -nr -vvv --include="**/file_11*.jpg" --exclude="*" /Storage/uploads/ /website/uploads/
[sender] hiding directory 1280000000 because of pattern *
[sender] hiding directory 1260000000 because of pattern *
[sender] hiding directory 1270000000 because of pattern *
The key concept here is that rsync applies the include/exclude patterns for each directory recursively. As soon as the first include/exclude is matched, the processing stops.
The first directory it evaluates is /Storage/uploads. Storage/uploads has 1280000000/, 1260000000/, 1270000000/ dirs/files. None of them match file_11*.jpg to include. All of them match * to exclude. So they are excluded, and rsync ends.
The solution is to include all dirs (*/) first. Then the first dir component will be 1260000000/, 1270000000/, 1280000000/ since they match */. The next dir component will be 1260000000/. In 1260000000/, file_11_00.jpg matches --include="file_11*.jpg", so it is included. And so forth.
$ rsync -nrv --include='*/' --include="file_11*.jpg" --exclude="*" /Storage/uploads/ /website/uploads/
./
1260000000/
1260000000/file_11_00.jpg
1260000000/file_11_01.jpg
1270000000/
1270000000/file_11_00.jpg
1270000000/file_11_01.jpg
1280000000/
1280000000/file_11_00.jpg
1280000000/file_11_01.jpg
https://download.samba.org/pub/rsync/rsync.1