I have disk usage problem with rsync and --link-dest
Incremental backup is taking full disk space:
#localhost media]$ ls
orig
----------------------------------------------------
localhost media]$ du -sh .
25M .
----------------------------------------------------
localhost media]$ rsync -avh orig/ full
----------------------------------------------------
#localhost media]$ du -sh .
49M .
----------------------------------------------------
localhost media]$ echo 1111 > orig/foo111
----------------------------------------------------
localhost media]$ rsync -avh --link-dest=full orig/ orig_1
----------------------------------------------------
localhost media]$ ls orig_1/foo111
orig_1/foo111
_____________________________________________________
localhost media]$ ls full/foo111
ls: cannot access full/foo111: No such file or directory
Everything looks good so far. The latest change is reflected in orig_1
But the directories aren't hard linked and they're all in full size.
-----------------------------------------------------
localhost media]$ du -sh .
74M .
---------------------------------------------
localhost media]$ du -sh orig_1/
25M orig_1/
--------------------------------------------
localhost media]$ du -sh orig
25M orig
---------------------------------------------
localhost media]$ du -sh full
25M full
Am I suppose to have the orig_1 size as 0? And stat command shows no hard links. What am I doing wrong?
When you ran rsync -avh --link-dest=full orig/ orig_1, you ignored this error message (it's more obvious if you remove -v):
--link-dest arg does not exist: full
If we then take a look at man rsync under --link-dest, we find:
If DIR is a relative path, it is relative to the destination directory.
And there it is. full is relative to the current directory. Relative to the destination directory, it would be ../full.
If you try again with rsync -avh --link-dest=../full orig/ orig_1, you get what you expect:
$ du -sh *
149M full
149M orig
232K orig_1
$ du -sh .
298M .
Note that, when counted individually, the directories still appear take up the full space:
$ du -sh orig_1
149M orig_1
This is because du keeps track of files it's already seen, and avoids counting them twice.
--link-dest takes a path relative to the destination. You want --link-dest=../orig.
Standard Unix filesystems do not allow hard links to directories, except for the special . and .. links. --link-dest only creates hard links for files, the rest of the directory structure is recreated as real directories.
And even if hard links were allowed to directories, du would still show the full size of each link. When using hard links, there's no distinction between the original and the link, they're each just names that refer to a particular inode, and du would scan them equivalently.
Related
I create a file using truncate -s 1024M a.test.
I was expecting a file size of a.test to be 1024M, but not getting correct size somehow.
Below is my code.
$du -sh a.test
4.0K a.test
When using ls -l a.test, it is ok:
$ ll a.test
-rw-rw-r-- 1 work work 1073741824 Jul 12 17:26 a.test
Can some one help me out with this issue.
du tells you how much actual disk space you use. Since your file does not have any data in it, the OS will store it as a sparse file, so actual disk usage is much smaller than the size of the file. If you check it with "du --apparent-size -sh a.test", then that will report what you expected.
In Linux, . and .. have the same inode number only in 2 situations:
. and .. in the root of the root filesystem has the same inode number.
. and .. in the mount directory of a newly mounted filesystem on a directory of root filesystem has the same inode number.
But when I check the inode on CentOS7 I find a mounted directory has the same inode number with the root directory of the root filesystem.
Such as below, the /boot directory has the same inode number as /. In addition, other directories such as /sys/bus/memory/drivers and /sys/kernel/debug/tracing/trace_stat/function20 are also has the same inode with /.(inode number is 64).
[root#localhost boot]# mount
/dev/mapper/centos-root on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/sda1 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
...
[root#localhost boot]# find / -inum 64
/
/boot
/sys/bus/memory/drivers
/sys/kernel/debug/tracing/trace_stat/function20
the tree joint part
[root#localhost ~]# ls -ai /
64 . 64 .. 64 boot
If they have the same inode number, how does kernel find the correct content? or these directories should point to the same content (but different files under these directories)? or it only happens with LVM directories?
Does anyone know why this happened? Do I make a mistake or get something wrong here?
From a linux box, I recently mounted a Windows share using cifs.
The intend was to "locally" use rsync to backup my windows machine.
The command line used to mount the Windows drive is something like:
mount \\192.168.1.74\share /cifs1 -t cifs -o noserverino,iocharset=utf8,ro
Please also note that the drive attached to the linux box is formatted ntfs.
When doing a sample backup, rsync was always re-copying the directories (names), but not the files. After looking more closely at the "ls -lh " output at both ends, I noticed that on the linux side the size of the directory level is always 0:
TTT-Admin#1080-Router:/tmp/mnt/RT-1080/tmp# ls -lh
drwxrwxrwx 1 TTT-Admi root 0 Feb 8 12:14 DeltaCopy
but the size of the directory level on the cifs side was always different from 0:
TTT-Admin#1080-Router:/cifs1/temp/Rsync-Packages# ls -lh
drwxr-xr-x 1 TTT-Admi root 8.0K Feb 8 12:14 DeltaCopy
This difference explains why rsync was always recopying directories, but not recopying the folders(which was correct, folder sizes and time stamps being the same on both ends).
EDIT: the rsync command is:
> rsync -av /cifs1/Temp/Rsync-Packages/DeltaCopy /mnt/RT-1080/tmp/rsync
-av /cifs1/Temp/Rsync-Packages/DeltaCopy /mnt/RT-1080/tmp/
Is a directory supposed to "have a size" or not ? What should I do to solve this discrepancy ?
Please see the image below.
Notice that my /home directory is already 100% but I still got plenty of space on /dev/mapper/vg00-srv. Is there a way that I can use that space for my /home or rather increase the size of my /home directory. This is a Debian box.
Your response is greatly appreciated.
sudo su -
umount /srv
resize2fs /dev/mapper/vg00-srv 800G
lvresize -L 810G /dev/mapper/vg00-srv
lvresize -L +80G /dev/mapper/vg00-home
resize2fs /dev/mapper/vg00-home
mount /srv
Just edit this below line in your /var/fstab file.
UUID=685a1898-6c19-4327-a383-1f8b840a9a3c /home ext4 defaults
Change ext4 to your desired home folder partition.
I have a backup system that creates directories named after Unix Timestamps, and then creates incremental backups using a hardlink system (--link-dest in rsync), so typically the first backup is very large, and then later backups are fractions as big.
This is my output of my current backups:
root#athos:/media/awesomeness_drive# du -sh lantea_home/*
31G lantea_home/1384197192
17M lantea_home/1384205953
17M lantea_home/1384205979
17M lantea_home/1384206056
17M lantea_home/1384206195
17M lantea_home/1384207349
3.1G lantea_home/1384207678
14M lantea_home/1384208111
14M lantea_home/1384208128
16M lantea_home/1384232401
15G lantea_home/1384275601
43M lantea_home/1384318801
Everything seems correct, however, take for example the last directory, lantea_home/1384318801:
root#athos:/media/awesomeness_drive# du -sh lantea_home/1384318801/
28G lantea_home/1384318801/
I consistently get this behavior, why is the directory considered 28G by the second du command?
Note - the output remains the same with the -P and -L flags.
Hardlinks are real references to the same file (represented by its inode). There is no difference between the "original" file and a hard link pointing to it as well. Both files have the same status, both are then references to this file. Removing one of them lets the other stay intact. Only removing the last hardlink will remove the file at last and free the disk space.
So if you ask du what it sees in one directory only, it does not care that there are hardlinks elsewhere pointing to the same contents. It simply counts all the files' sizes and sums them up. Only hardlinks within the considered directory are not counted more than once. du is that clever (not all programs necessarily need to be).
So in effect, directory A might have a du size of 28G, directory B might have a size of 29G, but together they still only occupy 30G and if you ask du of the size of A and B, you will get that number.
And with the switch "-l" du counts the hardlinks in every subdir too, so I can see, how big the whole backup is, not only the increment delta.