I have a server in which there's a file with 30GB inside the partition /, although df -h lists this partition as using 11GB. Why does that happen?
[root#APPSERVER21-S1 ~]# ls -l /etc/vinter/logs/
total 2046032
-rw-r--r-- 1 root root 3920496 Sep 11 14:35 PlusoftCRMIntegration-APISILVERNODE-1.log
-rw-r--r-- 1 root root 30823671719 Sep 11 15:13 rsp.appserver21-s1.apigoldnode-1.api-oi-gold.log
-rw-r--r-- 1 root root 406061056 Sep 11 15:13 rsp.appserver21-s1.apisilvernode-1.api-oi-silver.log
[root#APPSERVER21-S1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.6G 0 3.6G 0% /dev
tmpfs 3.6G 0 3.6G 0% /dev/shm
tmpfs 3.6G 137M 3.4G 4% /run
tmpfs 3.6G 0 3.6G 0% /sys/fs/cgroup
/dev/mapper/vg_main-lv_root 16G 5.0G 11G 32% /
/dev/mapper/vg_dados-lv_dados 20G 33M 20G 1% /datastorage
/dev/mapper/vg_dados-lv_docker 80G 128M 79G 1% /var/lib/docker
/dev/xvdb1 497M 161M 337M 33% /boot
tmpfs 722M 0 722M 0% /run/user/0
I have a server in which there's a file with 30GB inside the partition /, although df -h lists this partition as using 11GB.
Actually, it lists this filesystem as using 5 GB, not 11 GB.
Why does that happen?
This is impossible to answer without knowing what the contents of that file are, what the filesystem is, how the file was created, and so on.
There are filesystems that perform data-deduplication. If there are blocks with identical content, they will be replaced with links, so that they only take up the space of one block. So, if there is a lot of duplicate data on the filesystem, it is easily possible that the files take up much less space on-disk than their duplicated contents.
There are filesystems that perform compression. If there is a lot of redundancy on the filesystem, then the compression will reduce the data dramatically, and it is easily possible that the files take up much less space on-disk than their uncompressed contents.
Many filesystems support sparse files. Sparse files are files with "holes" in them. The holes will be read as a long string of binary zeroes, and are technically part of the content of the file, but they are not stored on-disk.
Related
I don't know actually if this is more a "classic" linux or a docker question but:
On an VM where some of my docker containers are running I've a strange thing. /var/lib/docker is an own partitionwith 20GB. When I look over the partition with df -h I see this:
eti-gwl1v-dockerapp1 root# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.8G 0 7.8G 0% /dev/shm
tmpfs 7.8G 815M 7.0G 11% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/sda2 12G 3.2G 8.0G 29% /
/dev/sda7 3.9G 17M 3.7G 1% /tmp
/dev/sda5 7.8G 6.8G 649M 92% /var
/dev/sdb2 20G 47M 19G 1% /usr2
/dev/sdb1 20G 2.9G 16G 16% /var/lib/docker
So usage is at 16%. But when I now navigate to /var/lib and do a du -sch docker I see this:
eti-gwl1v-dockerapp1 root# cd /var/lib
eti-gwl1v-dockerapp1 root# du -sch docker
19G docker
19G total
eti-gwl1v-dockerapp1 root#
So same directory/partition but two sizes? How is that going?
This is really a question for unix.stackexchange.com, but there is filesystem overhead that makes the partition larger than the total size of the individual files within it.
du and df show you two different metrics:
du shows you the (estimated) file space usage, i.e. the sum of all file sizes
df shows you the disk space usage, i.e. how much space on the disk is actually used
These are distinct values and can often diverge:
disk usage may be bigger than the mere sum of file sizes due to additional meta data: e.g. the disk usage of 1000 empty files (file size = 0) is >0 since their file names and permissions need to be stored
the space used by one or multiple files may be smaller than their reported file size due to:
holes in the file - block consisting of only null bytes are not actually written to disk, see sparse files
automatic file system compression
deduplication through hard links or copy-on-write
Since docker uses the image layers as a means of deduplication the latter is most probably the cause of your observation - i.e. the sum of the files is much bigger because most of them are shared/deduplicated through hard links.
du estimates filesystem usage through summing the size of all files in it. This does not deal well with the usage of overlay2: there will be many directories which contain the same files as contained in another, but overlaid with additional layers using overlay2. As such, du will show a very inflated number.
I have not tested this since my Docker daemon is not using overlay2, but using du -x to avoid going into overlays could give the right amount. However, this wouldn't work for other Docker drivers, like btrfs, for example.
I have a server running Centos 7. This is the result of df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.4G 0 7.4G 0% /dev
tmpfs 1.5G 139M 1.4G 10% /run
/dev/vda1 46G 44G 0 100% /
tmpfs 7.4G 0 7.4G 0% /dev/shm
tmpfs 7.4G 0 7.4G 0% /sys/fs/cgroup
/dev/vda15 99M 3.6M 95M 4% /boot/efi
/dev/mapper/LVMVolGroup-DATA_VOLUME 138G 17G 114G 13% /mnt/data
tmpfs 1.5G 0 1.5G 0% /run/user/0
Even if there are 2GB of free space on / , it shows that the filesystem is at 100% of usage, and I can't install new packages because it tells me there's no space left on device.
Besides, if I type sudo du -sh /* | sort -rh | head -15
the result is:
17G /mnt
1.1G /usr
292M /var
208M /root
139M /run
49M /boot
48M /tmp
32M /etc
28K /home
16K /lost+found
12K /anaconda-post.log
4.0K /srv
4.0K /opt
4.0K /media
0 /sys
So it seems that there are no big files filling up the disk, and the sum of the sizes of the directories is not even equal to 44GB.
Additional info: the only service running on the server is Jenkins, but its home is under /mnt/data/jenkins.
How can I solve the problem?
Found the solution.
The problem was related to some deleted files kept open by Jenkins.
Restarting the service the problem was solved.
The problem was related to the system cache/temp storage. Linux system created the cache files and its archive from time to time, especially when some long option is run like DB import or crone job etc.. or sometimes server up from sines long.
Restarting the service or server
so due to that, the cache/ temp files were deleted and the problem was solved.
even in windows, we faced that kind of performance issue when RAM is low, and restarting the system is the primary solution for that.
Just set a Fedora 22 system on VMWare with 60GB. When inputting the "df" command, the system displayed this:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/fedora-root 38440424 4140700 32324020 12% /
devtmpfs 2009804 0 2009804 0% /dev
tmpfs 2017796 92 2017704 1% /dev/shm
tmpfs 2017796 872 2016924 1% /run
tmpfs 2017796 0 2017796 0% /sys/fs/cgroup
tmpfs 2017796 532 2017264 1% /tmp
/dev/sda1 487652 79147 378809 18% /boot
/dev/mapper/fedora-home 18701036 49464 17678568 1% /home
What is the exact size of each 1K-blocks? Does the /dev/mapper/fedora-root contain the /dev/mapper/fedora-home?
I'm so confused with "df" command.
Thanks a lot.
You can see from the df output that /dev/mapper/fedora-home can currently be reached at /home which is its mount point. Because the mount point for /dev/mapper/fedora-root is at / farther up the directory tree, anything that /dev/mapper/fedora-root has in /home is not accessible by normal means until and unless /dev/mapper/fedora-home gets unmounted.
As David Schwartz noted, a 1K-block is a unit of one (binary) kilobyte, which is 1024 bytes. Because no one has bothered to change it since the time when it was important to performance, df still reports sizes in terms of the device's block size. That value does vary. There are still plenty of devices around that have a block size of 512 bytes. For output consistently in kilobytes, you can use df -k.
What is the exact size of each 1K-blocks?
1,024 bytes.
Does the /dev/mapper/fedora-root contains the /dev/mapper/fedora-home?
They're separate filesystems, that's why they appear on separate lines in the df output.
How to increase disk space of an instance without using EBS ? Root file system size is only showing 10 GB. Is there a way to create a bigger file system without EBS ?
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 9.9G 3.3G 6.1G 35% /
tmpfs 874M 0 874M 0% /lib/init/rw
udev 874M 84K 874M 1% /dev
tmpfs 874M 0 874M 0% /dev/shm
/dev/sdb 335G 12G 307G 4% /mnt
As you can see in the output, a much bigger partition is mounted at /mnt. You can move some of the things on the root filesystem there by either remounting it at the appropriate location or add symlinks. There is no other way to add more diskspace if you don't want to resort to EBS or a network filesystem.
We are using Amazon EBS to store a large number of small files (<10KB) in a 3-level directory structure.
~/lists# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 9.9G 3.9G 5.5G 42% /
tmpfs 854M 0 854M 0% /lib/init/rw
varrun 854M 64K 854M 1% /var/run
varlock 854M 0 854M 0% /var/lock
udev 854M 80K 854M 1% /dev
tmpfs 854M 0 854M 0% /dev/shm
/dev/sda2 147G 80G 60G 58% /mnt
/dev/sdj 197G 60G 128G 32% /vol
The partition in question is /vol (size: 200GB)
~/lists# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 655360 26541 628819 5% /
tmpfs 186059 3 186056 1% /lib/init/rw
varrun 186059 31 186028 1% /var/run
varlock 186059 2 186057 1% /var/lock
udev 186059 824 185235 1% /dev
tmpfs 186059 1 186058 1% /dev/shm
/dev/sda2 19546112 17573097 1973015 90% /mnt
/dev/sdj 13107200 13107200 0 100% /vol
~/lists# sudo /sbin/dumpe2fs /dev/sdj | grep "Block size"
dumpe2fs 1.41.4 (27-Jan-2009)
Block size: 4096
The number of inodes for the partition /vol are 13Million+. The block size is 4096. Taking the Block Size as 4096, the number of inodes the 200GB partition (ext3) should support is 52million+ (Maximum Inode Calculation: Volume size in bytes/2^12). So why does the partition only support 13million inode?
I'm pretty sure that inodes are allocated statically when you create the volume (using mfs.ext3 in this case). For whatever reason, mkfs.ext3 decided to reserve 13 Million inodes and now you can't create any more files.
See this 2001 discussion of inodes
The Wikipedia ext3 page has a footnote explaining this more concisely: wiki link
Also, inodes are allocated per file (not block), which is why there are only 13M inodes - mkfs.ext3 must have been configured with an average file size of 8 KB which would account for the issue you're seeing.