How to get disk usage from inside docker container - linux

I have started my container using the --privileged flag, so as far as I know, all disks should be available from inside the container - and that is partly true, but I somehow can't read the size of them.
lsblk on host (Ubuntu):
sda 8:0 1 59,6G 0 disk
└─sda1 8:1 1 59,6G 0 part /media/mauz/ESD-ISO
nvme0n1 259:0 0 953,9G 0 disk
├─nvme0n1p1 259:1 0 512M 0 part /boot/efi
├─nvme0n1p2 259:2 0 732M 0 part /boot
└─nvme0n1p3 259:3 0 952,7G 0 part
└─nvme0n1p3_crypt 253:0 0 952,6G 0 crypt
├─vgubuntu-root 253:1 0 930,4G 0 lvm /
└─vgubuntu-swap_1 253:2 0 976M 0 lvm [SWAP]
lsblk in container (Alpine):
sda 8:0 1 59.6G 0 disk
└─sda1 8:1 1 59.6G 0 part
nvme0n1 259:0 0 953.9G 0 disk
├─nvme0n1p1 259:1 0 512M 0 part
├─nvme0n1p2 259:2 0 732M 0 part
└─nvme0n1p3 259:3 0 952.7G 0 part
Both outputs are stripped from loop devices, but as you can see, there are 2 drives recognized in both.
Now, if I run the df command on the host:
Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 3261580 2564 3259016 1% /run
/dev/mapper/vgubuntu-root 959200352 137078032 773327904 16% /
tmpfs 16307884 215740 16092144 2% /dev/shm
tmpfs 5120 4 5116 1% /run/lock
/dev/nvme0n1p2 721392 364788 304140 55% /boot
/dev/nvme0n1p1 523248 76232 447016 15% /boot/efi
tmpfs 3261576 140 3261436 1% /run/user/1000
/dev/sda1 62519040 23118848 39400192 37% /media/mauz/ESD-ISO
And inside the container:
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 959200352 137078188 773327748 15% /
tmpfs 65536 0 65536 0% /dev
shm 65536 0 65536 0% /dev/shm
/dev/mapper/vgubuntu-root
959200352 137078188 773327748 15% /app
/dev/mapper/vgubuntu-root
959200352 137078188 773327748 15% /etc/os-release
/dev/mapper/vgubuntu-root
959200352 137078188 773327748 15% /etc/resolv.conf
/dev/mapper/vgubuntu-root
959200352 137078188 773327748 15% /etc/hostname
/dev/mapper/vgubuntu-root
959200352 137078188 773327748 15% /etc/hosts
Somehow, it does not show the correct drives in the second df output. Is there any way to make df show the correct output, even inside the container?
Or is there another way to get the correct disk sizes and usages from the host?

There is no "decent" solution that can accomplish what you want, but let me explain why.
You talk about "disk usage", but in reality there is no such thing as disk usage. As far as the disk (i.e. the device itself) is concerned, there is no concept of "usage". What you are looking for is rather filesystem disk usage, which is fundamentally different.
In order to know the "used" and "available" space of a filesystem, you will have to mount it. This allows the kernel to process filesystem metadata that can then be used to determine free and used filesystem blocks. Without mounting the filesystem this information is simply not available to the kernel (and therefore not available to df, for example).
In order for Docker to work, containers run with a different mount namsepace than the host. The core reason for this is that containers cannot in general safely share mount points with the host. Think for example what would happen if / in the host and / in the container referred to the same mount point: as soon as the container starts, it would likely break your system by touching sensitive files that it is not supposed to. So by default, Docker will "isolate" containers in their own mount namespace, so that they will only see the mount points they need, and there is no option to avoid this because of the above.
You could be able to get this information by reading raw data from the available block devices (without mounting them) and parsing the filesystem metadata (if any) from userspace within the container using some specialized tool, but this is a finicky solution as it would basically require one tool per possible filesystem type. See also Free space in unmounted partition at Unix & Linux SE.
You could also use bind mounts allowing the host and the container to share mount points, but this would have to be done on a per-mount basis, for example:
docker run --mount readonly,type=bind,source=/media/mauz/ESD-ISO,target=/container/path ...
$ df
...
/dev/sda1 62519040 23118848 39400192 37% /container/path
...
You say that for now you are "passing all mounted volumes manually", so I assume this is not different than whatever you are currently doing. On top of being pretty ugly, this solution would also have the limit of not being able to handle changes in devices or mount points on the host (e.g. if a new device is added and mounted).
The only "real" solution I can see here would be to run some application on the host, which periodically extracts the needed information and communicates it to the application running inside the container.

Using nsenter, this command should achieve what you wanted :
docker run -it --rm --privileged --pid host ubuntu nsenter -t 1 -m -u -n -i bash
# --privileged : run privileged
# --pid host : shares host's process id namespace
# nsenter - run program with namespaces of other processes
# -t 1 : Specify process 1 to get contexts from
# -m -u -n -i : Share mount, UTS, network, IPC namespace.

Related

Different sizes for /var/lib/docker

I don't know actually if this is more a "classic" linux or a docker question but:
On an VM where some of my docker containers are running I've a strange thing. /var/lib/docker is an own partitionwith 20GB. When I look over the partition with df -h I see this:
eti-gwl1v-dockerapp1 root# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.8G 0 7.8G 0% /dev/shm
tmpfs 7.8G 815M 7.0G 11% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/sda2 12G 3.2G 8.0G 29% /
/dev/sda7 3.9G 17M 3.7G 1% /tmp
/dev/sda5 7.8G 6.8G 649M 92% /var
/dev/sdb2 20G 47M 19G 1% /usr2
/dev/sdb1 20G 2.9G 16G 16% /var/lib/docker
So usage is at 16%. But when I now navigate to /var/lib and do a du -sch docker I see this:
eti-gwl1v-dockerapp1 root# cd /var/lib
eti-gwl1v-dockerapp1 root# du -sch docker
19G docker
19G total
eti-gwl1v-dockerapp1 root#
So same directory/partition but two sizes? How is that going?
This is really a question for unix.stackexchange.com, but there is filesystem overhead that makes the partition larger than the total size of the individual files within it.
du and df show you two different metrics:
du shows you the (estimated) file space usage, i.e. the sum of all file sizes
df shows you the disk space usage, i.e. how much space on the disk is actually used
These are distinct values and can often diverge:
disk usage may be bigger than the mere sum of file sizes due to additional meta data: e.g. the disk usage of 1000 empty files (file size = 0) is >0 since their file names and permissions need to be stored
the space used by one or multiple files may be smaller than their reported file size due to:
holes in the file - block consisting of only null bytes are not actually written to disk, see sparse files
automatic file system compression
deduplication through hard links or copy-on-write
Since docker uses the image layers as a means of deduplication the latter is most probably the cause of your observation - i.e. the sum of the files is much bigger because most of them are shared/deduplicated through hard links.
du estimates filesystem usage through summing the size of all files in it. This does not deal well with the usage of overlay2: there will be many directories which contain the same files as contained in another, but overlaid with additional layers using overlay2. As such, du will show a very inflated number.
I have not tested this since my Docker daemon is not using overlay2, but using du -x to avoid going into overlays could give the right amount. However, this wouldn't work for other Docker drivers, like btrfs, for example.

ec2 how to add more volume to exist device

I was trying to add more volume to my device
df -h
I get:
[root#ip-172-x-x-x ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.8G 44K 3.8G 1% /dev
tmpfs 3.8G 0 3.8G 0% /dev/shm
/dev/nvme0n1p1 7.8G 3.6G 4.2G 46% /
I wanna add all existing storage to /dev/nvme0n1p1
lsblk
I get
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 300G 0 disk
├─nvme0n1p1 259:1 0 8G 0 part /
└─nvme0n1p128 259:2 0 1M 0 part
I was trying to google around on aws instructions, still quite confuse. since most of the instruction is setting up brand new instance. While for my use case i cannot stop the instance.
i cannot do
mkfs
Also seems like the disk is already mount?? I guess i may misunderstand the meaning of mount...
since the filesystem is already there.
just wanna use all existing space.
Thanks for help in advance!!
your lsblk output shows that you have a 300G disk but your nvme0n1p1 is only 8G. You need to first grow your partition to fill the disk and then expand your filesystem to fill your partition:
Snapshot all ebs volumes you care about before doing any resize operations on them.
Install growpart
sudo yum install cloud-utils-growpart
Resize partiongrowpart /dev/nvme0n1 1
Reboot reboot now
Run lsblk and verify that the partition is now the full disk size
You may still have to run sudo resize2fs /dev/nvme0n1 to expand the filesystem

Not able to resize the AWS EC2 volume

I created an AWS EC2 Linux instance with 8GB root volume. Then I increased the EBS volume to 9GB and it went to the completed state. It's a small volume, so the resize took a couple of minutes to complete.
Now I try to extend extend the linux file system after resizing the volume using the instructions mentioned here. But, I get the below error message. I tried two times, the entire process. But it's all the same.
The filesystem is already 2096635 (4k) blocks long. Nothing to do!
Here is the screen shot of the image.
Can someone help me?
Just reboot the instance because it automatically resizes your root filesystem on boot.
I tried it myself. Here is the instance with an 8GB volume:
[ec2-user#ip-172-31-15-216 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
[ec2-user#ip-172-31-15-216 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 236M 56K 236M 1% /dev
tmpfs 246M 0 246M 0% /dev/shm
/dev/xvda1 7.8G 985M 6.7G 13% /
After modifying the EBS Volume:
[ec2-user#ip-172-31-15-216 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 9G 0 disk
└─xvda1 202:1 0 8G 0 part /
[ec2-user#ip-172-31-15-216 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 236M 56K 236M 1% /dev
tmpfs 246M 0 246M 0% /dev/shm
/dev/xvda1 7.8G 985M 6.7G 13% /
After the reboot:
[ec2-user#ip-172-31-15-216 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 9G 0 disk
└─xvda1 202:1 0 9G 0 part /
[ec2-user#ip-172-31-15-216 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 236M 56K 236M 1% /dev
tmpfs 246M 0 246M 0% /dev/shm
/dev/xvda1 8.8G 984M 7.7G 12% /
See also: increase EC2 EBS volume after cloning - resize2fs not working
# http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage_expand_partition.html
# Before goging to do this, detach and attach the root volume to anothe instance
# Note:
# 1) Before detach the volume, please make a note of device name which going to
# detch from the machine, why because we should mention same name when attaching back, otherwise data will be lost
# 2)
# Identifying device name which we want to expand
lsblk
# Running parted command on the device
sudo parted /dev/xvdf
# Changing the parted units of measure to sectors.
unit s
# Run the print command to list the partitions on the device
print
# if it shows warning, chose fix
# Delete the partition entry for the partition using the number (1) from the previous step
rm 1 # number 1 will change based the partition we want to delete
# Create a new partition that extends to the end of the volume
mkpart Linux 4096s 100%
# Run the print command again to verify your partition
print
# Check to see that any flags that were present earlier are still
# present for the partition that you expanded. In some cases the boot
# flag may be lost. If a flag was dropped from the partition when it was expanded,
# add the flag with the following command, substituting your partition number and the flag name.
# For example, the following command adds the boot flag to partition 1
set 1 boot on
#Run the quit command to exit parted.
quit
# verfiying the device
sudo e2fsck -f /dev/xvdf1

How does docker map host partitions?

I'm relatively new to docker, and when I started a container (an ubuntu base image), I noticed the following:
On the host,
$ df -h
...
/dev/sdc1 180M 98M 70M 59% /boot
/dev/sdc2 46G 20G 24G 46% /home
/dev/sdc5 37G 7.7G 27G 23% /usr
/dev/sdc6 19G 13G 5.3G 70% /var
$ lsblk
...
sdc 8:32 0 232.9G 0 disk
├─sdc1 8:33 0 190M 0 part /boot
├─sdc2 8:34 0 46.6G 0 part /home
├─sdc3 8:35 0 18.6G 0 part /
├─sdc4 8:36 0 1K 0 part
├─sdc5 8:37 0 37.3G 0 part /usr
├─sdc6 8:38 0 18.6G 0 part /var
├─sdc7 8:39 0 29.8G 0 part [SWAP]
└─sdc8 8:40 0 42.8G 0 part
On the container
$ df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 19G 13G 5.3G 70% /
none 19G 13G 5.3G 70% /
tmpfs 7.8G 0 7.8G 0% /dev
shm 64M 0 64M 0% /dev/shm
/dev/sdc6 19G 13G 5.3G 70% /etc/hosts
tmpfs 7.8G 0 7.8G 0% /proc/kcore
tmpfs 7.8G 0 7.8G 0% /proc/latency_stats
tmpfs 7.8G 0 7.8G 0% /proc/timer_stats
$ lsblk
sdc 8:32 0 232.9G 0 disk
|-sdc1 8:33 0 190M 0 part
|-sdc2 8:34 0 46.6G 0 part
|-sdc3 8:35 0 18.6G 0 part
|-sdc4 8:36 0 1K 0 part
|-sdc5 8:37 0 37.3G 0 part
|-sdc6 8:38 0 18.6G 0 part /var/lib/cassandra
|-sdc7 8:39 0 29.8G 0 part [SWAP]
`-sdc8 8:40 0 42.8G 0 part
Question 1: why is sdc6 mounted on different places between the host and the container?
Because the contents of the two mount points are different, so I assume docker must have done some kind of device mapping on the container, so sdc6 in the container isn't the same as the one on the host. However, the partition capacity and usage are the same, so I'm confused here.
Question 2: why is the container's root dir usage so high? The docker image doesn't have much stuff on it.
Thanks for any help.
Addition
The Dockerfile has a line
VOLUME /var/lib/cassandra
Question 1: why is sdc6 mounted on different places between the host and the container?
/dev/sdc6 on your host is /var, which is where /var/lib/docker resides and where Docker keeps certain data, such as the hosts file allocated to your container.
The hosts file is exposed as a bind mount inside the container, which is why you see:
/dev/sdc6 19G 13G 5.3G 70% /etc/hosts
Question 2: why is the container's root dir usage so high? The docker image doesn't have much stuff on it.
Take a look at the df output inside the container:
rootfs 19G 13G 5.3G 70% /
Now look at the df output on your host, and you'll see:
/dev/sdc6 19G 13G 5.3G 70% /var
The df inside the container is reflecting the state of the host filesystem. This suggests that you are using the aufs or overlay storage driver, both of which create "overlay" filesystems for containers on top of the host filesystem. The output from df would look different if you were using the devicemapper storage driver, relies on device mapper block devices instead of overlay filesystems.

My mounted EBS volume is not showing up

Trying to mount a 384G volume from old instance to a newly configure instance (8G). Attached 384G volume shows up on lsblk but on df -h it doesn't come up at all. What am I doing wrong?
[ec2-user#ip-10-111-111-111 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvdf 202:80 0 384G 0 disk
xvda1 202:1 0 8G 0 disk /
[ec2-user#ip-10-111-111-111 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.9G 1.5G 6.4G 19% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
Note: On EC2 instance dashboard it displays
Root device: /dev/sda1
Block devices: /dev/sda1 /dev/sdf
The df -k will only show mounted volumes.
You will need to mount your volume first, like this mount /dev/xvdf /mnt then you will be able to access it's content from /mnt and see it when typing df -k
For those landing here after not finding their xvdf devices on aws ec2 c5 or m5 instances, it's renamed to /dev/nvme... as per the docs
For C5 and M5 instances, EBS volumes are exposed as NVMe block
devices. The device names that you specify are renamed using NVMe
device names (/dev/nvme[0-26]n1). For more information, see Amazon EBS
and NVMe.
If this is windows follow this:
https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/recognize-expanded-volume-windows.html#recognize-expanded-volume-windows-disk-management

Resources