docker info command shows Storage Driver as overlay2 and is not showing information like when Storage Driver is devicemapper as shown below
Is there a way to see similar sort of information when Storage Driver is overlay2?
Reason:
docker info shows the following warnings when Storage Driver is devicemapper
WARNING: the devicemapper storage-driver is deprecated, and will be removed in a future release.
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
Device mapper used a dedicated block device (or loop mounted file as a device) as storage for the images and containers. The various utilization statistics applied to that block devices.
AUFS, Overlay, Overlay2, and other storage backends don't have this block device. They also don't separate between metadata and data storage, they're just different files in folders. To see the remaining capacity of the storage backend, you look at the remaining capacity of the backing filesystem. E.g.
df /var/lib/docker
Related
I create some pods with containers for which I set ephemeral-storage request and limit, like: (here 10GB)
Unfortunately, for some containers, the ephemeral-storage will be fully filled for unknown reasons. I would like to understand which dirs/files are responsible for filling it all, but I did not find a solution to do it.
I tried with df -h, but unfortunately, it will give stats for the whole node and not only for the particular pod/container.
Is there a way to retrieve the kubernetes container's ephemeral-storage usage details?
Pods use ephemeral local storage for scratch space, caching, and for logs. The kubelet can provide scratch space to Pods using local ephemeral storage to mount emptyDir volumes into containers.
Depending on your Kubernetes platform, You may not be able to easily determine where these files are being written, any filesystem can fill up, but rest assured that disk is being consumed somewhere (or worse, memory - depending on the specific configuration of your emptyDir and/or Kubernetes platform).
Refer to this SO link for more details on how by default & allocatable ephemeral-storage in a standard kubernetes environment is sourced from filesystem(mounted to /var/lib/kubelet).
And also refer to kubernetes documentation on how ephemeral storage can be managed & Ephemeral storage consumption management works.
I am assuming you're a GCP user, you can get a sense of your ephemeral-storage usage way:
Menu>Monitoring>Metrics Explorer>
Resource type: kubernetes node & Metric: Ephemeral Storage
Try the below commands to know kubernetes pod/container's ephemeral-storage usage details :
Try du -sh / [run inside a container] : du -sh will give the space consumed by your container files. Which simply returns the amount of disk space the current directory and all those stuff in it are using as a whole, something like: 2.4G.
Also you can check the complete file size using the du -h someDir command.
Inspecting container filesystems : You can use /bin/df as a tool to monitor ephemeral storage usage on the volume where ephemeral container data is located, which is /var/lib/kubelet and /var/lib/containers.
Is there any shared volume solution between kubernetes worker nodes , like gfs2 .
For example when i create single mysql server with persistent volume , if Worker node(which mysql pod is running on) Fail , pod is automatically recreating on other nodes , but data has losed becouse of first worker node has gone down and persistent volume was on this failed worker node.
Can i use gfs2 clustered storage on worker node ?
Agree with comments above and posting as a community wiki for future.
Relevant for 1.16 version.
As per official Persistent Volumes documentation there is a list of volumes supported ReadWriteMany access mode:
AzureFile
CephFS
Glusterfs
Quobyte
NFS
PortworxVolume
Suggest you use one of them.
I kicked off some Spark job on Kubernetes with quite big volume of data, and job failed that there is no enough space in /var/data/spark-xxx directory.
As the Spark documentation says on https://github.com/apache/spark/blob/master/docs/running-on-kubernetes.md
Spark uses temporary scratch space to spill data to disk during
shuffles and other operations. When using Kubernetes as the resource
manager the pods will be created with an emptyDir volume mounted for
each directory listed in SPARK_LOCAL_DIRS. If no directories are
explicitly specified then a default directory is created and
configured appropriately
Seems like /var/data/spark-xx directory is the default one for emptyDir. Thus, I tried to map that emptyDir to Volume (with bigger space) which is already mapped to Driver and Executors Pods.
I mapped it in the properties file and I can see that is mounted in the shell:
spark.kubernetes.driver.volumes.persistentVolumeClaim.checkvolume.mount.path=/checkpoint
spark.kubernetes.driver.volumes.persistentVolumeClaim.checkvolume.mount.readOnly=false
spark.kubernetes.driver.volumes.persistentVolumeClaim.checkvolume.options.claimName=sparkstorage
spark.kubernetes.executor.volumes.persistentVolumeClaim.checkvolume.mount.path=/checkpoint
spark.kubernetes.executor.volumes.persistentVolumeClaim.checkvolume.mount.readOnly=false
spark.kubernetes.executor.volumes.persistentVolumeClaim.checkvolume.options.claimName=sparkstorage
I am wondering if it's possible to mount emptyDir somehow on my persistent storage, so I can spill more data and avoid job failures?
I found that spark 3.0 has considered this problem and has completed the feature.
Spark supports using volumes to spill data during shuffles and other operations. To use a volume as local storage, the volume's name should starts with spark-local-dir-, for example:
--conf spark.kubernetes.driver.volumes.[VolumeType].spark-local-dir-[VolumeName].mount.path=<mount path>
--conf spark.kubernetes.driver.volumes.[VolumeType].spark-local-dir-[VolumeName].mount.readOnly=false
Reference:
https://issues.apache.org/jira/browse/SPARK-28042
https://github.com/apache/spark/pull/24879
Azure VM D4 is advertised as having 400GB of SSD backed storage. Provisioning the VM I end up with 127GB for the OS and 400GB for temp storage.
Is this normal? I need the full 400GB on the OS drive and dont see an obvious way to reconfigure storage.
That is correct. However, because the local SSD is not guaranteed to be persistent, you will not want this to be responsible for your OS drive.
In the D-series announcements, http://azure.microsoft.com/blog/2014/09/22/new-d-series-virtual-machine-sizes/
"Local Storage SSD Drive
On these new sizes, the temporary drive (D:\ on Windows, /mnt or /mnt/resource on Linux) are local SSDs. This high-speed local disk is best used for workloads that replicate across multiple instances, like MongoDB, or can leverage this high I/O disk for a local and temporary cache, like SQL Server 2014’s Buffer Pool Extensions. Note, these drives are not guaranteed to be persistent. Thus, while physical hardware failure is rare, when it occurs, the data on this disk may be lost, unlike your OS disk and any attached durable disks that are persisted in Azure Storage." (emphasis mine)
Found this post that explains how this is normal for the OS drive.
https://azure.microsoft.com/blog/2015/03/25/azure-vm-os-drive-limit-octupled/
So for marketplace images the guidance is to provision new data drives.
AWS Amazon offers 160GB space for small instance. On booting Suse linux the total root partition space I got is 10GB. On df -h I only see/dev/sda1 with 10GB space. Where is rest 150GB? How can I claim this space? I dont want to use EBS as it cost extra and 160GB space suffice my need. Please help.
The extra 150GB is given as an ephemeral storage, e.g. data on this storage won't survive reboots in contrast to the data on your root storage. During launching, you can select where your ephemeral disks should be made available as a device in your machine (this is the -boption when using the command line or in the "Instance Storage" tab when launching via the S3 console. You can then simply mount it in your running instance.