I am running out of inodes on my Debian 10 machine
I found the culprits
~# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 500578 317 500261 1% /dev
tmpfs 504337 547 503790 1% /run
/dev/sda1 3276800 3090639 186161 95% /
tmpfs 504337 1 504336 1% /dev/shm
tmpfs 504337 3 504334 1% /run/lock
tmpfs 504337 17 504320 1% /sys/fs/cgroup
overlay 3276800 3090639 186161 95% /var/lib/docker/overlay2/c238a2f5bcd5e7e193d572fd4b331510cd2438d113a66e9125f983c929cd8485/merged
overlay 3276800 3090639 186161 95% /var/lib/docker/overlay2/5f3ee0ad57f147f8dba95e1e06ea98fc697c72ff8eaaafa6494faeb8f0742096/merged
tmpfs 504337 11 504326 1% /run/user/10976
tmpfs 504337 11 504326 1% /run/user/10762
shm 504337 1 504336 1% /var/lib/docker/containers/135852b9e95d22ace0cf10b27752a3ad15d1fdfaebcde74f546c26ae4ef01ad8/mounts/shm
shm 504337 1 504336 1% /var/lib/docker/containers/cd9619f7703b1e9fee7c1338379dab5270260d23834b27f278c57500b5e50b5a/mounts/shm
docker overlay /var/lib/docker/overlay2/*/merged
Only 2 containers are running.
no dangling images on the machine as well.
~# docker info
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 19
Server Version: 20.10.18
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
runc version: v1.1.4-0-g5fd4c4d
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.7.0-0.bpo.2-amd64
Operating System: Debian GNU/Linux 10 (buster)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.848GiB
Name: name
ID: SNF2:X4SU:OLKG:ESHV:PX3U:UJGB:MADV:OZ3R:UCZP:KONV:K4ZV:ILL4
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
what can I do to reclaim inodes?
PS: Docker version 18.09.0, build 4d60db4
docker system prune and docker image prune does not help either.
I was trying to load my cgroup config with cgconfigparser, but it gives the following error:
cgconfigparser; error loading /etc/cgconfig.conf: Cgroup is not mounted
Then I tried grep cgroup /proc/mounts, which gives:
cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime 0 0
I'm in a docker container based on centos7, did some googling and looks like my cgroup is using cgroup2 instead of cgroup1? If this is the case how do I switch back to cgorup1 in docker without tempering with the docker image? If not, what can I do to resolve this error?
I am trying to setup AWS Greengrass core on Amazon Linux with respect to the documentation(followed all the prerequisites) of AWS(http://docs.aws.amazon.com/greengrass/latest/developerguide/start-core.html).
I am getting the below error on starting the Greengrass core daemon
./greengrassd start
Setting up greengrass daemon
Validating execution environment
Found cgroup subsystem: cpuset
Found cgroup subsystem: cpu
Found cgroup subsystem: cpuacct
Found cgroup subsystem: blkio
Found cgroup subsystem: memory
Found cgroup subsystem: devices
Found cgroup subsystem: freezer
Found cgroup subsystem: net_cls
Found cgroup subsystem: perf_event
Found cgroup subsystem: net_prio
Found cgroup subsystem: hugetlb
Found cgroup subsystem: pids
Starting greengrass daemon
Greengrass daemon 23933 failed to start
Failed to create overlay fs for container rootfs no such file or directory
I could not find any troubleshooting method for this and I am stuck in the middle of the process. It would be more helpful If any of you solved for me.
Have you tried executing the ./greengrassd start command using sudo ?
When I omit sudo I can an error that says this command needs to be run with using sudo e.g.
sudo ./greengrassd start
I want Docker to start with systemd cgroup driver. For some reason it is using only cgroupfs on my CentOS 7 server.
Here is startup config file.
# systemctl cat docker
# /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target
Wants=docker-storage-setup.service
Requires=docker-cleanup.timer
[Service]
Type=notify
NotifyAccess=all
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
Environment=DOCKER_HTTP_HOST_COMPAT=1
Environment=PATH=/usr/libexec/docker:/usr/bin:/usr/sbin
ExecStart=/usr/bin/dockerd-current \
--add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \
--default-runtime=docker-runc \
--exec-opt native.cgroupdriver=systemd \
--userland-proxy-path=/usr/libexec/docker/docker-proxy-current \
$OPTIONS \
$DOCKER_STORAGE_OPTIONS \
$DOCKER_NETWORK_OPTIONS \
$ADD_REGISTRY \
$BLOCK_REGISTRY \
$INSECURE_REGISTRY
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
Restart=on-abnormal
MountFlags=slave
[Install]
WantedBy=multi-user.target
# /etc/systemd/system/docker.service.d/docker-thinpool.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool \
--storage-opt=dm.use_deferred_removal=true --storage-opt=dm.use_deferred_deletion=true
EOF
When I start Docker, it's running like this:
# ps -fed | grep docker
root 8436 1 0 19:13 ? 00:00:00 /usr/bin/dockerd-current --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt=dm.use_deferred_removal=true --storage-opt=dm.use_deferred_deletion=true
root 8439 8436 0 19:13 ? 00:00:00 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc
Here is the output of docker info:
# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 1
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 185.6 MB
Data Space Total: 1.015 GB
Data Space Available: 829.4 MB
Metadata Space Used: 77.82 kB
Metadata Space Total: 8.389 MB
Metadata Space Available: 8.311 MB
Thin Pool Minimum Free Space: 101.2 MB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null bridge overlay host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-514.16.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 1
Total Memory: 992.7 MiB
Name: master
ID: 6CFR:H7SN:MEU7:PNJH:UMSO:6MNE:43Q5:SF4K:Z25I:BKHP:53U4:63SO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
127.0.0.0/8
Registries: docker.io (secure)
How can I make it run with systemd?
Thanks
SR
A solution that does not involve editing systemd units or drop-ins would be to create (or edit) the /etc/docker/daemon.json configuration file and to include the following:
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
After saving it, restart your docker service.
sudo systemctl restart docker
This solution obviously is only feasible if you would want to apply this system-wide.
Since I have two configuration file I need to add the entry in the second config file also -- /etc/systemd/system/docker.service.d/docker-thinpool.conf:
--exec-opt native.cgroupdriver=systemd \
Just to add, cgroupfs is dockers own control group manager. However, for the majority of Linux distributions ssytemd is the default init system now and systemd has tight integration with Linux control groups and In Kubernetes site, they recommend using systemd (see below) as using cgroupfs along with systemd seems to be non-optimal
So it is better to use systemd then for cgroup managment. kubelet is configured by default to use systemd. So it is easier and better to change Docker to use the systemd Cgroup driver
A history of this overlap is here https://lwn.net/Articles/676831/
In Kubernetes site, they recommend using systemd https://kubernetes.io/docs/setup/production-environment/container-runtimes/
Cgroup drivers When systemd is chosen as the init system for a Linux
distribution, the init process generates and consumes a root control
group (cgroup) and acts as a cgroup manager. Systemd has a tight
integration with cgroups and will allocate cgroups per process. It’s
possible to configure your container runtime and the kubelet to use
cgroupfs. Using cgroupfs alongside systemd means that there will then
be two different cgroup managers.
Control groups are used to constrain resources that are allocated to
processes. A single cgroup manager will simplify the view of what
resources are being allocated and will by default have a more
consistent view of the available and in-use resources. When we have
two managers we end up with two views of those resources. We have seen
cases in the field where nodes that are configured to use cgroupfs for
the kubelet and Docker, and systemd for the rest of the processes
running on the node becomes unstable under resource pressure.
OS: Centos 7.4 As kubernetes 1.23.1 recommend to use cgroup systemd, and docker 20.10.20 use cgroup cgroupfs. So, you have to change docker service file.
step1: Stop docker service
systemctl stop docker
step2: change on files /etc/systemd/system/multi-user.target.wants/docker.service and /usr/lib/systemd/system/docker.service
From :
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
TO:
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd
step3: start docker service and kubelet
systemctl start docker
kubeadm init phase kubelet-start
Make sure you are logged in as root and execute the below two commands :
echo '{"exec-opts": ["native.cgroupdriver=systemd"]}' >> /etc/docker/daemon.json
systemctl restart docker
Try to restart the docker service:
systemctl daemon-reload
systemctl restart docker.service
[root#host ~]# docker run 9e7de9390856
Timestamp: 2015-06-15 22:20:58.8367035 +1000 AEST
Code: System error
Message: [/usr/bin/tar -xf /var/lib/docker/tmp/cde0f3a199597ac2e18e7efc7744c84a6c134adef31fb88b6982a8732f45efa5090033894/_tmp.tar -C /var/lib/docker/devicemapper/mnt/cde0f3a199597ac2e18e7efc7744c84a6c134adef31fb88b6982a8732f45efa5/rootfs/tmp .] failed: /usr/bin/tar: ./was/fixPack/7.0.0-WS-WASSDK-LinuxX64-FP0000027.pak: Wrote only 4608 of 10240 bytes
/usr/bin/tar: ./was/fixPack/wasFixPackInstallResponseFile: Cannot write: No space left on device
.
.
Cannot write: No spaFATA[0141] Error response from daemon: : exit status 2
df -h:
Filesystem Size Used Avail Use% Mounted on
/dev/xvda2 6.0G 3.2G 2.9G 52% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.8G 0 1.8G 0% /dev/shm
tmpfs 1.8G 17M 1.8G 1% /run
tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup
/dev/xvdb1 99G 28G 67G 30% /var/lib/docker
docker info:
Containers: 2
Images: 34
Storage Driver: devicemapper
Pool Name: docker-202:17-2621441-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 15.89 GB
Data Space Total: 107.4 GB
Data Space Available: 76.3 GB
Metadata Space Used: 10.27 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.137 GB
Udev Sync Supported: true
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.1 (Maipo)
CPUs: 2
Total Memory: 3.452 GiB
Name: ip-10-100-128-182.localdomain
ID: 4ZZZ:BSQD:GBKL:4Y3N:J6BL:47QE:3HMQ:GLMY:FPUK:CEPM:3EBP:ZU7G
Debug mode (server): true
Debug mode (client): false
Fds: 13
Goroutines: 18
System Time: Mon Jun 15 22:48:24 AEST 2015
EventsListeners: 0
Init SHA1: 836be3a369bfc6bd4cbd3ade1eedbafcc1ea05d0
Init Path: /usr/libexec/docker/dockerinit
Docker Root Dir: /var/lib/docker
uname -a:
Linux ip-10-100-128-182.localdomain 3.10.0-229.el7.x86_64 #1 SMP Thu Jan 29 18:37:38 EST 2015 x86_64 x86_64 x86_64 GNU/Linux
Anyone can help me?
Not sure this information is enough. But tried couple of solutions, nothing worked.
docker version:
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 8aae715/1.6.0
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 8aae715/1.6.0
OS/Arch (server): linux/amd64
[root#host ~]# service docker status -l
Redirecting to /bin/systemctl status -l docker.service
docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled)
Active: active (running) since Tue 2015-06-16 00:31:46 AEST; 2min 2s ago
Docs: http://docs.docker.com
Main PID: 3306 (docker)
CGroup: /system.slice/docker.service
└─3306 /usr/bin/docker -d --storage-opt dm.basesize=30G --storage-opt dm.loopmetadatasize=4G
It sounds like you're trying to start a container from a 14GB image.
A Docker container, when using the devicemapper storage driver, only has 10GB of space available by default. You appear to be using the devicemapper driver, so this is probably the source of your problem.
This article discusses in detail the process you need to use to increase the amount of space available for container filesystems.
Filesystem-based drivers (like the overlay driver) to not have this same limitation (but they may of course suffer from other limitations).