I am running out of inodes on my Debian 10 machine
I found the culprits
~# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 500578 317 500261 1% /dev
tmpfs 504337 547 503790 1% /run
/dev/sda1 3276800 3090639 186161 95% /
tmpfs 504337 1 504336 1% /dev/shm
tmpfs 504337 3 504334 1% /run/lock
tmpfs 504337 17 504320 1% /sys/fs/cgroup
overlay 3276800 3090639 186161 95% /var/lib/docker/overlay2/c238a2f5bcd5e7e193d572fd4b331510cd2438d113a66e9125f983c929cd8485/merged
overlay 3276800 3090639 186161 95% /var/lib/docker/overlay2/5f3ee0ad57f147f8dba95e1e06ea98fc697c72ff8eaaafa6494faeb8f0742096/merged
tmpfs 504337 11 504326 1% /run/user/10976
tmpfs 504337 11 504326 1% /run/user/10762
shm 504337 1 504336 1% /var/lib/docker/containers/135852b9e95d22ace0cf10b27752a3ad15d1fdfaebcde74f546c26ae4ef01ad8/mounts/shm
shm 504337 1 504336 1% /var/lib/docker/containers/cd9619f7703b1e9fee7c1338379dab5270260d23834b27f278c57500b5e50b5a/mounts/shm
docker overlay /var/lib/docker/overlay2/*/merged
Only 2 containers are running.
no dangling images on the machine as well.
~# docker info
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 19
Server Version: 20.10.18
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
runc version: v1.1.4-0-g5fd4c4d
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.7.0-0.bpo.2-amd64
Operating System: Debian GNU/Linux 10 (buster)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.848GiB
Name: name
ID: SNF2:X4SU:OLKG:ESHV:PX3U:UJGB:MADV:OZ3R:UCZP:KONV:K4ZV:ILL4
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
what can I do to reclaim inodes?
PS: Docker version 18.09.0, build 4d60db4
docker system prune and docker image prune does not help either.
Related
We have deployed jenkins on docker container and recently we started seeing that our jenkins server is not coming up due to disk space issue. Below is the error we see in logs.
2022-09-17 21:41:32.567+0000 [id=32] INFO hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect V3LOCITY-SLAVE-02
/usr/local/bin/jenkins.sh: line 38: cannot create temp file for here-document: No space left on device
Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
Exception in thread "main" java.io.IOException: Jenkins has failed to create a temporary file in /tmp
at Main.extractFromJar(Main.java:498)
at Main._main(Main.java:310)
at Main.main(Main.java:151)
Caused by: java.io.IOException: No space left on device
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:2063)
at Main.extractFromJar(Main.java:495)
... 2 more
We assume issue with docker container running of out space, See below info for your reference.
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 1 1 572.5MB 0B (0%)
Containers 1 0 9.467GB 9.467GB (100%)
Local Volumes 0 0 0B 0B
Build Cache 0 0 0B 0B
Assuming container running of space we have increased it to 40 GB by adding below content in /etc/docker/daemon.json file and recreated the contained but still see the same issue after restart of container
{
"storage-driver": "devicemapper",
"storage-opts": [
"dm.basesize=40G"
]
}
See below docker info your reference.
Client:
Debug Mode: false
Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 1
Server Version: 19.03.11-ol
Storage Driver: devicemapper
Pool Name: docker-249:0-1140851221-pool
Pool Blocksize: 65.54kB
Base Device Size: 42.95GB
Backing Filesystem: xfs
Udev Sync Supported: true
Data file: /dev/loop0
Metadata file: /dev/loop1
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 10.82GB
Data Space Total: 107.4GB
Data Space Available: 96.56GB
Metadata Space Used: 6.877MB
Metadata Space Total: 2.147GB
Metadata Space Available: 2.141GB
Thin Pool Minimum Free Space: 10.74GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.170-RHEL7 (2020-03-24)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7eba5930496d9bbe375fdf71603e610ad737d2b2
runc version: 52de29d
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.1.12-124.65.1.2.el7uek.x86_64
Operating System: Oracle Linux Server 7.9
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 6.56GiB
Name: vm-app-docker-jenkinsqa
ID: TAII:OWLM:Y3BU:65DC:A3SK:SSJQ:H6H2:BLA2:HQA5:ODCP:Y7S5:KCJ2
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: the devicemapper storage-driver is deprecated, and will be removed in a future release.
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Registries:
You need to map jenkins home to an external folder (volume) and make sure the host has enough space.
See Jenkins docs for more details.
For example:
docker run --name jenkins -v /var/jenkins_home:/var/jenkins_home ...
I am running docker on a RHEL7.9 machine we hope to host webservices and a few other applications.
$ docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
scan: Docker Scan (Docker Inc., v0.8.0)
Server:
Containers: 22
Running: 22
Paused: 0
Stopped: 0
Images: 16
Server Version: 20.10.7
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux nvidia
Default Runtime: runc
Init Binary: docker-init
containerd version: 7eba5930496d9bbe375fdf71603e610ad737d2b2
runc version: v1.0.0-0-g84113ee
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-1160.24.1.el7.x86_64
Operating System: Red Hat Enterprise Linux
OSType: linux
Architecture: x86_64
CPUs: 80
Total Memory: 503.3GiB
Name: <not relevant>
ID: <not relevant>
Docker Root Dir: /var/lib/docker
Debug Mode: false
Username: <not relevant>
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: true
I have /var/lib/docker under it's own partition as part of security protocol. I did this after initial setup of the system.
$ grep '/var/lib/docker\s' /proc/mounts
/dev/mapper/afsys-var_lib_docker /var/lib/docker xfs rw,seclabel,relatime,attr2,inode64,sunit=512,swidth=512,noquota 0 0
$ mountpoint -- "$(docker info -f '{{ .DockerRootDir }}')"
/var/lib/docker is a mountpoint
I am unsure if things are configured correctly - specifically some of the overlay storage is showing up in separate mountpoints on filesystem. I'm unsure if this is expected.. or a byproduct of partitioning /var/lib/docker AFTER we setup the system and had previously built images/containers.
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
devtmpfs 263885104 0 263885104 0% /dev
tmpfs 263899860 0 263899860 0% /dev/shm
tmpfs 263899860 4181840 259718020 2% /run
tmpfs 263899860 0 263899860 0% /sys/fs/cgroup
/dev/mapper/sys-root 9763538944 135472276 9628066668 2% /
/dev/sdf1 972452 264664 707788 28% /boot
/dev/mapper/sys-maintenance 976087296 34336 976052960 1% /maintenance
/dev/mapper/sys-tmp 976087296 34472 976052824 1% /tmp
/dev/mapper/sys-var 976087296 54178732 921908564 6% /var
/dev/mapper/sys-var_lib_docker 524032000 62655660 461376340 12% /var/lib/docker
/dev/mapper/sys-var_log 976087296 2079404 974007892 1% /var/log
/dev/mapper/sys-var_log_audit 976087296 73968 976013328 1% /var/log/audit
/dev/mapper/sys-home 9763538944 36080988 9727457956 1% /home
tmpfs 52779976 0 52779976 0% /run/user/1001
tmpfs 52779976 0 52779976 0% /run/user/0
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/458fdb1acf9be0a10f3627ac8bffad5311542f6d66de976bed3f19b437f76d57/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/04015d24492d44b0b350a1b118904bbd620cb6554a4f10fb6000be1945b00e23/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/688ba6b06a96b2dbeb1602c91f36c69f4a2b55a731887c44b0d8ed496698099f/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/6cafdb8e46dd04a2b0bcc9982906f83ec706d8fe7980b62a20fbb45c7439be74/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/7d715bcebb32eb144166a48289816b7aad3247aff9a6289e78552f349ad32293/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/50beb5caa2817b62388fffe73cc736dbb80ef5553d5b881f6393316b22d3d415/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/0b5ce085bf279805aa3fb04329d1ff6c96c0ea487a81db0f6c62619b0ef12eab/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/7386a81809e579aac138c1e0449a32f23063258f5c4131df676deeb26924e5bb/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/f180488020c76514e0c4cf3ec651e31ac6b712d71e3dd066996c810f5c44cae6/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/e7aff65debb3b2200fe209b54e225419bf00f3d18e99caadde06249c67f70dce/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/3f5a54dae289b0169088e506229a5e75a54eb084a7e9eb7d191393bb0d922e1b/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/498b74db68c80bd88805bd4511c44c87624b00b53563250899fb821770a4c13c/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/e964f314751256feb5f0e2224d6306fabe500f4817bb5e2df2b9598f157032da/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/3ee10a1cb42e0028ef19072b878277f09c079440bdb9696d240ec7240aaf30f6/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/fc39cf63c7f11715ba366aa363b0bbe311109396bbad579d64cb8a86636f11f6/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/1dae92df5c219ca2fad777e8544101fce4c9d67da7004a1860ba3823b0e94f26/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/96450a2ec1c860f2b94d31347a8586a720bb72b4d75b30d716954f96bb3044a5/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/76a3e24abd07a441247d9ebd515c68001be8f146b1ed9d8e1ac9f03f290f6591/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/6cdf52c19bf11696c84190e4be40cc25ea553621670f142400f782324bda6d9a/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/c26d05d70bbf4e09900fc02b9a94e96b23b89c118f6a4b8eb840e22d9e2de34d/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/6426313243beafaa3059d43d7d6cb5c9954bdf9363012555dae59807657e58d5/merged
overlay 524032000 62655660 461376340 12% /var/lib/docker/overlay2/24d8c3c58b23f68c820bd624c8a7ec4902219ede1acdbb1336b055045e5d3c25/merged
Please forgive me if I am misinterpreting, but needed a sanity check and/or to be given advice on how to best configure so these overlays don't show up as separate mounts.
Why Docker uses overlayfs
Docker containers are composed of multiple layers. Docker needs to be able to efficiently combine layers, and add and remove those layers efficiently. To combine those layers, Docker uses a storage driver such as overlayfs or aufs.
These filesystems count as mounts, so they show up in tools such as mount or df.
I have /var/lib/docker under it's own partition as part of security protocol. I did this after initial setup of the system.
I believe Docker supports this. I see no reason why this wouldn't work. The only caveat I can think of is that if you had containers before creating this partition, then mounting that partition would shadow those containers, therefore making any containers created before the partition was created inaccessible.
Excluding overlay from df
If you want to avoid seeing these in the output of df, you can use this command:
df -x overlay
I am trying to install sentry docker on my linux .After clone it's repository:
git clone https://github.com/getsentry/onpremise
I run this
$ ./install.sh
but i got this error:
alt#mx-alt:/mnt/Software/Linux/sentry/onpremise
$ ./install.sh
Checking minimum requirements...
FAIL: Expected minimum RAM available to Docker to be 2400 MB but found MB
this is my docker info:
$ sudo docker info
Client:
Debug Mode: false
Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 1
Server Version: 19.03.13
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.19.0-12-amd64
Operating System: Debian GNU/Linux 10 (buster)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.63GiB
Name: mx-alt
ID: DRNU:OLX2:5VCT:GPNW:I3OV:4OHB:43UU:OVZL:OH5Y:5A2U:7MJA:SBHU
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
increase docker ram on linux?
I write my answer maybe help to someone!
1 - I pull busybox first:
$ docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
5f5dd3e95e9f: Pull complete
Digest: sha256:9f1c79411e054199210b4d489ae600a061595967adb643cd923f8515ad8123d2
Status: Downloaded newer image for busybox:latest
docker.io/library/busybox:latest
alt#mx-alt:~
alt#mx-alt:~
$ sudo docker run --rm busybox free -m 2
total used free shared buff/cache available
Mem: 16009 2176 11539 208 2293 13350
Swap: 8191 0 8191
alt#mx-alt:~
2- ./install.sh
Done.
Getting this error when run this command which was given by the web dashboard(changed the key to *)
sudo curl -Ls https://get.cloud.docker.com/ | sudo -H sh -s f74*************************
ERROR: Cannot detect Linux distribution or it's unsupported
Learn more: https://docs.docker.com/docker-cloud/tutorials/byoh/
while the docker info command gives the following output:
sudo docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.12.1
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 0
Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-37-generic
Operating System: Ubuntu 14.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.56 GiB
Name: G1
ID: WNIV:V2NZ:**********
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
Is docker cloud not supported on Ubuntu 14.04.1? Any tips on how to resolve this?
[root#host ~]# docker run 9e7de9390856
Timestamp: 2015-06-15 22:20:58.8367035 +1000 AEST
Code: System error
Message: [/usr/bin/tar -xf /var/lib/docker/tmp/cde0f3a199597ac2e18e7efc7744c84a6c134adef31fb88b6982a8732f45efa5090033894/_tmp.tar -C /var/lib/docker/devicemapper/mnt/cde0f3a199597ac2e18e7efc7744c84a6c134adef31fb88b6982a8732f45efa5/rootfs/tmp .] failed: /usr/bin/tar: ./was/fixPack/7.0.0-WS-WASSDK-LinuxX64-FP0000027.pak: Wrote only 4608 of 10240 bytes
/usr/bin/tar: ./was/fixPack/wasFixPackInstallResponseFile: Cannot write: No space left on device
.
.
Cannot write: No spaFATA[0141] Error response from daemon: : exit status 2
df -h:
Filesystem Size Used Avail Use% Mounted on
/dev/xvda2 6.0G 3.2G 2.9G 52% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.8G 0 1.8G 0% /dev/shm
tmpfs 1.8G 17M 1.8G 1% /run
tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup
/dev/xvdb1 99G 28G 67G 30% /var/lib/docker
docker info:
Containers: 2
Images: 34
Storage Driver: devicemapper
Pool Name: docker-202:17-2621441-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 15.89 GB
Data Space Total: 107.4 GB
Data Space Available: 76.3 GB
Metadata Space Used: 10.27 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.137 GB
Udev Sync Supported: true
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.1 (Maipo)
CPUs: 2
Total Memory: 3.452 GiB
Name: ip-10-100-128-182.localdomain
ID: 4ZZZ:BSQD:GBKL:4Y3N:J6BL:47QE:3HMQ:GLMY:FPUK:CEPM:3EBP:ZU7G
Debug mode (server): true
Debug mode (client): false
Fds: 13
Goroutines: 18
System Time: Mon Jun 15 22:48:24 AEST 2015
EventsListeners: 0
Init SHA1: 836be3a369bfc6bd4cbd3ade1eedbafcc1ea05d0
Init Path: /usr/libexec/docker/dockerinit
Docker Root Dir: /var/lib/docker
uname -a:
Linux ip-10-100-128-182.localdomain 3.10.0-229.el7.x86_64 #1 SMP Thu Jan 29 18:37:38 EST 2015 x86_64 x86_64 x86_64 GNU/Linux
Anyone can help me?
Not sure this information is enough. But tried couple of solutions, nothing worked.
docker version:
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 8aae715/1.6.0
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 8aae715/1.6.0
OS/Arch (server): linux/amd64
[root#host ~]# service docker status -l
Redirecting to /bin/systemctl status -l docker.service
docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled)
Active: active (running) since Tue 2015-06-16 00:31:46 AEST; 2min 2s ago
Docs: http://docs.docker.com
Main PID: 3306 (docker)
CGroup: /system.slice/docker.service
└─3306 /usr/bin/docker -d --storage-opt dm.basesize=30G --storage-opt dm.loopmetadatasize=4G
It sounds like you're trying to start a container from a 14GB image.
A Docker container, when using the devicemapper storage driver, only has 10GB of space available by default. You appear to be using the devicemapper driver, so this is probably the source of your problem.
This article discusses in detail the process you need to use to increase the amount of space available for container filesystems.
Filesystem-based drivers (like the overlay driver) to not have this same limitation (but they may of course suffer from other limitations).