My host machine OS is OEL7 with kernel
Linux ispaaaems1 3.10.0-123.el7.x86_64 #1 SMP Wed Jul 9 18:59:11 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux
And my docker info is
Containers: 4
Images: 124
Storage Driver: devicemapper
Pool Name: docker-253:0-88356-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 7.43 GB
Data Space Total: 107.4 GB
Data Space Available: 99.94 GB
Metadata Space Used: 9.302 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.138 GB
Udev Sync Supported: true
Library Version: 1.02.107-RHEL7 (2015-12-01)
Execution Driver: native-0.2
Kernel Version: 3.10.0-123.el7.x86_64
Operating System: Oracle Linux Server 7.2
CPUs: 2
Total Memory: 7.641 GiB
Name: ispaaaems1
ID: 6MUK:HS3D:OQTS:QMWY:WCKE:AZT6:COJP:F7EA:RPNX:7RHY:TKFB:D4LT
I am running a docker container with OS OEL6.6. I am sending a radius request at 1812-1813. All the packets are reaching the host machine, but few packets (3 out of 5) are getting dropped (not reaching inside the container).
Any help will be appreciated. Thanks in adavance.
Related
I have the same issue as in space issue on docker devmapper and CentOS7
It only specifies to clean up but not how I can increase the space and I dont have any images to clean. I tried several things with dm.min_free_space but nothing worked and want to increase the space.
OS Version/build: Red Hat Enterprise Linux Server release 7.3 (Maipo)
App version:
Client:
Version: 1.12.6
API version: 1.24
Package version: docker-common-1.12.6-11.el7.centos.x86_64
Go version: go1.7.4
Git commit: 96d83a5/1.12.6
Built: Tue Mar 7 09:23:34 2017
OS/Arch: linux/amd64
Server:
Version: 1.12.6
API version: 1.24
Package version: docker-common-1.12.6-11.el7.centos.x86_64
Go version: go1.7.4
Git commit: 96d83a5/1.12.6
Built: Tue Mar 7 09:23:34 2017
OS/Arch: linux/amd64
Steps to reproduce
I have no containers running currently and have some docker images pertaining to Kubernetes which will be used by the Kubernetes service.
sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[kubeuser4#kubenode4 Employee]$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/busybox latest 00f017a8c2a6 5 days ago 1.11 MB
registry.access.redhat.com/rhel7/pod-infrastructure latest 34d3450d733b 6 weeks ago 205 MB
docker.io/java 8 d23bdf5b1b1b 8 weeks ago 643.1 MB
gcr.io/google_containers/heapster_grafana v2.6.0-2 b43443930626 12 months ago 230 MB
When I try to create a docker image of my application that needs to be used, I get the below error.
devmapper: Thin Pool has 8783 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior
I tried the cleaning up as mentioned in the other forums, but not helped much and getting the same error. When I tried to run with this sudo docker --storage-opt dm.min_free_space=0%, seems like it starts as a daemon, but still it failed with another error "docker-runc not installed on system" and also I dont want to run it as a daemon.
Below are some command outputs
sudo dmsetup status
localvg00-lv_home: 0 20971520 linear
localvg00-lv_home: 20971520 20971520 linear
docker-251:5-134039-pool: 0 209715200 thin-pool 924 848/524288 1629226/1638400 - rw discard_passdown queue_if_no_space
localvg00-lv_tmp: 0 4194304 linear
localvg00-lv_swap: 0 8388608 linear
localvg00-lv_root: 0 2097152 linear
localvg00-lv_root: 2097152 20971520 linear
localvg00-lv_usr: 0 16777216 linear
localvg00-lv_var: 0 8388608 linear
localvg00-lv_var: 8388608 62914560 linear
sudo docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 4
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-251:5-134039-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 106.8 GB
Data Space Total: 107.4 GB
Data Space Available: 601.2 MB
Metadata Space Used: 3.473 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.144 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: overlay null bridge host
Swarm: inactive
Runtimes: runc docker-runc
Default Runtime: docker-runc
Security Options: seccomp
Kernel Version: 4.1.12-61.1.28.el7uek.x86_64
Operating System: Oracle Linux Server 7.3
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 2
Total Memory: 7.545 GiB
Name: kubenode4
I had also tried increasing all the physical volume size and logical volume size(lv_var) on my linux machine, but still it doesnt work.
sudo lvs
[sudo] password for kubeuser4:
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv_home localvg00 -wi-ao---- 20.00g
lv_root localvg00 -wi-ao---- 11.00g
lv_swap localvg00 -wi-ao---- 4.00g
lv_tmp localvg00 -wi-ao---- 2.00g
lv_usr localvg00 -wi-ao---- 8.00g
lv_var localvg00 -wi-ao---- 34.00g
sudo ls -lsh /var/lib/docker/devicemapper/devicemapper/data
2.3G -rw------- 1 root root 100G Mar 14 22:16 /var/lib/docker/devicemapper/devicemapper/data
Someone please let me know how it can be done.
Thanks,
It is better move away from devicemapper for a few reasons.
devicemapper in loopback unrecoverable storage issue: https://github.com/docker/docker/issues/3182 "devicemapper not recommended for production use".
I found it easy enough to switch to overlay storage driver, YMMV of course but hopefully not too much. 'rm -rf /var/lib/docker' is somewhat optional when switching but easy and I would highly recommend it as long as you can load your images back in. http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-storage-drivers/
systemctl stop docker
rm -rf /var/lib/docker
# if these files do not already exist . . . create them, otherwise you need to edit by hand, you can also just add -s overlay in the systemctl docker script
ls /etc/sysconfig/docker /etc/sysconfig/docker-storage
[[ $? != 0 ]] && {
echo OPTIONS='--selinux-enabled=false' > /etc/sysconfig/docker
echo "DOCKER_STORAGE_OPTIONS= -s overlay" > /etc/sysconfig/docker-storage
}
systemctl start docker
systemctl status docker
docker images
more reading:
https://docs.docker.com/engine/userguide/storagedriver/selectadriver/
https://integratedcode.us/2016/08/30/storage-drivers-in-docker-a-deep-dive/
Was able to get it working and have mentioned it in
https://forums.docker.com/t/devmapper-space-issue/29786/3
I'm running into a problem with docker. I've got here OpenSuse 13.2 with a self-built version of libseccomp library. it's fresh version 2.3.1 from couple of weeks ago. If i'm running any docker container, i get the following error:
hostname:/usr/lib/docker # docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
78445dd45222: Pull complete
Digest: sha256:c5515758d4c5e1e838e9cd307f6c6a0d620b5e07e6f927b07d05f6d12a1ac8d7
Status: Downloaded newer image for hello-world:latest
container_linux.go:247: starting container process caused "conditional filtering requires libseccomp version >= 2.2.1"
docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "conditional filtering requires libseccomp version >= 2.2.1".
ERRO[0002] error getting events from daemon: net/http: request canceled
Of course i can use an option --security-opt seccomp:unconfined when starting a container, but this is not my purpose.
# rpm -qa libseccomp
libseccomp-2.3.1-1.x86_64
docker info:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 1
Server Version: 1.13.0
Storage Driver: devicemapper
Pool Name: docker-254:2-655361-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: ext4
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 307.2 MB
Data Space Total: 107.4 GB
Data Space Available: 20.64 GB
Metadata Space Used: 806.9 kB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.147 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.03.01 (2011-10-15)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: oci runc
Default Runtime: runc
Init Binary: docker-init
containerd version: (expected: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e)
runc version: N/A (expected: 2f7393a47307a16f8cee44a37b262e8b81021e3e)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 3.16.7-53-desktop
Operating System: openSUSE 13.2 (Harlequin) (x86_64)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.868 GiB
Name: hostname
ID: DCOH:JZMG:ZUTM:5MSB:DVAG:SQXS:Z36N:5OXU:GQII:YTMO:RWDA:HYBJ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
WARNING: No kernel memory limit support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Seems the problem may be with runc. I'm currently running into the same identical error, on Leap 42.1 with docker & runc from OBS Virtualization:containers repo. My setup was running fine until recent set of package updates.
i | runc | package | 0.1.1+gitr2942_2f7393a-33.2 | x86_64 | Virtualization:containers (openSUSE_Leap_42.1)
i | docker | package | 1.13.0-182.1 | x86_64 | Virtualization:containers (openSUSE_Leap_42.1)
strings on /usr/sbin/runc show:
strings /usr/sbin/runc | grep 2.2.1
[..]
conditional filtering requires libseccomp version >= 2.2.1
[..]
Going down further, changelog shows:
* Fri Feb 24 2017
- update to docker-1.13.0 requirement
* Mon Dec 19 2016
- update runc to the version used in docker 1.12.5 (bsc#1016307).
And the source for that package has Godeps/_workspace/src/github.com/seccomp/libseccomp-golang/seccomp_internal.go with this on line 299:
return fmt.Errorf("conditional filtering requires libseccomp version >= 2.2.1")
Looks like there is now an official bug report, and this issue impacts a few different SUSE releases that use that repo:
https://bugzilla.opensuse.org/show_bug.cgi?id=1028639
I am trying to start any of my saved containers in docker but am unable to do it. I have started getting the Error response from daemon:
Error running DeviceCreate (createSnapDevice) dm_task_run failed
This started happening after committing a relatively big docker image and it seemed to have filled up all available docker data space, even though I had lots of space on the host machine. Now I am unable to free up the docker data space anymore, even after deleting the big image. Docker is unable to reclaim the space. I also tried the fix mentioned below so that I can start the docker container but was not successful. Is there anything I can do to fix existing Docker to run images again?
Related question: Can't run Docker container due device mapper error
Here is my host configuration. Data Space used and total has reached max and free is in 0.
# docker info
=========================================================
Containers: 49
Images: 23
Storage Driver: devicemapper
Pool Name: docker-8:3-4998488-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 107.4 GB
Data Space Total: 107.4 GB
Data Space Available: 0 B
Metadata Space Used: 60.36 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.087 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: Red Hat Enterprise Linux
CPUs: 4
Total Memory: 7.64 GiB
docker version
=========================================================
Client:
Version: 1.8.2
API version: 1.20
Go version: go1.4.2
Git commit: 0a8c2e3
Built: Thu Sep 10 19:08:45 UTC 2015
OS/Arch: linux/amd64
Server:
Version: 1.8.2
API version: 1.20
Go version: go1.4.2
Git commit: 0a8c2e3
Built: Thu Sep 10 19:08:45 UTC 2015
OS/Arch: linux/amd64
running
docker system df
will tell you disk usage and reclaimable space.
To free up space and keep your images:
Remove Volumes:
docker volume rm $(docker volume ls --format {{.ID}})
Remove Stopped Containers:
docker rm $(docker ps -aq)
To Just clean everything
optional: Deletes All images/containers/volumes/networks
docker system prune -af
On the RHEL docker host, The only way I could reclaim the space and start a docker image was to actually remove the devicemapper/data. No other docker command to remove containers/images/volumes seems to work. I backed up the image I wanted to a tar ball and did the removal to have it working again.
The command docker build fails with error :
Error getting container f43128eda488c88a3b2e111aafb30b80a44faaead33bcf02f8bffd7ae1832753 from driver devicemapper: Error mounting '/dev/mapper/docker-8:2-41159178-f43128eda488c88a3b2e111aafb30b80a44faaead33bcf02f8bffd7ae1832753' on '/var/lib/docker/devicemapper/mnt/f43128eda488c88a3b2e111aafb30b80a44faaead33bcf02f8bffd7ae1832753': no such file or directory
docker info
Containers: 7
Images: 148
Storage Driver: devicemapper
Pool Name: docker-8:2-41159178-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 27.96 GB
Data Space Total: 107.4 GB
Data Space Available: 79.42 GB
Metadata Space Used: 19.27 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.128 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.77 (2012-10-15)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.14.27-xxxx-grs-ipv6-64
Operating System: Ubuntu 14.04.2 LTS
CPUs: 4
Total Memory: 15.66 GiB
Name: libra
ID: KYU6:KECQ:GGF3:QL4W:SB35:C3UX:54EY:NN3A:U4RF:SFIK:5ULD:2THZ
Username: porfus
Registry: https://index.docker.io/v1/
I use root-server OVH Kimsufi and Ubuntu 14.04.2 (3.14.27-xxxx-grs-ipv6-64 #1 SMP Wed Dec 17 14:02:42 CET 2014 x86_64 x86_64 x86_64 GNU/Linux). I read the description of this bug on GitHub (https://github.com/docker/docker/issues/4036), but did not understand how to get rid of it.
Considering you have "Library Version: 1.02.77 (2012-10-15)", consider upgrading docker and that library to its latest version.
That would make sure all the fixes mentioned in issues/4036 are taken into account.
[root#host ~]# docker run 9e7de9390856
Timestamp: 2015-06-15 22:20:58.8367035 +1000 AEST
Code: System error
Message: [/usr/bin/tar -xf /var/lib/docker/tmp/cde0f3a199597ac2e18e7efc7744c84a6c134adef31fb88b6982a8732f45efa5090033894/_tmp.tar -C /var/lib/docker/devicemapper/mnt/cde0f3a199597ac2e18e7efc7744c84a6c134adef31fb88b6982a8732f45efa5/rootfs/tmp .] failed: /usr/bin/tar: ./was/fixPack/7.0.0-WS-WASSDK-LinuxX64-FP0000027.pak: Wrote only 4608 of 10240 bytes
/usr/bin/tar: ./was/fixPack/wasFixPackInstallResponseFile: Cannot write: No space left on device
.
.
Cannot write: No spaFATA[0141] Error response from daemon: : exit status 2
df -h:
Filesystem Size Used Avail Use% Mounted on
/dev/xvda2 6.0G 3.2G 2.9G 52% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.8G 0 1.8G 0% /dev/shm
tmpfs 1.8G 17M 1.8G 1% /run
tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup
/dev/xvdb1 99G 28G 67G 30% /var/lib/docker
docker info:
Containers: 2
Images: 34
Storage Driver: devicemapper
Pool Name: docker-202:17-2621441-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 15.89 GB
Data Space Total: 107.4 GB
Data Space Available: 76.3 GB
Metadata Space Used: 10.27 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.137 GB
Udev Sync Supported: true
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.1 (Maipo)
CPUs: 2
Total Memory: 3.452 GiB
Name: ip-10-100-128-182.localdomain
ID: 4ZZZ:BSQD:GBKL:4Y3N:J6BL:47QE:3HMQ:GLMY:FPUK:CEPM:3EBP:ZU7G
Debug mode (server): true
Debug mode (client): false
Fds: 13
Goroutines: 18
System Time: Mon Jun 15 22:48:24 AEST 2015
EventsListeners: 0
Init SHA1: 836be3a369bfc6bd4cbd3ade1eedbafcc1ea05d0
Init Path: /usr/libexec/docker/dockerinit
Docker Root Dir: /var/lib/docker
uname -a:
Linux ip-10-100-128-182.localdomain 3.10.0-229.el7.x86_64 #1 SMP Thu Jan 29 18:37:38 EST 2015 x86_64 x86_64 x86_64 GNU/Linux
Anyone can help me?
Not sure this information is enough. But tried couple of solutions, nothing worked.
docker version:
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 8aae715/1.6.0
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 8aae715/1.6.0
OS/Arch (server): linux/amd64
[root#host ~]# service docker status -l
Redirecting to /bin/systemctl status -l docker.service
docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled)
Active: active (running) since Tue 2015-06-16 00:31:46 AEST; 2min 2s ago
Docs: http://docs.docker.com
Main PID: 3306 (docker)
CGroup: /system.slice/docker.service
└─3306 /usr/bin/docker -d --storage-opt dm.basesize=30G --storage-opt dm.loopmetadatasize=4G
It sounds like you're trying to start a container from a 14GB image.
A Docker container, when using the devicemapper storage driver, only has 10GB of space available by default. You appear to be using the devicemapper driver, so this is probably the source of your problem.
This article discusses in detail the process you need to use to increase the amount of space available for container filesystems.
Filesystem-based drivers (like the overlay driver) to not have this same limitation (but they may of course suffer from other limitations).