Same docker image works in Docker for Windows but not in docker on linux - linux

What I want to achieve
I want to connect to an oracle database within an oracle container via sqlplus
What my problem is
I can connect to the database of the same image within Docker for Windows but not on a linux host
What I did
Windows
c:\>docker pull wnameless/oracle-xe-11g
c:\>docker run -d -p 49160:22 -p 49161:1521 wnameless/oracle-xe-11g
c:\>docker exec -it f79f7aa5222c /bin/bash
root#f79f7aa5222c:/# sqlplus system/oracle
Result:
SQL*Plus: Release 11.2.0.2.0 Production on Wed Jul 19 07:28:45 2017
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
Linux
[xx#yy Docker]$ sudo docker pull wnameless/oracle-xe-11g
[xx#yy Docker]$ sudo docker run --shm-size=2g -d -p 49160:22 -p 49161:1521 wnameless/oracle-xe-11g
[xx#yy Docker]$ sudo docker exec -it cea28583cb1c /bin/bash
root#cea28583cb1c:/# sqlplus system/oracle
Result:
SQL*Plus: Release 11.2.0.2.0 Production on Wed Jul 19 07:34:16 2017
Copyright (c) 1982, 2011, Oracle. All rights reserved.
ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
Infos
Linux system:
docker info
[xx#yy Docker]$ sudo docker info
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 33
Server Version: 1.13.1
Storage Driver: devicemapper
Pool Name: docker-8:17-4460587-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 4.741 GB
Data Space Total: 107.4 GB
Data Space Available: 23.21 GB
Metadata Space Used: 4.289 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.143 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use --storage-opt dm.thinpooldev to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.137 (2016-11-30)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: oci runc
Default Runtime: oci
Init Binary: /usr/libexec/docker/docker-init-current
containerd version: (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: N/A (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
WARNING: You're not using the default seccomp profile
Profile: /etc/docker/seccomp.json
selinux
Kernel Version: 4.11.9-300.fc26.x86_64
Operating System: Fedora 26 (Workstation Edition)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 8
Total Memory: 7.776 GiB
Name: yy.domain.de
ID: 4YVE:4FQ7:27SF:5JZJ:H6RA:P4H5:TK6O:FAGY:CKHS:RQGQ:3BDS:L7W7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://registry.fedoraproject.org/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Registries: registry.fedoraproject.org (secure), registry.access.redhat.com (secure), docker.io (secure)
Docker logs
[xx#yy Docker]$ sudo docker logs cea28583cb1c
Starting Oracle Net Listener.
Starting Oracle Database 11g Express Edition instance.
/usr/sbin/startup.sh: ignoring /docker-entrypoint-initdb.d/*
df
root#cea28583cb1c:/# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/docker-8:17-4460587-f479be876efbf45834ea013ac119d63629944a2b315c7cbe7da3a22c1b1e566e 10G 1.9G 8.2G 19% /
tmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sdb1 110G 89G 17G 85% /etc/hosts
shm 2.0G 0 2.0G 0% /dev/shm
tmpfs 3.9G 0 3.9G 0% /sys/firmware
stat
root#cea28583cb1c:/# stat /
File: '/'
Size: 253 Blocks: 0 IO Block: 4096 directory
Device: fd01h/64769d Inode: 131 Links: 22
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2017-07-11 07:12:36.676904944 +0000
Modify: 2017-07-19 07:33:44.026249772 +0000
Change: 2017-07-19 07:33:44.216248030 +0000
Birth: -
lsnrctrl
xx#yy:/etc/selinux# lsnrctl status
LSNRCTL for Linux: Version 11.2.0.2.0 - Production on 19-JUL-2017 11:30:37
Copyright (c) 1991, 2011, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC_FOR_XE)))
STATUS of the LISTENER
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.2.0 - Production
Start Date 19-JUL-2017 07:33:46
Uptime 0 days 3 hr. 56 min. 51 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Default Service XE
Listener Parameter File /u01/app/oracle/product/11.2.0/xe/network/admin/listener.ora
Listener Log File /u01/app/oracle/diag/tnslsnr/cea28583cb1c/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC_FOR_XE)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cea28583cb1c)(PORT=1521)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
What I already tried
I tried both users, oracle and root.
I changed permissions of /u01/app/oracle to 755
I checked selinux, this seems not to be active/installed. At least thats what the docker info tells and I cant find sestatus command and /etc/selinux only contains semanage.conf.
I tried several different oracle images, same result in every container.
EDIT
As requested, I tried starting oracle service myself:
service oracle-xe start
Result of alert_XE.log:
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on.
IMODE=BR
ILAT =19
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production.
Using parameter settings in server-side spfile /u01/app/oracle/product/11.2.0/xe/dbs/spfileXE.ora
System parameters with non-default values:
sessions = 176
sga_target = 576M
control_files = "/u01/app/oracle/oradata/XE/control.dbf"
compatible = "11.2.0.0.0"
db_recovery_file_dest = "/u01/app/oracle/fast_recovery_area"
db_recovery_file_dest_size= 10G
undo_management = "AUTO"
undo_tablespace = "UNDOTBS1"
remote_login_passwordfile= "EXCLUSIVE"
dispatchers = "(PROTOCOL=TCP) (SERVICE=XEXDB)"
shared_servers = 4
job_queue_processes = 4
audit_file_dest = "/u01/app/oracle/admin/XE/adump"
db_name = "XE"
open_cursors = 300
pga_aggregate_target = 195840K
diagnostic_dest = "/u01/app/oracle"

Solved the puzzle, finally.
It turned out that SELinux was active. I don't know how I missed that some weeks ago. I definitely checked that, but maybe I was within the container and not the host.
So I ran the container with the --privileged flag and suddenly I can connect to the db.
Thanks for your comments and sorry that I posted it here, you are right #jww. I will make sure to only post programming questions here in the future. Sorry for that.

You can also edit /etc/sysconfig/docker and remove --selinux-enabled from options that's usually added by default. Then restart docker service:
sudo systemctl restart docker
or
sudo service docker restart

Related

No network access from within Docker container on CentOS 7

I can't access internet from withing a docker container on CentOS 7.
I tried almost everything I can find online, even disabling firewalld does not do the trick.
I can ping from the host but not from inside the container.
ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=247 time=12.7 ms
docker run -it --rm alpine ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
^C
--- 1.1.1.1 ping statistics ---
4 packets transmitted, 0 packets received, 100% packet loss
Here is the result from docker info.
Client:
Debug Mode: false
Server:
Containers: 27
Running: 0
Paused: 0
Stopped: 27
Images: 9
Server Version: 19.03.6
Storage Driver: devicemapper
Pool Name: docker-253:0-135106576-pool
Pool Blocksize: 65.54kB
Base Device Size: 10.74GB
Backing Filesystem: xfs
Udev Sync Supported: true
Data file: /dev/loop0
Metadata file: /dev/loop1
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 1.371GB
Data Space Total: 107.4GB
Data Space Available: 37.14GB
Metadata Space Used: 2.908MB
Metadata Space Total: 2.147GB
Metadata Space Available: 2.145GB
Thin Pool Minimum Free Space: 10.74GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.158-RHEL7 (2019-05-13)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.703GiB
Name: aaa.bbb.fr
ID: PFCS:N6PF:2M5Z:VFSI:IK66:RRYB:GP5U:UU5R:ICX2:YSHP:2ZMR:DULU
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: the devicemapper storage-driver is deprecated, and will be removed in a future release.
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
And from uname -a.
Linux xxx.aaa-bbb.fr 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Server name has been "anonymized".
You could try run the container in interactive mode:
docker run -it alpine /bin/sh
And use route command to check routing tables:
/ # route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 172.17.0.1 0.0.0.0 UG 0 0 0 eth0
172.17.0.0 * 255.255.0.0 U 0 0 0 eth0
In my case, the "default" route uses the 172.17.0.1 gateway to connect the container to internet as the traceroute command shows:
/ # traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 46 byte packets
1 172.17.0.1 (172.17.0.1) 0.016 ms 0.012 ms 0.011 ms
2 192.168.2.1 (192.168.2.1) 0.821 ms 1.035 ms 0.909 ms
3 192.168.15.1 (192.168.15.1) 5.828 ms 2.388 ms 1.385 ms
4 * * *
5 201-156-180-113.reservada.static.axtel.net (201.156.180.113) 4.354 ms 3.901 ms 3.768 ms
6 189-211-110-77.static.axtel.net (189.211.110.77) 3.159 ms 3.586 ms 2.791 ms
...
15 74.125.50.198 (74.125.50.198) 30.514 ms 209.85.172.106 (209.85.172.106) 32.289 ms 74.125.50.198 (74.125.50.198) 30.217 ms
16 108.170.240.129 (108.170.240.129) 31.972 ms 108.170.252.161 (108.170.252.161) 33.039 ms 31.773 ms
17 216.239.63.239 (216.239.63.239) 30.631 ms 209.85.243.255 (209.85.243.255) 31.615 ms 216.239.42.99 (216.239.42.99) 31.734 ms
18 dns.google (8.8.8.8) 31.240 ms 30.674 ms 30.885 ms

docker change cgroup driver to systemd

I want Docker to start with systemd cgroup driver. For some reason it is using only cgroupfs on my CentOS 7 server.
Here is startup config file.
# systemctl cat docker
# /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target
Wants=docker-storage-setup.service
Requires=docker-cleanup.timer
[Service]
Type=notify
NotifyAccess=all
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
Environment=DOCKER_HTTP_HOST_COMPAT=1
Environment=PATH=/usr/libexec/docker:/usr/bin:/usr/sbin
ExecStart=/usr/bin/dockerd-current \
--add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \
--default-runtime=docker-runc \
--exec-opt native.cgroupdriver=systemd \
--userland-proxy-path=/usr/libexec/docker/docker-proxy-current \
$OPTIONS \
$DOCKER_STORAGE_OPTIONS \
$DOCKER_NETWORK_OPTIONS \
$ADD_REGISTRY \
$BLOCK_REGISTRY \
$INSECURE_REGISTRY
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
Restart=on-abnormal
MountFlags=slave
[Install]
WantedBy=multi-user.target
# /etc/systemd/system/docker.service.d/docker-thinpool.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool \
--storage-opt=dm.use_deferred_removal=true --storage-opt=dm.use_deferred_deletion=true
EOF
When I start Docker, it's running like this:
# ps -fed | grep docker
root 8436 1 0 19:13 ? 00:00:00 /usr/bin/dockerd-current --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt=dm.use_deferred_removal=true --storage-opt=dm.use_deferred_deletion=true
root 8439 8436 0 19:13 ? 00:00:00 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc
Here is the output of docker info:
# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 1
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 185.6 MB
Data Space Total: 1.015 GB
Data Space Available: 829.4 MB
Metadata Space Used: 77.82 kB
Metadata Space Total: 8.389 MB
Metadata Space Available: 8.311 MB
Thin Pool Minimum Free Space: 101.2 MB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null bridge overlay host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-514.16.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 1
Total Memory: 992.7 MiB
Name: master
ID: 6CFR:H7SN:MEU7:PNJH:UMSO:6MNE:43Q5:SF4K:Z25I:BKHP:53U4:63SO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
127.0.0.0/8
Registries: docker.io (secure)
How can I make it run with systemd?
Thanks
SR
A solution that does not involve editing systemd units or drop-ins would be to create (or edit) the /etc/docker/daemon.json configuration file and to include the following:
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
After saving it, restart your docker service.
sudo systemctl restart docker
This solution obviously is only feasible if you would want to apply this system-wide.
Since I have two configuration file I need to add the entry in the second config file also -- /etc/systemd/system/docker.service.d/docker-thinpool.conf:
--exec-opt native.cgroupdriver=systemd \
Just to add, cgroupfs is dockers own control group manager. However, for the majority of Linux distributions ssytemd is the default init system now and systemd has tight integration with Linux control groups and In Kubernetes site, they recommend using systemd (see below) as using cgroupfs along with systemd seems to be non-optimal
So it is better to use systemd then for cgroup managment. kubelet is configured by default to use systemd. So it is easier and better to change Docker to use the systemd Cgroup driver
A history of this overlap is here https://lwn.net/Articles/676831/
In Kubernetes site, they recommend using systemd https://kubernetes.io/docs/setup/production-environment/container-runtimes/
Cgroup drivers When systemd is chosen as the init system for a Linux
distribution, the init process generates and consumes a root control
group (cgroup) and acts as a cgroup manager. Systemd has a tight
integration with cgroups and will allocate cgroups per process. It’s
possible to configure your container runtime and the kubelet to use
cgroupfs. Using cgroupfs alongside systemd means that there will then
be two different cgroup managers.
Control groups are used to constrain resources that are allocated to
processes. A single cgroup manager will simplify the view of what
resources are being allocated and will by default have a more
consistent view of the available and in-use resources. When we have
two managers we end up with two views of those resources. We have seen
cases in the field where nodes that are configured to use cgroupfs for
the kubelet and Docker, and systemd for the rest of the processes
running on the node becomes unstable under resource pressure.
OS: Centos 7.4 As kubernetes 1.23.1 recommend to use cgroup systemd, and docker 20.10.20 use cgroup cgroupfs. So, you have to change docker service file.
step1: Stop docker service
systemctl stop docker
step2: change on files /etc/systemd/system/multi-user.target.wants/docker.service and /usr/lib/systemd/system/docker.service
From :
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
TO:
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd
step3: start docker service and kubelet
systemctl start docker
kubeadm init phase kubelet-start
Make sure you are logged in as root and execute the below two commands :
echo '{"exec-opts": ["native.cgroupdriver=systemd"]}' >> /etc/docker/daemon.json
systemctl restart docker
Try to restart the docker service:
systemctl daemon-reload
systemctl restart docker.service

Docker Devmapper space issue - increase size

I have the same issue as in space issue on docker devmapper and CentOS7
It only specifies to clean up but not how I can increase the space and I dont have any images to clean. I tried several things with dm.min_free_space but nothing worked and want to increase the space.
OS Version/build: Red Hat Enterprise Linux Server release 7.3 (Maipo)
App version:
Client:
Version: 1.12.6
API version: 1.24
Package version: docker-common-1.12.6-11.el7.centos.x86_64
Go version: go1.7.4
Git commit: 96d83a5/1.12.6
Built: Tue Mar 7 09:23:34 2017
OS/Arch: linux/amd64
Server:
Version: 1.12.6
API version: 1.24
Package version: docker-common-1.12.6-11.el7.centos.x86_64
Go version: go1.7.4
Git commit: 96d83a5/1.12.6
Built: Tue Mar 7 09:23:34 2017
OS/Arch: linux/amd64
Steps to reproduce
I have no containers running currently and have some docker images pertaining to Kubernetes which will be used by the Kubernetes service.
sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[kubeuser4#kubenode4 Employee]$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/busybox latest 00f017a8c2a6 5 days ago 1.11 MB
registry.access.redhat.com/rhel7/pod-infrastructure latest 34d3450d733b 6 weeks ago 205 MB
docker.io/java 8 d23bdf5b1b1b 8 weeks ago 643.1 MB
gcr.io/google_containers/heapster_grafana v2.6.0-2 b43443930626 12 months ago 230 MB
When I try to create a docker image of my application that needs to be used, I get the below error.
devmapper: Thin Pool has 8783 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior
I tried the cleaning up as mentioned in the other forums, but not helped much and getting the same error. When I tried to run with this sudo docker --storage-opt dm.min_free_space=0%, seems like it starts as a daemon, but still it failed with another error "docker-runc not installed on system" and also I dont want to run it as a daemon.
Below are some command outputs
sudo dmsetup status
localvg00-lv_home: 0 20971520 linear
localvg00-lv_home: 20971520 20971520 linear
docker-251:5-134039-pool: 0 209715200 thin-pool 924 848/524288 1629226/1638400 - rw discard_passdown queue_if_no_space
localvg00-lv_tmp: 0 4194304 linear
localvg00-lv_swap: 0 8388608 linear
localvg00-lv_root: 0 2097152 linear
localvg00-lv_root: 2097152 20971520 linear
localvg00-lv_usr: 0 16777216 linear
localvg00-lv_var: 0 8388608 linear
localvg00-lv_var: 8388608 62914560 linear
sudo docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 4
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-251:5-134039-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 106.8 GB
Data Space Total: 107.4 GB
Data Space Available: 601.2 MB
Metadata Space Used: 3.473 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.144 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: overlay null bridge host
Swarm: inactive
Runtimes: runc docker-runc
Default Runtime: docker-runc
Security Options: seccomp
Kernel Version: 4.1.12-61.1.28.el7uek.x86_64
Operating System: Oracle Linux Server 7.3
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 2
Total Memory: 7.545 GiB
Name: kubenode4
I had also tried increasing all the physical volume size and logical volume size(lv_var) on my linux machine, but still it doesnt work.
sudo lvs
[sudo] password for kubeuser4:
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv_home localvg00 -wi-ao---- 20.00g
lv_root localvg00 -wi-ao---- 11.00g
lv_swap localvg00 -wi-ao---- 4.00g
lv_tmp localvg00 -wi-ao---- 2.00g
lv_usr localvg00 -wi-ao---- 8.00g
lv_var localvg00 -wi-ao---- 34.00g
sudo ls -lsh /var/lib/docker/devicemapper/devicemapper/data
2.3G -rw------- 1 root root 100G Mar 14 22:16 /var/lib/docker/devicemapper/devicemapper/data
Someone please let me know how it can be done.
Thanks,
It is better move away from devicemapper for a few reasons.
devicemapper in loopback unrecoverable storage issue: https://github.com/docker/docker/issues/3182 "devicemapper not recommended for production use".
I found it easy enough to switch to overlay storage driver, YMMV of course but hopefully not too much. 'rm -rf /var/lib/docker' is somewhat optional when switching but easy and I would highly recommend it as long as you can load your images back in. http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-storage-drivers/
systemctl stop docker
rm -rf /var/lib/docker
# if these files do not already exist . . . create them, otherwise you need to edit by hand, you can also just add -s overlay in the systemctl docker script
ls /etc/sysconfig/docker /etc/sysconfig/docker-storage
[[ $? != 0 ]] && {
echo OPTIONS='--selinux-enabled=false' > /etc/sysconfig/docker
echo "DOCKER_STORAGE_OPTIONS= -s overlay" > /etc/sysconfig/docker-storage
}
systemctl start docker
systemctl status docker
docker images
more reading:
https://docs.docker.com/engine/userguide/storagedriver/selectadriver/
https://integratedcode.us/2016/08/30/storage-drivers-in-docker-a-deep-dive/
Was able to get it working and have mentioned it in
https://forums.docker.com/t/devmapper-space-issue/29786/3

Docker instance cannot run images anymore and unable to reclaim free space

I am trying to start any of my saved containers in docker but am unable to do it. I have started getting the Error response from daemon:
Error running DeviceCreate (createSnapDevice) dm_task_run failed
This started happening after committing a relatively big docker image and it seemed to have filled up all available docker data space, even though I had lots of space on the host machine. Now I am unable to free up the docker data space anymore, even after deleting the big image. Docker is unable to reclaim the space. I also tried the fix mentioned below so that I can start the docker container but was not successful. Is there anything I can do to fix existing Docker to run images again?
Related question: Can't run Docker container due device mapper error
Here is my host configuration. Data Space used and total has reached max and free is in 0.
# docker info
=========================================================
Containers: 49
Images: 23
Storage Driver: devicemapper
Pool Name: docker-8:3-4998488-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 107.4 GB
Data Space Total: 107.4 GB
Data Space Available: 0 B
Metadata Space Used: 60.36 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.087 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: Red Hat Enterprise Linux
CPUs: 4
Total Memory: 7.64 GiB
docker version
=========================================================
Client:
Version: 1.8.2
API version: 1.20
Go version: go1.4.2
Git commit: 0a8c2e3
Built: Thu Sep 10 19:08:45 UTC 2015
OS/Arch: linux/amd64
Server:
Version: 1.8.2
API version: 1.20
Go version: go1.4.2
Git commit: 0a8c2e3
Built: Thu Sep 10 19:08:45 UTC 2015
OS/Arch: linux/amd64
running
docker system df
will tell you disk usage and reclaimable space.
To free up space and keep your images:
Remove Volumes:
docker volume rm $(docker volume ls --format {{.ID}})
Remove Stopped Containers:
docker rm $(docker ps -aq)
To Just clean everything
optional: Deletes All images/containers/volumes/networks
docker system prune -af
On the RHEL docker host, The only way I could reclaim the space and start a docker image was to actually remove the devicemapper/data. No other docker command to remove containers/images/volumes seems to work. I backed up the image I wanted to a tar ball and did the removal to have it working again.

Docker run, no space left on device

[root#host ~]# docker run 9e7de9390856
Timestamp: 2015-06-15 22:20:58.8367035 +1000 AEST
Code: System error
Message: [/usr/bin/tar -xf /var/lib/docker/tmp/cde0f3a199597ac2e18e7efc7744c84a6c134adef31fb88b6982a8732f45efa5090033894/_tmp.tar -C /var/lib/docker/devicemapper/mnt/cde0f3a199597ac2e18e7efc7744c84a6c134adef31fb88b6982a8732f45efa5/rootfs/tmp .] failed: /usr/bin/tar: ./was/fixPack/7.0.0-WS-WASSDK-LinuxX64-FP0000027.pak: Wrote only 4608 of 10240 bytes
/usr/bin/tar: ./was/fixPack/wasFixPackInstallResponseFile: Cannot write: No space left on device
.
.
Cannot write: No spaFATA[0141] Error response from daemon: : exit status 2
df -h:
Filesystem Size Used Avail Use% Mounted on
/dev/xvda2 6.0G 3.2G 2.9G 52% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.8G 0 1.8G 0% /dev/shm
tmpfs 1.8G 17M 1.8G 1% /run
tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup
/dev/xvdb1 99G 28G 67G 30% /var/lib/docker
docker info:
Containers: 2
Images: 34
Storage Driver: devicemapper
Pool Name: docker-202:17-2621441-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 15.89 GB
Data Space Total: 107.4 GB
Data Space Available: 76.3 GB
Metadata Space Used: 10.27 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.137 GB
Udev Sync Supported: true
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.1 (Maipo)
CPUs: 2
Total Memory: 3.452 GiB
Name: ip-10-100-128-182.localdomain
ID: 4ZZZ:BSQD:GBKL:4Y3N:J6BL:47QE:3HMQ:GLMY:FPUK:CEPM:3EBP:ZU7G
Debug mode (server): true
Debug mode (client): false
Fds: 13
Goroutines: 18
System Time: Mon Jun 15 22:48:24 AEST 2015
EventsListeners: 0
Init SHA1: 836be3a369bfc6bd4cbd3ade1eedbafcc1ea05d0
Init Path: /usr/libexec/docker/dockerinit
Docker Root Dir: /var/lib/docker
uname -a:
Linux ip-10-100-128-182.localdomain 3.10.0-229.el7.x86_64 #1 SMP Thu Jan 29 18:37:38 EST 2015 x86_64 x86_64 x86_64 GNU/Linux
Anyone can help me?
Not sure this information is enough. But tried couple of solutions, nothing worked.
docker version:
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 8aae715/1.6.0
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 8aae715/1.6.0
OS/Arch (server): linux/amd64
[root#host ~]# service docker status -l
Redirecting to /bin/systemctl status -l docker.service
docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled)
Active: active (running) since Tue 2015-06-16 00:31:46 AEST; 2min 2s ago
Docs: http://docs.docker.com
Main PID: 3306 (docker)
CGroup: /system.slice/docker.service
└─3306 /usr/bin/docker -d --storage-opt dm.basesize=30G --storage-opt dm.loopmetadatasize=4G
It sounds like you're trying to start a container from a 14GB image.
A Docker container, when using the devicemapper storage driver, only has 10GB of space available by default. You appear to be using the devicemapper driver, so this is probably the source of your problem.
This article discusses in detail the process you need to use to increase the amount of space available for container filesystems.
Filesystem-based drivers (like the overlay driver) to not have this same limitation (but they may of course suffer from other limitations).

Resources