Failed to connect to containerd: failed to dial - linux

Just installed Docker CE following official instructions with the repository in Ubuntu 14.04
Installation went successfully, the daemon is running
$ ps aux | grep docker
[...] /usr/bin/dockerd --raw-logs [...]
My user is in the docker group:
$ groups
[...] docker
The cli can't seem to communicate (same with sudo)
$ docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock.
Is the docker daemon running?
The socket seems to have the correct permissions:
$ ls -l /var/run/docker.sock
srw-rw---- 1 root docker 0 Feb 4 16:21 /var/run/docker.sock
The log seems to claim about some issues though
$ sudo tail -f /var/log/upstart/docker.log
Failed to connect to containerd: failed to dial "/var/run/docker/containerd/docker-containerd.sock": dial unix:///var/run/docker/containerd/docker-containerd.sock: timeout
/var/run/docker.sock is up
time="2018-02-04T16:22:21.031459040+01:00" level=info msg="libcontainerd: started new docker-containerd process" pid=17147
INFO[0000] starting containerd module=containerd revision=89623f28b87a6004d4b785663257362d1658a729 version=v1.0.0
INFO[0000] setting subreaper... module=containerd
containerd: invalid argument
time="2018-02-04T16:22:21.056685023+01:00" level=error msg="containerd did not exit successfully" error="exit status 1" module=libcontainerd
Any advice to make this work ?
Relog and Docker restart already done of course

As #bobbear suggested and is actually mentioned in the official doc one of the prerequisites is:
Version 3.10 or higher of the Linux kernel. The latest version of the kernel available for you platform is recommended.
After having checked my Kernel version:
$ uname -a
Linux [...] 3.2.[...]-generic [...]-Ubuntu [...] x86_64
I searched for candidates:
$ apt-cache search linux-image
And installed my new_kernel:
$ sudo apt-get install \
linux-image-new_kernel \
linux-headers-new_kernel \
linux-image-extra-new_kernel

Same situation happend on me. IS because your linux kernel version too low !!! check it use command "uname -r" , if the version below "3.10" (for example: debian 7 whezzy default version is 3.2 ) ,even you install docker-ce suceessfully, you will still can not start docker daemon success.That why! All most answers on the web tell you to 'restart' bla bla bla... but they did not consider this problem.

Related

Cannot connect to the Docker daemon After I update the linux kernel

The docker daemon worked fine before, but after I update the linux kernel version from 4.10 to 4.12, I found that the Docker daemon can't work:
When I run docker ps comes the error:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
and I run systemctl status docker.service to see the details, it outputs the error:
systemd[1]: Failed to start Docker Application Container Engine.
and the detail error from dockerd':
ERRO[0001] [graphdriver] prior storage driver aufs failed: driver not supported
Error starting daemon: error initializing graphdriver: driver not supported
OS: linux Ubuntu 16.04 LTS kernel: v4.12, docker version :Docker version 17.07.0-ce, build 8784753

Remote LLDB debugging - Docker container

I'm trying to set up a remote debugging with LLDB 4.0.1.
There's a docker (17.06.0-ce) container with Arch linux.
Docker container is set in privileged mode, so now LLDB can be started in container.
Container contains core_service which is Rust executable.
Commands run inside container
(lldb) target create target/debug/core_service
Current executable set to 'target/debug/core_service' (x86_64).
(lldb) process launch
Process 182 launched: '/srv/core_service/target/debug/core_service' (x86_64)
Problem exists with remote debugging, lldb-server is started inside container with lldb-server platform --server --listen 0.0.0.0:1234.
I can connect from host lldb to container lldb-server, but I can't attach/create processes.
Commands run on host (lldb-server in container = localhost:1234)
(lldb) platform select remote-linux
Platform: remote-linux
Connected: no
(lldb) platform connect connect://localhost:1234
Platform: remote-linux
Triple: x86_64-*-linux-gnu
OS Version: 4.12.4 (4.12.4-1-ARCH)
Kernel: #1 SMP PREEMPT Fri Jul 28 18:54:18 UTC 2017
Hostname: 099bd76c07c9
Connected: yes
WorkingDir: /srv/core_service
(lldb) target create target/debug/core_service
Current executable set to 'target/debug/core_service' (x86_64).
(lldb) process launch
error: connect remote failed (Connection refused)
error: process launch failed: Connection refused
How can I fix it? Are there any docker, arch linux settings that would cause this error?
It seems, like there's some problem with lldb-server permissions in docker container.
Commands run on host (lldb-server in container)
(lldb) platform shell ps -A
PID TTY TIME CMD
1 ? 00:00:00 bash
9 ? 00:00:00 nginx
10 ? 00:00:00 nginx
11 ? 00:00:00 lldb-server
25 ? 00:00:00 core_service
59 ? 00:00:00 lldb-server
68 ? 00:00:00 ps
(lldb) platform shell kill -9 25
(lldb) platform process launch target/debug/core_service
error: connect remote failed (Connection refused)
error: Connection refused
(lldb) platform process launch anything
error: connect remote failed (Connection refused)
error: Connection refused
But I can't figure out what can it be. lldb-server is run as root in container, I can execute shell commands using lldb.
There is needed both platform port (1234 in your case) and gdbserver port (randomly generated by default). You can enforce the gdbserver port by lldb-server option --gdbserver-port.
Tested on Fedora 29 x86_64:
docker run --privileged -p 5000:5000 -p 5001:5001 fedora bash -c 'dnf -y install lldb;lldb-server platform --server --listen 0.0.0.0:5000 --gdbserver-port 5001'
and
echo 'int main(){}' >main.c;gcc -g -o main main.c;lldb -o 'platform select remote-linux' -o 'platform connect connect://localhost:5000' -o "target create ./main" -o 'b main' -o 'process launch'
(lldb) process launch
Process 45 stopped
* thread #1, name = 'main', stop reason = breakpoint 1.1
frame #0: 0x000000000040110f main`main at main.c:1
-> 1 int main(){}
Process 45 launched: '/root/main' (x86_64)
(lldb) _
This may be because the server cannot see any process on the host. It is still wrapped in its own PID namespace. When you launch the LLDB server, use a host pid name space
docker run --pid=host --privileged <yourimage>
Hopefully this will allow your container see all the host processes

What starts this docker process on my laptop?

Every time I boot up my Lubuntu 16.04 laptop I can see I have a running docker container:
$ ps -ef | grep docker
root 1724 1 3 21:17 ? 00:01:30 /usr/bin/dockerd -H fd://
root 1774 1724 0 21:17 ? 00:00:04 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc
root 4750 1774 0 21:17 ? 00:00:00 docker-containerd-shim 72541a4648b890132985daf2357d1130b8b5208cf12ede607b93ab2987629719 /var/run/docker/libcontainerd/72541a4648b890132985daf2357d1130b8b5208cf12ede607b93ab2987629719 docker-runc
stephane 10755 1793 0 22:07 pts/0 00:00:00 grep docker
It serves a Jenkins application on the port 80 and requesting localhost/ in the browser redirects to http://localhost/login?from=%2F and shows a Jenkins warning page:
Unlock Jenkins
To ensure Jenkins is securely set up by the administrator, a password has been written to the log (not sure where to find it?) and this file on the server:
A wget request shows:
$ wget localhost/
--2017-05-23 22:09:55-- http://localhost/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2017-05-23 22:09:55 ERROR 403: Forbidden.
How can I know which service is firing up this docker process ?
I looked in the /etc/init.d/ directory:
$ l /etc/init.d/
alsa-utils* checkroot-bootclean.sh* halt* mattermostd* nginxd* rc* single* uuidd*
anacron* checkroot.sh* hostname.sh* mountall-bootclean.sh* ntp* rc.local* skeleton whoopsie*
apachedsd* console-setup* httpd* mountall.sh* ondemand* rcS* ssh* x11-common*
apparmor* cron* hwclock.sh* mountdevsubfs.sh* openvpn* README tomcatd*
apport* cups* irqbalance* mountkernfs.sh* php-fpm* reboot* udev*
avahi-daemon* cups-browsed* keyboardd* mountnfs-bootclean.sh* plymouth* redis* ufw*
bluetooth* dbus* killprocs* mountnfs.sh* plymouth-log* resolvconf* umountfs*
bootmisc.sh* docker* kmod* mysqld* postfix* rsync* umountnfs.sh*
cgroupfs-mount* dropboxd* lightdm* networking* pppd-dns* rsyslog* umountroot*
checkfs.sh* grub-common* mariadbd* network-manager* procps* sendsigs* urandom*
The /etc/init.d/docker is mine and removing it from the directory, a reboot still comes up with a running docker process.
I removed the /etc/init.d/docker file, rebooted, and there is a docker process:
$ ps -ef | grep docker
root 1560 1 5 22:15 ? 00:00:06 /usr/bin/dockerd -H fd://
root 1645 1560 0 22:15 ? 00:00:00 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc
root 4644 1645 0 22:15 ? 00:00:00 docker-containerd-shim 069db46cca05d43c35f05ff50aaa836507cbf69e4e3d9443b6b859d0edb5b076 /var/run/docker/libcontainerd/069db46cca05d43c35f05ff50aaa836507cbf69e4e3d9443b6b859d0edb5b076 docker-runc
stephane 5520 1741 0 22:17 pts/0 00:00:00 grep docker
So I looked up for anything docker in all these files, but found nothing named docker:
$ cd /etc/init.d/
[stephane#stephane-ThinkPad-X301 init.d]
$ grep.sh docker
[stephane#stephane-ThinkPad-X301 init.d]
This docker process is there every time I start my laptop, even when off line.
What starts this docker process ?
Lubuntu 16.04 comes with systemd by default. At some point you must have started up a jenkins instance in docker - it's hard to tell exactly what started the process initially. However, systemd would be what is currently causing it to start. In order to stop it from running, run the following commands:
systemctl status docker <- Find out of systemctl thinks docker is running.
It'll likely show something like this:
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2017-05-21 22:59:46 EDT; 1 day 17h ago
Docs: http://docs.docker.com
Main PID: 1314 (dockerd-current)
Tasks: 14 (limit: 8192)
CGroup: /system.slice/docker.service
└─1314 /usr/bin/dockerd-current --add-runtime oci=/usr/libexec/docker/docker-runc-current --default-runtime=oci --containerd /run/containerd.sock --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --selinux-enabled --log-driver=journald
To stop it, run systemctl stop docker and then systemctl disable docker. As a last resort if this doesn't work, you can run systemctl mask docker.
Docker is being started by systemd in your environment. You can disable the entire engine by running:
sudo systemctl disable docker
sudo systemctl stop docker
You can also stop only the container that is running (the shim and Jenkins application):
sudo docker ps # lists the running containers along with their container id
sudo docker update --restart=no $container_id
sudo docker stop $container_id
If you know that you do not need this container and want to permanently delete it, you can run this instead of the above two last commands:
sudo docker rm -f $container_id
The -f switch also stops the container if it's currently running.
Edit: from your comment, your container is running under swarm mode which is redeploying it. To stop that first find the stack or service that is running it.
sudo docker stack ls
sudo docker service ls
If you see a stack listed, you can remove that with:
sudo docker stack rm $stack_name
If there are no stacks listed, or they don't apply to this container, you can delete the service with:
sudo docker service rm $service_name

Unable to start Docker Service in Ubuntu 16.04

I've been trying to use Docker (1.10) on Ubuntu 16.04 but installation fails because Docker Service doesn't start.
I've already tried to install docker by docker.io, docker-engine apt packages and curl -sSL https://get.docker.com/ | sh but it doesn't work.
My Host info is:
Linux Xenial 4.5.3-040503-generic #201605041831 SMP Wed May 4 22:33:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Here is systemctl status docker.service:
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since sáb 2016-05-14 15:17:31 CEST; 12min ago
Docs: https://docs.docker.com
Process: 22479 ExecStart=/usr/bin/docker daemon -H fd:// (code=exited, status=1/FAILURE)
Main PID: 22479 (code=exited, status=1/FAILURE)
may 14 15:17:30 Xenial docker[22479]: time="2016-05-14T15:17:30.103601523+02:00" level=info msg="New containerd process, pid: 22485\n"
may 14 15:17:31 Xenial docker[22479]: time="2016-05-14T15:17:31.149064723+02:00" level=error msg="devmapper: Unable to delete device: devicemapper: Can't set task name /dev/mapper/docker-8:6-2101297-pool"
may 14 15:17:31 Xenial docker[22479]: time="2016-05-14T15:17:31.149127439+02:00" level=warning msg="devmapper: Usage of loopback devices is strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or use `man docker` to refer to dm.thinpooldev section."
may 14 15:17:31 Xenial docker[22479]: time="2016-05-14T15:17:31.153010028+02:00" level=error msg="[graphdriver] prior storage driver \"devicemapper\" failed: devicemapper: Can't set task name /dev/mapper/docker-8:6-2101297-pool"
may 14 15:17:31 Xenial docker[22479]: time="2016-05-14T15:17:31.153130839+02:00" level=fatal msg="Error starting daemon: error initializing graphdriver: devicemapper: Can't set task name /dev/mapper/docker-8:6-2101297-pool"
may 14 15:17:31 Xenial systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
may 14 15:17:31 Xenial docker[22479]: time="2016-05-14T15:17:31+02:00" level=info msg="stopping containerd after receiving terminated"
may 14 15:17:31 Xenial systemd[1]: Failed to start Docker Application Container Engine.
may 14 15:17:31 Xenial systemd[1]: docker.service: Unit entered failed state.
may 14 15:17:31 Xenial systemd[1]: docker.service: Failed with result 'exit-code'.
Here is sudo docker daemon -D
DEBU[0000] docker group found. gid: 999
DEBU[0000] Listener created for HTTP on unix (/var/run/docker.sock)
INFO[0000] previous instance of containerd still alive (23050)
DEBU[0000] containerd connection state change: CONNECTING
DEBU[0000] Using default logging driver json-file
DEBU[0000] Golang's threads limit set to 55980
DEBU[0000] received past containerd event: &types.Event{Type:"live", Id:"", Status:0x0, Pid:"", Timestamp:0x57372cae}
DEBU[0000] containerd connection state change: READY
DEBU[0000] devicemapper: driver version is 4.34.0
DEBU[0000] devmapper: Generated prefix: docker-8:6-2101297
DEBU[0000] devmapper: Checking for existence of the pool docker-8:6-2101297-pool
DEBU[0000] devmapper: poolDataMajMin=7:0 poolMetaMajMin=7:1
DEBU[0000] devmapper: Major:Minor for device: /dev/loop0 is:7:0
DEBU[0000] devmapper: Major:Minor for device: /dev/loop1 is:7:1
DEBU[0000] devmapper: loadDeviceFilesOnStart()
DEBU[0000] devmapper: Skipping file /var/lib/docker/devicemapper/metadata/transaction-metadata
DEBU[0000] devmapper: loadDeviceFilesOnStart() END
DEBU[0000] devmapper: constructDeviceIDMap()
DEBU[0000] devmapper: constructDeviceIDMap() END
DEBU[0000] devmapper: Rolling back open transaction: TransactionID=1 hash= device_id=1
ERRO[0000] devmapper: Unable to delete device: devicemapper: Can't set task name /dev/mapper/docker-8:6-2101297-pool
WARN[0000] devmapper: Usage of loopback devices is strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or use `man docker` to refer to dm.thinpooldev section.
DEBU[0000] devmapper: Initializing base device-mapper thin volume
DEBU[0000] devicemapper: CreateDevice(poolName=/dev/mapper/docker-8:6-2101297-pool, deviceID=1)
DEBU[0000] devmapper: Error creating device: devicemapper: Can't set task name /dev/mapper/docker-8:6-2101297-pool
DEBU[0000] devmapper: Error device setupBaseImage: devicemapper: Can't set task name /dev/mapper/docker-8:6-2101297-pool
ERRO[0000] [graphdriver] prior storage driver "devicemapper" failed: devicemapper: Can't set task name /dev/mapper/docker-8:6-2101297-pool
DEBU[0000] Cleaning up old mountid : start.
FATA[0000] Error starting daemon: error initializing graphdriver: devicemapper: Can't set task name /dev/mapper/docker-8:6-2101297-pool
Here is ./check-config.sh output:
warning: /proc/config.gz does not exist, searching other paths for kernel config ...
info: reading kernel config from /boot/config-4.5.3-040503-generic ...
Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- apparmor: enabled and tools installed
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_DEVPTS_MULTIPLE_INSTANCES: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_MACVLAN: enabled (as module)
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled
Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_KMEM: missing
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: missing
(note that cgroup swap accounting is not enabled in your kernel config, you can enable it by setting boot option "swapaccount=1")
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CFQ_GROUP_IOSCHED: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_EXT3_FS: missing
- CONFIG_EXT3_FS_XATTR: missing
- CONFIG_EXT3_FS_POSIX_ACL: missing
- CONFIG_EXT3_FS_SECURITY: missing
(enable these ext3 configs if you are using ext3 as backing filesystem)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
- "overlay":
- CONFIG_VXLAN: enabled (as module)
- Storage Drivers:
- "aufs":
- CONFIG_AUFS_FS: missing
- "btrfs":
- CONFIG_BTRFS_FS: enabled (as module)
- "devicemapper":
- CONFIG_BLK_DEV_DM: enabled
- CONFIG_DM_THIN_PROVISIONING: enabled (as module)
- "overlay":
- CONFIG_OVERLAY_FS: enabled (as module)
- "zfs":
- /dev/zfs: missing
- zfs command: missing
- zpool command: missing
If someone could please help me I would be very thankful
Update
It seems that in newer versions of docker and Ubuntu the unit file for docker is simply masked (pointing to /dev/null).
You can verify it by running the following commands in the terminal:
sudo file /lib/systemd/system/docker.service
sudo file /lib/systemd/system/docker.socket
You should see that the unit file symlinks to /dev/null.
In this case, all you have to do is follow S34N's suggestion, and run:
sudo systemctl unmask docker.service
sudo systemctl unmask docker.socket
sudo systemctl start docker.service
sudo systemctl status docker
I'll also keep the original post, that answers the error log stating that the storage driver should be replaced:
Original Post
I had the same problem, and I tried fixing it with Salva Cort's suggestion, but printing /etc/default/docker says:
# THIS FILE DOES NOT APPLY TO SYSTEMD
So here's a permanent fix that works for systemd (Ubuntu 15.04 and higher):
create a new file /etc/systemd/system/docker.service.d/overlay.conf with the following content:
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// -s overlay
flush changes by executing:
sudo systemctl daemon-reload
verify that the configuration has been loaded:
systemctl show --property=ExecStart docker
restart docker:
sudo systemctl restart docker
The following unmasking commands worked for me (Ubuntu 18). Hope it helps someone out there... :-)
sudo systemctl unmask docker.service
sudo systemctl unmask docker.socket
sudo systemctl start docker.service
I had the same problem after upgrade docker from 17.05-ce to 17.06-ce via docker-machine
Update /etc/systemd/system/docker.service.d/10-machine.conf
replace
`docker daemon` => `dockerd`
example from
[Service]
ExecStart=
ExecStart=/usr/bin/docker deamon -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver aufs --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=generic
Environment=
to
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver aufs --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=generic
Environment=
flush changes by executing:
sudo systemctl daemon-reload
restart docker:
sudo systemctl restart docker
Well, finally I fixed it
Everything you have to do is to load a different storage-driver in my case I will use overlay:
Disable Docker service: sudo systemctl stop docker.service
Start Docker Daemon (overlay driver): sudo docker daemon -s overlay
Run Demo container: sudo docker run hello-world
In order to make these changes permanent, you must edit /etc/default/docker file and add the option:
DOCKER_OPTS="-s overlay"
Next time Docker service get loaded, it will run docker daemon -s overlay
I've been able to get it working after a kernel upgrade by following the directions in this blog.
https://mymemorysucks.wordpress.com/2016/03/31/docker-graphdriver-and-aufs-failed-driver-not-supported-error-after-ubuntu-upgrade/
sudo apt-get update
sudo apt-get install linux-image-extra-$(uname -r) linux-image-extra-virtual
sudo modprobe aufs
sudo service docker restart
After viewing some of the other answers it looks like the issue was that the service wasn't running with the -s overlay options.
I also happened to notice that docker tried to start up with ${DOCKER_OPTS} at the end of the call.
I was able to export DOCKER_OPTS="-s overlay" (bc by default DOCKER_OPTS was empty) and get docker running.
I had a similar issue on a new Docker installation (version 19.03.3-rc1) on Ubuntu 18.04.3 LTS. By default /etc/docker/daemon.json file does not exist on a new installation. Following a tutorial I changed the storage driver to devicemapper by creating a new daemon.json file. It worked but then I deleted the daemon.json file thinking that it would revert to the default but that did not work and the service would not start.
Creating the /etc/docker/daemon.json file again with the default storage driver fixed it for me.
{
"storage-driver": "overlay2"
}
sudo dockerd --debug will help to fix actual pain point I fixed the same error using this at ubuntu 20 LTS
As to me, I have get this error.
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
Finally I found, it the /etc/docker/daemon.json error, for I add registry-mirrors
{
"runtimes": {
"nvidia": {
"path": "/usr/bin/nvidia-container-runtime",
"runtimeArgs": []
}
}
# I forget to add a comma , here !!!!!!!
"registry-mirrors": ["https://docker.mirrors.ustc.edu.cn"]
}
After I add it , then systemctl restart docker, I solved it.
In my case I was getting the following error from journalctl -xe command
unable to configure the Docker daemon with file /etc/docker/daemon.json: invalid character 'â' looking for beginning of object key string
Just clean /etc/docker/daemon.json with
{
}
I had this issue today after an upgrade to the ubuntu kernel and tried numerous solutions above. However the only one that worked (Ubuntu 16.04.6 LTS) was to remove (or rename) the folder: /var/lib/docker
Please be aware, this will remove all your docker images, containers and volumes etc. So understand the implications before applying or take a backup!
There are more details here:
https://github.com/docker/for-linux/issues/162

How to identify orphaned veth interfaces and how to delete them?

When I start any container by docker run, we get a new veth interface. After deleting container, veth interface which was linked with container should be removed. However, sometimes it's fail ( oftern then container started with errors):
root#hostname /home # ifconfig | grep veth | wc -l
53
root#hostname /home # docker run -d -P axibase/atsd -name axibase-atsd-
28381035d1ae2800dea51474c4dee9525f56c2347b1583f56131d8a23451a84e
Error response from daemon: Cannot start container 28381035d1ae2800dea51474c4dee9525f56c2347b1583f56131d8a23451a84e: iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 33359 -j DNAT --to-destination 172.17.2.136:8883 ! -i docker0: iptables: No chain/target/match by that name.
(exit status 1)
root#hostname /home # ifconfig | grep veth | wc -l
55
root#hostname /home # docker rm -f 2838
2838
root#hostname /home # ifconfig | grep veth | wc -l
55
How I can identify which interfaces are linked with existing containers, and how I can remove extra interface which was linked with removed contrainers?
This way doesn't work (by root):
ifconfig veth55d245e down
brctl delbr veth55d245e
can't delete bridge veth55d245e: Operation not permitted
Extra interfaces now defined by transmitted traffic (if there are no activity, it's extra interface).
UPDATE
root#hostname ~ # uname -a
Linux hostname 3.13.0-53-generic #89-Ubuntu SMP Wed May 20 10:34:39 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root#hostname ~ # docker info
Containers: 10
Images: 273
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 502
Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-53-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 8
Total Memory: 47.16 GiB
Name: hostname
ID: 3SQM:44OG:77HJ:GBAU:2OWZ:C5CN:UWDV:JHRZ:LM7L:FJUN:AGUQ:HFAL
WARNING: No swap limit support
root#hostname ~ # docker version
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64
There are three problems here:
Starting a single container should not increase the count of veth interfaces on your system by 2, because when Docker creates a veth pair, one end of the pair is isolated in the container namespace and is not visible from the host.
It looks like you're not able to start a container:
Error response from daemon: Cannot start container ...
Docker should be cleaning up the veth interfaces automatically.
These facts make me suspect that there is something fundamentally wrong in your environment. Can you update your question with details about what distribution you're using, which kernel version, and which Docker version?
How I can identify which interfaces are linked with existing containers, and how I can remove extra interface which was linked with removed contrainers?
With respect to manually deleting veth interfaces: A veth interface isn't a bridge, so of course you can't delete one with brctl.
To delete a veth interface:
# ip link delete <ifname>
Detecting "idle" interfaces is a thornier problem, because if you just look at traffic you're liable to accidentally delete something that was still in use but that just wasn't seeing much activity.
I think what you would actually want to look for are veth interfaces whose peer is also visible in the global network namespace. You can find the peer of a veth interface using these instructions, and then it would be a simple matter of seeing if that interface is visible, and then deleting one or the other (deleting a veth interface will also remove its peer).
Fixed by upgrade docker to last version.
New version:
root#hostname ~ # docker version
Client:
Version: 1.8.1
API version: 1.20
Go version: go1.4.2
Git commit: d12ea79
Built: Thu Aug 13 02:35:49 UTC 2015
OS/Arch: linux/amd64
Server:
Version: 1.8.1
API version: 1.20
Go version: go1.4.2
Git commit: d12ea79
Built: Thu Aug 13 02:35:49 UTC 2015
OS/Arch: linux/amd64
Now interfaces remove together with containers. Old orphaned interfaces were deleted manually by following command:
# ip link delete <ifname>
Here is how you can delete them all together by pattern.
for name in $(ifconfig -a | sed 's/[ \t].*//;/^\(lo\|\)$/d' | grep veth)
do
echo $name
# ip link delete $name # uncomment this
done
In my case, all virtual ethernet network interface were created by Docker. For solving that, I've stopped all Docker services:
docker stop $(docker ps -q)
And the deleted all networks created by Docker:
docker network rm $(docker network ls -q)

Resources