docker daemon is not startable - linux

Question

Introduction
Today i had to move from my own virtual server to our companies linux server, where docker version 18.09.2 build 6247962 is installed. The server which provides me the error msgs is an ubuntu 18.04 lts server, my kernel: 4.4.0-042stab141.3
Unfortunately i can not talk to the old admin, because he left the company two years ago
What i did:
I need docker for my webcrawler. I’m not sure, if i already got the right image, so tried:
sudo docker pull scrapinghub/splash
After executing this command i got following error:
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
I also tried chmod 666 /usr/run/docker.sock, because this was a “solution” i read about on stackoverflow.
When i execute service docker status i get:
docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: failed (Result: start-limit-hit) since Fri 2020-10-23 11:46:30 CEST; 43min ago
Docs: https://docs.docker.com
Process: 7881 ExecStart=/usr/bin/dockerd -H fd:// (code=exited, status=1/FAILURE)
Main PID: 7881 (code=exited, status=1/FAILURE)
Also if i execute docker ps, to see my containers, i get:
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.39/containers/json: dial unix /var/run/docker.sock: connect: permission denied
when executing as root i get again:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
I know there are several topics about that error, but none of them helped me, so I am trying it now with my own error messages.
EDIT
systemctl status docker:
* docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: failed (Result: start-limit-hit) since Fri 2020-10-23 14:44:43 CEST; 21min ago
Docs: https://docs.docker.com
Process: 12126 ExecStart=/usr/bin/dockerd -H fd:// (code=exited, status=1/FAILURE)
Main PID: 12126 (code=exited, status=1/FAILURE)
dockerd
INFO[2020-10-23T15:16:41.905718923+02:00] libcontainerd: started new docker-containerd process pid=13326
INFO[2020-10-23T15:16:41.905847880+02:00] parsed scheme: "unix" module=grpc
INFO[2020-10-23T15:16:41.905859360+02:00] scheme "unix" not registered, fallback to default scheme module=grpc
INFO[2020-10-23T15:16:41.905903222+02:00] ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/docker-containerd.sock 0 <nil>}] module=grpc
INFO[2020-10-23T15:16:41.905918045+02:00] ClientConn switching balancer to "pick_first" module=grpc
INFO[2020-10-23T15:16:41.905969169+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc420601340, CONNECTING module=grpc
INFO[2020-10-23T15:16:41.935402002+02:00] starting containerd revision=9754871865f7fe2f4e74d43e2fc7ccd237edcbce version=docker-18.09.2
INFO[2020-10-23T15:16:41.935710291+02:00] loading plugin "io.containerd.content.v1.content"... type=io.containerd.content.v1
INFO[2020-10-23T15:16:41.935737945+02:00] loading plugin "io.containerd.snapshotter.v1.btrfs"... type=io.containerd.snapshotter.v1
WARN[2020-10-23T15:16:41.936336356+02:00] failed to load plugin io.containerd.snapshotter.v1.btrfs error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
INFO[2020-10-23T15:16:41.936354852+02:00] loading plugin "io.containerd.snapshotter.v1.aufs"... type=io.containerd.snapshotter.v1
WARN[2020-10-23T15:16:41.937768910+02:00] failed to load plugin io.containerd.snapshotter.v1.aufs error="modprobe aufs failed: "modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file '/lib/modules/4.4.0-042stab141.3/modules.builtin.bin'\nmodprobe: FATAL: Module aufs not found in directory /lib/modules/4.4.0-042stab141.3\n": exit status 1"
INFO[2020-10-23T15:16:41.937790278+02:00] loading plugin "io.containerd.snapshotter.v1.native"... type=io.containerd.snapshotter.v1
INFO[2020-10-23T15:16:41.937817682+02:00] loading plugin "io.containerd.snapshotter.v1.overlayfs"... type=io.containerd.snapshotter.v1
INFO[2020-10-23T15:16:41.937893773+02:00] loading plugin "io.containerd.snapshotter.v1.zfs"... type=io.containerd.snapshotter.v1
WARN[2020-10-23T15:16:41.938518477+02:00] failed to load plugin io.containerd.snapshotter.v1.zfs error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
INFO[2020-10-23T15:16:41.938534393+02:00] loading plugin "io.containerd.metadata.v1.bolt"... type=io.containerd.metadata.v1
WARN[2020-10-23T15:16:41.938549461+02:00] could not use snapshotter zfs in metadata plugin error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
WARN[2020-10-23T15:16:41.938557138+02:00] could not use snapshotter btrfs in metadata plugin error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
WARN[2020-10-23T15:16:41.938565014+02:00] could not use snapshotter aufs in metadata plugin error="modprobe aufs failed: "modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file '/lib/modules/4.4.0-042stab141.3/modules.builtin.bin'\nmodprobe: FATAL: Module aufs not found in directory /lib/modules/4.4.0-042stab141.3\n": exit status 1"
INFO[2020-10-23T15:16:41.938661276+02:00] loading plugin "io.containerd.differ.v1.walking"... type=io.containerd.differ.v1
INFO[2020-10-23T15:16:41.938679507+02:00] loading plugin "io.containerd.gc.v1.scheduler"... type=io.containerd.gc.v1
INFO[2020-10-23T15:16:41.938717611+02:00] loading plugin "io.containerd.service.v1.containers-service"... type=io.containerd.service.v1
INFO[2020-10-23T15:16:41.938730440+02:00] loading plugin "io.containerd.service.v1.content-service"... type=io.containerd.service.v1
INFO[2020-10-23T15:16:41.938744454+02:00] loading plugin "io.containerd.service.v1.diff-service"... type=io.containerd.service.v1
INFO[2020-10-23T15:16:41.938761110+02:00] loading plugin "io.containerd.service.v1.images-service"... type=io.containerd.service.v1
INFO[2020-10-23T15:16:41.938772688+02:00] loading plugin "io.containerd.service.v1.leases-service"... type=io.containerd.service.v1
INFO[2020-10-23T15:16:41.938784541+02:00] loading plugin "io.containerd.service.v1.namespaces-service"... type=io.containerd.service.v1
INFO[2020-10-23T15:16:41.938796278+02:00] loading plugin "io.containerd.service.v1.snapshots-service"... type=io.containerd.service.v1
INFO[2020-10-23T15:16:41.938807356+02:00] loading plugin "io.containerd.runtime.v1.linux"... type=io.containerd.runtime.v1
INFO[2020-10-23T15:16:41.938875766+02:00] loading plugin "io.containerd.runtime.v2.task"... type=io.containerd.runtime.v2
INFO[2020-10-23T15:16:41.938921789+02:00] loading plugin "io.containerd.monitor.v1.cgroups"... type=io.containerd.monitor.v1
INFO[2020-10-23T15:16:41.939301371+02:00] loading plugin "io.containerd.service.v1.tasks-service"... type=io.containerd.service.v1
INFO[2020-10-23T15:16:41.939328814+02:00] loading plugin "io.containerd.internal.v1.restart"... type=io.containerd.internal.v1
INFO[2020-10-23T15:16:41.939371309+02:00] loading plugin "io.containerd.grpc.v1.containers"... type=io.containerd.grpc.v1
INFO[2020-10-23T15:16:41.939383717+02:00] loading plugin "io.containerd.grpc.v1.content"... type=io.containerd.grpc.v1
INFO[2020-10-23T15:16:41.939395037+02:00] loading plugin "io.containerd.grpc.v1.diff"... type=io.containerd.grpc.v1
INFO[2020-10-23T15:16:41.939405310+02:00] loading plugin "io.containerd.grpc.v1.events"... type=io.containerd.grpc.v1
INFO[2020-10-23T15:16:41.939439881+02:00] loading plugin "io.containerd.grpc.v1.healthcheck"... type=io.containerd.grpc.v1
INFO[2020-10-23T15:16:41.939451270+02:00] loading plugin "io.containerd.grpc.v1.images"... type=io.containerd.grpc.v1
INFO[2020-10-23T15:16:41.939461540+02:00] loading plugin "io.containerd.grpc.v1.leases"... type=io.containerd.grpc.v1
INFO[2020-10-23T15:16:41.939477843+02:00] loading plugin "io.containerd.grpc.v1.namespaces"... type=io.containerd.grpc.v1
INFO[2020-10-23T15:16:41.939488529+02:00] loading plugin "io.containerd.internal.v1.opt"... type=io.containerd.internal.v1
INFO[2020-10-23T15:16:41.939543415+02:00] loading plugin "io.containerd.grpc.v1.snapshots"... type=io.containerd.grpc.v1
INFO[2020-10-23T15:16:41.939558389+02:00] loading plugin "io.containerd.grpc.v1.tasks"... type=io.containerd.grpc.v1
INFO[2020-10-23T15:16:41.939568889+02:00] loading plugin "io.containerd.grpc.v1.version"... type=io.containerd.grpc.v1
INFO[2020-10-23T15:16:41.939579026+02:00] loading plugin "io.containerd.grpc.v1.introspection"... type=io.containerd.grpc.v1
INFO[2020-10-23T15:16:41.939748944+02:00] serving... address="/var/run/docker/containerd/docker-containerd-debug.sock"
INFO[2020-10-23T15:16:41.939799183+02:00] serving... address="/var/run/docker/containerd/docker-containerd.sock"
INFO[2020-10-23T15:16:41.939810589+02:00] containerd successfully booted in 0.005619s
INFO[2020-10-23T15:16:41.946915966+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc420601340, READY module=grpc
INFO[2020-10-23T15:16:41.950501665+02:00] parsed scheme: "unix" module=grpc
INFO[2020-10-23T15:16:41.950522807+02:00] scheme "unix" not registered, fallback to default scheme module=grpc
INFO[2020-10-23T15:16:41.950564242+02:00] parsed scheme: "unix" module=grpc
INFO[2020-10-23T15:16:41.950575481+02:00] scheme "unix" not registered, fallback to default scheme module=grpc
INFO[2020-10-23T15:16:41.950676109+02:00] ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/docker-containerd.sock 0 <nil>}] module=grpc
INFO[2020-10-23T15:16:41.950697518+02:00] ClientConn switching balancer to "pick_first" module=grpc
INFO[2020-10-23T15:16:41.950746317+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc420601570, CONNECTING module=grpc
INFO[2020-10-23T15:16:41.950889604+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc420601570, READY module=grpc
INFO[2020-10-23T15:16:41.950962148+02:00] ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/docker-containerd.sock 0 <nil>}] module=grpc
INFO[2020-10-23T15:16:41.950978801+02:00] ClientConn switching balancer to "pick_first" module=grpc
INFO[2020-10-23T15:16:41.951015340+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc420601810, CONNECTING module=grpc
INFO[2020-10-23T15:16:41.951146523+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc420601810, READY module=grpc
ERRO[2020-10-23T15:16:41.952596133+02:00] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded. storage-driver=overlay2
ERRO[2020-10-23T15:16:41.953602275+02:00] AUFS was not found in /proc/filesystems storage-driver=aufs
ERRO[2020-10-23T15:16:41.954528718+02:00] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded. storage-driver=overlay
INFO[2020-10-23T15:16:41.956786508+02:00] Graph migration to content-addressability took 0.00 seconds
WARN[2020-10-23T15:16:41.958146162+02:00] Your kernel does not support oom control
WARN[2020-10-23T15:16:41.958167327+02:00] Your kernel does not support memory swappiness
WARN[2020-10-23T15:16:41.958182470+02:00] Your kernel does not support kernel memory limit
WARN[2020-10-23T15:16:41.958198841+02:00] Unable to find cpu cgroup in mounts
WARN[2020-10-23T15:16:41.958238330+02:00] Unable to find cpuset cgroup in mounts
WARN[2020-10-23T15:16:41.958309720+02:00] mountpoint for pids not found
INFO[2020-10-23T15:16:41.963433348+02:00] stopping event stream following graceful shutdown error="context canceled" module=libcontainerd namespace=plugins.moby
INFO[2020-10-23T15:16:41.963543844+02:00] stopping healthcheck following graceful shutdown module=libcontainerd
INFO[2020-10-23T15:16:41.967297616+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc420601810, TRANSIENT_FAILURE module=grpc
INFO[2020-10-23T15:16:41.967319558+02:00] pickfirstBalancer: HandleSubConnStateChange: 0xc420601810, CONNECTING module=grpc
Error starting daemon: Devices cgroup isn't mounted
-- Logs begin at Fri 2020-10-23 11:03:38 CEST. --
Oct 23 14:44:39 dockerd[12126]: time="2020-10-23T14:44:39.944764478+02:00" level=warning msg="Your kernel does not support memory swappiness"
Oct 23 14:44:39 dockerd[12126]: time="2020-10-23T14:44:39.944777538+02:00" level=warning msg="Your kernel does not support kernel memory limit"
Oct 23 14:44:39 dockerd[12126]: time="2020-10-23T14:44:39.944793390+02:00" level=warning msg="Unable to find cpu cgroup in mounts"
Oct 23 14:44:39 dockerd[12126]: time="2020-10-23T14:44:39.944824929+02:00" level=warning msg="Unable to find cpuset cgroup in mounts"
Oct 23 14:44:39 dockerd[12126]: time="2020-10-23T14:44:39.944869519+02:00" level=warning msg="mountpoint for pids not found"
Oct 23 14:44:39 dockerd[12126]: time="2020-10-23T14:44:39.947106531+02:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
Oct 23 14:44:39 dockerd[12126]: time="2020-10-23T14:44:39.947131964+02:00" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
Oct 23 14:44:39 dockerd[12126]: time="2020-10-23T14:44:39.951205781+02:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42401c7d0, TRANSIENT_FAILURE" module=grpc
Oct 23 14:44:39 dockerd[12126]: time="2020-10-23T14:44:39.951239041+02:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42401c7d0, CONNECTING" module=grpc
Oct 23 14:44:40 dockerd[12126]: Error starting daemon: Devices cgroup isn't mounted
Also if i try docker info i get following error:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?```

Answer 1

Since I didn't get a reply to my queries from the chat in the link provided https://chat.stackoverflow.com/rooms/223520/discussion-between-y-y-and-moritz-sauter
Therefore, my answer below has some assumptions.
It seems you have been using AUFS as the docker storage driver in your "virtual server", which can be confirmed from docker info or in the daemon.json file usually found in this path /etc/docker/.
Given the above is true, when you moved to the new "company Linux server" which has a fairly newer Ubuntu and kernel, therefore there is high possibility that AUFS driver is not enabled by default, you can verify it by grep aufs /proc/filesystems.
My introspection comes from your docker logs, it has a warning
WARN[2020-10-23T15:16:41.937768910+02:00] failed to load plugin io.containerd.snapshotter.v1.aufs error="modprobe aufs failed: "modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file '/lib/modules/4.4.0-042stab141.3/modules.builtin.bin'\nmodprobe: FATAL: Module aufs not found in directory /lib/modules/4.4.0-042stab141.3\n": exit status 1"
Now if all of the above is true how to resolve?
You will probably have to move away from AUFS to overlay2 check this answer. Unless you have strict reasons to stick to AUFS, in that case you will have to switch to a different kernel this answer may help
Note: if you using some non standard kernel or older kernel please do so at your own risk, since there maybe some vulnerabilities which you may have to patch back

Answer 2

As per all the docker command outputs, it looks like the docker-daemon is down.
You can try to start the docker-daemon using one of the follow commands.
$ sudo systemctl start docker
or
$ sudo service docker start
After starting that, your commands will start to work.
And if it is failing, you can reset the limit and restart.
sudo systemctl reset-failed docker.service