I was trying to load my cgroup config with cgconfigparser, but it gives the following error:
cgconfigparser; error loading /etc/cgconfig.conf: Cgroup is not mounted
Then I tried grep cgroup /proc/mounts, which gives:
cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime 0 0
I'm in a docker container based on centos7, did some googling and looks like my cgroup is using cgroup2 instead of cgroup1? If this is the case how do I switch back to cgorup1 in docker without tempering with the docker image? If not, what can I do to resolve this error?
Related
I am trying to setup AWS Greengrass core on Amazon Linux with respect to the documentation(followed all the prerequisites) of AWS(http://docs.aws.amazon.com/greengrass/latest/developerguide/start-core.html).
I am getting the below error on starting the Greengrass core daemon
./greengrassd start
Setting up greengrass daemon
Validating execution environment
Found cgroup subsystem: cpuset
Found cgroup subsystem: cpu
Found cgroup subsystem: cpuacct
Found cgroup subsystem: blkio
Found cgroup subsystem: memory
Found cgroup subsystem: devices
Found cgroup subsystem: freezer
Found cgroup subsystem: net_cls
Found cgroup subsystem: perf_event
Found cgroup subsystem: net_prio
Found cgroup subsystem: hugetlb
Found cgroup subsystem: pids
Starting greengrass daemon
Greengrass daemon 23933 failed to start
Failed to create overlay fs for container rootfs no such file or directory
I could not find any troubleshooting method for this and I am stuck in the middle of the process. It would be more helpful If any of you solved for me.
Have you tried executing the ./greengrassd start command using sudo ?
When I omit sudo I can an error that says this command needs to be run with using sudo e.g.
sudo ./greengrassd start
I am trying to start docker on a Centos-6-ish OS. It is failing for cgroups reasons. I believe the mount is correctly structured (docker recommends https://github.com/tianon/cgroupfs-mount/blob/master/cgroupfs-mount) so the final error message is unclear to me.
thrashin(bash):/base/data/tmp# ./cgroups-mount
thrashin(bash):/base/data/tmp# grep cgroup /proc/mounts
cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0
thrashin(bash):/base/data/tmp# cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 4 1 1
blkio 5 1 1
thrashin(bash):/base/data/tmp# dockerd &
[1] 7201
thrashin(bash):/base/data/tmp# WARN[0000] could not change group /var/run/docker.sock to docker: group docker not found
INFO[0000] libcontainerd: new containerd process, pid: 7214
WARN[0000] containerd: low RLIMIT_NOFILE changing to max current=1024 max=4096
WARN[0001] unable to modify root key limit, number of containers could be limited by this quota: open /proc/sys/kernel/keys/root_maxkeys: no such file or directory
INFO[0001] [graphdriver] using prior storage driver: overlay2
INFO[0001] Graph migration to content-addressability took 0.00 seconds
WARN[0001] Your kernel does not support cgroup memory limit
WARN[0001] Unable to find cpu cgroup in mounts
WARN[0001] Your kernel does not support cgroup blkio throttle.read_bps_device
WARN[0001] Your kernel does not support cgroup blkio throttle.write_bps_device
WARN[0001] Your kernel does not support cgroup blkio throttle.read_iops_device
WARN[0001] Your kernel does not support cgroup blkio throttle.write_iops_device
WARN[0001] mountpoint for pids not found
Error starting daemon: Devices cgroup isn't mounted
^C
[1]+ Exit 1 dockerd
cgroup device is mounted.
Is the failure due to the warning that the cpu subsystem is not provided? If so, how do I provide this? Is this a kernel build option?
Solved this by rebuilding the kernel to provide the cpu cgroup subsystem.
It's not immediately clear that that should be the solution to solving
Error starting daemon: Devices cgroup isn't mounted
Given that there are other cgroup warnings, aside from the warning about the cpu subsystem.
I want Docker to start with systemd cgroup driver. For some reason it is using only cgroupfs on my CentOS 7 server.
Here is startup config file.
# systemctl cat docker
# /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target
Wants=docker-storage-setup.service
Requires=docker-cleanup.timer
[Service]
Type=notify
NotifyAccess=all
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
Environment=DOCKER_HTTP_HOST_COMPAT=1
Environment=PATH=/usr/libexec/docker:/usr/bin:/usr/sbin
ExecStart=/usr/bin/dockerd-current \
--add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \
--default-runtime=docker-runc \
--exec-opt native.cgroupdriver=systemd \
--userland-proxy-path=/usr/libexec/docker/docker-proxy-current \
$OPTIONS \
$DOCKER_STORAGE_OPTIONS \
$DOCKER_NETWORK_OPTIONS \
$ADD_REGISTRY \
$BLOCK_REGISTRY \
$INSECURE_REGISTRY
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
Restart=on-abnormal
MountFlags=slave
[Install]
WantedBy=multi-user.target
# /etc/systemd/system/docker.service.d/docker-thinpool.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool \
--storage-opt=dm.use_deferred_removal=true --storage-opt=dm.use_deferred_deletion=true
EOF
When I start Docker, it's running like this:
# ps -fed | grep docker
root 8436 1 0 19:13 ? 00:00:00 /usr/bin/dockerd-current --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt=dm.use_deferred_removal=true --storage-opt=dm.use_deferred_deletion=true
root 8439 8436 0 19:13 ? 00:00:00 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc
Here is the output of docker info:
# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 1
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 185.6 MB
Data Space Total: 1.015 GB
Data Space Available: 829.4 MB
Metadata Space Used: 77.82 kB
Metadata Space Total: 8.389 MB
Metadata Space Available: 8.311 MB
Thin Pool Minimum Free Space: 101.2 MB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null bridge overlay host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-514.16.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 1
Total Memory: 992.7 MiB
Name: master
ID: 6CFR:H7SN:MEU7:PNJH:UMSO:6MNE:43Q5:SF4K:Z25I:BKHP:53U4:63SO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
127.0.0.0/8
Registries: docker.io (secure)
How can I make it run with systemd?
Thanks
SR
A solution that does not involve editing systemd units or drop-ins would be to create (or edit) the /etc/docker/daemon.json configuration file and to include the following:
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
After saving it, restart your docker service.
sudo systemctl restart docker
This solution obviously is only feasible if you would want to apply this system-wide.
Since I have two configuration file I need to add the entry in the second config file also -- /etc/systemd/system/docker.service.d/docker-thinpool.conf:
--exec-opt native.cgroupdriver=systemd \
Just to add, cgroupfs is dockers own control group manager. However, for the majority of Linux distributions ssytemd is the default init system now and systemd has tight integration with Linux control groups and In Kubernetes site, they recommend using systemd (see below) as using cgroupfs along with systemd seems to be non-optimal
So it is better to use systemd then for cgroup managment. kubelet is configured by default to use systemd. So it is easier and better to change Docker to use the systemd Cgroup driver
A history of this overlap is here https://lwn.net/Articles/676831/
In Kubernetes site, they recommend using systemd https://kubernetes.io/docs/setup/production-environment/container-runtimes/
Cgroup drivers When systemd is chosen as the init system for a Linux
distribution, the init process generates and consumes a root control
group (cgroup) and acts as a cgroup manager. Systemd has a tight
integration with cgroups and will allocate cgroups per process. It’s
possible to configure your container runtime and the kubelet to use
cgroupfs. Using cgroupfs alongside systemd means that there will then
be two different cgroup managers.
Control groups are used to constrain resources that are allocated to
processes. A single cgroup manager will simplify the view of what
resources are being allocated and will by default have a more
consistent view of the available and in-use resources. When we have
two managers we end up with two views of those resources. We have seen
cases in the field where nodes that are configured to use cgroupfs for
the kubelet and Docker, and systemd for the rest of the processes
running on the node becomes unstable under resource pressure.
OS: Centos 7.4 As kubernetes 1.23.1 recommend to use cgroup systemd, and docker 20.10.20 use cgroup cgroupfs. So, you have to change docker service file.
step1: Stop docker service
systemctl stop docker
step2: change on files /etc/systemd/system/multi-user.target.wants/docker.service and /usr/lib/systemd/system/docker.service
From :
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
TO:
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd
step3: start docker service and kubelet
systemctl start docker
kubeadm init phase kubelet-start
Make sure you are logged in as root and execute the below two commands :
echo '{"exec-opts": ["native.cgroupdriver=systemd"]}' >> /etc/docker/daemon.json
systemctl restart docker
Try to restart the docker service:
systemctl daemon-reload
systemctl restart docker.service
I'm building some CentOS VM with VMWare, with no access to internet, so I've downloaded and made local repositories, including this one
Then I have installed docker-engine.x86_64, and when starting the docker daemon, I get the following errors :
[root]# dockerd
DEBU[0000] docker group found. gid: 993
...
...
DEBU[0001] Error retrieving the next available loopback: open /dev/loop-control: no such device
ERRO[0001] **There are no more loopback devices available.**
ERRO[0001] [graphdriver] prior storage driver "devicemapper" failed: loopback attach failed
DEBU[0001] Cleaning up old mountid : start.
FATA[0001] Error starting daemon: error initializing graphdriver: loopback attach failed
After manually add the loop module which control loop device with this command :
insmod /lib/modules/3.10.0-327.36.2.el7.x86_64/kernel/drivers/block/loop.ko
The error changes to :
[graphdriver] prior storage driver "devicemapper" failed: devicemapper: Error running deviceCreate (CreatePool) dm_task_run failed
I've read that it could be because I have not enough space disk, I think it's not that, any idea?
[root]# df -k .
Filesystem blocs de 1K Used Available Used Mounted on
/dev/mapper/centos-root 51887356 2436256 49451100 5% /
I got the "There are no more loopback devices available" error, which stopped dockerd from running.
I fixed it by ensuring the storage driver was 'overlay':
# /usr/bin/dockerd -D --storage-driver=overlay
This was on Debian Jessie and docker running as a systemd service/unit.
To make it permanent, I created a systemd drop-in:
$ cat /etc/systemd/system/docker.service.d/docker.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --storage-driver=overlay
All of the similar questions I see are resolved by cleaning up the images or containers or orphaned volumes but I am not having any of those problems. I even completely deleted /var/lib/docker and still nothing.
Relevant output:
[N] ⋊> ~/W/W/cocagne on master ⨯ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock:ro -v /var/lib/docker:/var/lib/docker martin/docker-cleanup-vol
umes
docker: Error response from daemon: Container command '/usr/local/bin/docker-cleanup-volumes.sh' not found or does not exist..
[N] ⋊> ~/W/W/cocagne on master ⨯ docker-compose build 11:56:23
mysql uses an image, skipping
Building vitess
Traceback (most recent call last):
File "/usr/bin/docker-compose", line 9, in <module>
load_entry_point('docker-compose==1.7.1', 'console_scripts', 'docker-compose')()
File "/usr/lib/python3.5/site-packages/compose/cli/main.py", line 58, in main
command()
File "/usr/lib/python3.5/site-packages/compose/cli/main.py", line 109, in perform_command
handler(command, command_options)
File "/usr/lib/python3.5/site-packages/compose/cli/main.py", line 213, in build
force_rm=bool(options.get('--force-rm', False)))
File "/usr/lib/python3.5/site-packages/compose/project.py", line 300, in build
service.build(no_cache, pull, force_rm)
File "/usr/lib/python3.5/site-packages/compose/service.py", line 718, in build
buildargs=build_opts.get('args', None),
File "/usr/lib/python3.5/site-packages/docker/api/build.py", line 54, in build
path, exclude=exclude, dockerfile=dockerfile, gzip=gzip
File "/usr/lib/python3.5/site-packages/docker/utils/utils.py", line 103, in tar
t.add(os.path.join(root, path), arcname=path, recursive=False)
File "/usr/lib/python3.5/tarfile.py", line 1938, in add
self.addfile(tarinfo, f)
File "/usr/lib/python3.5/tarfile.py", line 1966, in addfile
copyfileobj(fileobj, self.fileobj, tarinfo.size)
File "/usr/lib/python3.5/tarfile.py", line 244, in copyfileobj
dst.write(buf)
File "/usr/lib/python3.5/tempfile.py", line 483, in func_wrapper
return func(*args, **kwargs)
OSError: [Errno 28] No space left on device
[I] ⋊> ~/W/W/cocagne on master ⨯ docker ps -a 11:56:30
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[I] ⋊> ~/W/W/cocagne on master ⨯ docker ps -q 11:57:25
[I] ⋊> ~/W/W/cocagne on master ⨯ docker image -q 11:57:28
docker: 'image' is not a docker command.
See 'docker --help'.
[I] ⋊> ~/W/W/cocagne on master ⨯ docker images -a 11:57:39
REPOSITORY TAG IMAGE ID CREATED SIZE
martin/docker-cleanup-volumes latest 8c41df286c03 12 weeks ago 22.12 MB
[I] ⋊> ~/W/W/cocagne on master ⨯ df -h 11:57:41
Filesystem Size Used Avail Use% Mounted on
dev 3.9G 0 3.9G 0% /dev
run 3.9G 832K 3.9G 1% /run
/dev/sda4 27G 9.1G 17G 36% /
tmpfs 3.9G 64M 3.8G 2% /dev/shm
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
tmpfs 3.9G 32K 3.9G 1% /tmp
/dev/sda1 42G 16G 25G 39% /home
/dev/sda2 42G 9.4G 30G 24% /var
/dev/sda5 1.3G 32M 1.3G 3% /boot
tmpfs 790M 12K 790M 1% /run/user/1000
[I] ⋊> ~/W/W/cocagne on master ⨯ 11:57:54
docker info
[I] ⋊> ~/W/W/cocagne on master ⨯ docker info 12:01:55
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.11.2
Storage Driver: devicemapper
Pool Name: docker-8:2-2359321-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 11.8 MB
Data Space Total: 107.4 GB
Data Space Available: 34.57 GB
Metadata Space Used: 581.6 kB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.147 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.131 (2016-07-15)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 4.6.4-1-ARCH
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.706 GiB
Name: crockford
ID: HO2U:ELWR:LDB3:PMEY:5YOJ:D7YJ:2HJA:PVYG:45K2:J6KI:D6WO:4RUE
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
One thing that makes my issue a little different (Where I think the root of the issue comes from)
Before I created a separate partition for /var, it was on my root partition, which eventually maxed out. Once it maxed out, I shrunk my home partition, create a /var partition, copied my root's /var to my new /var, and removed my old /var. But for some reason, docker still think's it's maxed out? I have no idea.
I also tried to resinstall docker with sudo pacman -S docker but nothing.
Edit: I just tried it with a normal docker build . and that works fine. Somehow docker-compose thinks it's out of memory though?
The python stack trace from docker-compose indicates that it can't seem to create a temporary file. This would indicate there's no space left in /tmp.
OP mentioned that his RAM is completely consumed when he runs docker-compose in the comments. Given that and the fact that /tmp is mounted on tmpfs it makes sense that there is no space left for Python/docker-compose to create any temporary files in /tmp.
The possible solutions are:
Temporarily switch the default tempfile generation location by setting one of the following environment variables: TMPDIR, TEMP, TMP (ref: Python doc)
Change /tmp to not use tmpfs and use disk instead.
Increase the amount of RAM/Swap space on your machine. (You can increase swap without messing with your partitions like so). tmpfs is backed by volatile storage, which means both RAM and Swap should theoretically work.
Note, most of these cases will result in a slowdown of your application, especially if the docker build process is I/O heavy.
Try this:
mount -o remount,size=4G,noatime /tmp