I am trying to set hard CPU and memory limits in a Docker Swarm consisting of three VMs. I am using the cpu and memory limit configurations suggested by docker documentation in my docker-compose.yml file. My docker-compose.yml file looks like
version: "3"
services:
app:
# replace username/repo:tag with your name and image details
image: user/testing:part2
deploy:
replicas: 10
resources:
limits:
cpus: "0.5"
memory: 4M
restart_policy:
condition: on-failure
The CPU and memory resources of my host machine and the VMs are shown in the figure below. My host machine has 4 CPUs and all the VMs have 1 CPU each.
To figure out if Docker containers can limit their resources, I am running a test program with an infinite loop in my swarm. One of the snapshots from my experiments is shown below. It shows docker stats results on the three VMs (VM1: bottom left, VM2: top right, VM3: bottom right).
Looking at the results, I have a few questions.
How is the CPU limited to 50% for each container?
Each VM has 1 CPU then how come on one VM, docker containers CPU% sum exceeds 100%?
In the image above, I have 7 docker containers running and sum of CPU% for all of them is 22+21+88+52+66+78+76 = 403 approx. which
means that the swarm is using 4 cores instead of 3. Is it
possible that docker allows the swarm to use CPU resources of the
host machine as well if needed?
Can anybody please guide me with these questions? Thank you.
Related
AMD 24 CORE Threadripper and 200GB RAM
Ubuntu 20
Docker Latest Version
Docker Swarm Mode (but the only Host)
I have my docker stack compose file.
Scaling the service up I don't have any problems to 249 Containers, but then I have a bottleneck and don't know why
Do I need to change Settings somewhere to remove the bottleneck?
I already have
fs.inotify.max_queued_events = 100000000
fs.inotify.max_user_instances = 100000000
fs.inotify.max_user_watches = 100000000
in /etc/sysctl.conf
as I had a bottleneck with inotify instances by nearly 100 containers, solved that problem with that.
But I cant scale past 249 Containers
One issue is certainly going to be IP availability if this is docker swarm, as overlay networks - the default on swarm - are automatically /24 and thus limited to ~255 hosts.
So:
a. attach the service to a manually created network that you have scoped to be larger than /24
b. ensure that the services endpoint mode is dnsrr as vip's cannot be used safely with networks with larger than /24 address space.
I'm using Spark v3.1.2 and the standalone cluster manager but the total "Memory in use" reported in the UI is incorrect. It is reporting "1024.0 MiB Total" even though the single worker has been limited to 768MiB. Is this a bug when workers are running inside a container (docker or ECS)?
Screenshot of master UI showing memory
Setup:
docker-compose.yml file
1 x master service (container)
1 x worker service (container)
worker memory constrained to 768MiB and 1 CPU
Relevant section in docker-compose.yml to limit the worker container memory:
deploy:
mode: replicated
replicas: 2
resources:
limits:
cpus: "1.0"
memory: 768M
Verification docker limited the memory using "docker container inspect":
"Memory": 805306368
I've tried spinning up another worker and the UI reports a total of 2GiB in memory as it thinks each worker has 1024Mib, verified in the log:
INFO Master: Registering worker 172.22.0.4:7078 with 1 cores, 1024.0 MiB RAM
I've also tried this in ECS and observed similar behaviour.
I've also tried allocating > 1GiB to the worker in docker e.g. 5GiB and the master keeps reporting 1GiB lower? When the worker has been allocated 5GiB, the master registers the worker as having 4GiB:
Registering worker 172.22.0.3:7078 with 1 cores, 4.0 GiB RAM
Is this a problem with the Spark standalone cluster manager or the Spark Java worker code not able to determine the correct memory when running within a container?
Is there a way to specify the total RAM and total CPUs that a docker can use? My machine's size is 16 GB RAM and 4 CPUs. How can I make the docker-machine to utilize the complete RAM and underlying CPUs?
version: '3'
services:
prisma:
image: prismagraphql/prisma:1.34
restart: always
ports:
- "4466:4466"
environment:
JAVA_OPTS: '-Xmx8192m'
PRISMA_CONFIG: |
port: 4466
databases:
default:
connector: mongo
uri: 'mongodb+srv://abc:test#abc.net/default_default?retryWrites=true&w=majority'
What you are asking is the default behavior.
From docs
By default, a container has no resource constraints and can use as much of a given resource as the host’s kernel scheduler allows.
What you can do to ensure a container gets minimum amount of resources and is not at the mercy of scheduler is to use the resource reservations to ensure the container always has minimum required resources and optionally limits if you also want to enforce upper threshold for the resource usage.
See Resources section in docs
version: "3.8"
services:
redis:
image: redis:alpine
deploy:
resources:
limits:
cpus: '0.50'
memory: 50M
reservations:
cpus: '0.25'
memory: 20M
In this general example, the redis service is constrained to use no more than 50M of memory and 0.50 (50% of a single core) of available processing time (CPU), and has 20M of memory and 0.25 CPU time reserved (as always available to it).
Getting started with Kubernetes so have the following question:
Say a microservice has the following C# code snippet:
var tasks = _componentBuilders.Select(b =>
{
return Task.Factory.StartNew(() => b.SetReference(context, typedModel));
});
Task.WaitAll(tasks.ToArray());
On my box, I understand that each thread be executed on a vCPU. So if I have 4 cores with hyperthreading enabled I will be able to execute 8 tasks concurrently. Therefore, if I have about 50000 tasks, it will take roughly
(50,000/8) * approximate time per task
to complete this work. This ignores context switch, etc.
Now, moving to the cloud and assuming this code is in a docker container managed by Kubernetes Deployment and we have a single docker container per VM to keep this simple. How does the above code scale horizontally across the VMs in the deployment? Can not find very clear guidance on this so if anyone has any reference material, that would be helpful.
You'll typically use a Kubernetes Deployment object to deploy application code. That has a replicas: setting, which launches some number of identical disposable Pods. Each Pod has a container, and each pod will independently run the code block you quoted above.
The challenge here is distributing work across the Pods. If each Pod generates its own 50,000 work items, they'll all do the same work and things won't happen any faster. Just running your application in Kubernetes doesn't give you any prebuilt way to share thread pools or task queues between Pods.
A typical approach here is to use a job queue system; RabbitMQ is a popular open-source option. One part of the system generates the tasks and writes them into RabbitMQ. One or more workers reads jobs from the queue and runs them. You can set this up and demonstrate it to yourself without using container technology, then repackage it in Docker or Kubernetes just changing the RabbitMQ broker address at deploy time.
In this setup I'd probably have the worker run jobs serially, one at a time, with no threading. That will simplify the implementation of the worker. If you want to run more jobs in parallel, run more workers; in Kubernetes, increase the Deployment replica: count.
In Kubernetes, when we deploy containers as Pods we can include the resources.limits.cpu and resources.requests.cpu fields for each container in the Pod's manifest:
resources:
requests:
cpu: "1000m"
limits:
cpu: "2000m"
In the example above we have a request for 1 CPU and a limit for a maximum of 2 CPUs. This means the Pod will be scheduled to a worker node which can satisfy above resource requirements.
One cpu, in Kubernetes, is equivalent to 1 vCPU/Core for cloud providers and 1 hyperthread on bare-metal Intel processors.
We can vertically scale by increasing / decreasing the values for the requests and limits fields. Or we can horizontally scale by increasing / decreasing the number of replicas of the pod.
For more details about resource units in Kubernetes here
I'm currently trying to deploy a node.js app on docker containers. I need to deploy 30 of them but they begin to have a weird behavior at some point, some of them freeze.
I am currently running Docker version for windows 18.03.0-ce, build 0520e24302, my computer specs (cpu and memory):
I5 4670 K
24 GB of ram
My docker default machine resource allocation is the following :
Allocated RAM : 10 Gb
Allocated vCPUs : 4
My node application is running on apline3.8 and node.js 11.4 and mostly do http requests every 2-3 seconds.
When i try to deploy 20 containers everything is running like a charm, my application do the job and i can see that there is an activity on every on my containers through the logs, activity stats.
The problem comes when i try to deploy more containers, more than 20, i can notice that some of the previously deployed containers start to stop their activities (0% cpu using, logs freezing). When everything is deployed (30 containers), Docker start to block the activity of some of them and unblock them at some point to block some others (blocking/unblocking is random). It seems to be sequential. I tried to wait and see what happened and the result is that some of the containers are able to poursue their activities and some others are stuck forever (still running but no more activity).
It's important to notice that i used the following resources restrictions on each of my containers :
MemoryReservation : 160mb
Memory soft limit : 160mb
NanoCPUs : 250000000 (0.25 cpus)
I had to increase my docker default machine resource allocation and decrease container's ressource allocation because it was using almost 100% of my cpu, maybe i did a mistake in my configuration. I tried to tweak those values, but no success i still have some containers freezing.
I'm kind of lost right know.
Any help would be appreciated even a little one, thank you in advance !