How much memory and cpu nginx and nodejs in each container needs? - node.js

I want to create a task definition in aws ecs.
How much memory and cpu I need to run nginx in conatiner? and nodejs in another container?
nginx - just a proxy from 80 to 3000.
nodejs - simple services that call to atlas mongodb

I will never recommend a hard memory limit while running container in ECS.
Plus you can not determine memory for the idle state of the container so better to have look on some benchmark for Nginx while node memory vary from application to the application also poor code might consume more memory than good and managed application.
NGINX used one worker, 15% CPU and 1MB of memory to serve 11,500 requests per second.
Benchmarks have shown NGINX lightweight
Now based on your traffic EXPECTED_REQUST/11500 = Required memory
While the memory of Nodejs is really critical and totally depend on your code if the application does not close file or request properly it will hit the max memory sooner then expected, so go for memory reservation.
memoryReservation
The soft limit (in MiB) of memory to reserve for the container. When system memory is under contention, Docker attempts to keep the container memory to this soft limit; however, your container can consume more memory when needed
For example, if your container normally uses 128 MiB of memory, but occasionally bursts to 256 MiB of memory for short periods of time, you can set a memoryReservation of 128 MiB, and a memory hard limit of 300 MiB. This configuration would allow the container to only reserve 128 MiB of memory from the remaining resources on the container instance, but also allow the container to consume more memory resources when needed.
ECS memoryReservation
So better to do not set hard limit that is called memory.
The amount (in MiB) of memory to present to the container. If your container attempts to exceed the memory specified here, the container is killed. The total amount of memory reserved for all containers within a task must be lower than the task memory value, if one is specified.

The easiest way to determine this, is empirically, using docker stats -
because the utilized resources vary proportionally to the requests per seconds your app will receive.
At idle expect ~10MB of memory for both containers and ~0 CPU usage :)
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
4be92e1cd6bf elastic_germain 0.00% 6.68MiB / 15.65GiB 0.04% 5.68kB / 0B 0B / 0B 11
7c5552053782 modest_black 0.00% 4.543MiB / 15.65GiB 0.03% 6.37kB / 0B 0B / 8.19kB 2
You should not exceed 128MB for the nginx, unless you plan to enable content caching.
As for the node container, 512MB should be more than enough.
These values are relative to the real traffic the app shall receive: trying to forecast these the numbers ahead of time is just an educated guess if the expected traffic is unknown or vaguely guessed.
The good news is that the maximum theoretical limit is just as much as your host’s kernel scheduler allows. A stateless app can go beyond that point, using horizontal scaling, across multiple docker nodes.

Related

Varnish RAM problems

I have problems with RAM on servers which use varnish and only it (no other apps there). Each machine has 64GB RAM available for caching and has three separated varnish services for different backends. Currently, the sum of RAM allocated to varnish from all services is 24GB RAM on each server. I want to increase this value up to 48GB (75% of the whole memory available) but I have some problems.
When I tried to allocate 8GB more just for one service (32GB from all), the committed memory got a peak to 70GB RAM(?). What's more, the increased service has restarted a few times after getting 100% ram allocated to its limit (error msg: child not responding to CLI, killing it/died signal=6/Panic message: Assert error in vbf_fetch_thread()). In addition, services use a lot of VSZ (virtual memory size), it that okay?
This could be Transient memory which is uncapped by default and we'll use malloc as storage.
In Transinet stevedore Varnish stores object with a TTL < 10s, therefore if you have many of those that's what you see.
The solution is to either increase the TTLs or cap the Transient storage.
I've changed jemalloc parameters lg_dirty_mult and lg_chunk. Now I was able to assign 42 GB ram to varnish and committed memory is around 60 GB now. The main varnish task is to cash images and TTL is set to 365d.

Node web app running in Fargate crashes under load with memory and CPU relatively untaxed

We are running a Koa web app in 5 Fargate containers. They are pretty straightforward crud/REST API's with Koa over Mongo Atlas. We started doing capacity testing, and noticed that the node servers started to slow down significantly with plenty of headroom left on CPU (sitting at 30%), Memory (sitting at or below 20%), and Mongo (still returning in < 10ms).
To further test this, we removed the Mongo operations and just hammered our health-check endpoints. We did see a lot of throughput, but significant degradation occurred at 25% CPU and Node actually crashed at 40% CPU.
Our fargate tasks (containers) are CPU:2048 (2 "virtual CPUs") and Memory 4096 (4 gigs).
We raised our ulimit nofile to 64000 and also set the max-old-space-size to 3.5 GB. This didn't result in a significant difference.
We also don't see significant latency in our load balancer.
My expectation is that CPU or memory would climb much higher before the system began experiencing issues.
Any ideas where a bottleneck might exist?
The main issue here was that we were running containers with 2 CPUs. Since Node only effectively uses 1 CPU, there was always a certain amount of CPU allocation that was never used. The ancillary overhead never got the container to 100%. So node would be overwhelmed on its 1 cpu while the other was basically idle. This resulted in our autoscaling alarms never getting triggered.
So adjusted to 1 cpu containers with more horizontal scale out (ie more instances).

Cassandra not utilizing CPU and RAM

We are running a Cassandra 5 nodes cluster (3.10) with 8 cores, 32 memory and 2TB disk each.
The cluster is running in k8s over google cloud.
Recently our disk size was increased from 400GB to ~ 800GB in each node, at that point we start suppering from many read/write timeouts.
when checking the usage of the node in their resources we notice that their CPU is at 1.5 - 2, ram is 17GB.
it seems like they are bound from some reason and the only observation we saw is that there a reverse correlation between disk size and used cpu, the higher disk usage the lower the cpu usage.
is there a way to see what's blocking the CPU and RAM from utilizing 100% of their resources?

Identify Memory Leakage Issue in NodeJS Code

I am using swagger-express-mw for my REST API application with express. However I am observing continuous memory increase in my production environment.
My Servers are Linux 4C,16 GB under an Application Load Balancer(ALB). Currently there are 2 servers under the ALB and memory usage has increased from 2% to 6%. I am not sure if GC has been executed on it or not yet.
Below is sample snapshot of memory. Here the app is using approximately 100 MB for process but buffers are increasing. Is it a memory leakage issue?

Kubernetes doesn't take into account total node memory usage when starting Pods

What I see: Kubernetes takes into account only the memory used by its components when scheduling new Pods, and considers the remaining memory as free, even if it's being used by other system processes outside Kubernetes. So, when creating new deployments, it attempts to schedule new pods on a suffocated node.
What I expected to see: Kubernetes automatically take in consideration the total memory usage (by kubernetes components + system processes) and schedule it on another node.
As a work-around, is there a configuration parameter that I need to set or is it a bug?
Yes, there are few parameters to allocate resources:
You can allocate memory and CPU for your pods and allocate memory and CPU for your system daemons manually. In documentation you could find how it works with the example:
Example Scenario
Here is an example to illustrate Node Allocatable computation:
Node has 32Gi of memory, 16 CPUs and 100Gi of Storage
--kube-reserved is set to cpu=1,memory=2Gi,ephemeral-storage=1Gi
--system-reserved is set to cpu=500m,memory=1Gi,ephemeral-storage=1Gi
--eviction-hard is set to memory.available<500Mi,nodefs.available<10%
Under this scenario, Allocatable will be 14.5 CPUs, 28.5Gi of memory and 98Gi of local storage. Scheduler ensures that the total memory requests across all pods on this node does not exceed 28.5Gi and storage doesn’t exceed 88Gi. Kubelet evicts pods whenever the overall memory usage across pods exceeds 28.5Gi, or if overall disk usage exceeds 88GiIf all processes on the node consume as much CPU as they can, pods together cannot consume more than 14.5 CPUs.
If kube-reserved and/or system-reserved is not enforced and system daemons exceed their reservation, kubelet evicts pods whenever the overall node memory usage is higher than 31.5Gi or storage is greater than 90Gi
You can allocate as many as you need for Kubernetes with flag --kube-reserved and for system with flag -system-reserved.
Additionally, if you need stricter rules for spawning pods, you could try to use Pod Affinity.
Kubelet has the parameter --system-reserved that allows you to make a reservation of cpu and memory for system processes.
It is not dynamic (you reserve resources only at launch) but is the only way to tell Kubelet not to use all resources in node.
--system-reserved mapStringString
A set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=500Mi,ephemeral-storage=1Gi) pairs that describe resources reserved for non-kubernetes components. Currently only cpu and memory are supported. See http://kubernetes.io/docs/user-guide/compute-resources for more detail. [default=none]

Resources