I have problems with RAM on servers which use varnish and only it (no other apps there). Each machine has 64GB RAM available for caching and has three separated varnish services for different backends. Currently, the sum of RAM allocated to varnish from all services is 24GB RAM on each server. I want to increase this value up to 48GB (75% of the whole memory available) but I have some problems.
When I tried to allocate 8GB more just for one service (32GB from all), the committed memory got a peak to 70GB RAM(?). What's more, the increased service has restarted a few times after getting 100% ram allocated to its limit (error msg: child not responding to CLI, killing it/died signal=6/Panic message: Assert error in vbf_fetch_thread()). In addition, services use a lot of VSZ (virtual memory size), it that okay?
This could be Transient memory which is uncapped by default and we'll use malloc as storage.
In Transinet stevedore Varnish stores object with a TTL < 10s, therefore if you have many of those that's what you see.
The solution is to either increase the TTLs or cap the Transient storage.
I've changed jemalloc parameters lg_dirty_mult and lg_chunk. Now I was able to assign 42 GB ram to varnish and committed memory is around 60 GB now. The main varnish task is to cash images and TTL is set to 365d.
Related
I've setup a 3 machine VoltDB cluster with more or less default settings. However there seems to be a constant problem with voltdb eating up all of the RAM heap and not freeing it. The heap size is recommended 2GB.
Things that I think might be bad in my setup:
I've set 1 min async snapshots
Most of my queries are AdHoc
Event though it might not be ideal, I don't think it should lead to a problem where memory doesn't get freed.
I've setup my machines accordingly to 2.3. Configure Memory Management.
On this image you can see sudden drops in memory usage. These are server shutdowns.
Heap filling warnings
DB Monitor, current state of leader server
I would also like to note that this server is not heavily loaded.
Sadly, I couldn't find anyone with a similar problem. Most of the advice were targeted on fixing problems with optimizing memory use or decreasing the amount of memory allocated to voltdb. No one seems to have this memory leak lookalike.
I've deployed the project on AWS & we are using local Redis. We have around 150+ Redis keys. having a lot of data. We have 16 GB RAM on the EC2 instance & in Redis config, we defined maxmemory 10 GB.
But we are getting bellow error.
--> redis-cli
--> memory doctor
Sam, I detected a few issues in this Redis instance memory implants:
Peak memory: In the past this instance used more than 150% the memory that is currently using. The allocator is normally not able to release memory after a peak, so you can expect to see a big fragmentation ratio, however this is actually harmless and is only due to the memory peak, and if the Redis instance Resident Set Size (RSS) is currently bigger than expected, the memory will be used as soon as you fill the Redis instance with more data. If the memory peak was only occasional and you want to try to reclaim memory, please try the MEMORY PURGE command, otherwise the only other option is to shutdown and restart the instance.
I'm here to keep you safe, Sam. I want to help you.
That would be great if anyone can help us to resolve this as soon as possible.
Please let us know.
I want to create a task definition in aws ecs.
How much memory and cpu I need to run nginx in conatiner? and nodejs in another container?
nginx - just a proxy from 80 to 3000.
nodejs - simple services that call to atlas mongodb
I will never recommend a hard memory limit while running container in ECS.
Plus you can not determine memory for the idle state of the container so better to have look on some benchmark for Nginx while node memory vary from application to the application also poor code might consume more memory than good and managed application.
NGINX used one worker, 15% CPU and 1MB of memory to serve 11,500 requests per second.
Benchmarks have shown NGINX lightweight
Now based on your traffic EXPECTED_REQUST/11500 = Required memory
While the memory of Nodejs is really critical and totally depend on your code if the application does not close file or request properly it will hit the max memory sooner then expected, so go for memory reservation.
memoryReservation
The soft limit (in MiB) of memory to reserve for the container. When system memory is under contention, Docker attempts to keep the container memory to this soft limit; however, your container can consume more memory when needed
For example, if your container normally uses 128 MiB of memory, but occasionally bursts to 256 MiB of memory for short periods of time, you can set a memoryReservation of 128 MiB, and a memory hard limit of 300 MiB. This configuration would allow the container to only reserve 128 MiB of memory from the remaining resources on the container instance, but also allow the container to consume more memory resources when needed.
ECS memoryReservation
So better to do not set hard limit that is called memory.
The amount (in MiB) of memory to present to the container. If your container attempts to exceed the memory specified here, the container is killed. The total amount of memory reserved for all containers within a task must be lower than the task memory value, if one is specified.
The easiest way to determine this, is empirically, using docker stats -
because the utilized resources vary proportionally to the requests per seconds your app will receive.
At idle expect ~10MB of memory for both containers and ~0 CPU usage :)
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
4be92e1cd6bf elastic_germain 0.00% 6.68MiB / 15.65GiB 0.04% 5.68kB / 0B 0B / 0B 11
7c5552053782 modest_black 0.00% 4.543MiB / 15.65GiB 0.03% 6.37kB / 0B 0B / 8.19kB 2
You should not exceed 128MB for the nginx, unless you plan to enable content caching.
As for the node container, 512MB should be more than enough.
These values are relative to the real traffic the app shall receive: trying to forecast these the numbers ahead of time is just an educated guess if the expected traffic is unknown or vaguely guessed.
The good news is that the maximum theoretical limit is just as much as your host’s kernel scheduler allows. A stateless app can go beyond that point, using horizontal scaling, across multiple docker nodes.
I am using swagger-express-mw for my REST API application with express. However I am observing continuous memory increase in my production environment.
My Servers are Linux 4C,16 GB under an Application Load Balancer(ALB). Currently there are 2 servers under the ALB and memory usage has increased from 2% to 6%. I am not sure if GC has been executed on it or not yet.
Below is sample snapshot of memory. Here the app is using approximately 100 MB for process but buffers are increasing. Is it a memory leakage issue?
we re facing OOM error when trying to execute multiple SQL query session via scheduled job .
Detailed error:
The error message is: org.postgresql.util.PSQLException:ERROR: Out of memory (seg6 slice5 sungpmsh0:40002 pid=13610)
Detail: VM protect failed to allocate 65584 bytes from system, VM Protect 5835 MB available
We tried
After reading the pivotal support doc, we are doing basic troubleshoot here
validated two memory parameters here
current setting in GPdb
GPDB vmprotect limit :8 GB
GPB statemen_mem: based on the vmprotect limit.as per reading it is responsible for running the query in the segment.
Test 2 Did Tuning the SQL queries. also, what should I tune here please guide?
Based on source
https://discuss.pivotal.io/hc/en-us/articles/201947018-Pivotal-Greenplum-GPDB-Memory-Configuration
https://discuss.pivotal.io/hc/en-us/articles/204268778-What-are-VM-Protect-failed-to-allocate-d-bytes-d-MB-available-error-
But still getting the same OOM error.
Do we need to increase the vmprotect limit? if Yes, then by which amount should we increase it?
How to handle concurrency at gpdb?
How much swap we need to add here when we are already running with 30 GB RAM.
currently, we have added 15GB swap here? is that ok ?
What is the query to identify host connection with Greenplum database ?
Thanks in advance
Do we need to increase the vmprotect limit? if Yes, then by which amount should we increase it?
There is a nice calculator on setting gp_vmem_protect_limit on Greenplum.org. The setting depends on how much memory, swap, and segments per host you have.
http://greenplum.org/calc/
You can be getting OOM errors for several reasons.
Bad query
Bad table distribution (skew)
Bad settings (like gp_vmem_protect_limit)
Not enough resources (RAM)
How to handle concurrency at gpdb?
More RAM, less segments per host, and workload management to limit the number of concurrent queries running.
How much swap we need to add here when we are already running with 30 GB RAM. currently, we have added 15GB swap here? is that ok ?
Only 30GB of RAM? That is pretty small. You can add more swap but it will slow down the queries compared to real RAM. I wouldn't use much more than 8GB of swap.
I recommend using 256GB of RAM or more especially if you are worried about concurrency.
What is the query to identify host connection with Greenplum database
select * from pg_stat_activity;