I've just noticed that pm2 process consumes a ton of memory, but when I tried to take a heap snapshot to figure it out the heap snapshot was 20x more times less.
This is what was inspected:
Then I read some articles covering how to debug heap snaphots and all of them were in vitro experiments. I doubt anyone code like this. All of the heap snaphots I did were healthy, just like any other process with low memory consumption. Does nodejs produce something like runtime cache or functions' calculations results in a form of weak map which is detached from heap snapshot?
Is there any way to restrict nodejs memory usage?
Related
I need to investigate a memory leak or at least a constant increase of the memory on the server. So I took multiple memory dumps and I see a very big sized object in the Gen1 Objects but it tells me NoPtrs.
So does it mean that it is unreferenced but still in memory could it be the source of my memory leak? Do you have some advices on how I could continue on it and identify what is creating those byte[]?
The website in made in .Net 6
Here is a screenshot of it.
Regards,
I have a NodeJS server running on a small VM with 256MB of RAM and I notice the memory usage keeps growing as the server receives new requests. I read that an issue on small environments is that Node doesn't know about the memory constraints and therefore doesn't try to garbage collect until much later (so for instance, maybe it would only want to start garbage collecting once it reaches 512MB of used RAM), is it really the case?
I also tried using various flags such as --max-old-space-size but didn't see much change so I'm not sure if I have an actual memory leak or if Node just doesn't GC as soon as possible?
This might not be a complete answer, but it's coming from experience and might provide some pointers. Memory leak in NodeJS is one of the most challenging bugs that most developers could ever face.
But before we talk about memory leak, to answer your question - unless you explicitly configure --max-old-space-size, there are default memory limits that would take over. Since certain phases of Garbage collection in node are expensive (and sometimes blocking) steps, depending upon how much memory is available to it, it would delay (e.g. mark-sweep collection) some of the expensive GC cycles. I have seen that in a Machine with 16 GB of memory it would easily let the memory go as high as 800 MB before significant Garbage Collections would happen. But I am sure that doesn't make ~800 MB any special limit. It would really depend on how much available memory it has and what kind of application are you running. E.g. it is totally possible that if you have some complex computations, caches (e.g. big DB Connection Pools) or buggy logging libraries - they would themselves always take high memory.
If you are monitoring your NodeJs's memory footprint - sometime after the the server starts-up, everything starts to warm up (express loads all the modules and create some startup objects, caches warm up and all of your high memory consuming modules became active), it might appear as if there is a memory leak because the memory would keep climbing, sometimes as high as ~1 gb. Then you would see that it stabilizes (this limit used to be lesser in <v8 versions).
But sometimes there are actual memory leaks (which might be hard to spot if there is no specific pattern to it).
In your case, 256 MB seems to be meeting just the minimum RAM requirements for nodejs and might not really be enough. Before you start getting anxious of memory leak, you might want to pump it up to 1.5 GB and then monitor everything.
Some good resources on NodeJS's memory model and memory leak.
Node.js Under the Hood
Memory Leaks in NodeJS
Can garbage collection happen while the main thread is
busy?
Understanding and Debugging Memory Leaks in Your Node.js Applications
Some debugging tools to help spot the memory leaks
Node inspector |
Chrome
llnode
gcore
I have a NodeJS application running on a k8s pod. The actual size of the pod is 2GB, but in environment variables, we set this value to 4GB --max-old-space-size=4096 (which will not be true in my case - for some tenants we do allocate 4GB but most pods have 2GB).
Now I tried 2 ways to detect the memory usage and the total memory and both are providing different stats.
I'm fetching memory usage from this system file: /sys/fs/cgroup/memory/memory.usage_in_bytes and total memory from this file : /sys/fs/cgroup/memory/memory.limit_in_bytes
limit_in_bytes is returning 2GB correctly, but the value of usage_in_bytes is fluctuating too much, It's around 1gb in few mins and spikes to 2GB in next minute even though nothing changed in that minute(no stress on the system).
Stats of a process
Memory Usage POD: 2145124352
shlog - memLimit 214748364
The second option I tried is using this V8 built in node js library to get heap statistics : https://nodejs.org/api/v8.html#v8getheapstatistics.
Usage:
const initialStats = v8.getHeapStatistics();
console.log("heap_size_limit: ", (initialStats.heap_size_limit)); // total memory
console.log("total_heap_size: ", (initialStats.total_heap_size)); // current usage
Now in total memory, it's return 4G, which is not right in my case. but the current usage here seems much true.
Stats of the same process
total_heap_size: 126312448,
heap_size_limit: 4320133120,
Complete response of v8 getHeapStatistics method:
HeapStats: {
total_heap_size: 126312448,
total_heap_size_executable: 1097728,
total_physical_size: 124876920,
total_available_size: 4198923736,
used_heap_size: 121633632,
heap_size_limit: 4320133120,
malloced_memory: 73784,
peak_malloced_memory: 9831240,
does_zap_garbage: 0,
number_of_native_contexts: 1,
number_of_detached_contexts: 0
}
My goal is to detect the memory usage according to the total memory of the pod, and so some throttling when the memory consumption reached 85%. I'm willing to use the first method but please tell me why there is so much difference in the memory usage and how to get the accurate memory usage of the pod.
Really looking forward to get some help on this. Thank you.
To get overall memory consumption of a process, look to (and trust) the operating system's facilities.
Node's v8.getHeapStatistics tell you about the managed (a.k.a. garbage-collected) heap where all the JavaScript objects live. But there can be quite a bit of other, non-garbage-collected memory in the process, for example Node buffers and certain strings, and various general infrastructure that isn't on the managed heap. In the average Chrome renderer process, the JavaScript heap tends to be around a third of total memory consumption, but with significant outliers in both directions; for Node apps it depends a lot on what your app is doing.
Setting V8's max heap size (which, again, is just the garbage-collected part of the process' overall memory usage) to a value larger than the amount of memory available to you doesn't make much sense: it invites avoidable crashes, because V8 won't spend as much time on garbage collection when it thinks there's lots of available memory left, but the OS/pod as a whole could already be facing an out-of-memory situation at that point. That's why I linked the other answer: you very likely want to set the max heap size to something a little less than available memory, to give the garbage collector the right hints about when to work harder to stay under the limit.
I am currently running node.js using pm2.
And recently, I was able to check "custom metrics" using the pm2 monit command.
Here, information such as Heap size, used heap size, and active requests are shown.
I don't know how the heap size is determined. Actually, I checked pm2 running on different servers.
Each was set to 95mib / 55mib, and accordingly, the used heap size was different.
Also, is the heap usage closer to 100% the better?
While searching on "StackOverflow" to find related information, I saw the following article.
What does Heap Usage mean in PM2
Also what means active requests ? It is continuously zero.
Thank you!
[Edit]
env : ubuntu18.04 [ ec2 - t3.micro ]
node version : v10.15
[Additional]
server memory : 1GB [ 40~50% used ]
cpu : vCPU (2) [ 1~2% used ]
The heap is the RAM used by the program you're asking PM2 to manage and monitor. Heap space, in Javascript and similar language runtimes, is allocated when your program creates objects and released upon garbage collection. Your runtime asks your OS for more heap space whenever it needs it: when active allocations exceed the free space. So your heap size will probably grow as your program starts up. That's normal.
Most programs allocate and release lots of objects as they do their work, so you should not try to optimize the % usage of your heap. When your program is running at a steady state – that is, after it has started up — you'll find the % utilization creeping up until garbage collection happens, and then dropping back. For example, a nodejs/express web server allocates req and res objects for each incoming request, then uses them, then drops them so the garbage collector can reclaim their RAM.
If your allocated heap size keeps growing, over minutes or hours, you probably have a memory leak. That is a programming bug: a problem you should do your best to solve. You should look up how that works for your application language. Other than that, don't worry too much about heap usage.
Active requests count work being done via various asynchronous objects like file writers and TCP connections. Unless your program is very busy it stays near zero.
Keep an eye on loop delay if your program does computations. If it creeps up, some computation function is hogging Javascript.
How can I calculate the real memory usage of a single process? I am not talking about the virtual memory, because it just keeps growing. For instance, there are proc files like smaps, where you can get the mappings of a process. But this is virtual memory and the values of that file just keeps growing for running process. But I would like to reflect the real memory usage of a process. E.g. if you plot the memory usage of a process it should represent the allocations of memory and also the freeing of memory. So the plot should be like an up and down movement instead of a linear function, that just keeps growing for a running process.
So, how could I calculate the real memory usage? I would appreciate any helpful answer.
It's actually kind of a complicated question. The two most common metrics for a program's memory usage at the OS level are virtual size and resident set size. (These show in the output of ps -u as the VSZ and RSS columns.) Roughly speaking, these tell the total memory the program has assigned to it, versus how much it is currently actively using.
Further complicating the question is that when you use malloc (or the C++ new operator) to allocate memory, memory is allocated from a pool in your process which is built by occasionally requesting an allocation of memory from the operating system. But when you free memory, the memory goes back into this pool, but it is typically not returned to the OS. So as your program allocates and frees memory, you typically will not see its memory footprint go up and down. (However, if it frees a lot of memory and then doesn't allocate it any more, eventually you may see its rss go down.)