I am currently running node.js using pm2.
And recently, I was able to check "custom metrics" using the pm2 monit command.
Here, information such as Heap size, used heap size, and active requests are shown.
I don't know how the heap size is determined. Actually, I checked pm2 running on different servers.
Each was set to 95mib / 55mib, and accordingly, the used heap size was different.
Also, is the heap usage closer to 100% the better?
While searching on "StackOverflow" to find related information, I saw the following article.
What does Heap Usage mean in PM2
Also what means active requests ? It is continuously zero.
Thank you!
[Edit]
env : ubuntu18.04 [ ec2 - t3.micro ]
node version : v10.15
[Additional]
server memory : 1GB [ 40~50% used ]
cpu : vCPU (2) [ 1~2% used ]
The heap is the RAM used by the program you're asking PM2 to manage and monitor. Heap space, in Javascript and similar language runtimes, is allocated when your program creates objects and released upon garbage collection. Your runtime asks your OS for more heap space whenever it needs it: when active allocations exceed the free space. So your heap size will probably grow as your program starts up. That's normal.
Most programs allocate and release lots of objects as they do their work, so you should not try to optimize the % usage of your heap. When your program is running at a steady state – that is, after it has started up — you'll find the % utilization creeping up until garbage collection happens, and then dropping back. For example, a nodejs/express web server allocates req and res objects for each incoming request, then uses them, then drops them so the garbage collector can reclaim their RAM.
If your allocated heap size keeps growing, over minutes or hours, you probably have a memory leak. That is a programming bug: a problem you should do your best to solve. You should look up how that works for your application language. Other than that, don't worry too much about heap usage.
Active requests count work being done via various asynchronous objects like file writers and TCP connections. Unless your program is very busy it stays near zero.
Keep an eye on loop delay if your program does computations. If it creeps up, some computation function is hogging Javascript.
Related
I've just noticed that pm2 process consumes a ton of memory, but when I tried to take a heap snapshot to figure it out the heap snapshot was 20x more times less.
This is what was inspected:
Then I read some articles covering how to debug heap snaphots and all of them were in vitro experiments. I doubt anyone code like this. All of the heap snaphots I did were healthy, just like any other process with low memory consumption. Does nodejs produce something like runtime cache or functions' calculations results in a form of weak map which is detached from heap snapshot?
Is there any way to restrict nodejs memory usage?
I have a NodeJS server running on a small VM with 256MB of RAM and I notice the memory usage keeps growing as the server receives new requests. I read that an issue on small environments is that Node doesn't know about the memory constraints and therefore doesn't try to garbage collect until much later (so for instance, maybe it would only want to start garbage collecting once it reaches 512MB of used RAM), is it really the case?
I also tried using various flags such as --max-old-space-size but didn't see much change so I'm not sure if I have an actual memory leak or if Node just doesn't GC as soon as possible?
This might not be a complete answer, but it's coming from experience and might provide some pointers. Memory leak in NodeJS is one of the most challenging bugs that most developers could ever face.
But before we talk about memory leak, to answer your question - unless you explicitly configure --max-old-space-size, there are default memory limits that would take over. Since certain phases of Garbage collection in node are expensive (and sometimes blocking) steps, depending upon how much memory is available to it, it would delay (e.g. mark-sweep collection) some of the expensive GC cycles. I have seen that in a Machine with 16 GB of memory it would easily let the memory go as high as 800 MB before significant Garbage Collections would happen. But I am sure that doesn't make ~800 MB any special limit. It would really depend on how much available memory it has and what kind of application are you running. E.g. it is totally possible that if you have some complex computations, caches (e.g. big DB Connection Pools) or buggy logging libraries - they would themselves always take high memory.
If you are monitoring your NodeJs's memory footprint - sometime after the the server starts-up, everything starts to warm up (express loads all the modules and create some startup objects, caches warm up and all of your high memory consuming modules became active), it might appear as if there is a memory leak because the memory would keep climbing, sometimes as high as ~1 gb. Then you would see that it stabilizes (this limit used to be lesser in <v8 versions).
But sometimes there are actual memory leaks (which might be hard to spot if there is no specific pattern to it).
In your case, 256 MB seems to be meeting just the minimum RAM requirements for nodejs and might not really be enough. Before you start getting anxious of memory leak, you might want to pump it up to 1.5 GB and then monitor everything.
Some good resources on NodeJS's memory model and memory leak.
Node.js Under the Hood
Memory Leaks in NodeJS
Can garbage collection happen while the main thread is
busy?
Understanding and Debugging Memory Leaks in Your Node.js Applications
Some debugging tools to help spot the memory leaks
Node inspector |
Chrome
llnode
gcore
I have a NodeJS application running on a k8s pod. The actual size of the pod is 2GB, but in environment variables, we set this value to 4GB --max-old-space-size=4096 (which will not be true in my case - for some tenants we do allocate 4GB but most pods have 2GB).
Now I tried 2 ways to detect the memory usage and the total memory and both are providing different stats.
I'm fetching memory usage from this system file: /sys/fs/cgroup/memory/memory.usage_in_bytes and total memory from this file : /sys/fs/cgroup/memory/memory.limit_in_bytes
limit_in_bytes is returning 2GB correctly, but the value of usage_in_bytes is fluctuating too much, It's around 1gb in few mins and spikes to 2GB in next minute even though nothing changed in that minute(no stress on the system).
Stats of a process
Memory Usage POD: 2145124352
shlog - memLimit 214748364
The second option I tried is using this V8 built in node js library to get heap statistics : https://nodejs.org/api/v8.html#v8getheapstatistics.
Usage:
const initialStats = v8.getHeapStatistics();
console.log("heap_size_limit: ", (initialStats.heap_size_limit)); // total memory
console.log("total_heap_size: ", (initialStats.total_heap_size)); // current usage
Now in total memory, it's return 4G, which is not right in my case. but the current usage here seems much true.
Stats of the same process
total_heap_size: 126312448,
heap_size_limit: 4320133120,
Complete response of v8 getHeapStatistics method:
HeapStats: {
total_heap_size: 126312448,
total_heap_size_executable: 1097728,
total_physical_size: 124876920,
total_available_size: 4198923736,
used_heap_size: 121633632,
heap_size_limit: 4320133120,
malloced_memory: 73784,
peak_malloced_memory: 9831240,
does_zap_garbage: 0,
number_of_native_contexts: 1,
number_of_detached_contexts: 0
}
My goal is to detect the memory usage according to the total memory of the pod, and so some throttling when the memory consumption reached 85%. I'm willing to use the first method but please tell me why there is so much difference in the memory usage and how to get the accurate memory usage of the pod.
Really looking forward to get some help on this. Thank you.
To get overall memory consumption of a process, look to (and trust) the operating system's facilities.
Node's v8.getHeapStatistics tell you about the managed (a.k.a. garbage-collected) heap where all the JavaScript objects live. But there can be quite a bit of other, non-garbage-collected memory in the process, for example Node buffers and certain strings, and various general infrastructure that isn't on the managed heap. In the average Chrome renderer process, the JavaScript heap tends to be around a third of total memory consumption, but with significant outliers in both directions; for Node apps it depends a lot on what your app is doing.
Setting V8's max heap size (which, again, is just the garbage-collected part of the process' overall memory usage) to a value larger than the amount of memory available to you doesn't make much sense: it invites avoidable crashes, because V8 won't spend as much time on garbage collection when it thinks there's lots of available memory left, but the OS/pod as a whole could already be facing an out-of-memory situation at that point. That's why I linked the other answer: you very likely want to set the max heap size to something a little less than available memory, to give the garbage collector the right hints about when to work harder to stay under the limit.
It is known that some applications aren't aware of Linux kernel isolation and virtualization features such as cgroups. This includes system utils like top, free and ps, but also platforms like Java.
I've recently read an article which suggests that when running JVMs in Kubernetes, you should enforce manual limits on the Java heap size to avoid errors.
I cannot find anywhere whether this is also true for NodeJS. Do I need to implement something like above to set --max_old_space_size=XXX on my NodeJS application in Kubernetes?
A NodeJS process will try an allocate memory regardless of the container limits, just like Java.
Setting a limit on the process will help stop the OS from killing the process, particularly in constrained environments where Node might try allocate past the memory limit even though Node could probably run inside the limit.
If you are running an app that is close to using the memory limit then adding the memory limit settings just changes the failure scenario. NodeJS and the JVM will have a chance to exit with an out of memory error (OOM) rather than be killed by the operating system. The process will likely slow to a crawl as it nears the memory limit and the garbage collector tries the best it can to keep the process below the limit.
Note that the old space is only one of multiple memory spaces in NodeJS. Only the new space (semi spaces) and old space can be limited.
--max_semi_space_size (max size of a semi-space (in MBytes), the new space consists of two semi-spaces)
type: int default: 0
--max_old_space_size (max size of the old space (in Mbytes))
type: int default: 0
The other heap spaces are generally small and static enough not to worry about.
Modules that run native code can allocate memory outside the heap and can't be limited by an option.
I have been running a crypto-intensive application that was generating pseudo-random strings, with special structure and mathematical requirements. It has generated around 1.7 million voucher numbers per node in over the last 8 days. The generation process was CPU intensive, with very low memory requirements.
Mnesia running on OTP-14B02 was the storage database and the generation was done within each virtual machine. I had 3 nodes in the cluster with all mnesia tables disc_only_copies type. Suddenly, as activity on the Solaris boxes increased (other users logged on remotely and were starting webservers, ftp sessions, and other tasks), my bash shell started reporting afork: not enough space error.
My erlang Vms also, went down with this error below:
Crash dump was written to: erl_crash.dump
temp_alloc: Cannot reallocate 8388608 bytes of memory (of type "root_set").
Usually, we get memory allocation errors and not memory re-location errors and normally memory of type "heap" is the problem. This time, the memory type reported is type "root-set".
Qn 1. What is this "root-set" memory?
Qn 2. Has it got to do with CPU intensive activities ? (why am asking this is that when i start the task, the Machine reponse to say mouse or Keyboard interrupts is too slow meaning either CPU is too busy or its some other problem i cannot explain for now)
Qn 3. Can such an error be avoided? and how ?
The fork: not enough space message suggests this is a problem with the operating system setup, but:
Q1 - The Root Set
The Root Set is what the garbage collector uses as a starting point when it searches for data that is live in the heap. It usually starts off from the registers of the VM and off from the stack, if the stack has references to heap data that still needs to be live. There may be other roots in Erlang I am not aware of, but these are the basic stuff you start off from.
That it is a reallocation error of exactly 8 Megabyte space could mean one of two things. Either you don't have 8 Megabyte free in the heap, or that the heap is fragmented beyond recognition, so while there are 8 megabytes in it, there are no contiguous such space.
Q2 - CPU activity impact
The problem has nothing to do with the CPU per se. You are running out of memory. A large root set could indicate that you have some very deep recursions going on where you keep around a lot of pointers to data. You may be able to rewrite the code such that it is tail-calling and uses less memory while operating.
You should be more worried about the slow response times from the keyboard and mouse. That could indicate something is not right. Does a vmstat 1, a sysstat, a htop, a dstat or similar show anything odd while the process is running? You are also on the hunt to figure out if the kernel or the C libc is doing something odd here due to memory being constrained.
Q3 - How to fix
I don't know how to fix it without knowing more about what the application is doing. Since you have a crash dump, your first instinct should be to take the crash dump viewer and look at the dump. The goal is to find a process using a lot of memory, or one that has a deep stack. From there on, you can seek to limit the amount of memory that process is using. either by rewriting the code so it can give memory up earlier, by tuning the garbage collection setup for the process (see the spawn options in the erlang man pages for this), or by adding more memory to the system.