So I have an express server that tries to send back a lot of JSON data (625,763,181 chars if stringified) with res.send(data). This crashes the application with
`FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory``
(more info can be found on my last question on this topic)
Here's a few snapshots of the heap before and after it tries to send the data:
As you can see the memory goes from 28MB to 700MB and 97% of it is allocated as strings (and inspecting it I see that it's the JSON data stringified).
So my question is: Am I doing anything wrong here, should I compress the JSON somehow or use something other than res.send? or can I just not send that much data?
OBS: I know I can add --max-old-space-size=8192 but I'm trying to host the project on a server with limited space that runs out with or without that flag.
EDIT: I'm leaving this up because there might be optimisations I hadn't thought of, and I'm really interested. But I found a bug in my data causing it to grow in O(n*m) instead of just O(n+m), reducing the size from 625,763,181 chars to 131,700 chars.
Related
Context:
I have an Node.js application which memory seems to very high, I don't know if that is memory leak or not, because it is reduced after certain period time, but some times on heavy load it keeps on increasing takes much longer to get reduced.
So going through the articles and couple of videos, i figured that i have to heap snapshot and analyse what is causing the memory leak.
Steps:
I have taken 4 snap shots as of now in my local to reproduce the memory leak.
Snapshot 1: 800MB
Snapshot 2: 1400MB
Snapshot 3: 1600MB
Snapshot 4: 2000+MB
When i uploaded the heapdump files to chrome dev tools I see there a lot of information but i don't know how to proceed from there.
Please check below screenshot, it says there is constructor [array] which has 687206 as shallow Size & Retained Size is 721414 in the columns, so when expanded that constructor i can see there are 4097716 constructors created ( refer the second screenshot attached below ).
Question
What does internal array [] means ? Why is there 4097716 created ?
How can a filter out the constructor which created by my app and showing me that instead of some system/v8 engine constructor ?
In the same screenshot one of the constructor uses global variable called tenantRequire function, this is custom global function which is being used internally in some places instead of normal Node.js require, I see the variable across all the constructor like "Array", "Object". This is that global tenantRequire code for reference. It is just patched require function with trycatch. Is this causing the memory leak somehow ?
Refer screenshot 3, [string] constructor it has 270303848 as shallow size. When i expanded it shows modules loaded by Node.js. Question why is this taking that much size ? & Why is my lodash modules are repeated in that string constructor ?
Without knowing much about your app and the actions that cause the high memory usage, it's hard to tell what could be the issue. Which tool did you use to record the heap snapshot? What is the sequence of operations you did when you recorded the snapshot? Could you add this information to your question?
A couple of remarks
You tagged the question with node.js and showed Chrome DevTools. That's ok. You can totally take a heap snapshot of a Node.js application and analyze it in Chrome DevTools. But since both Node.js and Chrome use the same JS engine (V8) and the same garbage collector (Orinoco), it might be a bit confusing for someone who reads the question. Just to make sure I understand it correctly: the issue is in a Node.js app, not in a browser app. And you are using Chrome just to analyze the heap snapshot. Right?
Also, you wrote that you took the snapshots to reproduce the memory leak. That's not correct. You performed some action which you thought would cause a high memory usage, recorded a heap snapshot, and later loaded the snapshot in Chrome DevTools to observe the supposed memory leak.
Trace first, profile second
Every time you suspect a performance issue, you should first use tracing to understand which functions in your applications are problematic (i.e. slow, create a lot of objects that have to be garbage-collected, etc).
Then, when you know which functions to focus on, you can profile them.
Try these visual tools
There are a few tools that can help you with tracing/profiling your app. Have a look a FlameScope (a web app) and node-clinic (a suite of tools). There is also Perfetto, but I think it's for Chrome apps, not Node.js apps.
I also highly recommend the V8 blog.
Recently I started my first project with node.js, and I can definitely say I'm loving it. Very powerful with all the modules; however, it seems I'm having a "slight" memory leak that causes my server to crash after about an hour (hit's 99-100% CPU). I've been trying to fix this problem for a while now.
Luckily, after a bit of searching, I found a popular tool called memwatch. I of course installed the module, and started logging memory usage/storage of my server's process.
Eventually, after looking through the logs, I have found the likely cause.
{
"what": "String",
"size_bytes": 9421368,
"size": "8.98 mb",
"+": 16635,
"-": 533
}
Of course, within thirty seconds this little bugger managed an increase of 9mb (very unusual). This is nice and dandy to know that my memory leak seems to be of type string, but where exactly do I go from here? Is there any way I can get more accurate results?
I looked through my code, but there really isn't a string in my code that could possibly grow like this. Is there a possibility this string isn't actually a part of my code, and more a part of node or the Socket.IO module?
Right approach. Use StrongOps (Previously Nodefly) to profile memory. Isolate type of leaking object. Look at heap retained sizes as well as the instance counts. Growing Instance counts with steady workload will point at few smoking guns.
I believe StrongOps uses memwatch + some V8 profiler/GC code under the hood. Better automation. See link - http://strongloop.com/node-js-performance/strongops/
Then used node-heapdump module, that their co-founder (core contributor Ben Noordhuis) wrote to isolate the leak down to collection object, GC roots and line of code.
See blog from Ben - http://strongloop.com/strongblog/how-to-heap-snapshots/
You can use node-heapdump module, to make a dump of the V8 heap for later inspection, so you will be able to see more accurate results.
After you will make a heapdump, analyze it with Chrome DevTools:
https://developers.google.com/chrome-developer-tools/docs/javascript-memory-profiling
As Shubhra suggested, another tool to consider in helping you diagnose your memory leak is the heap profiler of StrongOps monitoring. You can easily get started in a few steps here: http://docs.strongloop.com/display/DOC/Setting+up+StrongOps+monitoring
This will save you time from having to dig through logs and gives you a visual of what's going on in your applications heap over time, as well as comparing the String to other likely culprits causing your memory leak.
You can find more information here: http://docs.strongloop.com/display/DOC/Profiling#Profiling-Memoryprofiler
I'm creating bower package search site (everything is open sourced) and I hit the wall. I have some memory leak (or I think I have) and I honestly don't know why it is there.
You can download it and run on Your own, but simple hint will help me greatly.
I have narrowed it down to this function call here https://github.com/kamilbiela/bowereggs-backend/blob/master/main.js#L14 ( nest.fetchAndSave() ) which is all defined here: https://github.com/kamilbiela/bowereggs-backend/blob/master/lib/nest.js
Basically it downloads a package list from internet, Json.parse and inserts it into database, plus some when.js promises.
Running this function few times creates a 30mb of memory per run, that is not cleaned by garbage collector. Also note that this is my first "real" node.js project, so I'll be really grateful for any tip.
For anyone having the same problem:
https://github.com/c4milo/node-webkit-agent
After making few heap dumps I discovered that objects are garbage collected and the real memory usage isn't tied to it. I think that real memory usage is bigger because of using mongo and other non node.js stuff. Also real memory usage stabilizes at ~300mb, heap dumps at ~35mb.
I am looking after an app built with Node JS that's producing some interesting issues. It was originally running on Node JS v0.3.0 and I've since upgraded to v0.10.12. We're using Node JS to render charts on the server and we've noticed the memory usage keeps climbing chart after chart.
Q1: I've been monitoring the RES column in top for the Node JS process, is this correct or should I be monitoring something else?
I've been setting variables to null to try and reallocate memory back to the system resources (I read this somewhere as a solution) and it makes only a slight difference.
I've pushed the app all the way to 1.5gb and it then ceases to function and the process doesn't appear to die. No error messages which I found odd.
Q2: Is there anything else I can do?
Thanks
Steve
That is a massive jump in versions. You may want to share what code changes you may have made to get it working on latest stable. The api is not the same as back in v0.3, so that may be part of the problem.
If not then the issue you see it more likely from heap fragmentation than from an actual leak. In later v8 versions garbage collection is more liberal with cleanup to improve performance. (see http://code.google.com/p/chromium/issues/detail?id=112386 for some discussion on this)
You may try running the application with --max_old_space_size=32 which will limit the amount of memory v8 can use to around 32MB. Note the docs say "max size of the old generation", so it won't be exactly 32MB. Just around it, for lack of a better technical explanation.
Also you can track the amount of external memory usage with --trace_external_memory. This will allow you to know if external memory (i.e. Buffers) are being retained in your application.
You're note on the application hanging around 1.5GB would tell me you're probably on a 64-bit system. You only mentioned it ceases to function, but didn't note if the CPU is spinning during that time. Also since I don't have example code I'm not sure of what might be causing this to happen.
I'd try running on latest development (v0.11.3 at the time of this writing) and see if the issue is fixed. A lot of performance/memory enhancements are being worked on that may help your issue.
I guess you have somewhere a memory leak (in form of a closure?) that keeps the (not longer used?) diagrams(?) somewhere in memory.
The v8 sometimes needs a bit tweaking when it comes to > 1 GB of memory. Try out --noincremental_marking and/or --max_old_space_size=81920000 (if you have 8 GB available).
Check for more options with node --v8-options and go through the --trace*-parameters to find out what slows down/stops node.
I have a web application deployed in Oracle iPlanet web server 7. Website is used actively in Internet.
After deploying, heap size is growing and after 2 or 3 weeks, OutOfMemory error is thrown.
So I began to use profiling tool. I am not familiar with heap dump. All I noticed that char[], hashmap and String objects occupy too much at heap. How can I notice what causes memory leak from heap dump? My assumptations about my memory leak;
I do so much logging in code using log4j for keeping in log.txt file. Is there a problem with it?
may be an error removing inactive sessions?
some static values like cities, gender type stored in static hashmap ?
I have a login mechanism but no logout mechanism. When site is opened again, new login needed. (silly but not implemented yet.) ?
All?
Do you have an idea about them or can you add another assumptions about memory leak?
Since Java has garbage collection a "memory leak" would usually be the result of you keeping references to some objects when they shouldn't be kept alive.
You might be able to see just from the age of the objects which ones are potentially old and being kept around when they shouldn't.
log4j shouldn't cause any problems.
The hashmap should be okay, since you actually want to keep these values around.
Inactive sessions might be the problem if they're stored in memory and if something keeps references to them.
There is one more thing you can try: new project, Plumbr, which aims to find memory leaks in java applications. It is in beta stage, but should be stable enough to give it a try.
As a side node, Strings and char[] are almost always on top of the profilers' data. This rarely means any real problem.