So I have this node application running on Ubuntu. And I noticed that there're lots of threads showing up form pstree -a
└─node /bin/node --expose-gc -max-old-space-size=256 main.js
└─process.title
├─sh ...
├─sudo ...
│... bunch of scripts i'm doing
├─66*[{process.title}]
└─5*[{node}]
Sometimes there're tens of them but it could go up to hundreds. And I have no idea how are they created, what are they doing. But for sure they are eating up system resources.
This project has complex package dependencies, so it becomes extream hard for me to locate the root cause of this problem. It will be very appreciated if someone could shed some light for me on this situation.
Thanks for #jfriend00 's comment, it really helped me to narrow this down.
So asyncawait and its underlying node-fibers turn out to be the root cause in my case.
While I'm still not quite sure how this is happening (as this is a widely used module and doesn't seem to find anyone talking about this). But taking some time replacing all asyncawait with node native async brings the thread count back to 4.
Related
I'm having bit trouble with a NodeJS/Express/React application that is on production as we speak.
The problem is, that it keeps climbing up on memory usage and it just doesn't stop. It is slow and steady, and eventually Node crashes. I have several heapdumps that I have been creating with the help of node-heapdump, however, I don't know how to properly identify the leak.
I will share an image of my snapshot. Please note that I sorted by shallow size so supposedly one of those objects/types that appear on top must be the problem:
As I can see below, there is this "Promis in #585" that I see in many places and that could be the one, but I'm unable to identify that line, function or component.
Anybody could help? I can share more screenshots if you want.
Thanks.
I found the problem.
I'm using React Body Classname in my app so when we load different routes we can change the body class from client side. This npm module needs to be used with the Rewind() funcion when you do server side rendering in order to avoid memory leaks:
This is the module I'm talking about:
https://github.com/iest/react-body-classname
And, in order to avoid the memory leak, we are calling:
BodyClassName.rewind()
In the render function of our main App.js container component. This way, doesn't matter what url a user is landing on, Rewind() will always be called and so the data that can be garbage collected will be properly freed in the future.
Now our app stays at a nice and steady 120mb of memory usage.
Thanks anyway :D
I'm writing a crawler module which is calling it-self recursively to download more and more links depending on a depth option parameter passed.
Besides that, I'm doing more tasks on the returned resources I've downloaded (enrich/change it depending on the configuration passed to the crawler). This process is going on recursively until it's done which might take a-lot of time (or not) depending on the configurations used.
I wish to optimize it to be as fast as possible and not to hinder on any Node.js application that will use it.I've set up an express server that one of its routes launch the crawler for a user defined (query string) host. After launching a few crawling sessions for different hosts, I've noticed that I can sometimes get real slow responses from other routes that only return simple text.The delay can be anywhere from a few milliseconds to something like 30 seconds, and it's seems to be happening at random times (well nothing is random but I can't pinpoint the cause).I've read an article of Jetbrains about CPU profiling using V8 profiler functionality that is integrated with Webstorm, but unfortunately it only shows on how to collect the information and how to view it, but it doesn't give me any hints on how to find such problems, so I'm pretty much stuck here.
Could anyone help me with this matter and guide me, any tips on what could hinder the express server that my crawler might do (A lot of recursive calls), or maybe how to find those hotspots I'm looking for and optimize them?
It's hard to say anything more specific on how to optimize code that is not shown, but I can give some advice that is relevant to the described situation.
One thing that comes to mind is that you may be running some blocking code. Never use deep recursion without using setTimeout or process.nextTick to break it up and give the event loop a chance to run once in a while.
For over a month I'm struggling with a very annoying memory leak issue and I have no clue how to solve it.
I'm writing a general purpose web crawler based on: http, async, cheerio and nano. From the very beginning I've been struggling with memory leak which was very difficult to isolate.
I know it's possible to do a heapdump and analyse it with Google Chrome but I can't understand the output. It's usually a bunch of meaningless strings and objects leading to some anonymous functions telling me exactly nothing (it might be lack of experience on my side).
Eventually I came to a conclusion that the library I had been using at the time (jQuery) had issues and I replaced it with Cheerio. I had an impression that Cheerio solved the problem but now I'm sure it only made it less dramatic.
You can find my code at: https://github.com/lukaszkujawa/node-web-crawler. I understand it might be lots of code to analyse but perhaps I'm doing something stupid which can be obvious strait away. I'm suspecting the main agent class which does HTTP requests https://github.com/lukaszkujawa/node-web-crawler/blob/master/webcrawler/agent.js from multiple "threads" (with async.queue).
If you would like to run the code it requires CouchDB and after npm install do:
$ node crawler.js -c conf.example.json
I know that Node doesn't go crazy with garbage collection but after 10min of heavy crawling used memory can go easily over 1GB.
(tested with v0.10.21 and v0.10.22)
For what it's worth, Node's memory usage will grow and grow even if your actual used memory isn't very large. This is for optimization on behalf of the V8 engine. To see your real memory usage (to determine if there is actually a memory leak) consider dropping this code (or something like it) into your application:
setInterval(function () {
if (typeof gc === 'function') {
gc();
}
applog.debug('Memory Usage', process.memoryUsage());
}, 60000);
Run node --expose-gc yourApp.js. Every minute there will be a log line indicating real memory usage immediately after a forced garbage collection. I've found that watching the output of this over time is a good way to determine if there is a leak.
If you do find a leak, the best way I've found to debug it is to eliminate large sections of your code at a time. If the leak goes away, put it back and eliminate a smaller section of it. Use this method to narrow it down to where the problem is occurring. Closures are a common source, but also check for anywhere else references may not be cleaned up. Many network applications will attach handlers for sockets that aren't immediately destroyed.
I have been asked to debug, and improve, a complex multithreaded app, written by someone I don't have access to, that uses concurrent queues (both GCD and NSOperationQueue). I don't have access to a plan of the multithreaded architecture, that's to say a high-level design document of what is supposed to happen when. I need to create such a plan in order to understand how the app works and what it's doing.
When running the code and debugging, I can see in Xcode's Debug Navigator the various threads that are running. Is there a way of identifying where in the source-code a particular thread was spawned? And is there a way of determining to which NSOperationQueue an NSOperation belongs?
For example, I can see in the Debug Navigator (or by using LLDB's "thread backtrace" command) a thread's stacktrace, but the 'earliest' user code I can view is the overridden (NSOperation*) start method - stepping back earlier in the stack than that just shows the assembly instructions for the framework that invokes that method (e.g. __block_global_6, _dispatch_call_block_and_release and so on).
I've investigated and sought various debugging methods but without success. The nearest I got was the idea of method swizzling, but I don't think that's going to work for, say, queued NSOperation threads. Forgive my vagueness please: I'm aware that having looked as hard as I have, I'm probably asking the wrong question, and probably therefore haven't formed the question quite clearly in my own mind, but I'm asking the community for help!
Thanks
The best I can think of is to put breakpoints on dispatch_async, -[NSOperation init], -[NSOperationQueue addOperation:] and so on. You could configure those breakpoints to log their stacktrace, possibly some other info (like the block's address for dispatch_async, or the address of the queue and operation for addOperation:), and then continue running. You could then look though the logs when you're curious where a particular block came from and see what was invoked and from where. (It would still take some detective work.)
You could also accomplish something similar with dtrace if the breakpoints method is too slow.
My application is doing some heavy IO on raw /dev/sdb block device using pread64/pwrite64. Sometimes it doing just fine. Call to pread64/pwrite64 usually takes as little as 50-100us. But sometimes it takes a whole lot more, up to several seconds.
What can you recommend to find the cause of such problem?
I have not used it but I have heard about a tool called latencytop.
When it's hung like that, grab a stackshot. Another option is pstack or lsstack.
And as #Zan pointed out, latencytop could also give you that information.
That might not fully answer your question, but at least you'll know with certainty what it was trying to do when it was hung.