I am running an express application which uses imagemagick to take a screenshot of a webpage and upload the result to S3 and insert a row into our database.
The service works well however after a few requests eventually the server just hangs upon request. There is no response, but the server is technically not 'down'.
Is there any way for me to see logs, or anything that can be causing this such as memory leak?
You should profile your application to check when the bottleneck happens and where it happens in your code. A good starting point is with node's integrated V8 stack profiling module.
Related
I have looked at the request trace for several requests that resulted in the same outcome.
What will happen is I'll get a HttpModule="iisnode", Notification="EXECUTE_REQUEST_HANDLER", HttpStatus=500, HttpReason="Internal Server Error", HttpSubstatus=1013, ErrorCode="The pipe has been ended. (0x6d)"
This is a production API. Fewer than 1% of requests get this result but it's not the requests themselves - I can reissue the same request and it'll work.
I log telemetry for every API request - basics on the way in, things like http status and execution time as the response is on its way out.
None of the requests that get this error are in telemetry which makes me think something is happening somewhere between IIS and iisnode.
If anyone has resolved this or has solid thoughts on how to pin down what the root issue is I'd appreciate it.
Well for me, what's described here covered the bulk of the issue: github.com/Azure/iisnode/issues/57 Setting keepAliveTimeout to 0 on the express server reduced the 500s significantly.
Once the majority of the "noise" was eliminated it was much easier to correlate the remaining 500s that would occur to things I could see in my logs. For example, I'm using a 3rd party node package to resize images and a couple of the "images" that were loaded into the system weren't images. Instead of gracefully throwing an exception, the package seems to exit out of the running node process. True story. So on Azure, it would get restarted, but while that was happening requests would get a 500 internal server error.
i have a nodejs application, which using express.
My problem:
The app working well on localhost, until i stop that
I bought a shared hosting on Namecheap.com and deployed my app
Everything working, except one thing, this is the 24/7 availability
After half hour of idle (When server does not get request(s)) application shuts down, until it gets new request, and this causes high load time
There are no errors
My question is, is there anything that i can do to prevent this?
Like a custom code that does not let server to go idle?
Application uses: Express, MySql, Express-session, nodecache, Body and cookie parser, gz compression, nodemailer, path, fs
I am using pm2.
Code is long so i won't paste here, if you have suggestions i'll check and provide you more info!
(App registered in cPanel Application Manager powered by Phusion Passanger, NodeEnv=Production)
I have an express server inside a Cloud Run Docker container.
I'm getting those logs:
Are those generated by the express package? Or are those logs somehow generated by cloud run?
Here is what express docs says: https://expressjs.com/en/guide/debugging.html
Express uses the debug module internally to log information about route matches, middleware functions that are in use, application mode, and the flow of the request-response cycle.
But it does not give much detail on what those logs are and how you enable or disable them.
Should I leave them on? Won't it hurt my server's performance if it's going to log every request like that? This is NODE_ENV === "production.
These logs entry are generated by the Cloud Run runtime platform. It's not your Express server. The performance aren't impacted by this log, and in any cases, you can't deactivate them.
You could exclude them to save space in logging capacity (and save money), but I don't recommend this. They brings 3 of 4 golden signals of your application (error rate, latency, traffic). Very important for production monitoring
We have a C# Web API server and a Node Express server. We make hundreds of requests from the C# server to a route on the Node server. The route on the Node server does intensive work and often doesn't return for 6-8 seconds.
Making hundreds of these requests simultaneously seems to cause the Node server to fail. Errors in the Node server output include either socket hang up or ECONNRESET. The error from the C# side says
No connection could be made because the target machine actively refused it.
This error occurs after processing an unpredictable number of the requests, which leads me to think it is simply overloading the server. Using a Thread.Sleep(500) on the C# side allows us to get through more requests, and fiddling with the wait there leads to more or less success, but thread sleeping is rarely if ever the right answer, and I think this case is no exception.
Are we simply putting too much stress on the Node server? Can this only be solved with Load Balancing or some form of clustering? If there is an another alternative, what might it look like?
One path I'm starting to explore is the node-toobusy module. If I return a 503 though, what should be the process in the following code? Should I Thread.Sleep and then re-submit the request?
It sounds like your node.js server is getting overloaded.
The route on the Node server does intensive work and often doesn't return for 6-8 seconds.
This is a bad smell - if your node process is doing intense computation, it will halt the event loop until that computation is completed, and won't be able to handle any other requests. You should probably have it doing that computation in a worker process, which will run on another cpu core if available. cluster is the node builtin module that lets you do that, so I'll point you there.
One path I'm starting to explore is the node-toobusy module. If I return a 503 though, what should be the process in the following code? Should I Thread.Sleep and then re-submit the request?
That depends on your application and your expected load. You may want to refresh once or twice if it's likely that things will cool down enough during that time, but for your API you probably just want to return a 503 in C# too - better to let the client know the server's too busy and let them make their own decision then to keep refreshing on its behalf.
I'm developing a node web application. And, while testing around, one of the client chrome browser went into hung state. The browser entered into an infinite loop where it was continuously downloading all the JavaScript files referenced by the html page. I rebooted the webserver (node.js), but once the webserver came back online, it continued receiving tons of request per second from the same browser in question.
Obviously, I went ahead and terminated the client browser so that the issue went away.
But, I'm concerned, once my web application go live/public, how to handle such problem-client-connections from the server side. Since I will have no access to the clients.
Is there anything (an npm module/code?), that can make best guess to handle/detect such bad client connections from within my webserver code. And once detected, ignore any future requests from that particular client instance. I understand handling within the Node server might not be the best approach. But, at least I can save my cpu/network by not rendering to the bad requests.
P.S.
Btw, I'm planning to deploy my node web application onto Heroku with a small budget. So, if you know of any firewall/configuration that could handle the above scenario please do recommend.
I think it's important to know that this is a pretty rare case. If your application has a very large user base or there is some other reason you are concerned with DOS/DDOS related attacks, it looks like Heroku provides some DDOS security for you. If you have your own server, I would suggest looking into Nginx or HAProxy as load balancers for your app combined with fail2ban. See this tutorial.