We have a NodeJS app running as a Azure Web App Service on a Linux based App Service Plan. (configured to be running as always on).
Setup:
NodeJS 16
App Service Plan (Linux)
Redis (Azure managed hosted service)
Application Insights (Azure managed hosted service)
Packages:
express 4.17.2
dotenv 14.2.0
redis 4.0.2
applicationinsights 2.2.0
The web service does basic data calculations returning a result as a REST API service. Redis is used to store previously calculated results.
Application Insights has been enabled on the App Service level in the portal.
For additional fault monitoring, we added the NPM Package applicationinsights version 2.2.0 in code.
Application insights is configured at start up of the app using:
const appInsights = require("applicationinsights");
appInsights.setup(process.env.APPLICATIONINSIGHTS_CONNECTION_STRING)
appInsights.start()
The app service runs for some time but then crashes unexpectedly with the following in the KUDU logs:
2022-01-20T00:41:19.028838008Z events.js:377
2022-01-20T00:41:19.029056811Z throw er; // Unhandled 'error' event
2022-01-20T00:41:19.029073211Z ^
2022-01-20T00:41:19.029079111Z
2022-01-20T00:41:19.029084211Z SocketClosedUnexpectedlyError: Socket closed unexpectedly
2022-01-20T00:41:19.029089512Z at TLSSocket.<anonymous> (/home/site/wwwroot/node_modules/#node-redis/client/dist/lib/client/socket.js:184:118)
2022-01-20T00:41:19.029095412Z at Object.onceWrapper (events.js:520:26)
2022-01-20T00:41:19.029100512Z at TLSSocket.emit (events.js:412:35)
2022-01-20T00:41:19.029105412Z at net.js:675:12
2022-01-20T00:41:19.029110212Z at TCP.done (_tls_wrap.js:563:7)
2022-01-20T00:41:19.029115112Z Emitted 'error' event on Commander instance at:
2022-01-20T00:41:19.029128012Z at RedisSocket.<anonymous> (/home/site/wwwroot/node_modules/#node-redis/client/dist/lib/client/index.js:338:14)
2022-01-20T00:41:19.029149012Z at RedisSocket.emit (events.js:400:28)
2022-01-20T00:41:19.029154512Z at RedisSocket._RedisSocket_onSocketError (/home/site/wwwroot/node_modules/#node-redis/client/dist/lib/client/socket.js:207:10)
2022-01-20T00:41:19.029159212Z at TLSSocket.<anonymous> (/home/site/wwwroot/node_modules/#node-redis/client/dist/lib/client/socket.js:184:107)
2022-01-20T00:41:19.029164013Z at Object.onceWrapper (events.js:520:26)
2022-01-20T00:41:19.029168413Z [... lines matching original stack trace ...]
2022-01-20T00:41:19.029172813Z at TCP.done (_tls_wrap.js:563:7)
I then removed the use of Redis to test the scenario without a external connection, but after some time running, the application still crashes without triggering try/catch code.
I was able to trace the following debug information:
arg0:OperationalError {cause: Error: read ECONNRESET
at TCP.onStreamRead…nternal/stream_base_commons:220:20)
at TC…, isOperational: true, errno: -4077, code: 'ECONNRESET', syscall: 'read', …}
cause:Error: read ECONNRESET\n at TCP.onStreamRead (node:internal/stream_base_commons:220:20)\n at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {errno: -4077, code: 'ECONNRESET', syscall: 'read', stack: 'Error: read ECONNRESET\n at TCP.onStreamRea…Trampoline (node:internal/async_hooks:130:17)', message: 'read ECONNRESET'}
code:'ECONNRESET'
errno:-4077
isOperational:true
syscall:'read'
message:'read ECONNRESET'
name:'Error'
stack:'Error: read ECONNRESET\n at TCP.onStreamRead (node:internal/stream_base_commons:220:20)\n at TCP.callbackTrampoline (node:internal/async_hooks:130:17)'
My local debug console points me to the file: /node_modules\diagnostic-channel-publishers\dist\src\console.pub.js:43:39 which as I understand is used to log console log events to Application Insights*.
I then removed Application Insights and the Web App has been running stable without any crashes. I re-enabled the use of Redis and no issues traced thus far. This points me to the issue being Application Insights not being able to gracefully handle a break in TCP Socket connection to the Application Insights Service.
Any way to confirm this or prevent the app to crash?
Error: read ECONNRESET\n at TCP.onStreamRead (node:internal/stream_base_commons:220:20)\n at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {errno: -4077, code: 'ECONNRESET', syscall: 'read', stack: 'Error: read ECONNRESET\n at TCP.onStreamRea…Trampoline (node:internal/async_hooks:130:17)', message: 'read ECONNRESET'}
"ECONNRESET" is commonly thrown when the other end of a TCP connection closes its end because of any protocol-related issues and since no one is listening to the 'error' event it gets thrown. To cope with it, you need set up a listener that can handle such an erroneous condition.
Application Insights not being able to gracefully handle a break in TCP Socket connection
The number of outbound connections that can be made is limited. The maximum number of outbound connections is determined by the size of the worker used.
For more information, please refer this MSFT documentation
Related
I am testing a nodejs app on GCP functions and a MongoDB instance in GCP cloud run(I know this is not a good idea) but again this is a test. I am able to get the mongo image running on cloud run and I am given a service endpoint as https://mongodb.foo.bar.run.app and I have the container port in this cloud run service as 27017.
When I try and edit the mongoose.connect() method to include this service uri I am not able to connect with the app. I have the mongodb cloud run instance open to receive traffic from all traffic(Again not a good idea), but I still can't get the node app to connect.
I have setup the connection in node as mongoose.connect('mongodb://mongodb.foo.bar.run.app:27017/test');
but I get this error
Error: connect ETIMEDOUT <IP ADDRESS GOES HERE>:27017
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1146:16)
at TCPConnectWrap.callbackTrampoline (internal/async_hooks.js:129:14)
Emitted 'error' event on NativeConnection instance at:
at NativeConnection.Connection.error (/Users/foo/nodejs/bar/node_modules/mongoose/lib/connection.js:443:8)
at /Users/foo/nodejs/bar/node_modules/mongoose/lib/connection.js:472:15
at /Users/foo/nodejs/bar/node_modules/mongoose/lib/drivers/node-mongodb-native/connection.js:59:21
at /Users/foo/nodejs/bar/node_modules/mongodb/lib/db.js:232:14
at Server.<anonymous> (/Users/foo/nodejs/bar/node_modules/mongodb/lib/server.js:240:9)
at Object.onceWrapper (events.js:421:26)
at Server.emit (events.js:314:20)
at Pool.<anonymous> (/Users/foo/nodejs/bar/node_modules/mongodb-core/lib/topologies/server.js:308:68)
at Pool.emit (events.js:314:20)
at Connection.<anonymous> (/Users/foo/nodejs/bar/node_modules/mongodb-core/lib/connection/pool.js:115:12)
at Object.onceWrapper (events.js:421:26)
at Connection.emit (events.js:314:20)
at Socket.<anonymous> (/Users/foo/nodejs/bar/node_modules/mongodb-core/lib/connection/connection.js:144:49)
at Object.onceWrapper (events.js:421:26)
at Socket.emit (events.js:314:20)
at emitErrorNT (internal/streams/destroy.js:100:8) {
name: 'MongoError'
}
I tried to curl the service uri as
curl https://mongodb.foo.bar.run.app
and I get a 200 response in the mongo logs GET 200 466B but if I
curl mongodb.foo.bar.run.app
I get a 302 response code GET 302 0B.
I looked at the mongoose documentation but I couldn't figure what I was doing wrong. I know the documentation states user and password on the uri string, but when I run it locally on my docker engine I did not use a user and password and it worked.
As mentioned in the comments, it's not possible to run MongoDB on Cloud Run since as mentioned here:
Cloud Run can only receive HTTP requests or Pub/Sub push events.
Connecting to MongoDB requires a TCP protocol rather than HTTP, that's why every call to it would fail even if the port is reachable.
Cloud Run is in this sense more similar to AWS Lambda than it's to AWS Fargate as it's even billed on a per request basis rather than on resources consumed.
I have a lambda function accessed via API gateway. The function receives username and password and performs authentication on a given active directory.
If the user is found then it calls the database to fetch the user by username > generates JWT and returns it in the response.
For last few days I have been getting CORS errors on the login page. On further inspection of the CloudWatch logs I see these errors
{
"errno": "ETIMEDOUT",
"code": "ETIMEDOUT",
"syscall": "connect",
"address": "10.1.2.39",
"port": 389
}
error.code Error: connect ETIMEDOUT 10.1.2.39:389
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
{ errno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'connect', address: '10.1.2.39', port: 389}
ERROR: aws-serverless-express connection error
ERROR Error: socket hang up
at connResetException (internal/errors.js:609:14)
at Socket.socketOnEnd (_http_client.js:458:23)
at Socket.emit (events.js:326:22)
at endReadableNT (_stream_readable.js:1241:12)
at processTicksAndRejections (internal/process/task_queues.js:84:21) { code: 'ECONNRESET'}
The weird thing is, when I try to log into the application for the first time, I get these errors, but let's say I wait for some 15-20 minutes and then try logging again, no error appears and I am logged in smoothly.
The AD is maintained by the customer and they claim nothing is wrong on their end. This issue started happening recently even though the application has been in production for the last six months.
I am using the activedirectory module for AD authentication. If I run a simple authenticate code locally with the same username and password then it works so there is no problem with AD authentication or the credentials.
I am using Node 12 in my lambda. Also I am using aws-serverless-koa and not express so I am assuming that the error is coming from some dependency (probably activedirectory).
I am working on a project where I am using angular in frontend and node on the backend. Everything working fine on the local machine. But on production sometimes I am getting the below error on node side and everything stops working.
I have used https://www.npmjs.com/package/sync-request module in my application
uncaughtException: nodeNC failed:
events.js:292
throw er; // Unhandled 'error' event
^
Error: connect ECONNREFUSED 127.0.0.1:35701
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1141:16)
Emitted 'error' event on Socket instance at:
at emitErrorNT (internal/streams/destroy.js:92:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
at processTicksAndRejections (internal/process/task_queues.js:84:21) {
errno: 'ECONNREFUSED',
code: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 35701
}
Error: connect ECONNREFUSED 127.0.0.1:35701 means you may not have started your server and it is not listening to the request
From node.js docs:
ECONNREFUSED (Connection refused): No connection could be made because the target machine actively refused it. This usually results from trying to connect to a service that is inactive on the foreign host.
This is expected. Since you are making HTTP requests using a synchronous package, which states in the docs:
N.B. You should not be using this in a production application. In a node.js application you will find that you are completely unable to scale your server. In a client application you will find that sync-request causes the app to hang/freeze. Synchronous web requests are the number one cause of browser crashes. For production apps, you should use then-request, which is exactly the same except that it is asynchronous.
I would not personally use a package that has been updated for over two years, when during the same period node.js, JS, and browser technologies have updated their APIs rapidly.
The suggestion I can only give you is to use an asynchronous package, in line with the asynchronous architecture of node, that does that in a non-blocking, event-driven way. Choice is yours but bear in mind that you WILL run into these issues from time to time becasue of these design decisions.
If you need additional help, you need to show your server side code so we can redesign the HTTP handler to be asynchronous and eliminate the ECONNREFUSED error at runtime.
Does this help?
I'm trying to setup a Node.js Websocket server over at my webhost (Namecheap). However, I am getting this error:
events.js:292
throw er; // Unhandled 'error' event
^
Error: Unexpected server response: 301
at ClientRequest.<anonymous> (D:\****\node_modules\ws\lib\websocket.js:576:7)
at ClientRequest.emit (events.js:315:20)
at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:596:27)
at HTTPParser.parserOnHeadersComplete (_http_common.js:119:17)
at TLSSocket.socketOnData (_http_client.js:469:22)
at TLSSocket.emit (events.js:315:20)
at addChunk (_stream_readable.js:295:12)
at readableAddChunk (_stream_readable.js:271:9)
at TLSSocket.Readable.push (_stream_readable.js:212:10)
at TLSWrap.onStreamRead (internal/stream_base_commons.js:186:23)
Emitted 'error' event on WebSocket instance at:
at abortHandshake (D:\****\node_modules\ws\lib\websocket.js:694:15)
at ClientRequest.<anonymous> (D:\****\node_modules\ws\lib\websocket.js:576:7)
[... lines matching original stack trace ...]
at TLSSocket.Readable.push (_stream_readable.js:212:10)
The code for the client and server came from here:
https://github.com/websockets/ws#Simple server
https://github.com/websockets/ws#sending-and-receiving-text-data
with ws://www.host.com/path changed to ws://<mydomainname>/<dir1>/<dir2>
Both client and server code runs properly when run locally. Http related code runs properly when used on my webhost. I am trying to get the client code to run locally and connect via websockets to my server hosted at Namecheap.
I suspect something related to the server is preventing connection. I'd ask support but I doubt if they can resolve this. Maybe there's a locked feature or something? Is there something else I could have missed?
Websockets cannot work on Namecheap's hosted servers. Their incoming ports are blocked and cannot be opened for security reasons per their customer's support.
If anyone is looking for a solution just in case, try out a Heroku free account.
I have my MongoDB server running on localhost:27017, and while I can usually run my Node.js app fine, when I disconnect from the internet Mongoose throws the error
Error: failed to connect to [localhost:27017]
Note that I can still connect to the MongoDB server from the Mongo shell client. Also, if I start up my app first and then lose internet connection, my app can access the database fine offline. So why can't it start up without internet?
EDIT: here is the error in full
events.js:85
throw er; // Unhandled 'error' event
^
Error: failed to connect to [localhost:27017]
at null.<anonymous> (<My App>\node_modules\mongoose\
node_modules\mongodb\lib\mongodb\connection\server.js:555:74)
at emit (events.js:118:17)
at null.<anonymous> (<My App>\node_modules\mongoose\
node_modules\mongodb\lib\mongodb\connection\connection_pool.js:156:15)
at emit (events.js:110:17)
at Socket.<anonymous> (<My App>\node_modules\mongoos
e\node_modules\mongodb\lib\mongodb\connection\connection.js:534:10)
at Socket.emit (events.js:107:17)
at net.js:923:16
at process._tickCallback (node.js:355:11)
[nodemon] app crashed - waiting for file changes before starting...
Edit: wording
Use 127.0.0.1 instead of localhost. By turning off your wifi interface the OS is no longer able to resolve localhost.
Consider node-offline-localhost.
Add the following before the breaking code:
require('node-offline-localhost').always();
And it just works (hopefully), at least until RFC 3493 gets fixed.
Full disclosure: I authored this package to streamline https in my dev environment when offline.