aws-serverless-express connection error - socket hangup - node.js

I have a lambda function accessed via API gateway. The function receives username and password and performs authentication on a given active directory.
If the user is found then it calls the database to fetch the user by username > generates JWT and returns it in the response.
For last few days I have been getting CORS errors on the login page. On further inspection of the CloudWatch logs I see these errors
{
"errno": "ETIMEDOUT",
"code": "ETIMEDOUT",
"syscall": "connect",
"address": "10.1.2.39",
"port": 389
}
error.code Error: connect ETIMEDOUT 10.1.2.39:389
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
{ errno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'connect', address: '10.1.2.39', port: 389}
ERROR: aws-serverless-express connection error
ERROR Error: socket hang up
at connResetException (internal/errors.js:609:14)
at Socket.socketOnEnd (_http_client.js:458:23)
at Socket.emit (events.js:326:22)
at endReadableNT (_stream_readable.js:1241:12)
at processTicksAndRejections (internal/process/task_queues.js:84:21) { code: 'ECONNRESET'}
The weird thing is, when I try to log into the application for the first time, I get these errors, but let's say I wait for some 15-20 minutes and then try logging again, no error appears and I am logged in smoothly.
The AD is maintained by the customer and they claim nothing is wrong on their end. This issue started happening recently even though the application has been in production for the last six months.
I am using the activedirectory module for AD authentication. If I run a simple authenticate code locally with the same username and password then it works so there is no problem with AD authentication or the credentials.
I am using Node 12 in my lambda. Also I am using aws-serverless-koa and not express so I am assuming that the error is coming from some dependency (probably activedirectory).

Related

ECONNRESET - Does not gracefully throw error, but crashes web app

We have a NodeJS app running as a Azure Web App Service on a Linux based App Service Plan. (configured to be running as always on).
Setup:
NodeJS 16
App Service Plan (Linux)
Redis (Azure managed hosted service)
Application Insights (Azure managed hosted service)
Packages:
express 4.17.2
dotenv 14.2.0
redis 4.0.2
applicationinsights 2.2.0
The web service does basic data calculations returning a result as a REST API service. Redis is used to store previously calculated results.
Application Insights has been enabled on the App Service level in the portal.
For additional fault monitoring, we added the NPM Package applicationinsights version 2.2.0 in code.
Application insights is configured at start up of the app using:
const appInsights = require("applicationinsights");
appInsights.setup(process.env.APPLICATIONINSIGHTS_CONNECTION_STRING)
appInsights.start()
The app service runs for some time but then crashes unexpectedly with the following in the KUDU logs:
2022-01-20T00:41:19.028838008Z events.js:377
2022-01-20T00:41:19.029056811Z throw er; // Unhandled 'error' event
2022-01-20T00:41:19.029073211Z ^
2022-01-20T00:41:19.029079111Z
2022-01-20T00:41:19.029084211Z SocketClosedUnexpectedlyError: Socket closed unexpectedly
2022-01-20T00:41:19.029089512Z at TLSSocket.<anonymous> (/home/site/wwwroot/node_modules/#node-redis/client/dist/lib/client/socket.js:184:118)
2022-01-20T00:41:19.029095412Z at Object.onceWrapper (events.js:520:26)
2022-01-20T00:41:19.029100512Z at TLSSocket.emit (events.js:412:35)
2022-01-20T00:41:19.029105412Z at net.js:675:12
2022-01-20T00:41:19.029110212Z at TCP.done (_tls_wrap.js:563:7)
2022-01-20T00:41:19.029115112Z Emitted 'error' event on Commander instance at:
2022-01-20T00:41:19.029128012Z at RedisSocket.<anonymous> (/home/site/wwwroot/node_modules/#node-redis/client/dist/lib/client/index.js:338:14)
2022-01-20T00:41:19.029149012Z at RedisSocket.emit (events.js:400:28)
2022-01-20T00:41:19.029154512Z at RedisSocket._RedisSocket_onSocketError (/home/site/wwwroot/node_modules/#node-redis/client/dist/lib/client/socket.js:207:10)
2022-01-20T00:41:19.029159212Z at TLSSocket.<anonymous> (/home/site/wwwroot/node_modules/#node-redis/client/dist/lib/client/socket.js:184:107)
2022-01-20T00:41:19.029164013Z at Object.onceWrapper (events.js:520:26)
2022-01-20T00:41:19.029168413Z [... lines matching original stack trace ...]
2022-01-20T00:41:19.029172813Z at TCP.done (_tls_wrap.js:563:7)
I then removed the use of Redis to test the scenario without a external connection, but after some time running, the application still crashes without triggering try/catch code.
I was able to trace the following debug information:
arg0:OperationalError {cause: Error: read ECONNRESET
at TCP.onStreamRead…nternal/stream_base_commons:220:20)
at TC…, isOperational: true, errno: -4077, code: 'ECONNRESET', syscall: 'read', …}
cause:Error: read ECONNRESET\n at TCP.onStreamRead (node:internal/stream_base_commons:220:20)\n at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {errno: -4077, code: 'ECONNRESET', syscall: 'read', stack: 'Error: read ECONNRESET\n at TCP.onStreamRea…Trampoline (node:internal/async_hooks:130:17)', message: 'read ECONNRESET'}
code:'ECONNRESET'
errno:-4077
isOperational:true
syscall:'read'
message:'read ECONNRESET'
name:'Error'
stack:'Error: read ECONNRESET\n at TCP.onStreamRead (node:internal/stream_base_commons:220:20)\n at TCP.callbackTrampoline (node:internal/async_hooks:130:17)'
My local debug console points me to the file: /node_modules\diagnostic-channel-publishers\dist\src\console.pub.js:43:39 which as I understand is used to log console log events to Application Insights*.
I then removed Application Insights and the Web App has been running stable without any crashes. I re-enabled the use of Redis and no issues traced thus far. This points me to the issue being Application Insights not being able to gracefully handle a break in TCP Socket connection to the Application Insights Service.
Any way to confirm this or prevent the app to crash?
Error: read ECONNRESET\n at TCP.onStreamRead (node:internal/stream_base_commons:220:20)\n at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {errno: -4077, code: 'ECONNRESET', syscall: 'read', stack: 'Error: read ECONNRESET\n at TCP.onStreamRea…Trampoline (node:internal/async_hooks:130:17)', message: 'read ECONNRESET'}
"ECONNRESET" is commonly thrown when the other end of a TCP connection closes its end because of any protocol-related issues and since no one is listening to the 'error' event it gets thrown. To cope with it, you need set up a listener that can handle such an erroneous condition.
Application Insights not being able to gracefully handle a break in TCP Socket connection
The number of outbound connections that can be made is limited. The maximum number of outbound connections is determined by the size of the worker used.
For more information, please refer this MSFT documentation

Getting error in node js on running application

I am working on a project where I am using angular in frontend and node on the backend. Everything working fine on the local machine. But on production sometimes I am getting the below error on node side and everything stops working.
I have used https://www.npmjs.com/package/sync-request module in my application
uncaughtException: nodeNC failed:
events.js:292
throw er; // Unhandled 'error' event
^
Error: connect ECONNREFUSED 127.0.0.1:35701
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1141:16)
Emitted 'error' event on Socket instance at:
at emitErrorNT (internal/streams/destroy.js:92:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
at processTicksAndRejections (internal/process/task_queues.js:84:21) {
errno: 'ECONNREFUSED',
code: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 35701
}
Error: connect ECONNREFUSED 127.0.0.1:35701 means you may not have started your server and it is not listening to the request
From node.js docs:
ECONNREFUSED (Connection refused): No connection could be made because the target machine actively refused it. This usually results from trying to connect to a service that is inactive on the foreign host.
This is expected. Since you are making HTTP requests using a synchronous package, which states in the docs:
N.B. You should not be using this in a production application. In a node.js application you will find that you are completely unable to scale your server. In a client application you will find that sync-request causes the app to hang/freeze. Synchronous web requests are the number one cause of browser crashes. For production apps, you should use then-request, which is exactly the same except that it is asynchronous.
I would not personally use a package that has been updated for over two years, when during the same period node.js, JS, and browser technologies have updated their APIs rapidly.
The suggestion I can only give you is to use an asynchronous package, in line with the asynchronous architecture of node, that does that in a non-blocking, event-driven way. Choice is yours but bear in mind that you WILL run into these issues from time to time becasue of these design decisions.
If you need additional help, you need to show your server side code so we can redesign the HTTP handler to be asynchronous and eliminate the ECONNREFUSED error at runtime.
Does this help?

Nodejs Error "EPROTO" when using GitHub Webhook to forward to Jenkins using dockerimage

I'm using a jenkins server behind a firewall. I used smee-client smee.io to get the webhooks from GitHub through the firewall.
I used the dockerimage from deltaprojects/smee-client. It is running and connects to smee.io/xyz to get the webhooks. But if GitHub sending a webhook (configured sending it to smee.io/xyz) it was successfull with a 200 Response.
But the smee-client ist throwing some EPROTO Errors from nodejs. (see output below)
Config Github webhook:
Payload url https://smee.io/xyz
Content type application/json
Enable SSL verification
* Send me everything
[*] active
Webhooks seems to work and get a 200 HTML Response
The smee-client is showing the following Error:
{ Error: write EPROTO 140483050982248:error:1408F10B:SSL
routines:ssl3_get_record:wrong version
number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:
at WriteWrap.afterWrite [as oncomplete] (net.js:788:14)
errno: 'EPROTO',
code: 'EPROTO',
syscall: 'write',
response: undefined }
{ Error: write EPROTO 140483050982248:error:1408F10B:SSL
routines:ssl3_get_record:wrong version
number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:
at WriteWrap.afterWrite [as oncomplete] (net.js:788:14)
errno: 'EPROTO',
code: 'EPROTO',
syscall: 'write',
response: undefined }
I tried to build the image myself but with the same result in Error massage.
I'm not that fammiliar with ssl certificates or even if this problem is related to ssl.
Maybe someone faced this problem as well and know a hint what i'm doing wrong? That would be really nice
Got solved by forwarding from smee-client to jenkins with http:// instead of https://
This Error message was kind of misleading

Node.js http server crashes with unknown IP address displayed

I am running a Node.js HTTP server, and suddenly the following message is displayed, and the server crashes:
UPDATED ERROR LOG: (after edit)
Uncaught exception in main server
{ Error: connect ETIMEDOUT 139.99.8.126:80
at Object.exports._errnoException (util.js:1018:11)
at exports._exceptionWithHostPort (util.js:1041:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1086:14)
code: 'ETIMEDOUT',
errno: 'ETIMEDOUT',
syscall: 'connect',
address: '139.99.8.126',
port: 80 }
Uncaught exception in main server
{ Error: socket hang up
at createHangUpError (_http_client.js:253:15)
at Socket.socketCloseListener (_http_client.js:285:23)
at emitOne (events.js:101:20)
at Socket.emit (events.js:188:7)
at TCP._handle.close [as _onclose] (net.js:497:12) code: 'ECONNRESET' }
This is a foreign IP address not used by us.
I tried to grep into all node modules for this IP address and see nothing.
Appears to be a hacking attempt but I have no clue where to start looking.
Any help will be appreciated. I have blocked the IP address in the firewall.
As per node.js docs about errors say:
ETIMEDOUT (Operation timed out): A connect or send request failed because the connected party did not properly respond after a period of time. Usually encountered by http or net — often a sign that a socket.end() was not properly called.
You should create a handler for errors, so your server won't crash:
netSv.on('error', function (error) {
if( error.message.code === 'ETIMEDOUT' ){
// mail the error with some additional data to you or do something with it
}
})

zoho email imap connection using node-imap is getting timeout in aws instance

I am using node-imap to connect and read zoho mail from my nodejs application, I am getting timeout err as below in AWS ubuntu instance, But in my local it is getting connected fine.
{ [Error: connect ETIMEDOUT]
code: 'ETIMEDOUT',
errno: 'ETIMEDOUT',
syscall: 'connect',
source: 'socket' }
Do i need to open any socket in AWS to use imap?
If so how i can do that?
You help regarding this is highly appreciated.
Thanks.

Resources