How to debug ECONNRESET with socket.io and express encountered when running a Nessus scan? - node.js

Question

I'm encountering ECONNRESET errors that are crashing my node server when I run a Nessus Essentials basic network scan:
node:events:505
throw er; // Unhandled 'error' event
^
Error: read ECONNRESET
at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
Emitted 'error' event on Socket instance at:
at emitErrorNT (node:internal/streams/destroy:157:8)
at emitErrorCloseNT (node:internal/streams/destroy:122:3)
at processTicksAndRejections (node:internal/process/task_queues:83:21) {
errno: -54,
code: 'ECONNRESET',
syscall: 'read'
}
I don't know exactly what Nessus is doing and I haven't managed to reproduce the error any other way, but I've identified two interesting facts:
The problem only occurs when running on Node versions < 16.16.0
It also goes away if I remove socket.io, or if I instantiate socket.io using a different port or HTTP server than my express server
For context, I've simplified my application down to a minimal project still exhibits the error:
import express from 'express';
import { Server } from 'socket.io';
import morgan from 'morgan';
import winston from 'winston';
const logger = winston.createLogger({
level: 'debug',
format: winston.format.combine(
winston.format.timestamp({ format: 'YYYY-MM-DD HH:mm:ss:ms' }),
winston.format.splat(),
winston.format.prettyPrint(),
winston.format(info => {
info.level = `[${info.level.toUpperCase()}]`;
return info;
})(),
winston.format.printf(info => {
if (typeof info.message === 'object') {
info.message = JSON.stringify(info.message, null, 3);
}
return `${info.timestamp} ${info.level} ${info.message}`;
}),
),
transports: [new winston.transports.Console()]
});
const morganMiddleware = morgan(
':remote-addr :method :url :status :response-time ms',
{
stream: {
write: message => logger.http(message.trim()),
},
});
const app = express();
app.use(morganMiddleware);
const port = 3001;
const httpServer = app.listen(port, () => {
logger.info(
`Server is now listening on port ${port} ✓`,
);
new Server(httpServer);
});
export default app;
If I remove the new Server(httpServer); line, the app doesn't crash. So the error seems to be linked to socket.io and Express sharing a connection, and something going wrong during the vulnerability scan (something causing the socket to be dropped?), but I haven't managed to debug it further.
I have tried all sorts of ways of catching the error, but the only way that works is using process.on('uncaughtException'). That's not helpful however, because at that point, there is no safe way to recover from the error. The error bypasses all the error handling of both Express and socket.io.
I could of course "solve" the problem by upgrading to a more recent version of node, but I need to understand the problem in order to be sure that it's actually fixed and won't surface again in some other form. Also, I would like to be able to make my app more resilient by catching this sort of error if they were to occur in the future.
Or perhaps there's some way to separate socket.io from express that without using separate ports (which wouldn't be practical in my use-case). Can I use proxy websocket related requests through Express to socket.io without sharing an HTTP server?
Any suggestions, either to help understand/debug the problem, or to work around it, would be welcome.

Answer 1

After much investigation, I figured out what was going on.
The ECONNRESET error I was getting occurred when Socket.io received an HTTP2 upgrade request: upgrade events are intercepted by Socket.io so that websocket upgrades can be handled, but for HTTP2 upgrades, the socket was incorrectly being disconnected without any error handler to catch the error, which in turn led to the the application crashing.
I was able to fix the bug, which was released as engine.io 6.2.1.
The reason I was not seeing the issue with versions of Node older than 16.16.0 was actually unrelated: Nessus was, for some reason, sending malformed HTTP requests for the HTTP2 upgrade, using LF as line separators instead of CRLF. Starting with v16.16.0, Node now rejects such requests with a 400 Bad Request, which avoids the Socket.io issue.