I am writing a simple port scanner using core net module from Node.js. I am getting a 'Callback was already called' error with my code. Can you spot please where the error is coming from? Below is my code:
const net = require('net')
const async = require('async')
function findPortStatus(host, port, timeout, cb) {
const socket = new net.Socket()
socket.setTimeout(timeout, () => {
// couldn't establish a connection because of timeout
socket.destroy()
return cb(null, null)
})
socket.connect(port, host, () => {
// connection established
return cb(null, port)
})
socket.on('error', (err) => {
// couldn't establish a connection
return cb(null, null)
})
}
const funcs = []
for (let port = 0; port <= 80; port++) {
funcs.push(function(callback) {
findPortStatus('192.30.253.112', port, 4000, (err, port) => {
if (!err) {
return callback(null, port)
}
})
})
}
async.parallel(funcs, (err, ports) => {
if (err) {
console.error(err.message)
} else {
for (let port of ports) {
if (port) {
console.log(port)
}
}
}
})
Not sure if this is related, but you really should pass something to the callback when you call it. null,null isn't very useful for debugging. What I would suggest is timeout events in your context are probably not errors, but they are informative. You could just cb(null, 'timeout') or cb(null, {state: 'timedOut', port: port}) or something to better keep track of what worked and what didn't.
The most likely candidate for your actual error, though, is if your socket emits an error or timeout event after the connect event was already successful. Dropped connection or the like. If all you're looking for is a 'ping'-like functionality (across more than just ICMP obviously), then you should probably close the connection as soon as you get a connect and/or remove the other event listeners as part of the connect listener's handler.
Finally, the node docs suggest you not call socket.connect() directly, unless implementing a custom socket (which it doesn't seem like you are), but to use net.createConnection() instead; not sure that'll help you but it's worth noting.
It looks like the successfully connected sockets are subsequently timing out (which makes sense, as you connect but then do nothing with the connection, so it times out).
If you disconnect from a socket once you have recorded a successful connection, then that should clear up the error.
Related
I have a node js service that consumes messages from Kafka and processes it through various steps of transformation logic. During the processing, services use Redis and mongo for storage and caching purposes. In the end, it sends the transformed message to another destination via UDP packets.
On startup, it starts consuming message from Kafka after a while, it crashes down with the unhandled error: ERR_CANNOT_SEND unable to send data(see below picture).
restarting the application resolves the issue temporarily.
I initially thought it might have to do with the forwarding through UDP sockets, but the forwarding destinations are reachable from the consumer!
I'd appreciate any help here. I'm kinda stuck here.
Consumer code:
const readFromKafka = ({host, topic, source}, transformationService) => {
const logger = createChildLogger(`kafka-consumer-${topic}`);
const options = {
// connect directly to kafka broker (instantiates a KafkaClient)
kafkaHost: host,
groupId: `${topic}-group`,
protocol: ['roundrobin'], // and so on the other kafka config.
};
logger.info(`starting kafka consumer on ${host} for ${topic}`);
const consumer = new ConsumerGroup(options, [topic]);
consumer.on('error', (err) => logger.error(err));
consumer.on('message', async ({value, offset}) => {
logger.info(`recieved ${topic}`, value);
if (value) {
const final = await transformationService([
JSON.parse(Buffer.from(value, 'binary').toString()),
]);
logger.info('Message recieved', {instanceID: final[0].instanceId, trace: final[1]});
} else {
logger.error(`invalid message: ${topic} ${value}`);
}
return;
});
consumer.on('rebalanced', () => {
logger.info('cosumer is rebalancing');
});
return consumer;
};
Consumer Service startup and error handling code:
//init is the async function used to initialise the cache and other config and components.
const init = async() =>{
//initialize cache, configs.
}
//startConsumer is the async function that connects to Kafka,
//and add a callback for the onMessage listener which processes the message through the transformation service.
const startConsumer = async ({ ...config}) => {
//calls to fetch info like topic, transformationService etc.
//readFromKafka function defn pasted above
readFromKafka( {topicConfig}, transformationService);
};
init()
.then(startConsumer)
.catch((err) => {
logger.error(err);
});
Forwarding code through UDP sockets.
Following code throws the unhandled error intermittently as this seemed to work for the first few thousands of messages, and then suddenly it crashes
const udpSender = (msg, destinations) => {
return Object.values(destinations)
.map(({id, host, port}) => {
return new Promise((resolve) => {
dgram.createSocket('udp4').send(msg, 0, msg.length, port, host, (err) => {
resolve({
id,
timestamp: Date.now(),
logs: err || 'Sent succesfully',
});
});
});
});
};
Based on our comment exchange, I believe the issue is just that you're running out of resources.
Throughout the lifetime of your app, every time you send a message you open up a brand new socket. However, you're not doing any cleanup after sending that message, and so that socket stays open indefinitely. Your open sockets then continue to pile up, consuming resources, until you eventually run out of... something. Perhaps memory, perhaps ports, perhaps something else, but ultimately your app crashes.
Luckily, the solution isn't too convoluted: just reuse existing sockets. In fact, you can just reuse one socket for the entirety of the application if you wanted, as internally socket.send handles queueing for you, so no need to do any smart hand-offs. However, if you wanted a little more concurrency, here's a quick implementation of a round-robin queue where we've created a pool of 10 sockets in advance which we just grab from whenever we want to send a message:
const MAX_CONCURRENT_SOCKETS = 10;
var rrIndex = 0;
const rrSocketPool = (() => {
var arr = [];
for (let i = 0; i < MAX_CONCURRENT_SOCKETS; i++) {
let sock = dgram.createSocket('udp4');
arr.push(sock);
}
return arr;
})();
const udpSender = (msg, destinations) => {
return Object.values(destinations)
.map(({ id, host, port }) => {
return new Promise((resolve) => {
var sock = rrSocketPool[rrIndex];
rrIndex = (rrIndex + 1) % MAX_CONCURRENT_SOCKETS;
sock.send(msg, 0, msg.length, port, host, (err) => {
resolve({
id,
timestamp: Date.now(),
logs: err || 'Sent succesfully',
});
});
});
});
};
Be aware that this implementation is still naïve for a few reasons, mostly because there's still no error handling on the sockets themselves, only on their .send method. You should look at the docs for more info about catching events such as error events, especially if this is a production server that's supposed to run indefinitely, but basically the error-handling you've put inside your .send callback will only work... if an error occurs in a call to .send. If between sending messages, while your sockets are idle, some system-level error outside of your control occurs and causes your sockets to break, your socket may then emit an error event, which will go unhandled (like what's happening in your current implementation, with the intermittent errors that you see prior to the fatal one). At that point they may now be permanently unusable, meaning they should be replaced/reinstated or otherwise dealt with (or alternatively, just force the app to restart and call it a day, like I do :-) ).
I'd like to get the number of connections of a few servers running on my local machine.
I've successfully used server.getConnections() on a server created via net.createServer(), however I don't know how to use it on an already started server.
I tried obtaining the server instance by connecting to it, using net.connect(). The connection is created successfully and I get a new net.Socket object, however my understanding is that I actually need a net.Server in order to use getConnections().
So my question is, how do I get a net.Server instance of an already running server?
I realize my question is an XY problem, so apologies to all who tried to answer it.
I suspect the answer to the literal question, "how do I get an instance of an existing server", is: "you can't".
I should have added more details to the question, especially what I was trying to achieve.
My application is a load balancer / reverse proxy server. Initially I was able to use getConnections() because I would start the proxy server and a few dummy servers from the same script. However I wanted to make the dummy servers and the proxy separate from each other, so even though I did have complete control over them, I needed to pretend that I didn't actually own the servers.
The solution I found to my specific case, in the end, was to keep a hash list of servers I can connect to (via the reverse proxy), and increment the connection counters every time I connect to a specific server:
let servers = [
{ port: 4000, connectionsCounter: 0 },
{ port: 5000, connectionsCounter: 0 },
{ port: 6000, connectionsCounter: 0 },
];
let myProxyServer = net.createServer((socket) => {
// Open a connection to the first server in the list
net.connect(servers[0].port, () => {
// Once connected, increment the connections counter
socket.on('connect', () => {
servers[0].connectionsCounter++;
});
// When the connection ends, decrement the counter
socket.on('close', () => {
servers[0].connectionsCounter--;
});
});
});
I hope this will be helpful to someone.
If you want to just use the server, you can probably store it as a variable when you call net.createServer()
const my_server = net.createServer();
// do what you want with it
my_server.getConnections();
my_server.listen();
you can make instance of net.createServer() and then get your number of connections from server.on('connection', <callback>):
server.on('connection', (socket) => {
// someone connected
console.log("New active connection");
server.getConnections((err, count) => {
if(err){
console.log(err);
} else {
console.log("Currently " + count + " active connection(s)");
}
});
});
i hope this complete example code help you:
const net = require('net');
const uuid = require('uuid/v1');
const server = net.createServer((socket) => {
socket.uuid = uuid();
socket.on('data', (data) => {
//const response = JSON.parse(data.toString('utf8'));
});
socket.on('error', (err) => {
console.log('A client has left abruptly !');
server.getConnections((err, count) => {
if(err){
console.log(err);
} else {
console.log("Currently " + count + " active connection(s)");
}
});
});
socket.on('end', () => {
console.log("A client has left");
server.getConnections((err, count) => {
if(err){
console.log(err);
} else {
console.log("Currently " + count + " active connection(s)");
}
});
});
});
server.on('error', (err) => {
// handle errors here
console.log("Error:", err);
});
server.on('connection', (socket) => {
// someone connected
console.log("New active connection");
server.getConnections((err, count) => {
if(err){
console.log(err);
} else {
console.log("Currently " + count + " active connection(s)");
}
});
});
// port number.
server.listen(3000, () => {
console.log('opened server on', server.address());
});
or you can use netstat for get number of connection, https://www.npmjs.com/package/node-netstat nodejs module for this solution:
const netstat = require('node-netstat');
myObject = {
protocol: 'tcp',
};
setInterval(function () {
let count = 0;
netstat({
filter: {
local: {port: 3000, address: '192.168.1.1'}
}
}, item => {
// console.log(item);
count++;
console.log(count);
});
}, 1000);
Your application is a bit unclear.
Only the server socket can report how many connections it has. This is why if you create the net.Server you can access that information from it.
If you want to connect to an application and query the number of clients connected to it, the application that you connect to needs to provide that information to you when you ask. This is not information that the socket provides - the application itself has to provide that information.
If you are writing the application that created the net.Server, you can create another net.Server on a different port that you can then connect to and query it for information about the other clients on its other sockets.
If you are trying to generically find the number of connections to a particular application that has a socket, that application needs to be able to tell you, or, as #root mentioned, you need to ask the OS the application is running on. This function will be OS dependent and will likely require elevated privileges. But consider connecting to a socket on a router or IoT device: that application may not be running on any OS at all.
I am using nodejs for websocket connection(https://github.com/websockets/ws). I defines a connect method to return Promise to caller. This is the code:
connect = () => {
return new Promise((resolve, reject) => {
try {
this.ws = new WebSocket(this.url, {});
this.ws.on("open", () => {
console.log("connected:", this.ws.readyState);
resolve();
});
this.ws.on("close", function close() {
console.log("disconnected");
// how to throw error exception here?
});
} catch (err) {
reject(err);
}
});
};
the caller will call this method and try to reconnect if the connection failed.
connect().then(() => {
...
}).catch(() => {
// do reconnect
})
it works fine if it fails to connect on the first time. But what if the connection drops. How can I let caller knows that there is an error happens?
One solution I can think of is to create an event emitter to notify caller that we need to reconnect. But if the code is used in many places in my project, I have to pass the event emitter to many places. Is there any better way to do that?
It's unclear for me which npm package you're using but if it's ws you can refer to this part of its README and adapt to your use case if necessary:
https://github.com/websockets/ws/blob/master/README.md#how-to-detect-and-close-broken-connections
On Node.js version 10+
Say we have a server listening on a path (unix domain sockets):
const server = net.createServer(socket => {
});
const p = path.resolve(process.env.HOME + '/unix.sock');
server.listen(p);
is there a way to determine if some other server is already listening on path p, before running the above code?
An alternative to my existing answer would be creating a client and trying to connect to the server.
Based on the result, you can identify whether the unix socket is taken or not.
function isPortTaken(port, fn) {
var net = require('net');
var client = new net.Socket();
client.connect(port, function(err) {
if(err)
return fn(false)
client.destroy();
fn(true);
});
}
Warning: In case the server triggers some action on a new connection, this will trigger it.
The easy but kind-of dirty way would be to try and use the port, and see if it throws the EADDRINUSE error.
function isPortTaken(port, fn) {
var net = require('net')
var tester = net.createServer()
.once('error', function (err) {
if (err.code != 'EADDRINUSE')
fn(true)
})
.once('listening', function () {
tester.once('close', function () {
fn(false)
})
.close()
})
.listen(port)
}
The callback will give a boolean value.
I was going to write this script myself, but then found it somewhere and made small change to it. You can find the original script here:
https://gist.github.com/timoxley/1689041
I want my application (lets say a simple node file for now) to work as it is even if redis is not available. I'm not able to do it the correct way. This is what I've tried.
var redis = require('redis');
var redisClient = null;
var getRedisClient = function(){
if(redisClient){
return redisClient;
}
try {
redisClient = redis.createClient({connect_timeout : 5000, max_attempts : 1});
redisClient.on("error", function(err) {
console.error("Error connecting to redis", err);
redisClient = null;
});
return redisClient;
} catch(ex){
console.log("error initialising redis client " + ex);
return null;
}
};
try {
var client = getRedisClient();
console.log("done!");
} catch (ex){
console.log("Exception");
}
However, with this code my application exits if redis is not available (it shouldn't because i've not given a process.exit() command).
How can I solve this?
Checking for Successful Connection on Start
Using a promise, you could guarantee that at least initially, you were able to connect to redis without error within a specified time period:
const redis = require('redis');
const Promise = require('bluebird');
function getRedisClient(timeoutMs){
return new Promise((resolve, reject) => {
const redisClient = redis.createClient();
const timer = setTimeout(() => reject('timeout'), timeoutMs);
redisClient.on("ready", () => {
clearTimeout(timer);
resolve(redisClient);
});
redisClient.on("error", (err) => {
clearTimeout(timer);
reject(err);
});
});
};
const redisReadyTimeoutMs = 10000;
getRedisClient(redisReadyTimeoutMs)
.then(redisClient => {
// the client has connected to redis sucessfully
return doSomethingUseful();
}, error => {
console.log("Unable to connect to redis", error);
});
You Need Proper Error Handling
The redis client being non-null does NOT guarantee using it won't throw an error.
you could experience infrastructure misfortune e.g. crashed redis process, out of memory or network being down.
a bug in your code could cause an error e.g. invalid or missing arguments to a redis command.
You should be handling redis client errors as a matter of course.
DON'T null the Redis Client on Error
It won't give you much but it will force you to check for null every time you try and use it.
The redis client also has inbuilt reconnect and retry mechanisms that you'll miss out on if you null it after the first error. See the redis package docs, look for retry_strategy.
DO Wrap your redis client code with try .. catch ... or use .catch in your promise chain.
DO Make use of a retry_strategy.