The App crashes if I do not call client.close in the program below.
The send of message and receiving of data works fine. But if I exit the function and comes back to it again later, the App crashes and I can not receive message anymore. I restart the Smart Phone and it works again only for the first time the function runs.
If I put client.close() inside the client.on('message',, I only get the first data from the host or source, because the socket will close prematurely. Also the App do not crash.
If I remove the client.close(), I get all the data from multiple sources saved in the array I provided let RawMessageUDP = [].
Also I confirmed that the callback function of the client.on('message', will not be executed when there are no more message in the socket.
How can I determine that there are no more message in the socket, so I can close it?
There are two hosts which receives the message and reply data string back to this App. There are no issues in the host I confirmed since they close the connection after sending.
Send_UDP_Multicast = async () => {
const message = Buffer.from('Some bytes');
const client = dgram.createSocket('udp4');
let RawMessageUDP = []
let countMessage = 0
client.on('error', (err) => {
console.log(err.stack)
client.close()
})
client.on('message', (data, rinfo) => { //Console: socket-x, bound to address: 0.0.0.0, port: 65000 max
RawMessageUDP[countMessage] = data.toString()
console.log('Receiving remote data.' + RawMessageUDP[countMessage])
countMessage++
//client.close()
})
client.send(message, 0, message.length, 1900, '239.255.255.250', (err) => {
if (err) {
console.log(err);
client.close();
}
})
}
https://github.com/jurniores/SocketUDP resolve your problem. This lib you will find how to disconnect the clients and if the clients fall down it will disconnect them and will not overload your server.
Related
I have a node js service that consumes messages from Kafka and processes it through various steps of transformation logic. During the processing, services use Redis and mongo for storage and caching purposes. In the end, it sends the transformed message to another destination via UDP packets.
On startup, it starts consuming message from Kafka after a while, it crashes down with the unhandled error: ERR_CANNOT_SEND unable to send data(see below picture).
restarting the application resolves the issue temporarily.
I initially thought it might have to do with the forwarding through UDP sockets, but the forwarding destinations are reachable from the consumer!
I'd appreciate any help here. I'm kinda stuck here.
Consumer code:
const readFromKafka = ({host, topic, source}, transformationService) => {
const logger = createChildLogger(`kafka-consumer-${topic}`);
const options = {
// connect directly to kafka broker (instantiates a KafkaClient)
kafkaHost: host,
groupId: `${topic}-group`,
protocol: ['roundrobin'], // and so on the other kafka config.
};
logger.info(`starting kafka consumer on ${host} for ${topic}`);
const consumer = new ConsumerGroup(options, [topic]);
consumer.on('error', (err) => logger.error(err));
consumer.on('message', async ({value, offset}) => {
logger.info(`recieved ${topic}`, value);
if (value) {
const final = await transformationService([
JSON.parse(Buffer.from(value, 'binary').toString()),
]);
logger.info('Message recieved', {instanceID: final[0].instanceId, trace: final[1]});
} else {
logger.error(`invalid message: ${topic} ${value}`);
}
return;
});
consumer.on('rebalanced', () => {
logger.info('cosumer is rebalancing');
});
return consumer;
};
Consumer Service startup and error handling code:
//init is the async function used to initialise the cache and other config and components.
const init = async() =>{
//initialize cache, configs.
}
//startConsumer is the async function that connects to Kafka,
//and add a callback for the onMessage listener which processes the message through the transformation service.
const startConsumer = async ({ ...config}) => {
//calls to fetch info like topic, transformationService etc.
//readFromKafka function defn pasted above
readFromKafka( {topicConfig}, transformationService);
};
init()
.then(startConsumer)
.catch((err) => {
logger.error(err);
});
Forwarding code through UDP sockets.
Following code throws the unhandled error intermittently as this seemed to work for the first few thousands of messages, and then suddenly it crashes
const udpSender = (msg, destinations) => {
return Object.values(destinations)
.map(({id, host, port}) => {
return new Promise((resolve) => {
dgram.createSocket('udp4').send(msg, 0, msg.length, port, host, (err) => {
resolve({
id,
timestamp: Date.now(),
logs: err || 'Sent succesfully',
});
});
});
});
};
Based on our comment exchange, I believe the issue is just that you're running out of resources.
Throughout the lifetime of your app, every time you send a message you open up a brand new socket. However, you're not doing any cleanup after sending that message, and so that socket stays open indefinitely. Your open sockets then continue to pile up, consuming resources, until you eventually run out of... something. Perhaps memory, perhaps ports, perhaps something else, but ultimately your app crashes.
Luckily, the solution isn't too convoluted: just reuse existing sockets. In fact, you can just reuse one socket for the entirety of the application if you wanted, as internally socket.send handles queueing for you, so no need to do any smart hand-offs. However, if you wanted a little more concurrency, here's a quick implementation of a round-robin queue where we've created a pool of 10 sockets in advance which we just grab from whenever we want to send a message:
const MAX_CONCURRENT_SOCKETS = 10;
var rrIndex = 0;
const rrSocketPool = (() => {
var arr = [];
for (let i = 0; i < MAX_CONCURRENT_SOCKETS; i++) {
let sock = dgram.createSocket('udp4');
arr.push(sock);
}
return arr;
})();
const udpSender = (msg, destinations) => {
return Object.values(destinations)
.map(({ id, host, port }) => {
return new Promise((resolve) => {
var sock = rrSocketPool[rrIndex];
rrIndex = (rrIndex + 1) % MAX_CONCURRENT_SOCKETS;
sock.send(msg, 0, msg.length, port, host, (err) => {
resolve({
id,
timestamp: Date.now(),
logs: err || 'Sent succesfully',
});
});
});
});
};
Be aware that this implementation is still naïve for a few reasons, mostly because there's still no error handling on the sockets themselves, only on their .send method. You should look at the docs for more info about catching events such as error events, especially if this is a production server that's supposed to run indefinitely, but basically the error-handling you've put inside your .send callback will only work... if an error occurs in a call to .send. If between sending messages, while your sockets are idle, some system-level error outside of your control occurs and causes your sockets to break, your socket may then emit an error event, which will go unhandled (like what's happening in your current implementation, with the intermittent errors that you see prior to the fatal one). At that point they may now be permanently unusable, meaning they should be replaced/reinstated or otherwise dealt with (or alternatively, just force the app to restart and call it a day, like I do :-) ).
I am utilizing socket.io V3.1.1 and am trying to figure out how to get the client to reconnect after the connection is disconnected due to a phone going asleep. When I run the code, it connects, refreshes an order, and then I press the power button on the iPhone to cause it to sleep. I then press the home key, and I see a disconnect event, followed by a connect event which refreshes the order. If I repeat the process, I see it disconnect, but I never see a connect event.
The question is, how does one reliably connect/reconnect and/or why don't I get additional connect events?
Here's a sample console.log output:
connect
refreshOrder 1234
disconnect due to transport error
connect
refreshOrder 1234
disconnect due to transport error
and then no more events
Here's the primary code that's involved:
const socket = io("wss://xyz.org:3000");
socket.on('connect', () => {
console.log("connect");
if (order_id !== null){
refreshOrder(order_id);
}
});
socket.on("disconnect", (reason) => {
console.log("disconnect due to " + reason);
socket.connect();
});
I also have the following event handlers, but they never get called:
socket.on('reconnect', () => {
console.log('reconnect');
if (order_id !== null){
refreshOrder(order_id);
}
});
socket.on("connect_error", (err) => {
console.log("connect_error: " + err);
setTimeout(() => {
console.log("socket.connect retry");
socket.connect();
}, 1000);
});
The answer is that you should ignore the disconnect and let the built-in reconnect code handle it. I believe that my problem was that I had looked at examples of pre-V3 code and didn't understand that it was handled differently in V3. So, a better disconnect event handler would be
socket.on("disconnect", (reason) => {
console_log("disconnect due to " + reason);
if (reason === "io server disconnect") {
// disconnect initiated by server. Manually reconnect
socket.connect();
}
});
I have a connected object that works on 3G network and that is sending data to my IP on a selectable port.
I would like to read the data sent from this object on a node.js program because all my API on my server is on node.js
For this I tried to create a 'net' server like this :
var net = require('net');
var server = net.createServer(function (socket) {
socket.on('data', function (data) {
var StrData = data.toString('ascii');
fs.appendFile("test.txt","data received : "+ StrData, function (err) {
if (err) throw err;
console.log(StrData);
});
});
});
server.listen(2000, '0.0.0.0');
And I am getting all the data sent by the object to my "test.txt" file as wanted. The problem is that it seems that the connection is closed by my server after receiving the first data, so my connected object have to create another connection every time and just can send the first data.
My question is:
How can I let the connection opened so the object can send all the data?
I also tried 'socket.io' and 'http' but I can't get what I want.
It is hard to debug the issue with the given code, can you try to add an error listener to your socket and see if any errors are triggered:
socket.on('error', (err) => {
console.log(err);
}
My guess is your client is closing the socket, thinking the request is timed out (since you are not returning a success response of some sort). net.createServer creates a TCP server which expects a response for all requests.
I'm using a service written in Node.js to receive messages via MQTT (https://www.npmjs.com/package/mqtt) which then writes to a database (SQL Server using mssql).
This will work very nicely when everything is functioning normally, I create the mqtt listener and subscribe to new message events.
However, if the connection to the DB fails (this may happen periodically due to a network outage etc.), writing the message to the database will fail and the message will be dropped on the floor.
I would like to tell the MQTT broker - "I couldn't process the message, keep it in the buffer until I can."
var mqtt = require('mqtt')
var client = mqtt.connect('mymqttbroker')
client.on('connect', function () {
client.subscribe('messagequeue')
})
client.on('message', function (topic, message) {
writeMessageToDB(message).then((result) => {console.log('success'};).catch((err) => {/* What can I do here ?*/});
})
Maybe set a timeout on a resend function? Probably should be improved to only try n times before dropping the message, but it's definitely a way to do it. This isn't tested, obviously, but it should hopefully give you some ideas...
var resend = function(message){
writeMessageToDB(message).then((result) => {
console.log('Resend success!')
})
.catch((err) => {
setTimeout(function(message){
resend(message);
}, 60000);
});
}
client.on('message', function (topic, message) {
writeMessageToDB(message).then((result) => {
console.log('success')
})
.catch((err) => {
resend(message);
});
});
I am writing a simple port scanner using core net module from Node.js. I am getting a 'Callback was already called' error with my code. Can you spot please where the error is coming from? Below is my code:
const net = require('net')
const async = require('async')
function findPortStatus(host, port, timeout, cb) {
const socket = new net.Socket()
socket.setTimeout(timeout, () => {
// couldn't establish a connection because of timeout
socket.destroy()
return cb(null, null)
})
socket.connect(port, host, () => {
// connection established
return cb(null, port)
})
socket.on('error', (err) => {
// couldn't establish a connection
return cb(null, null)
})
}
const funcs = []
for (let port = 0; port <= 80; port++) {
funcs.push(function(callback) {
findPortStatus('192.30.253.112', port, 4000, (err, port) => {
if (!err) {
return callback(null, port)
}
})
})
}
async.parallel(funcs, (err, ports) => {
if (err) {
console.error(err.message)
} else {
for (let port of ports) {
if (port) {
console.log(port)
}
}
}
})
Not sure if this is related, but you really should pass something to the callback when you call it. null,null isn't very useful for debugging. What I would suggest is timeout events in your context are probably not errors, but they are informative. You could just cb(null, 'timeout') or cb(null, {state: 'timedOut', port: port}) or something to better keep track of what worked and what didn't.
The most likely candidate for your actual error, though, is if your socket emits an error or timeout event after the connect event was already successful. Dropped connection or the like. If all you're looking for is a 'ping'-like functionality (across more than just ICMP obviously), then you should probably close the connection as soon as you get a connect and/or remove the other event listeners as part of the connect listener's handler.
Finally, the node docs suggest you not call socket.connect() directly, unless implementing a custom socket (which it doesn't seem like you are), but to use net.createConnection() instead; not sure that'll help you but it's worth noting.
It looks like the successfully connected sockets are subsequently timing out (which makes sense, as you connect but then do nothing with the connection, so it times out).
If you disconnect from a socket once you have recorded a successful connection, then that should clear up the error.