NodeJS and RabbitMQ, how to be sure my message is processed - node.js

I am building a kind of micro service application and using RabbitMQ to communicate between my services.
I have a nodeJS app that is supposed to receive messages from RabbitMQ and execute commands when a particular message comes in. So here is what the following code does:
Connects to RabbitMQ
Listens to symfony_messages queue
If a message identified by product.created comes in, the script executes a particular command using spawn from child_process.
My question is: Sometimes, I am going to "restart" my script. How can I be sure that at the moment of restarting the script is not in the middle of processing an event? How can I be sure that the process is not going to consume a message and stop before spawning the process?
The possible solution that came to my mind is:
Send a signal to the nodeJS process to tell him "Process a last message and stop". But how can I send such a signal?
And here is the code (you do not need to read if you already get the question):
const amqp = require('amqplib/callback_api')
const { spawn } = require('child_process')
amqp.connect('amqp://guest:guest#127.0.0.1:5672', (err, conn) => {
if (err) {
console.log(err)
return
}
conn.createChannel((err, channel) => {
let q = 'symfony_messages'
channel.assertQueue(q, {
durable: false
})
console.log(" [*] Waiting for messages in %s. To exit press CTRL+C", q);
channel.consume(q, (msg) => {
let event = JSON.parse(msg.content.toString())
if (event.name === 'product.created') {
console.log('Indexing order...')
let cp = spawn('php', [path.join(__dirname, '..', '..', 'bin', 'console'), 'elastic:index:orders', event.payload.product_id])
cp.stdout.on('data', (data) => {
console.log(`stdout: ${data}`);
})
cp.stderr.on('data', (data) => {
console.log(`stderr: ${data}`);
})
cp.on('close', (code) => {
console.log(`child process exited with code ${code}`);
})
}
}, {noAck: true});
})
})

Wouldn't it be a good pattern to use the channel.ack(message) function on the message once the message has been processed successfully? You've set the noAck option to true, but you can use the ACK mechanism to ensure messages are only taken off the queue once they are successfully processed.
Likewise, you can use the Nack function to deliberately tell RabbitMQ that the message was not processed, I normally do this in the process function error handler (or promise.catch).
I use a similar mechanism in a service that writes messages to a database. I only ACK the message once the message is written to the db. It's also useful to setup a dead letter exchange / queue within RabbitMQ so that any message that is Nacked ends up there. You can then inspect these messages and see why they couldn't be processed (or automatically attempt to re-process once the error condition that caused the problem is resolved.)

Related

RabbitMQ data lost on crash

I'm using RabbitMQ to store and retrieve data. I referred this article. I have set the durable flag to true and the noAck flag to false (i need to store the messages on the queue even after consuming).
I created these scenarios:
I updated stock data 3 times with consumers off state (inactive). Then I activated the consumer.It consumed all the three messages from the queue. [Works good.]
Now I again produced three messages (consumer inactive again) then I turned off the rabbitmq server. When I restarted the server and activated the consumer. It doesn't seem to be consuming the data (are the messages that were on the queue has been lost?)
Consumer :
connection.createChannel(function (error1, channel) {
if (error1) {
throw error1;
}
var queue = "updateStock2";
channel.assertQueue(queue, {
durable: true,
});
console.log(
" [*] Waiting for stockData messages in %s. To exit press CTRL+C",
queue
);
channel.consume(
queue,
function (data) {
stock = JSON.parse(data.content.toString());
console.log(" [x] Received Stock:", stock.name + " : " + stock.value);
},
{
noAck: false,
}
);
Producer :
connection.createChannel(function (error1, channel) {
if (error1) {
throw error1;
}
var queue = "updateStock2";
channel.assertQueue(queue, {
durable: true,
});
channel.sendToQueue(queue, Buffer.from(data));
console.log(" [x] Sent %s", data);
});
setTimeout(function () {
connection.close();
//process.exit(0);
}, 500);});
Aren't they persistent? If the server crashes all the messages in the queue are gone forever?
How to retrieve data that were in the queue when the server crashes?
Thanks in advance.
Why your messages have lost?
Regret to say, you did not declare {persistent: true} when you send message.Check https://www.rabbitmq.com/tutorials/tutorial-two-javascript.html, so you should use channel.sendToQueue(queue, Buffer.from(msg), {persistent: true});
Aren't they persistent?
Durable queues will be recovered on node boot, including messages in them published as persistent. Messages published as transient will be discarded during recovery, even if they were stored in durable queues.
Which middleware maybe better for you?
If you want a middleware which can persist messages even if consumed by consumers, you maybe need kafka

Exit custom function with loop process on node.js shutdown signal

I have a job that is executed ones per day. My app is running on Heroku, and dyno is restarted ones a day.
So what can happen is that during job execution Heroku starts the restart of dyno.
That itself is not a problem as I can start job two times per day, but what is a problem is to stop the job in the middle of task when it is not in stable status.
I would like now somehow to send this signal to job function so I can break any loops and stop function execution in safe way.
I know how to get signal:
process
.on('SIGTERM', shutdown('SIGTERM'))
.on('SIGINT', shutdown('SIGINT'))
.on('uncaughtException', shutdown('uncaughtException'));
function shutdown(signal) {
console.log(`${ signal }...`);
return (err) => {
if (err) console.error(err.stack || err);
setTimeout(() => {
console.log('...waited 5s, exiting.');
process.exit(err ? 1 : 0);
}, 5000).unref();
};
}
but how to send this signal to my job function and to break from it safely?
Thank you.
So the best solution I came up with is following.
// Manage signals
let shutDownSignal = false;
process
.on('SIGTERM', shutdown('SIGTERM'))
.on('SIGINT', shutdown('SIGINT'))
.on('uncaughtException', shutdown('uncaughtException'));
function shutdown(signal) {
return (err) => {
shutDownSignal = true;
console.log(`Received signal: ${ signal }...`);
if (err) console.error(err.stack || err);
setTimeout(() => {
console.log('...waited 15s, exiting.');
process.exit(err ? 1 : 0);
}, 15000).unref();
};
}
module.exports.getShutDownSingnal = function(){ return shutDownSignal; }
then with getShutDownSingnal() anywhere I can check whether shutdown is initiated.
One more thing. It is necessary to put Procfile in app root with
web: node index.js
in it (or app.js depending what are you using).
This is necessary so that SIGTERM and SIGKILL signals are transferred correctly to node (for example if using npm, it will not transfer this signal correctly). More about this on Heroku docs
Maybe this will be useful for someone.

How to log stack trace on node.js process error event

My node process is dying and I can't seem to log to a file when the process exits. It is a long running process invoked directly with node index.js:
// index.js
const fs = require('fs');
exports.getAllCars = (process => {
if (require.main === module) {
console.log(`Running process: ${process.getgid()}.`);
let out = fs.createWriteStream(`${__dirname}/process.log`);
// trying to handle process events here:
process.on('exit', code => out.write(`Exit: ${code}`));
return require('./lib/cars').getAllCars();
} else {
return require('./lib/cars').getAllCars;
}
})(process);
Also tried creating event handlers for error, uncaughtException. Nothing works when killing my process manually (with kill {pid}). The file process.log is created but nothing is there. Do writeable streams require a stream.end() to be called on completion?
According to Node.js documentation:
The 'exit' event is emitted when the Node.js process is about to exit
as a result of either:
The process.exit() method being called explicitly.
The Node.js event loop no longer having any additional work to perform.
So, if you start a process that should never end, it will never trigger.
Also, writable streams do not require to be closed:
If autoClose(an option from createWriteStream) is set to true (default
behavior) on error or end the file descriptor will be closed
automatically.
however, the createWriteStream function opens the file with flag 'w' by default, which means that the file will be overwritten every time (maybe this is the reason why you always see it empty). I suggest to use
fs.appendFileSync(file, data)
Here are the events that want to listen:
//catches ctrl+c event
//NOTE:
//If SIGINT has a listener installed, its default behavior will be removed (Node.js will no longer exit).
process.on('SIGINT', () => {
fs.appendFileSync(`${__dirname}/process.log`, `Received SIGINT\n`);
process.exit()
});
//emitted when an uncaught JavaScript exception bubbles
process.on('uncaughtException', (err) => {
fs.appendFileSync(`${__dirname}/process.log`, `Caught exception: ${err}\n`);
});
//emitted whenever a Promise is rejected and no error handler is attached to it
process.on('unhandledRejection', (reason, p) => {
fs.appendFileSync(`${__dirname}/process.log`, `Unhandled Rejection at: ${p}, reason: ${reason}\n`);
});
I suggest you put the code in a try catch block to find out whether its the code or some external cause which results in program termination.
and then check the log after the event...
try {
//your code
}catch(e) {
console.log(e.stack);
}

How to control commit of a consumed kafka message using kafka-node

I'm using Node with kafka for the first time, using kafka-node. Consuming a message requires calling an external API, which might even take a second to response. I wish to overcome sudden failures of my consumer, in a way that if a consumer fails, another consumer that will consume that will replace it will receive the same message that its work was not completed.
I'm using kafka 0.10 and trying to use ConsumerGroup.
I thought of setting autoCommit: false in options, and committing the message only once its work has been completed (as I previously done with some Java code some time ago).
However, I can't seem to be sure how should I correctly commit the message only once it is done. How should I commit it?
Another worry I have is that it seems, because of the callbacks, that the next message is being read before the previous one had finished. And I'm afraid that if message x+2 have finished before message x+1, then the offset will be set at x+2, thus in case of failure x+1 will never be re-executed.
Here is basically what I did so far:
var options = {
host: connectionString,
groupId: consumerGroupName,
id: clientId,
autoCommit: false
};
var kafka = require("kafka-node");
var ConsumerGroup = kafka.ConsumerGroup;
var consumerGroup = new ConsumerGroup(options, topic);
consumerGroup.on('connect', function() {
console.log("Consuming Kafka %s, topic=%s", JSON.stringify(options), topic);
});
consumerGroup.on('message', function(message) {
console.log('%s read msg Topic="%s" Partition=%s Offset=%d', this.client.clientId, message.topic, message.partition, message.offset);
console.log(message.value);
doSomeStuff(function() {
// HOW TO COMMIT????
consumerGroup.commit(function(err, data) {
console.log("------ Message done and committed ------");
});
});
});
consumerGroup.on('error', function(err) {
console.log("Error in consumer: " + err);
close();
});
process.once('SIGINT', function () {
close();
});
var close = function() {
// SHOULD SEND 'TRUE' TO CLOSE ???
consumerGroup.close(true, function(error) {
if (error) {
console.log("Consuming closed with error", error);
} else {
console.log("Consuming closed");
}
});
};
One thing you can do here is to have a retry mechanism for every message you process.
You can consult my answer on this thread:
https://stackoverflow.com/a/44328233/2439404
I consume messages from Kafka using kafka-consumer, batch them together using async/cargo and put them in async/queue (in-memory queue). The queue takes a worker function as an arguement to which I am passing a async/retryable.
For your problem, you can just use retryable to do processing on your messages.
https://caolan.github.io/async/docs.html#retryable
This may solve your problem.

Prevent sending data to stdin if spawn fails

In my Node.js (v0.10.9) code I'm trying to detect 2 cases:
an external tool (dot) is installed - in that case I want to send some data to stdin of created process
the external tool is not installed - in that case I want to display warning and I don't want to send anything to process' stdin
My problem is that I don't know how to send data to child's stdin if and only if the process was spawned successfully (i.e. stdin is ready for writing).
Following code works fine if dot is installed, but otherwise it tries to send data to the child although the child wasn't spawned.
var childProcess = require('child_process');
var child = childProcess.spawn('dot');
child.on('error', function (err) {
console.error('Failed to start child process: ' + err.message);
});
child.stdin.on('error', function(err) {
console.error('Working with child.stdin failed: ' + err.message);
});
// I want to execute following lines only if child process was spawned correctly
child.stdin.write('data');
child.stdin.end();
I'd need something like this
child.on('successful_spawn', function () {
child.stdin.write('data');
child.stdin.end();
});
From the node.js docs: http://nodejs.org/api/child_process.html#child_process_child_process_spawn_command_args_options
Example of checking for failed exec:
var spawn = require('child_process').spawn,
child = spawn('bad_command');
child.stderr.setEncoding('utf8');
child.stderr.on('data', function (data) {
if (/^execvp\(\)/.test(data)) {
console.log('Failed to start child process.');
}
});
Have a look at core-worker:
https://www.npmjs.com/package/core-worker
This package makes it a lot easier to handle processes.
I think what you want to do is something like that (from the docs):
import { process } from "core-worker";
const simpleChat = process("node chat.js", "Chat ready");
setTimeout(() => simpleChat.kill(), 360000); // wait an hour and close the chat
simpleChat.ready(500)
.then(console.log.bind(console, "You are now able to send messages."))
.then(::simpleChat.death)
.then(console.log.bind(console, "Chat closed"))
.catch(() => /* handle err */);
So if the process is not started correctly, none of the .then statements are executed which is exactly what you want to do, right?

Resources