My scenario: I want my app to publish logs to RabbitMQ and have another process consume those logs and write to a DB. Additionally, logs should persist in RabbitMQ even if there is no consumer at the moment. However, with the code I have now, my logs don't show up in RabbitMQ unless I start a consumer. What am I doing wrong?
My code:
var amqp = require('amqp');
var connection = amqp.createConnection({
host: "localhost",
port: 5672
});
connection.on('ready', function() {
// Immediately publish
setTimeout(function() {
connection.publish('logs',
new Buffer('hello world'), {},
function(err, res) {
console.log(err, '|', res);
});
}, 0);
// Wait a second to subscribe
setTimeout(function() {
connection.queue('logs', function(q) {
q.subscribe(function(message) {
console.log(message.data);
});
});
}, 1000);
});
Many times the general set up with rabbit MQ is for the publisher to declare and exchange and publish to it. Then the consumer declares the same exchange (which will just ensure it exists if it is already there and create it if the consumer starts first). This is incorrect for your usage. You need to have the queue created from the moment that you start publishing to it.
The publisher must create the exchange and the queue, the queue needs to be
autodelete=false
, durable only helps if you plan to restart your RabbitMQ server. It then publishes to the exchange and the messages will be delivered to the queue where they will wait for your consumer to connect to it and then read all the messages that it missed. It must use the exact same queue declare parameters as the producer did when it declared the queue. As it is
autodelete=false
It will ensure that no matter when the the consumer comes up and down it will stay alive and retain the messages.
Related
I am trying to publish data to a frontend angular application using socket.io in feathersjs. I have created a channel as follow and created mockData stream.
function mockData(app) {
app.emit('testEvent', { test: 'Something happened'});
};
setInterval(mockData.bind(null, app), 1000);
app.publish('testEvent', (data) => {
console.log('test');
return app.channel('testStream1');}
)
And when client is connected, I have added the client to testStream1 channel as follows.
app.on('connection', connection => {
// On a new real-time connection, add it to the anonymous channel
app.channel('testStream1');
app.channel('anonymous').join(connection);
app.channel('testStream1').join(connection);
});
Here nothing in app.publish callback executes, but if I try to directly emit from socketio object then I can receive values in client.
function mockData(app) {
app.emit('testEvent', { test: 'Something happened'});
app.io.emit('testStream1', { text: 'A client connected!' }); // Adding this line, I am able to get values from socketio client
};
Seems that app.publish([event,] fn) doesn't register the publishing function for custom event testEvent. How to get the data using app.publish ?
It might be neat feature in the future but currently, publishers and channels are only intended for service events, not global events. Although the app object is an event emitter, it won't send anything to the client. If you create a custom service event named testEvent you will be able to do
app.service('myservice').emit('testEvent', 'this will be published to clients');
You can also emit any Socket.io event on app.io directly but it will not go through Feathers channel/publishing mechanism.
Quoting the book, RabbitMQ in Depth:
A Basic.Ack request is sent to a publisher when a message that it has
published has been directly consumed by consumer applications on all
queues it was routed to or that the message was enqueued and persisted
if requested.
Confused with Has been directly consumed, does it mean when consumer send ack to broker publisher will be informed that consumer process message successfully? or it means that publisher will be notified when consumer just receive message from the queue?
or that the message was enqueued and persisted if requested. Is this like conjuction or publisher will be informed when either of those happens? (In that case publisher would be notified twice)
Using node.js and amqplib wanted to check what is happening actually:
// consumer.js
amqp.connect(...)
.then(connection => connection.createChannel())
.then(() => { assert exchange here })
.then(() => { assert queue here })
.then(() => { bind queue and exchange here })
.then(() => {
channel.consume(QUEUE, (message) => {
console.log('Raw RabbitMQ message received', message)
// Simulate some job to do
setTimeout(() => {
channel.ack(message, false)
}, 5000})
}, { noAck: false })
})
// publisher.js
amqp.connect(...)
.then(connection => connection.createConfirmChannel())
.then(() => { assert exchange here })
.then(() => {
channel.publish(exchange, routingKey, new Buffer(...),{}, (err, ok) => {
if (err) {
console.log('Error from handling confirmation on publisher side', err)
} else {
console.log('From handling confirmation on publisher side', ok)
}
})
})
Running the example, i can see following logs:
From handling confirmation on publisher side undefined
Raw RabbitMQ message received
Time to ack the message
As far as i see, at least by this log, publisher will be notified only when message was enqueued? (So having consumer acking the message will not influence publisher in any way)
Quoting further:
If a message cannot be routed, the broker will send a Basic.Nack RPC
request indicating the failure. It is then up to the publisher to
decide what to do with the message.
Changing the above example, where i only changed the routing key of the message to something that should not be routed anywhere (there are no bindings that would match routing key), from logs i can see only following.
From handling confirmation on publisher side undefined
Now i'm more confused, about what publisher is notified exactly here? I would understand that it receive an error, like Can't route anywhere, that would be aligned with quote above. But as you can see err is not defined and as side question even if amqplib in their official docs are using (err, ok), in no single case i see those defined. So here output is same like in above example, how one can differ between above example and un-routable message.
So what im up to here, when exactly publisher will be notified about what is happening with the message? Any concrete example in which one would use PublisherConfirms? From logging above, i would conclude that is nice to have it in cases where you want to be 100% sure that message was enqueued.
After searching again and again i have found this
http://www.rabbitmq.com/blog/2011/02/10/introducing-publisher-confirms/
The basic rules are as follows:
An un-routable mandatory or immediate message is confirmed right after the basic.return
transient message is confirmed the moment it is enqueued
Persistent message is confirmed when it is persisted to disk or when it is consumed on every queue.
If more than one of these conditions are met, only the first causes a
confirm to be sent. Every published message will be confirmed sooner
or later and no message will be confirmed more than once.
by default publishers don't know anything about consumers.
PublisherConfirms is used to check if the message reached the broker, but not if the message has been enqueued.
you can use mandatory flag to be sure the message has been routed
see this https://www.rabbitmq.com/reliability.html
To ensure messages are routed to a single known queue, the producer
can just declare a destination queue and publish directly to it. If
messages may be routed in more complex ways but the producer still
needs to know if they reached at least one queue, it can set the
mandatory flag on a basic.publish, ensuring that a basic.return
(containing a reply code and some textual explanation) will be sent
back to the client if no queues were appropriately bound.
I'm not entirely sure about the notification on ack/nack question, but check out the BunnyBus Node library for a simpler api and RabbitMQ management :)
https://github.com/xogroup/bunnybus
const BunnyBus = require('bunnybus');
const bunnyBus = new BunnyBus({
user: 'your-user',
vhost: 'your-vhost', // cloudamqp defaults vhost to the username
password: 'your-password',
server: 'your.server.com'
});
const handler = {
'test.event': (message, ack) => {
// Do your work here.
// acknowledge the message off of the bus.
return ack();
}
};
// Create exchange and queue if they do not already exist and then auto connect.
return bunnyBus.subscribe('test', handler)
.then(() => {
return bunnyBus.publish({event: 'test.event', body: 'here\'s the thing.'});
})
.catch(console.log);
Hello I am using mosca mqtt broker I want to read the message that a client publish to a topic.
Is there any way to do that?
In published event I log the packet.payload and it prints only clientid and topic.
server.on('published', function(packet, client) {
console.log('Published', packet.payload);
});
Thank you
You could use the callback "on published " to log the published message either in a file or maybe in your database for accessing whenever you need it, like:
server.on('published', function(packet, client) {
// Do what you want with your message here
var msg = packet.payload.toString('utf8');
});
Be careful of internal messages though. this callback does not filter for you so you have to do it yourself.
Refer to this answer here
server.on('published', function(packet, client) {
console.log('Published: ', packet.payload.toString('utf8'));
});
I'm using the redis-sentinel-client library to manage a connection to a Redis sentinel group. The issue I have is that upon connecting I need to process records which may or may not already be present in the Redis store.
As I have two clients (due to the fact that one is a subscriber) I am not sure the best way to organise my event listeners so that I guarantee that both clients are ready prior to attempting any operations.
At the moment I have the following:
var sentinelSubscriberClient = RedisSentinel.createClient(opts);
var sentinelPublisherClient = RedisSentinel.createClient(opts);
sentinelSubscriberClient.on('ready', function redisSubscriberClientReady() {
sentinelPublisherClient.removeAllListeners('ready');
sentinelPublisherClient.on('ready', function () {
supportedChannels.forEach(function (channel) {
sentinelSubscriberClient.subscribe(channel);
});
// Includes reading + publishing via `sentinelPublisherClient`
processUnprocessed();
});
});
(there are also error listeners but I've removed them to make the code easier to read)
This current approach falls over if the publisher client emits ready before the subscriber client. My question is how can I organise the event listeners so that I can safely call .subscribe() on the subscriber client and various methods (.lrange(), .publish() etc.) of the publisher listener?
Thanks!
Simply move client creation into the ready callback function.
var sentinelSubscriberClient = RedisSentinel.createClient(opts);
var sentinelPublisherClient = null;
sentinelSubscriberClient.on('ready', function redisSubscriberClientReady() {
sentinelPublisherClient = RedisSentinel.createClient(opts);
sentinelPublisherClient.on('ready', function () {
supportedChannels.forEach(function (channel) {
sentinelSubscriberClient.subscribe(channel);
});
// Includes reading + publishing via `sentinelPublisherClient`
processUnprocessed();
});
});
Do I really need to instantiate the queue "queue = connection.queue(...)" every time I want to use it?
Server (node.js):
queue = connection.queue('incoming', { // <--- DO I REALLY NEED THAT FOR EVERY REQUEST?
durable : true,
exclusive : false }, function() {});
queue.subscribe(function(msg) {
// Unsubcribe here. Maybe there is something like a once listener?
console.log("RECEIVED: "+msg)
//res.send(msg.data);
queue.unsubscribe(ctag);
res.redirect('/home');
}).addCallback(function(ok) { ctag = ok.consumerTag; });
exchange.publish('msg_queue', 'functional!', { // request
replyTo: 'incoming'
});
If queue = connection.queue(...); is instantiated with the server, the first request using the queue is successful, but the followings requests throw an error:
Error: NOT_FOUND - no queue 'incoming' in vhost '/'
at Queue._onMethod (/Users/cassiomelo/code/cubix/cubix_nodejs/node_modules/amqp/amqp.js:1720:15)
For an rpc example ,that keeps the reply queue hanging around, please have a look at How to create REP/REQ on Rabbit.js