RabbitMQ messages disappearing mysteriously - node.js

I have a node worker waiting for RabbitMQ messages. Today I made some changes to it, and now when I connect the dev instance, everything works—it appears to connect—but then the consume functions don't fire. Messages still disappear though like they've been consumed.
Rabbit.queue('consume', function (q) {
q.bind('consume');
// This fires
console.log('Listening to queue...');
q.subscribe(function (message) {
// This doesn't fire but message gets removed from queue
console.log(message);
init(message);
});
});

I had this same behavior using the RabbitMQ client in a C# console application:
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
//Various RabbitMQ client magic that doesn't work
}
}
Console.WriteLine(" Press [enter] to exit.");
Console.ReadLine();
Note those last two lines. They keep the console open and running. Otherwise the app would have instantly closed. The problem was I put those two lines of code outside the using statement. Once you step passed the closing bracket of the using statement, the "used" object is disposed. So I was closing my connection and channel almost immediately after opening them. The solution was to put the Console lines INSIDE the using brackets:
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
//Various RabbitMQ client magic that astounds and amazes
Console.WriteLine(" Press [enter] to exit.");
Console.ReadLine();
}
}
Obvious and trivial once you see it, but I wasted an hour on it. I don't know if this is analogous to anything you are doing in NodeJS, but perhaps it will help some other frustrated coder.

Related

socket.io how to send multiple messages sequentially?

I'm using socket.io like this
Client:
socket.on('response', function(i){
console.log(i);
});
socket.emit('request', whateverdata);
Server:
socket.on('request', function(whateverdata){
for (i=0; i<10000; i++){
console.log(i);
socket.emit('response', i);
}
console.log("done!");
});
I need output like this when putting the two terminals side by side:
Server Client
0 0
1 1
. (etc) .
. .
9998 9998
9999 9999
done!
But instead I am getting this:
Server Client
0
1
. (etc)
.
9998
9999
done!
0
1
.
. (etc)
9998
9999
Why?
Shouldn't Socket.IO / Node emit the message immediately, not wait for the loop to complete before emitting any of them?
Notes:
The for loop is very long and computationally slow.
This question is referring to the socket.io library, not websockets in general.
Due to latency, waiting for confirmation from the client before sending each response is not possible
The order that the messages are received is not important, only that they are received as quickly as possible
The server emits them all in a loop and it takes a small bit of time for them to get to the client and get processed by the client in another process. This should not be surprising.
It is also possible that the single-threaded nature of Javascript in node.js prevents the emits from actually getting sent until your Javascript loop finishes. That would take detailed examination of socket.io code to know for sure if that is an issue. As I said before if you want to 1,1 then 2,2 then 3,3 instead of 1,2,3 sent, then 1,2,3 received you have to write code to force that.
If you want the client to receive the first before the server sends the 2nd, then you have to make the client send a response to the first and have the server not send the 2nd until it receives the response from the first. This is all async networking. You don't control the order of events in different processes unless you write specific code to force a particular sequence.
Also, how do you have client and server in the same console anyway? Unless you are writing out precise timestamps, you wouldn't be able to tell exactly what event came before the other in two separate processes.
One thing you could try is to send 10, then do a setTimeout(fn, 1) to send the next 10 and so on. That would give JS a chance to breathe and perhaps process some other events that are waiting for you to finish to allow the packets to get sent.
There's another networking issue too. By default TCP tries to batch up your sends (at the lowest TCP level). Each time you send, it sets a short timer and doesn't actually send until that timer fires. If more data arrives before the timer fires, it just adds that data to the "pending" packet and sets the timer again. This is referred to as the Nagle's algorithm. You can disable this "feature" on a per-socket basis with socket.setNoDelay(). You have to call that on the actual TCP socket.
I am seeing some discussion that Nagle's algorithm may already be turned off for socket.io (by default). Not sure yet.
In stepping through the process of socket.io's .emit(), there are some cases where the socket is marked as not yet writable. In those cases, the packets are added to a buffer and will be processed "later" on some future tick of the event loop. I cannot see exactly what puts the socket temporarily in this state, but I've definitely seen it happen in the debugger. When it's that way, a tight loop of .emit() will just buffer and won't send until you let other events in the event loop process. This is why doing setTimeout(fn, 0) every so often to keep sending will then let the prior packets process. There's some other event that needs to get processed before socket.io makes the socket writable again.
The issue occurs in the flush() method in engine.io (the transport layer for socket.io). Here's the code for .flush():
Socket.prototype.flush = function () {
if ('closed' !== this.readyState &&
this.transport.writable &&
this.writeBuffer.length) {
debug('flushing buffer to transport');
this.emit('flush', this.writeBuffer);
this.server.emit('flush', this, this.writeBuffer);
var wbuf = this.writeBuffer;
this.writeBuffer = [];
if (!this.transport.supportsFraming) {
this.sentCallbackFn.push(this.packetsFn);
} else {
this.sentCallbackFn.push.apply(this.sentCallbackFn, this.packetsFn);
}
this.packetsFn = [];
this.transport.send(wbuf);
this.emit('drain');
this.server.emit('drain', this);
}
};
What happens sometimes is that this.transport.writable is false. And, when that happens, it does not send the data yet. It will be sent on some future tick of the event loop.
From what I can tell, it looks like the issue may be here in the WebSocket code:
WebSocket.prototype.send = function (packets) {
var self = this;
for (var i = 0; i < packets.length; i++) {
var packet = packets[i];
parser.encodePacket(packet, self.supportsBinary, send);
}
function send (data) {
debug('writing "%s"', data);
// always creates a new object since ws modifies it
var opts = {};
if (packet.options) {
opts.compress = packet.options.compress;
}
if (self.perMessageDeflate) {
var len = 'string' === typeof data ? Buffer.byteLength(data) : data.length;
if (len < self.perMessageDeflate.threshold) {
opts.compress = false;
}
}
self.writable = false;
self.socket.send(data, opts, onEnd);
}
function onEnd (err) {
if (err) return self.onError('write error', err.stack);
self.writable = true;
self.emit('drain');
}
};
Where you can see that the .writable property is set to false when some data is sent until it gets confirmation that the data has been written. So, when rapidly sending data in a loop, it may not be letting the event come through that signals that the data has been successfully sent. When you do a setTimeout() to let some things in the event loop get processed that confirmation event comes through and the .writable property gets set to true again so data can again be sent immediately.
To be honest, socket.io is built of so many abstract layers across dozens of modules that it's very difficult code to debug or analyze on GitHub so it's hard to be sure of the exact explanation. I did definitely see the .writable flag as false in the debugger which did cause a delay so this seems like a plausible explanation to me. I hope this helps.

Akka.net Ask timeout when used in Azure WebJob

At work we have some code in a Azure WebJob where we use Rabbit
The basic workflow is this
A message arrives on RabbitMQ Queue
We have a message handler for the incoming message
Within the message handler we start a top level (user) supervisor actor where we "ask" it to handle the message
The supervisor actor hierarchy is like this
And the relevant top level code is something like this (this is the WebJob code)
static void Main(string[] args)
{
try
{
//Bootstrap akka IoC resolver well ahead of any actor usages
new AutoFacDependencyResolver(ContainerOperations.Instance.Container, ContainerOperations.Instance.Container.Resolve<ActorSystem>());
var system = ContainerOperations.Instance.Container.Resolve<ActorSystem>();
var busQueueReader = ContainerOperations.Instance.Container.Resolve<IBusQueueReader>();
var dateTime = ContainerOperations.Instance.Container.Resolve<IDateTime>();
busQueueReader.AddHandler<ProgramCalculationMessage>("RabbitQueue", x =>
{
//This is code that gets called whenever we have a RabbitMQ message arrive
//This is code that gets called whenever we have a RabbitMQ message arrive
//This is code that gets called whenever we have a RabbitMQ message arrive
//This is code that gets called whenever we have a RabbitMQ message arrive
//This is code that gets called whenever we have a RabbitMQ message arrive
try
{
//SupervisorActor is a singleton
var supervisorActor = ContainerOperations.Instance.Container.ResolveNamed<IActorRef>("SupervisorActor");
var actorMessage = new SomeActorMessage();
var supervisorRunTask = runModelSupervisorActor.Ask(actorMessage, TimeSpan.FromMinutes(25));
//we want to wait this guy out
var supervisorRunResult = supervisorRunTask.GetAwaiter().GetResult();
switch (supervisorRunResult)
{
case CompletedEvent completed:
{
break;
}
case FailedEvent failed:
{
throw failed.Exception;
}
}
}
catch (Exception ex)
{
_log.Error(ex, "Error found in Webjob");
//throw it for the actual RabbitMqQueueReader Handler so message gets NACK
throw;
}
});
Thread.Sleep(Timeout.Infinite);
}
catch (Exception ex)
{
_log.Error(ex, "Error found");
throw;
}
}
And this is the relevant IOC code (we are using Autofac + Akka.NET DI for Autofac)
builder.RegisterType<SupervisorActor>();
_actorSystem = new Lazy<ActorSystem>(() =>
{
var akkaconf = ActorUtil.LoadConfig(_akkaConfigPath).WithFallback(ConfigurationFactory.Default());
return ActorSystem.Create("WebJobSystem", akkaconf);
});
builder.Register<ActorSystem>(cont => _actorSystem.Value);
builder.Register(cont =>
{
var system = cont.Resolve<ActorSystem>();
return system.ActorOf(system.DI().Props<SupervisorActor>(),"SupervisorActor");
})
.SingleInstance()
.Named<IActorRef>("SupervisorActor");
The problem
So the code is working fine and doing what we want it to, apart from the Akka.Net "ask" timeout shown above in the WebJob code.
Annoyingly this seems to work fine if I try and run the webjob locally. Where I can simulate a "ask" timeout by providing a new supervisorActor that simply doesn't EVER respond with a message back to the "Sender".
This works perfectly running on my machine, but when we run this code in Azure, we DO NOT see a Timeout for the "ask" even though one of our workflow runs exceeded the "ask" timeout by a mile.
I just don't know what could be causing this behavior, does anyone have any ideas?
Could there be some Azure specific config value for the WebJob that I need to set.
The answer to this was to use the async rabbit handlers which apparently came out in V5.0 of the C# rabbit client. The offical docs still show the sync usage (sadly).
This article is quite good : https://gigi.nullneuron.net/gigilabs/asynchronous-rabbitmq-consumers-in-net/
Once we did this, all was good

node-kafka pause() method on consumer. Any working version?

I can't make following code to work:
"use strict";
let kafka = require('kafka-node');
var conf = require('./providers/Config');
let client = new kafka.Client(conf.kafka.connectionString, conf.kafka.clientName);
let consumer = new kafka.HighLevelConsumer(client, [ { topic: conf.kafka.readTopic } ], { groupId: conf.kafka.clientName, paused: true });
let threads = 0;
consumer.on('message', function(message) {
threads++;
if (threads > 10) consumer.pause();
if (threads > 50) process.exit(1);
console.log(threads + " >>> " + message.value);
});
consumer.resume();
I see 50 messages in console and process exits by termination statement.
What I'm trying to understand, is that is it my code broken or package broken? Or maybe I'm just doing something wrong? Does anyone was able to make kafka consumer work with pause/resume? I tried several versions of kafka-node, but all of them behave same way. Thanks!
You are already using pause and resume in your code, so obviously they work. ;)
It's because pause doesn't pause the consumption of messages. It pauses the fetching of messages. I'm guessing you already fetched the first 50 in one throw before you receive the first message and call pause.
For kicks, I just tested pause() and resume() in the Node REPL and they work as expected:
var kafka = require('kafka-node');
var client = new kafka.Client('localhost:2181');
var consumer = new kafka.HighLevelConsumer(client, [{topic: 'MyTest'}]);
consumer.on('message', (msg) => { console.log(JSON.stringify(msg)) });
Then I go into another window and run:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic MyTest
And type some stuff, and it shows up in the first window. Then in the first window, I type: consumer.pause(); And type some more in the second window. Nothing appears in the first window. Then I run consumer.resume()in the first window, and the delayed messages appear.
BTW, you should be able to play with the Kafka config property fetch.message.max.bytes and control how many messages can be fetched at one time. For example, if you had fixed-width messages of 500 bytes, set fetch.message.max.bytes to something less than 1000 (but greater than 500!) to only receive a single message per fetch. But note that this might not fix the problem entirely -- I am fairly new to Node, but it is asynchronous, and I suspect a second fetch could get kicked off before you processed the first fetch completely (or at all).

Performing Operations while Closing NodeJS

I have a Firebase Connection in nodejs that pushes data to a url while the connection is persistent, when it closes, I want to remove that data (think, I push "hey I'm here", and when I leave, the text disappears)
I made a "runnable" that shows an example of it:
http://web-f6176e84-c073-416f-93af-62a9a9fbfabd.runnable.com
basically, hit "ctrl + c" and it prints out "trying to remove reference" but never actually deletes the data ( the documents say that remove() is equivalent to set(null) which it basically sets the data to null, and since it's null, the entire element should be gone.)
However it's not removing it, I don't see the data ever "disappear". (I'm using a temp Firebase URL, you should be able to duplicate with any URL you can access if this url stops existing).
this is the code I'm using.
var FB_URL = 'https://cuhiqgro1t3.firebaseio-demo.com/test_code';
var Firebase = require('firebase');
var myRootRef = new Firebase(FB_URL);
console.log("created Firebase URL");
process.stdin.resume(); //so the program will not close instantly
function delete_fb_entries() {
return function() {
console.log("Trying to remove reference");
myRootRef.remove();
process.exit();
}
}
//do something when app is closing
process.on('exit', delete_fb_entries());
//catches ctrl+c event
process.on('SIGINT', delete_fb_entries());
//catches uncaught exceptions
process.on('uncaughtException', delete_fb_entries());
EDIT: Additional Information as to the "why", I push my local IP address out to my Firebase URL cause I'm lazy and it's easier to just have a webpage setup I can always access that will show the url of particular devices (and I know using the routers tables would be easier), I actually also have other purposes for this usage as well (if I happen to be inside my network, I can just select a particular device from my webpage and access the data I need, either way, it works, but I just can't get it to remove itself correctly, this used to work at one point in time I believe, so I can only assume the API has changed or something).
EDIT 2: OK removed process.exit() as suggested, and the runnable seemed to delete the data in question, I tried it on my local data (and after some cleaning up and commenting out), it removed the data, however when I hit Ctrl + C it no longer exits the program.....so yay.
I need to figure out if "process.exit()" is necessary or unnecessary at this point.
Edit 3: Ok so I need to use process.exit (as far as I can tell, Ctrl + C no longer exits the program, I have to ctrl + Z, and reboot). I tried adding it right after, but I realized that removing a firebase element is not a synchronus operation, so when I close it I tried (the next attempt) was to use the on complete handler for the remove function (so remove(onComplete), and then adding the process.exit() to the onComplete function).
So finally it looks like this and it seems to be working with my application
var FB_URL = 'https://cuhiqgro1t3.firebaseio-demo.com/test_code';
var Firebase = require('firebase');
var myRootRef = new Firebase(FB_URL);
console.log("created Firebase URL");
function onComplete() {
process.exit();
]
process.stdin.resume(); //so the program will not close instantly
function delete_fb_entries() {
return function() {
console.log("Trying to remove reference");
myRootRef.remove(onComplete);
}
}
//do something when app is closing
process.on('exit', delete_fb_entries());
//catches ctrl+c event
process.on('SIGINT', delete_fb_entries());
//catches uncaught exceptions
process.on('uncaughtException', delete_fb_entries());
EDIT 4: In response to comments below, So I tried modifying a simple program to be the following:
function delete_fb_entries (){
return function () {
console.log("I should quit soon");
}
}
process.stdin.resume(); //so the program will not close instantly
//catches ctrl+c event
process.on('SIGINT', delete_fb_entries());
My program never exited. I don't understand why node would not close in this case, changing to add a process.exit() after the console.log causes nodejs to quit. This is not an async function, so why is it not exiting in this case? (Is this a bug, or a misunderstanding of how this works by me?)
You cannot perform asynchronous operations in a process's exit event handler, only synchronous operations, since the process is exited once all exit event handlers have been executed.

How can I pop objects from Redis as they are added realtime?

I want to get a Node.js process running as it's checking a Redis server for anything new to pop.
Another process will be doing the pushing sporadically, and the Node process will be trying to pop whatever that comes in. The Node process will stay running.
Can anyone point me to a good direction?
I'm trying to figure out how to listen for such event. Sure, I can pop it once, but how do I get Node process to keep listening for any addition to Redis server?
You'll want to use a blocking pop: http://redis.io/commands/brpop
function waitForPush () {
client.brpop(['list','otherlist',0], function (listName, item) {
// do stuff
waitForPush();
});
}
This seems like a good use case for pub/sub: http://redis.io/topics/pubsub
Your Node.js process that sporadically pushes to Redis can also publish to a channel each time it pushes something. Like this:
var pushClient = redis.createClient();
//push something onto Redis
pushClient.publish("pubsub-channel", "Just pushed something onto Redis");
Then your other process would subscribe to that channel. Each time the message event is fired, you pop off whatever was just pushed:
var client = redis.createClient();
client.subscribe("pubsub-channel");
client.on("message", function(channel, message){
//pop off new item
});
How about a modified version of recursive function with a process.nextTick(...)
function waitForPush () {
client.brpop(['list','otherlist',0], function (listName, item) {
// do stuff
process.nextTick(waitForPush);
});
}

Resources