I have implemented a message bus in Linux for IPC using ZeroMQ (more specifically CZMQ). Here is what I have implemented.
My question is, how do I know that send dropped the message when the publisher buffer is full?
In my simple test setup, I am using a publisher-subscriber with a proxy. I have a fast sender and a very slow receiver causing messages to hit HWM and drop on send. My exception is that send would fail with 'message dropped' error, but it is not the case. the zmq_msg_send() is not giving me any error even though the messages get dropped (I can verify this by seeing gaps in messages in subscriber end).
How can I know when the messages get dropped? If this is the intended behaviour and ZeroMQ does not let us know that, what is a workaround to find if my send dropped the message?
What you appear to be asking for is fault tolerance for which PUB/SUB isn't ideal. Not only may the HWM be reached, but consider what happens if a subscribing client dies and gets restarted - it will miss messages sent by the publisher for the duration. FWIW. In ZMQ v2, the default HWM was infinite for PUB/SUB, but got changed to 1000 in v3 because systems were choking for memory due to messages being queued faster than they could be sent. The 1000 seemed like a reasonable value for bursts of messages when the average message rate was within the network bandwidth. YMMV.
If you just want to know when messages get dropped, it's as simple as adding an incrementing message number to the message and having the subscribers monitor that. You could choose to place this number in it's own frame or not; overall simplicity will be the decider. I don't believe it's possible to determine when messages get dropped specifically because the HWM has been reached.
By default zeromq pub/sub from recent versions defaults to a high-water mark ZMQ_SNDHWM/ZMQ_RCVHWM of 1000 messages.
What this means is if you burst in a tight loop more than 1000 messages it will prob drop some. It is simple to write a test and give each message a payload with a sequence number.
One option is to set both the HWMs to 0. This will mean it's infinite.
You can play about with this using some examples I wrote recently:
https://gist.github.com/easytiger/992b3a29eb5c8545d289
https://gist.github.com/easytiger/e382502badab49856357
The will pub and sub on a tport in a burst of messages. If you play with the HWM you can see in big bursts that if it isn't 0 it will drop a great many
Related
I have a queue and 3 consumers bind to the queue. Each consumer has a prefetch_count of 250(or say X) and manual acknowledgement is done prefetch_count(X)/2(i.e 125) messages - meaning consumer manually acknowledges 125 messages in a single go (which helps to reduce round-trip time and hence increases performance). Everything is working fine as expected but the only issue arises when there are no new messages in the queue and the consumers have some unacknowledged messages whose count is less 125.
As the acknowledgement is only sent when the count is 125, these unacknowledged messages keeps requeuing. How can I solve this ?
How can I know that my consumer has no new messages to process and can acknowledge all the remaining messages waiting to be acknowledged.
If I understand your scenario correctly, it sounds as though you have a series of messages that get published all at once, and then you process them in batches of 250 at a time until you have none left. The problem is, that if you don't have a number of messages that is divisible by 125, then your final batch never gets acknowledged. Clearly this is a logical problem, but it sounds like you are wondering if there is an easy way to deal with it.
Your question "How can I know that my consumer has no new messages to process?" is based upon a premise which RabbitMQ does not support -- namely, the "end" of a sequence of messages. RabbitMQ consumers expect to continue to receive messages indefinitely, so from their perspective, there is no such thing as "done."
Thus, any such concept must be implemented elsewhere, higher up in your application logic. Here are some options for you to consider:
If you know in advance how many messages will be processed, then send that count first and store. Send the final ack once you have processed that number (assuming no duplicates were processed).
Monitor the in-memory collection at the consumer (all pre-fetched messages reside here until they are actually processed). When it drops below 125, you know that you have a batch size less than that.
Similar to #1, send a special "last message" that the consumer can receive and know to acknowledge upon receipt.
Caveat: I would argue that you have some deeper design problem going on that is leading down the path where it would ever be desirable to do this in the first place. Each message should be 100% independent of any other message. If that assumption is violated, you will have a very fragile system.
In which order does QueueClient.Receive() deliver messages?
I have been running some tests and what I can see a few of the messages (top-most ones I guess) are delivered over and over again if you don't Complete() them.
Is there a way to force it deliver in a Round Robin manner?
When you have a message that is delivered, but not completed, it's expected for message to show up on the queue once LockDuration expires and be consumed again. You have to complete the message. If you don't, it will eventually go into DLQ, but prior to that your consumer(s) will receive it multiple times.
QueueClient.Receive gets whatever is available on the broker (server). I'm not following Round Robin delivery idea, because it's a queue. You get what's in the queue. As a rule of thumb, I would suggest not to rely on the order of messages.
Saying that, there's an ASB Session feature that can preserve and guarantee an ordered of delivery. In case you're looking for sessions, similar question was asked before.
When you create the QueueClient you can specify the receive mode and set it to ReceiveAndDelete:
QueueClient.CreateFromConnectionString(connectionString, path, ReceiveMode.ReceiveAndDelete);
This will enable you to remove the message from the queue as soon as you receive it instead of having to call Complete.
If you don't call Complete or use ReceiveAndDelete, the order will be:
Get a message (locking it for X seconds)
Get the next message in order (locking it for X seconds)
First message lock expired so you get it again and relock it.
Same for the second message and so on forever.
I'm working on what's basically a highly-available distributed message-passing system. The system receives messages from someplace over HTTP or TCP, perform various transformations on it, and then sends it to one or more destinations (also using TCP/HTTP).
The system has a requirement that all messages sent to a given destination are in-order, because some messages build on the content of previous ones. This limits us to processing the messages sequentially, which takes about 750ms per message. So if someone sends us, for example, one message every 250ms, we're forced to queue the messages behind each other. This eventually introduces intolerable delay in message processing under high load, as each message may have to wait for hundreds of other messages to be processed before it gets its turn.
In order to solve this problem, I want to be able to parallelize our message processing without breaking the requirement that we send them in-order.
We can easily scale our processing horizontally. The missing piece is a way to ensure that, even if messages are processed out-of-order, they are "resequenced" and sent to the destinations in the order in which they were received. I'm trying to find the best way to achieve that.
Apache Camel has a thing called a Resequencer that does this, and it includes a nice diagram (which I don't have enough rep to embed directly). This is exactly what I want: something that takes out-of-order messages and puts them in-order.
But, I don't want it to be written in Java, and I need the solution to be highly available (i.e. resistant to typical system failures like crashes or system restarts) which I don't think Apache Camel offers.
Our application is written in Node.js, with Redis and Postgresql for data persistence. We use the Kue library for our message queues. Although Kue offers priority queueing, the featureset is too limited for the use-case described above, so I think we need an alternative technology to work in tandem with Kue to resequence our messages.
I was trying to research this topic online, and I can't find as much information as I expected. It seems like the type of distributed architecture pattern that would have articles and implementations galore, but I don't see that many. Searching for things like "message resequencing", "out of order processing", "parallelizing message processing", etc. turn up solutions that mostly just relax the "in-order" requirements based on partitions or topics or whatnot. Alternatively, they talk about parallelization on a single machine. I need a solution that:
Can handle processing on multiple messages simultaneously in any order.
Will always send messages in the order in which they arrived in the system, no matter what order they were processed in.
Is usable from Node.js
Can operate in a HA environment (i.e. multiple instances of it running on the same message queue at once w/o inconsistencies.)
Our current plan, which makes sense to me but which I cannot find described anywhere online, is to use Redis to maintain sets of in-progress and ready-to-send messages, sorted by their arrival time. Roughly, it works like this:
When a message is received, that message is put on the in-progress set.
When message processing is finished, that message is put on the ready-to-send set.
Whenever there's the same message at the front of both the in-progress and ready-to-send sets, that message can be sent and it will be in order.
I would write a small Node library that implements this behavior with a priority-queue-esque API using atomic Redis transactions. But this is just something I came up with myself, so I am wondering: Are there other technologies (ideally using the Node/Redis stack we're already on) that are out there for solving the problem of resequencing out-of-order messages? Or is there some other term for this problem that I can use as a keyword for research? Thanks for your help!
This is a common problem, so there are surely many solutions available. This is also quite a simple problem, and a good learning opportunity in the field of distributed systems. I would suggest writing your own.
You're going to have a few problems building this, namely
2: Exactly-once delivery
1: Guaranteed order of messages
2: Exactly-once delivery
You've found number 1, and you're solving this by resequencing them in redis, which is an ok solution. The other one, however, is not solved.
It looks like your architecture is not geared towards fault tolerance, so currently, if a server craches, you restart it and continue with your life. This works fine when processing all requests sequentially, because then you know exactly when you crashed, based on what the last successfully completed request was.
What you need is either a strategy for finding out what requests you actually completed, and which ones failed, or a well-written apology letter to send to your customers when something crashes.
If Redis is not sharded, it is strongly consistent. It will fail and possibly lose all data if that single node crashes, but you will not have any problems with out-of-order data, or data popping in and out of existance. A single Redis node can thus hold the guarantee that if a message is inserted into the to-process-set, and then into the done-set, no node will see the message in the done-set without it also being in the to-process-set.
How I would do it
Using redis seems like too much fuzz, assuming that the messages are not huge, and that losing them is ok if a process crashes, and that running them more than once, or even multiple copies of a single request at the same time is not a problem.
I would recommend setting up a supervisor server that takes incoming requests, dispatches each to a randomly chosen slave, stores the responses and puts them back in order again before sending them on. You said you expected the processing to take 750ms. If a slave hasn't responded within say 2 seconds, dispatch it again to another node randomly within 0-1 seconds. The first one responding is the one we're going to use. Beware of duplicate responses.
If the retry request also fails, double the maximum wait time. After 5 failures or so, each waiting up to twice (or any multiple greater than one) as long as the previous one, we probably have a permanent error, so we should probably ask for human intervention. This algorithm is called exponential backoff, and prevents a sudden spike in requests from taking down the entire cluster. Not using a random interval, and retrying after n seconds would probably cause a DOS-attack every n seconds until the cluster dies, if it ever gets a big enough load spike.
There are many ways this could fail, so make sure this system is not the only place data is stored. However, this will probably work 99+% of the time, it's probably at least as good as your current system, and you can implement it in a few hundred lines of code. Just make sure your supervisor is using asynchronous requests so that you can handle retries and timeouts. Javascript is by nature single-threaded, so this is slightly trickier than normal, but I'm confident you can do it.
Scenario
There is a Linux-powered device connected to a CAN bus. The device periodically transmits the CAN message. The nature of the data carried by this message is like measurement rather than command, i.e. only the most recent one is actually valid, and if some messages are lost that is not an issue as long as the latest one was received successfully.
Then the device in question is being disconnected from the CAN bus for some amount of time that is much longer than the interval between subsequent message transmissions. The device logic is still trying to transmit the messages, but since the bus is disconnected the CAN controller is unable to transmit any of them so the messages are being accumulated in the TX queue.
Some time later the CAN bus connection is restored, and all the accumulated messages are being kicked on the bus one by one.
Problem
When the CAN bus connection is restored, undefined amount of outdated messages will be transmitted from the TX queue.
While the CAN bus connection is still not available but TX queue is already full, transmission of some most recent messages (i.e. the only valid messages) will be discarded.
Once the CAN bus connection is restored, there would be short term traffic burst while the TX queue is being flushed. This can alter the Time Triggered Bus Scheduling if one is used (it is in my case).
Question
My application uses SocketCAN driver, so basically the question should be applied to SocketCAN, but other options are considered too if there are any.
I see two possible solutions: define a message transmission timeout (if a message was not transmitted during some predefined amount if time, it will be discarded automatically), or abort transmission of outdated messages manually (though I doubt it is possible at all with socket API).
Since the first option seems to be most real to me, the question is:
How does one define TX timeout for CAN interface under Linux?
Are there other options exist to solve the problems described above, aside from TX timeouts?
My solution for this problem was shutting down and bringing the device up again:
void
clear_device_queue
(void)
{
if (!queue_cleared)
{
const char
*dev = getenv("MOTOR_CAN_DEVICE");
char
cmd[1024];
sprintf(cmd, "sudo ip link set down %s", dev);
system(cmd);
usleep(500000);
sprintf(cmd, "sudo ip link set up %s", dev);
system(cmd);
queue_cleared = true;
}
}
I don't know the internals of SocketCAN, but I think the larger part of the problem should be solved on a more general, logical level.
Before, there is one aspect to clarify:
The question includes tag safety-critical...
If the CAN communication is not relevant to implement a safety function, you can pick any solution you find useful. There may be parts of the second alternative which are useful for you in this case too, but those are not mandatorx.
If the communication is, however used in a safety-relevant context, there must be a concept that takes into account the requirements imposed by IEC 61508 (safety of programmable electronic systems in general) and IEC 61784-x/62280 (safe communcation protocols).
Those standards usually lead to some protocol measures that come in handy with any embedded communication, but especially for the present problem:
Add a sequence counter to the protocol frames.
The receiver shall monitor that it the counter values it sees don't make larger "jumps" than allowed (e.g., if you allow to miss 2 frames along the way, max. counter increment may be +3. CAN bus may redouble a frame, so a counter increment of +0 must be tolerated, too.
The receiver must monitor that every received frame is followed by another within a timeout period. If your CAN connection is lost and recovered in the meantime, it depends if the interruption was longer or within the timeout.
Additionally, the receiver may monitor that a frame doesn't follow the preceding one too early, but if the frames include the right data, this usually isn't necessary.
[...] The nature of the data carried by this message is like measurement rather than command, i.e. only the most recent one is actually valid, and if some messages are lost that is not an issue as long as the latest one was received successfully.
Through CAN, you shall never communicate "commands" in the meaning that every one of them can trigger a change, like "toggle output state" or "increment set value by one unit" because you never know whether the frame reduplication hits you or not.
Besides, you shall never communicate "anything safety-relevant" through a single frame because any frame may be lost or broken by an error. Instead, "commands" shall be transferred (like measurements) as a stream of periodical frames with measurement or set value updates.
Now, in order to get the required availability out of the protocol design, the TX queue shouldn't be long. If you actually feel as you need that queue, it could be that the bus is overloaded, compared to the timing requirements it faces. From my point of view, the TX "queue" shouldn't be longer than one or two frames. Then, the problem of recovering the CAN connection is nearly fixed...
I've recently been investigating and playing around with linux message queues and have come across something that I don't quite understand why it happens!
If we have two programs running that are both using msgrcv() in an infinite for loop to check for messages and then send two messages, the first program running will receive the 1st message, and the second program the 2nd message? If you keep sending messages it then alternates between each receiver.
Obviously, I understand that as soon as one program has read the message it is removed from the queue but who/how is it decided who will receive the message if they are all infinitely checking?
Any help would be appreciated!
The short answer is that the kernel decides.
The long answer is that this is handled by the do_msgrcv() call within the Linux kernel. If there is no message available, the caller gets put on a queue until a message is available. It's not guaranteed to go back and forth like you describe, since it all depends on the timing of each msgrcv() call, but in your case, it will probably behave that way virtually all of the time.