I have 2 services: producer and consumer.
As far as I understand, message.ts is the time the producer produced the message (not the time the kafka-broker received the message).
Questions
When the consumer consume the message, how can I know how much time it was inside the kafka-broker (without the network latency: from producer to kafka-broker and from kafka-broker to consumer)?
I did a ping from my consumer vm to the kafka broker. the ping result was 0.7ms (millisecond). Does the network latency from each side to the kafka broker is 0.3ms? I assume kafka transport is TCP so there is a "ACK" message for everything. And I assume that each side won't do nothing without "ACK" so I conclude that the network latency on each size is the same as the ping result: 0.7ms (millisecond). Am I correct?
It's a little more complicated than that. Many variables go into how long it takes to process a message. I suggest you look into Distributed Tracing. Something like Zipkin works like magic and is very easy to setup and use. Here's a tutorial on how to setup Zipkin tracing with Spring Boot. You can even use it with Kafka Connect with an interceptor, here's the one I use: brave-kafka-interceptor.
Zipkin produces a trace for every message including all producers and consumers that processed it. Those traces end up lookin something like this:
You can see how much time a message took to be processed, and how much time it took to be consumed afte being produced, which is what you're looking for.
I tested manually this by producing and consuming from the same vm to a kafka (which was inside my cluster). The result was 1.3-1.5 ms.
It means that the procssing time took 0.1 ms on average.
I produced a new message every 1 second to avoid delay while consuming.
This is not the best solution, but it is suffient to my research.
Related
I am noticing something weird happening with my system. So, I am using Kafka to send and receive messages between different systems. I have around 6 or 7 topics each with 10 partitions.
I have an external system that is sending messages on my Kafka topics. So this external system will send messages initially to a topic for eg. "XYZ" and will wait for a response from the Server. Once the Server reads and responds back to the external system then only it will continue further.
Now in our scenario when the external system sends messages to topic "XYZ" it is always sending on partition no 6. This is happening even after restarting the entire system multiple times. Messages on XYZ topic are always being sent to Partition 6.
Now on the Server side, I am using kafka-node to create clients, consumer and producer to consume and produce the messages to kafka. But in this case, it is not consuming from the topic "XYZ".
As a workaround, I tried to test everything by deleting the topics and creating them again but only with a single partition, and this time it worked fine. The entire system worked without any problem.
and creating them again but only with a single partition, and this time it worked fine
Unclear what this scenario is testing... If you wanted 1 partition, then why did the topics get created with 10?
The only reason, in theory, why this would happen is if you are closing and re-creating the producer instance, and it is not correctly randomly seeding the round-robin distribution of the sent events, and always picking the same value. Or, you have defined a key for your records, and it's always hashed to partition 6.
in this case, it is not consuming from the topic "XYZ".
Only one consumer can be active on any partition at a time. If all data ends up in partition 6, then you can only have one consumer... So, sounds like something is reading it, just not what you expect.
I found that non-persistent messages are lost sometimes even though the my pulsar client is up and running.
Those non-persistent messages are lost when the throughput is high (more than 1000 messages within a very short period of time. I personally think that this is not high).
If I increase the parameter receiverQueueSize or change the message type to persistent message, the problem is gone.
I check the Pulsar source code (I am not sure this is the latest one)
https://github.com/apache/pulsar/blob/35f0e13fc3385b54e88ddd8e62e44146cf3b060d/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentDispatcherMultipleConsumers.java#L185
and I think that Pulsar simply ignore those non-persistent messages if no consumer is available to handle the newly arrived non-persistent messages.
"No consumer" here means
no consumer subscribe the topic
OR all consumers are busy on processing messages received before
Is my understanding correct?
The Pulsar broker does not do any buffering of messages for the non-persistent topics, so if consumers are not connected or are connected but not keeping up with the producers, the messages are simply discarded.
This is done because any in-memory buffering would be anyway very limited and not sufficient to change any of the semantics.
Non-persistent topics are really designed for use cases where data loss is an acceptable situation (eg: sensors data which gets updates every 1sec and you just care about last value). For all the other cases, a persistent topic is the way to go.
For every service which read/write from/to topic in Kafka/Redis, there are some basic metrics which we want to have in Prometheus:
How "fast" the writes are for every topic
How "fast" the reads are for every topic
In Kafka, I may want to determine how "fast" each group-id reads.
To determine the "speed" of reading from a topic, one can think of a mechanism where someone publish the same message in intervals of 10 seconds and the consumer sends to Prometheus when it fully processed that message. If the graph show that the message was read every 12 seconds, it means that we have a lag of 2 seconds when reading any messages.
It looks like a lot of repeated manual work on every topic there is on the system.
Question
Is my proposal makes any sense? Are there any best-practice/tools on how to determine "lags"/"speed" of reading/writing from every topic in redis/kafka/... in Prometheus?
I had the exact same issue once.
Maintaining the each topic metrics manually is very tiring and not at all scalable.
I switched over to using kafka_consumergroup_lag metric from the kafka_exporter
This along with the consumergroup,topic labels were enough to let us know to know which topic was not being read/lagging behind and by which consumer group.
Also has other metrics like the rate of meassages being read.
As for converting this lag in terms of time, either attach an produce time to kafka message and read it at the other end of the kafka pipeline and export the difference in time via micrometer from the application to Prometheus.
Or better still:- use tracing to track each message in the piepline using OpenTracing tools like Jaeger
Use this for Redis monitoring.
All these exporters send the data in the Prometheus format and can be directly integrated.
Everything I can find about performance of Amazon Simple Queue Service (SQS), including their own documentation, suggests that getting high throughput requires multiple threads. And I've verified this myself using the JS API with Node 12. If I create multiple threads, I get about the same throughput on each thread, so the total throughput increase is pretty much linear. But I'm running this on a nice machine with lots of cores. When I run in Lambda on a single core, multiple threads don't improve the performance, and generally this is what I would expect of multi-threaded apps.
But here's what I don't understand - there should be very little going on here in the way of CPU, most of the time is spent waiting on web requests. The AWS SQS API appears to be asynchronous in that all of the methods use callbacks for the responses, and I'm using Promises to "asyncify" all of the API calls, with multiple tasks running concurrently. Normally doing this with any kind of async IO is handled great by Node, and improves throughput hugely, I do it all the time with database APIs, multiple streams, etc. But SQS definitely isn't behaving that way, it's behaving as though its IO is actually synchronous and blocking threads on the network calls, which would be outrageous for any modern API.
Has anyone had success getting high SQS message throughput in a single Node thread? The max I'm seeing is about 50 to 100 messages/sec for FIFO queues (send, receive, and delete, all of which are calling the batch methods with the max batch size of 10). And this is running in lambda, i.e. on their own network, which is only slightly faster than running it on my laptop over the Internet, another surprising find. Amazon's documentation says FIFO queues should support up to 3000 messages per second when batching, which would be just fine for me. Does it really take multiple threads on multiple cores or virtual CPUs to achieve this? That would be ridiculous, I just can't believe that much CPU would be used, it should be mostly IO time, which should be asynchronous.
Edit:
As I continued to test, I found that the linear improvement with the number of threads only happened when each thread was processing a different queue. If the threads are all processing the same queue, there is no improvement by adding threads. So it behaves as though each queue is throttled by Amazon. But the throughput to which it seems to be throttling is way below what I found documented as the max throughput. Really confused and disappointed right now!
Michael's comments to the original question were right on. I was sending all messages to the same message group. I had previously been working with AMQP message queues, in which messages will be ordered in the queue in the order they're sent, and they'll be distributed to subscribers in that order. But when multiple listeners are consuming the AMQP queue, because of varying network latencies, there is no guarantee that they'll be received in that order chronologically.
So that's actually a really cool feature of SQS, the guarantee that messages will be chronologically received in the order they were sent within the same message group. In my case, I don't care about the receipt order. So now I'm setting a unique message group ID on each message, and scaling up performance by increasing the number of async message receive loops, still just in one thread, and the throughput is amazing!
So the bottom line: If exact receipt order of messages isn't important for your FIFO queue, set the message group ID to a unique value on each message, and scale out with more receiver tasks to get the best throughput performance. If you do need guaranteed message ordering, it looks like around 50 messages per second is about the best you'll do.
I use Kafka in production. Many services send and read messages into it.
All work fine but I had a bug in one service.
For a weird reason this one sends millions messages by second to Kafka.
Due to this bug, my Kafka crashes.
It's not a Kafka bug but how can I protect it against potential flood ?
Upgrade to Kafka 0.9 and enable Quotas.
You need some kinda throttling on the service not to flood Kafka. One of the option would be to use Apigee.
Indeed with Apigee you can throttle the response. This can be done in two ways;
With a 'Spike Arest', which smoothes out the requests over time, for instance if you define an amount of requests per second, its smoothed over milliseconds, and when an amount of reuqests defined per minute it is smoother per second.
A Qouta defines a maximum amount of requests per minute, day, etc. After that requests are rejected.
Apigee also provides a 'concurrent rate' policy which does exactly that; tracks the amount of concurrent calls, and rejects in case more are sent. (HTTP 503).
You might like:
http://apigee.com/docs/api-services/content/comparing-quota-spike-arrest-and-concurrent-rate-limit-policies
Also, in case you want to keep it custom, check out Volos, or Apigee 127, which is a great open source project created by Apigee as well.
hope this helps.
Actually, on a final note; I'd solve the root cause; why do you get 1mio requests? Or is that expected bevahior?