I am using Spring Integration with mqtt-paho version 4.0.4 For receiving MQTT messages on specified topic.
When application is receiving huge load I found that, sometimes application is dropping connection with IMA (mqtt) and this was happened three times in a span of 1 Lac record.
But it regains the connectivity and started consuming messages received there after. There were no issue in IMA re-connectivity.
There is some other issue which I faced during this testing.
When there is continuous load on application, at some point application stops receiving messages and we can see one message flashed on screen i.e.
May 04, 2015 2:45:29 PM org.eclipse.paho.client.mqttv3.internal.ClientState checkForActivity
SEVERE: gvjIpONtSpP: Timed out as no activity, keepAlive=60,000 lastOutboundActivity=1,430,730,869,017 lastInboundActivity=1,430,730,929,151
After this we can see that there are no messages received on application even if continuous load is pushed through utility.
This behavior I found it three times.
At around 40K.
At around 90K.
At around 145K.
There is no consistent point or figures where application actually stops receiving messages.
Please let me know if anybody has faced and solved this before .
We had the same issue during performance testing and during MQTT Paho client performance/durability testing, before moving to production. The issue was on broker side, after settings adjustment, the IMA broker was able to consume millions of messages with no rejection.
Please look into max buffer parameter on IMA configuration web console. And overlimit behavior policy (what to do with messages published over specified threshold): reject, rollover etc.
Related
I'm trying to use GCP Pub/Sub StreamingPull using the NodeJs client and I understand that the pub sub is designed for 100% error rate as mentioned in Docs.
So do I have to restart the listener if I face errors in the errorHandler and also please tell what error code should I be looking for to see if the streaming connection is closed. Here is the ref Error Codes
const errorHandler=(error)=>{
if(errorCodeCheckCondition){
subscription.on('message', messageHandler);
subscription.removeListener('message', messageHandler);
}
}
subscription.on('error', errorHandler);
I'm using GCP Pub/Sub StreamingPull for first time, so please guide.
You do need to re-establish the streaming pull connection after you get any error.
According to the rpc StreamingPull
The server will close the stream and return the status on any error. The server may close the stream with status UNAVAILABLE to reassign server-side resources, in which case, the client should re-establish the stream. Flow control can be achieved by configuring the underlying RPC channel.
Since You know about StreamingPull has a 100% error rate, I believe you must have also gone through the Diagnosing StreamingPull errors.
The Pub/Sub client library will re-establish the underlying streaming pull connection when it disconnects for a retriable reason, e.g., an UNAVAILABLE error. You can see in the StreamingPull config in the library the set of errors that are retried internally.
The errors you would typically get back at the application level would be ones where some additional intervention is likely necessary, e.g., a PERMISSION_DENIED error (where the subscriber does not have permission to receive messages on the subscription) or a NOT_FOUND error (where the subscription does not exist. Retrying on these types of errors is likely just to result in the error reoccurring until the underlying issue is resolved.
You could decide that retrying is what you want to do because you want the subscriber to start working again without having to manually restart it once other steps are taken to fix the problem, but you'll want to make sure you have some way to discover these types of issues, perhaps through some kind of Cloud Monitoring alerting on streaming pull errors or on a large number of unprocessed messages building up.
To clarify the title: I am using ActiveMQ 5.15.15 (NOT the Artemis engine), and I am using AMQP 1.0 without official JMS libraries. And to be more specific, I am using the AmazonMQ version of this, which will soon upgrade to 5.16.2. I could force the upgrade, if needed.
I'm using an AMQP 1.0 compatible library (rhea) that has served us well so far, but I'm not finding any documentation for how to get ActiveMQ's redelivery plugin to work with my library. The library maintainers are unaware with how this is exposed via ActiveMQ, as well.
I've not been able to get the redelivery plugin to work, despite trying to add various headers, delivery annotations, message annotations, or application properties. I do have schedulerSupport="true" in my broker element for the server config.
These are the keys I've tried, and the values are numeric. E.g., 30000 for 30 seconds before allowing a consumer/subscriber see the message in the queue. I saw them in various docs, and it didn't hurt to try them.
AMQ_SCHEDULED_DELAY
x-opt-delivery-delay
_AMQ_SCHED_DELIVERY
I have also released the message from the client, meaning it failed to deliver (also passing a value that signals the failure to the broker and increases the attempted delivery count). While the number of delivery attempts increased, the delay and exponential backoff have not seemed to be working at the broker level.
I see that the STOMP protocol allows for headers when publishing, which allow setting options a bit more clearly. However, I don't want to switch everything over unless it makes sense to do so.
I also saw another ability to send a delayed message as a topic via the REST API, but I'm not sure if that was intended to be a production use case.
So right now, I'm either looking at:
hold the message in memory for a bit and attempt to republish or release it after a delay
Investigate STOMP, see if the redelivery plugin works with that
But I'm hoping someone knows where to implement this.
My redeliveryPolicy is basic:
<!--
The Redelivery plugin extends the capabilities of destination policies with respect to message redelivery.
For more information, see http://activemq.apache.org/message-redelivery-and-dlq-handling.html
-->
<redeliveryPlugin fallbackToDeadLetter="true" sendToDlqIfMaxRetriesExceeded="true">
<redeliveryPolicyMap>
<redeliveryPolicyMap>
<redeliveryPolicyEntries>
<!--<redeliveryPolicy maximumRedeliveries="4" queue="SpecialQueue" redeliveryDelay="10000"/>-->
</redeliveryPolicyEntries>
<defaultEntry>
<!-- 5s -> 15s -> 45s -> 135s -> 405s -->
<redeliveryPolicy backOffMultiplier="3" initialRedeliveryDelay="5000" maximumRedeliveries="5"/>
</defaultEntry>
</redeliveryPolicyMap>
</redeliveryPolicyMap>
</redeliveryPlugin>
Update
I am using the auth plugin, and there's an entry that seems like it's for a built-in process. I think this came from a sample/default config. There doesn't appear to be a whole lot of documentation around this from a quick search. I can try opening access to other users, but each update/restart can take up to 15 minutes with the current setup.
<authorizationEntry admin="administrators" queue="SchedulingProcessor.>" write="scheduling-processor"/>
Comment Clarifications
My main objective is to delay redeliveries, so consumers don't see a failed message that was placed back into the queue for n seconds.
I started with no special headers/properties/annotations + the redelivery plugin, which also didn't work.
There is a distinction around message delivery delay and message redelivery delay that I think you are confusing or at least the question is mixing up.
A sender can request that a message be delivered after some delay from an AMQP client using the Message Annotations section of the sent message and adding in 'x-opt-delivery-delay' or 'x-opt-delivery-time' annotation assuming the broker has enabled scheduled deliveries. Some examples of this can be found in the ActiveMQ unit test. The delay annotation indicates a relative delay from time of receipt in milliseconds while the delivery time annotation indicates a time in UTC to deliver the message.
The ActiveMQ 5 redelivery policy affects messages that have been explicitly tagged as not deliverable on the client and therefore the AMQP Released outcome is not the right choice to trigger this behavior as it simply indicates that the client isn't going to process it and the remote should consider it undelivered and send it elsewhere. You would need to use one of Rejected or Modified(undeliverableHere=true) to "poison" the message and trigger the redelivery policy. This should if things go right trigger a redelivery after some delay although since ActiveMQ 5 has a relatively basic AMQP protocol head it will likely resend to the same consumer even if you've explicitly set the undeliverable here flag. I don't know how much that bit has been tested if any so your mileage may vary.
I have a namespace in ASB with a single topic and subscription. I can see the message count from when I was testing the consumer (ruby client), the first spike, but after I stopped testing and nothing in the client was running there were 10 incoming requests from 'somewhere'. The second spike in the graph. The machine was off at this point so it must have come from Azure.
Within half an hour of turning on my machine there were 6 incoming requests, the third spike but nothing is running as it's a commandline client so I assume it's Azure again.
I don't have anything else in Azure (functions apps etc). Literally nothing is running/polling/pulling/peeking etc.
Is it possible to identify where these requests are coming from? Graph is below.
Incoming requests are not incoming messages. When there are no messaging and you receive nothing, there are still operations taking place. This is due to the fact that the client (assuming you're using client with a message handler rather than trying to retrieve messages yourself), the client is performing long-polling. I.e. the client will poll for new messages rather than the broker to push those. Unlike RabbitMQ, which will push to a client when messages become available.
Incoming Requests is a 'Request Metric' that shows number of requests made to the "Service Bus service" over a specified period. Even without any message flow within the ServiceBus Namespace there can be incoming requests.
Probably it is not possible to find the origin of the incoming request.
Problem
We are developing a Azure Service Bus based Cloud Service, but after 24 hours the queue clients seem to get closed automatically.
Can someone confirm this behavior or give advise how to fix it?
At the moment we close the clients after 24 hours manually and recreate them to avoid this effect, but this can't be the only solution.
Sessions dropping intermittently is a normal occurrence. The AMQP protocol and stack in the client is newer and generally more resilient against this. The only reason not to use AMQP is if you are using transactions. Also, unless you have a good reason to run your own receive loop, use OnMessage.
You are getting ‘OperationCanceledException’ when the link fails for any reason and any in-flight requests will fail with this exception. However, this is transient, so you should be able to reuse the same QueueClient to issue receives and those should (eventually) work as the client recovers. OnMessage will hide all of that from you.
I followed this article to try eventhub with Apache Storm, But when I run the Storm topology it was receiving events for a minute and then it stopped receiving. So I've restarted my program and then it was receiving the remaining messages. Every time I run the program after a minute it couldn't receive from eventhub. Please help me with the possibilities of the issue...
Should I change any configurations at Storm or Zookeeper.
The above jar contains a fix for a known issue in the QPID JMS client, which is used by the Event Hub spout implementation. When the service sends an empty frame (heartbeat to keep connection alive), a decoding error occurs in the client and that causes the client to stop processing commands. Details of this issue can be found here: https://issues.apache.org/jira/browse/QPID-6378