I have a question regarding JMS and Spring Integration.
I have 3 queues, let's call them QUEUE_SOURCE, QUEUE_TARGET and QUEUE_ERROR.
A DefaultMessageListenerContainer is used to read the messages from the QUEUE_SOURCE.
I have configured a JMS Transaction manager for these queues.
When I read from the QUEUE_SOURCE, but an error occurs when posting the message to QUEUE_TARGET, I
can see that the message is retried a few times before an exception is thrown, triggering a rollback.
I guess what happens is this:
1) Start transaction
2) Read message
3) Try to post message on target queue QUEUE_TARGET, but fail
4) Do not commit because of error, so message is not removed from QUEUE_SOURCE
OR
1) Start transaction
2) Read message
3) Remove message from QUEUE_SOURCE
4) Try to post message on target queue QUEUE_TARGET, but fail
5) Put message back on QUEUE_SOURCE
After all redeliveries fail, a rollback is triggered.
However, what happens during this rollback confuses me.
I have no clue where the message ends up, it certainly does not end up on the source queue, and it does not end up on the error queue either. So what is really happening during this rollback?
You would need to show your exact configuration but your speculation #1 is close; the broker delivers the message but does not remove it from the queue until the commit. If your code throws and exception, the framework will execute session.rollback() and the broker will redeliver. If the delivery is successful, the framework will commit the transaction and only then will the message be removed.
Related
I've been recently having problems with my Service Bus queue. Random messages (one can pass and the other not) are placed on the deadletter queue with the error message saying:
"DeadLetterReason": "Moved because of Unable to get Message content There was an error deserializing the object of type System.String. The input source is not correctly formatted."
"DeadLetterErrorDescription": "Des"
This happens even before my consumer has the chance to receive the message from the queue.
The weird part is that when I requeue the message through Service Bus Explorer it passes and is successfully received and handled by my consumer.
I am using the same version of Service Bus either for sending and receiving the messages:
Azure.Messaging.ServiceBus, version: 7.2.1
My message is being sent like this:
await using var client = new ServiceBusClient(connString);
var sender = client.CreateSender(endpointName);
var message = new ServiceBusMessage(serializedMessage);
await sender.SendMessageAsync(message).ConfigureAwait(true);
So the solution I have for now for the described issue is that I implemented a retry policy for the messages that land on the dead-letter queue. The message is cloned from the DLQ and added again to the ServiceBus queue and for the second time there is no problems and the message completes successfully. I suppose that this happens because of some weird performance issues I might have in the Azure infrastructure. But this approach bought me some time to investigate further.
I have dot net code which read loan-numbers from Azure service queue and calls my API for each loan-number.
This is my code which calls the api
private async Task ProcessMessagesAsync(Message message, CancellationToken token)
{
try
{
string loanNumber = Encoding.UTF8.GetString(message.Body);
_logger.LogInformation($"Received message: SequenceNumber:{message.SystemProperties.SequenceNumber} LoanNumber:{loanNumber}");
//API CALL HERE
await _apiClient.getResult(loanNumber);
await _queueClient.CompleteAsync(message.SystemProperties.LockToken);
}
catch (Exception ex)
{
//sending failed messages to Dead Letter queue
await _queueClient.AbandonAsync(message.SystemProperties.LockToken);
}
}
Failed loan-numbers are successfully sent to Dead Letter queue. When the server is down or bad request from API response.
I want to call the api after certain duration on the loan-numbers which are in Dead Letter Queue.
Is there any way to convert the messages in Dead Letter queue to active messages after some interval??
I am new to azure. Please help me to resolve the issue.
Thanks in advance.
Is there any way to convert the messages in Dead Letter queue to
active messages after some interval??
Automatically, no. However what you could do is read the messages from dead letter queue and then post them as new message in your main queue.
As far as automating the whole process, one possible solution would be to run a timer triggered Azure Function that reads messages from a dead letter queue periodically and post them in your main queue.
#akhil, it's worth noting that these messages will automatically be re-queued on the main queue until their DeliveryCount exceeds the MaxDeliveryCount of your main queue. The default value for MaxDeliveryCount is 10 so any failed requests to the API be retried ten times by this handler before being moved to the DLQ.
If you wanted to be a bit 'smarter' about this you could delay the retries using the ScheduledEnqueueTimeUtc property on the message in your catch block.
As #Gaurav Mantri has said, the framework offers nothing to process dead-letter messages 'for free'; you'll have to code a handler yourself as you would for a normal queue.
No, there is no way to convert the DLQ message to active message.
you need to process the DLQ and fix the reason as to why it ended in DLQ and then re transmit it back to your queue.
A sample from Github can be useful here.
https://github.com/Azure/azure-service-bus/tree/master/samples/DotNet/Microsoft.ServiceBus.Messaging/DeadletterQueue
(This sample shows how to move messages to the Dead-letter queue, how to retrieve messages from it, and resubmit corrected message back into the main queue.)
This could be an old answer but we can use ServiceBusExplorer from Microsoft. It can pull the messages from dead letter queue and then those can be requeued using
Let me know if you have any further questions.
We are using Azure Queue for our printing job but when deleting message from queue by queue.DeleteMessage(message), the method throws below exception.
The remote server returned an error: (404) Not Found
Above exception was handled but still looking for workaround.
Can anyone please suggest how to fix it.
Thanks,
Sneh
According to this article, we can find that:
After a client retrieves a message with the Get Messages operation,
the client is expected to process and delete the message. To delete
the message, you must have two items of data returned in the response
body of the Get Messages operation:
The message ID, an opaque GUID value that identifies the message in the queue.
A valid pop receipt, an opaque value that indicates that the message has been retrieved.
If a message with a matching pop receipt is not found, the service returns error code 404 (Not Found). And Pop receipts remain valid until one of the following events occurs:
The message has expired.
The message has been deleted using the last pop receipt received
either from Get Messages or Update Message.
The invisibility timeout has elapsed and the message has been
dequeued by a Get Messages request. When the invisibility timeout
elapses, the message becomes visible again. If it is retrieved by
another Get Messages request, the returned pop receipt can be used
to delete or update the message.
The message has been updated with a new visibility timeout. When the
message is updated, a new pop receipt will be returned.
I ran into this issue today and the root cause was ownership issues between two different queues. We had setup two queues, one to hold our message awaiting processing and one for messages that had errored out. The problem came with the logic of how the message was moved between queues.
If our processing failed, we would perform the following logic:
_errorQueue.AddMessage(msg);
_queue.DeleteMessage(msg);
The DeleteMessage would also return a (404) Not Found because the msg had been moved to the errorQueue. There were two solutions that I found to this issue:
1. Switch Logic
If you switch the logic than the msg will be deleted before being added to the errorQueue which will avoid the ownership swap.
_queue.DeleteMessage(msg);
_errorQueue.AddMessage(msg);
2. Insert Copy of Message
Solution #1 has the potential to lose the message if something happens between deletion and insertion (small chance but a chance nonetheless). The solution I went with inserted a copy of the msg with the same payload so it didn't run into this ownership issue because it was a different object.
_errorQueue.AddMessage(new CloudQueueMessage(msg.AsString));
_queue.DeleteMessage(msg);
Debugging Tip
One useful tip I encountered while debugging it making sure the exception your catching isn't the default Exception. Catch the StorageException instead to get access to Azure Storage related error information.
try
{
_queue.DeleteMessage(msg);
}
catch (StorageException ex) //use this instead of base Exception
{
var info = ex.RequestInformation; //has useful information
}
If can provide more information to help you debug your real issue.
I'm using Azure WebJobs to poll a queue and then process the message.
Part of the message processing includes a hit to 3rd party HTTP endpoint. (e.g. a Weather api or some Stock market api).
Now, if the hit to the api fails (network error, 500 error, whatever) I try/catch this in my code, log whatever and then ... what??
If I continue .. then I assume the message will be deleted by the WebJobs SDK.
How can I:
1) Say to the SDK - please don't delete this message (so it will be retried automatically at the next queue poll and when the message is visible again).
2) Set the invisibility time value, when the SDK pops a message off the queue for processing.
Thanks!
Now, if the hit to the api fails (network error, 500 error, whatever) I try/catch this in my code, log whatever and then ... what??
The Webjobs SDK behaves like this: If your method throws an uncaught exception, the message is returned to the Queue with its dequeueCount property +1. Else, if all is well, the message is considered successfully processed and is deleted from the Queue - i.e. queue.DeleteMessage(retrievedMessage);
So don't gracefully catch the HTTP 500, throw an exception so the SDK gets the hint.
If I continue .. then I assume the message will be deleted by the WebJobs SDK.
From https://github.com/Azure/azure-content/blob/master/articles/app-service-web/websites-dotnet-webjobs-sdk-get-started.md#contosoadswebjob---functionscs---generatethumbnail-method:
If the method fails before completing, the queue message is not deleted; after a 10-minute lease expires, the message is released to be picked up again and processed. This sequence won't be repeated indefinitely if a message always causes an exception. After 5 unsuccessful attempts to process a message, the message is moved to a queue named {queuename}-poison. The maximum number of attempts is configurable.
If you really dislike the hardcoded 10-minute visibility timeout (the time the message stays hidden from consumers), you can change it. See this answer by #mathewc:
From https://stackoverflow.com/a/34093943/4148708:
In the latest v1.1.0 release, you can now control the visibility timeout by registering your own custom QueueProcessor instances via JobHostConfiguration.Queues.QueueProcessorFactory. This allows you to control advanced message processing behavior globally or per queue/function.
https://github.com/Azure/azure-webjobs-sdk-samples/blob/master/BasicSamples/MiscOperations/CustomQueueProcessorFactory.cs#L63
protected override async Task ReleaseMessageAsync(CloudQueueMessage message, FunctionResult result, TimeSpan visibilityTimeout, CancellationToken cancellationToken)
{
// demonstrates how visibility timeout for failed messages can be customized
// the logic here could implement exponential backoff, etc.
visibilityTimeout = TimeSpan.FromSeconds(message.DequeueCount);
await base.ReleaseMessageAsync(message, result, visibilityTimeout, cancellationToken);
}
I'm seeing weird behaviour when sending data through my channels. I'm using SI gateway when sending a message to be processed. The gateway is setup as below
<integration:gateway id="rptPubGateway"
default-request-channel="rptPubInChannel"
default-reply-channel="rptOutputAvailableChannel"
default-reply-timeout="60000"
service-interface="xxxx.RptPubGateway" />
The reply channel is being set up as a publish/subscribe channel
<integration:publish-subscribe-channel id="rptOutputAvailableChannel" />
The last service that processes the message is being declared as below
<integration:service-activator input-channel="rptOutputAvailableChannel" ref="requestMessageHandler" method="rptIsDone" output-channel="nullChannel"/>
Now, the issue that i have is that while the code works fine most of the time, it fails sometimes. When everything works fine the last component processing my message is PublishSubsChannel
PublishSubscribeChannel - preSend on channel 'rptOutputAvailableChannel'
but when it fails the last component becomes BridgerHandler
BridgeHandler#e851a798' sending reply Message:
I should mention that there are no exceptions being thrown while processing my message. (after the failure I can always resend the same message and everything will work OK)
I'm not sure how that BridgerHandler gets created. From what I know this bridge gets created for pub/subs channels but then why I don't see it in the log when everything works fine ?
I'd appreciate any help
When you say "fails" what do you mean?
What are the other consumers on rptOutputAvailableChannel?
The BridgeHandler you see is an internal bridge to the message's replyChannel header, which is where the reply ultimately must go. When explicitly sending to rptOutputAvailableChannel, the bridge handler is always invoked to get the reply to the message's temporary reply channel.