I'm working on a project where we would like to use ServiceBus like a classical queue, for messaging. Very often I see messages in queue that I can't get. These messages are active (not dead letters). I don't receive any exceptions, it waits for timeout and then message is null without any issues. I've already checked size of messages but all are less then 1.5 KB.
try
...
QueueClient _queueClient = QueueClient.CreateFromConnectionString(Properties.Settings.Default.ServiseBusConnectionString, Properties.Settings.Default.StatQueueName);
var count = 1000;
for (var i = 1; i <= count; i++)
{
var message = _queueClient.Receive();
//timeout
if (message == null)
{
//message is null :(
continue;
}
}
...
catch
Read How to receive messages from a queue and make sure you use _queueClient.Complete() or _queueClient.Abandon() to finish with each message.
Related
I implemented 2 simple services in Service Fabric, which communicate over Event Hub and I encounter very strange behavior.
The listener service reads the messages using PartitionReciever with ReceiveAsync method. It reads the messages always from the start of the partition, but even though the maxMessageCount parameter is set to very high number, which definitely exceeds the number of messages in the partition, it reads only "random" amount of messages but never the full list. It always starts to read correctly from the beginning of the partition but it almost never reads the full list of messages which should be present there...
Did I miss something in documentation and this is normal behavior, or am I right, that this is very strange bahviour?
A code snippet of my receiver service:
PartitionReceiver receiver = eventHubClient.CreateReceiver(PartitionReceiver.DefaultConsumerGroupName, Convert.ToString(partition), PartitionReceiver.StartOfStream);
ServiceEventSource.Current.Write("RecieveStart");
IEnumerable<EventData> ehEvents = null;
int i = 0;
do
{
try
{
ehEvents = await receiver.ReceiveAsync(1000);
break;
}
catch (OperationCanceledException)
{
if (i == NUM_OF_RETRIES-1)
{
await eventHubClient.CloseAsync();
StatusCode(500);
}
}
i++;
} while (i < NUM_OF_RETRIES);
I'm trying to get a queue's retry logic working and I'm having an issue. the minBackoff variable doesn't seem to actually work. I see in my logs, a message gets received then fails then retries almost immediately. My minBackoff is set to 600 seconds.
Here's the code that sets up the query:
NamespaceManager nsManager = NamespaceManager.CreateFromConnectionString(connectionString);
nsManager.Settings.RetryPolicy = new RetryExponential(minBackoff: TimeSpan.FromSeconds(5),
maxBackoff: TimeSpan.FromSeconds(30),
maxRetryCount: 3);
if (!nsManager.QueueExists(queueName))
{
nsManager.CreateQueue(queueName);
}
else
{
nsManager.DeleteQueue(queueName);
nsManager.CreateQueue(queueName);
}
QueueClient client = QueueClient.CreateFromConnectionString(connectionString, queueName);
client.RetryPolicy = new RetryExponential(minBackoff: TimeSpan.FromSeconds(15),
maxBackoff: TimeSpan.FromSeconds(600),
maxRetryCount: 3);
for (int i = 0; i < 2000; i++)
{
UserCreationSubmitted creationMessage = new UserCreationSubmitted()
{
CreationStatus = "Step 1",
Id = Guid.NewGuid(),
UserName = "user number " + i,
Userid = Guid.NewGuid()
};
BrokeredMessage message = new BrokeredMessage(creationMessage);
client.Send(message);
}
Here's the code that's not working how I think it should...
client.RetryPolicy = new RetryExponential(minBackoff: TimeSpan.FromSeconds(15),
maxBackoff: TimeSpan.FromSeconds(600),
maxRetryCount: 3);
client.OnMessage(message =>
{
UserCreationSubmitted msg = message.GetBody<UserCreationSubmitted>();
Console.WriteLine("------------------------------");
Console.WriteLine($"Body {msg.UserName}");
Random rnd = new Random();
int ranNum = rnd.Next(0, 9);
if (msg.UserName.Contains(ranNum.ToString()))
{
Console.WriteLine("!!!Error!!!");
Console.WriteLine("------------------------------");
throw new Exception();
}
});
Does anyone have an idea as to why the minbackoff and maxbackoff's don't seem to actually work here? Oddly enough the maxRetryCount's working like a trooper so I imagine it's definitely something in my implementation that's causing the others to not work.
RetryExponential is used by ASB client for retries when a failure to receive a message is happening. In your code, exception is thrown during processing within OnMessage API callback, after the fact that message was received. OnMessage API will abandon the message, causing it to show up right away.
There a few options you could take:
Clone a message, set ScheduledEnqueueTimeUtc to the delay you want, send it and then complete the original message.
Defer your message, but then you can only receive it by SequenceNumber. In that case, you could create a new message with that would contain original message sequence number as a payload, schedule the new message and send it. That way you'd have the delivery count on the original message have an accurate count.
Ideally, would be nice to abandon a message with a timespan, but that's not possible with the current API.
SubscriptionClient receiver = messageFactory.CreateSubscriptionClient("NewTopic", subscriberId);
TimeSpan e = new TimeSpan(0, 0, 5, 0, 0);
RetryExponential x = new RetryExponential(e,e,e,e,2);
OnMessageOptions options = new OnMessageOptions();
options.AutoComplete = false;
//options.AutoRenewTimeout = TimeSpan.FromMinutes(1);
options.ExceptionReceived += options_ExceptionReceived;
receiver.OnMessage(receivedMessage =>
{
try
{
Console.WriteLine(receivedMessage.Label);
bool t = receivedMessage.IsBodyConsumed;
Console.WriteLine(string.Format("Message received: {0}", receivedMessage.GetBody<string>()));
Console.WriteLine(receivedMessage.SequenceNumber);
Console.WriteLine(receivedMessage.TimeToLive);
Console.WriteLine(receivedMessage.To);
Console.WriteLine(receivedMessage.DeliveryCount);
receivedMessage.Abandon();
}
catch (Exception)
{
// Indicates a problem, unlock message in subscription.
receivedMessage.Abandon();
}
}, options);
Hi All,
In the retryExponential Constructor i set the maxRetryCount as 2.
And i delibretly Abandon the message in Onmessage to check the max retry count. Even after setting the retry count to 2 i am receiving the message more than 2 times.
--TIA
It looks like you're confusing transient fault handling with dead lettering.
The retry mechanism you're using is to cope with distributed computing issues. Like throttling and service unavailability.
If you're unable to handle an incoming message, you usually move it to the DLQ after some attempts.
(and you're not using the 'x' variable)
I want to be able to remove selected messages from my deadletter queue.
How is this accomplished?
I constantly get an error:
The operation cannot be completed because the RecieveContext is Null
I have tried every approach I can think of and read about, here is where I am now:
public void DeleteMessageFromDeadletterQueue<T>(string queueName, long sequenceNumber)
{
var client = GetQueueClient(queueName, true);
var messages = GetMessages(client);
foreach(var m in messages)
{
if(m.SequenceNumber == sequenceNumber)
{
m.Complete();
}
else
{
m.Abandon();
}
}
}
/// <summary>
/// Return a list of all messages in a Queue
/// </summary>
/// <param name="client"></param>
/// <returns></returns>
private IEnumerable<BrokeredMessage> GetMessages(QueueClient client)
{
var peekedMessages = client.PeekBatch(0, peekedMessageBatchCount).ToList();
bool getmore = peekedMessages.Count() == peekedMessageBatchCount ? true : false;
while (getmore)
{
var moreMessages = client.PeekBatch(peekedMessages.Last().SequenceNumber, peekedMessageBatchCount);
peekedMessages.AddRange(moreMessages);
getmore = moreMessages.Count() == peekedMessageBatchCount ? true : false;
}
return peekedMessages;
}
Not sure why this seems to be such a difficult task.
The issue here is that you've called PeekBatch which returns the messages that are just peeked. There is no Receive context which you can then use to either complete or abandon the message. The Peek and PeekBatch operations only return the messages and does not lock them at all, even if the receivedMode is set to PeekLock. It's mostly for skimming the queue, but you can't take an action on them. Note the docs for both Abandon and Complete state, "must only be called on a message that has been received by using a receiver operating in Peek-Lock ReceiveMode." It's not clear here, but Peek and PeekBatch operations don't count in this as they don't actually get a receive context. This is why it fails when you attempt to call abandon. If you actually found the one you were looking for it would throw a different error when you called Complete.
What you want to do is use a ReceiveBatch operation instead in PeekBatch RecieveMode. This will actually pull a batch of the messages back and then when you look through them to find the one you want you can actually affect the message complete. When you fire the abandon it will immediately release the message that isn't what you want back to the queue.
If your deadletter queue is pretty small usually this won't be bad. If it is really large then taking this approach isn't the most efficient. You're treating the dead letter queue more like a heap and digging through it rather than processing the messages "in order". This isn't uncommon when dealing with dead letter queues which require manual intervention, but if you have a LOT of these then it may be better to have something processing the dead letter queue into a different type of store where you can more easily find and destroy messages, but can still recreate the messages that are okay to push to a different queue to reprocess.
There may be other options, such as using Defer, if you are manually dead lettering things. See How to use the MessageReceiver.Receive method by sequenceNumber on ServiceBus.
I was unsuccessful with MikeWo's suggestion, because when I used the combination of instantiating the DLQ QueueClient with ReceiveMode.PeekLock and pulling messages with ReceiveBatch, I was using the versions of Receive/ReceiveBatch requesting the message by its SequenceNumber.
[aside: in my app, I peek all the messages and list them, and have another handler to re-queue to the main queue a dead-lettered message based on it's specific sequence number...]
But the call to Receive(long sequenceNumber) or ReceiveBatch(IEnumerable sequenceNumber) on the DLQClient always throws the exception, "Failed to lock one or more specified messages. The message does not exist." (even when I only passed 1 and it is definitely in the queue).
Additionally, for reasons that aren't clear, using ReceiveBatch(int messageCount), always only returns the next 1 message in the queue no matter what value is used as the messageCount.
What finally worked for me was the following:
QueueClient queueClient, deadLetterClient;
GetQueueClients(qname, ReceiveMode.PeekLock, out queueClient, out deadLetterClient);
BrokeredMessage msg = null;
var mapSequenceNumberToBrokeredMessage = new Dictionary<long, BrokeredMessage>();
while (msg == null)
{
#if UseReceive
var message = deadLetterClient.Receive();
#elif UseReceiveBatch
var messageEnumerable = deadLetterClient.ReceiveBatch(CnCountOfMessagesToPeek).ToList();
if ((messageEnumerable == null) || (messageEnumerable.Count == 0))
break;
else if (messageEnumerable.Count != 1)
throw new ApplicationException("Invalid expectation that there'd always be only 1 msg returned by ReceiveBatch");
// I've yet to get back more than one in the deadletter queue, but...
var message = messageEnumerable.First();
#endif
if (message.SequenceNumber == lMessageId)
{
msg = message;
break;
}
else if (mapSequenceNumberToBrokeredMessage.ContainsKey(message.SequenceNumber))
{
// this means it's started the list over, so we didn't find it...
break;
}
else
mapSequenceNumberToBrokeredMessage.Add(message.SequenceNumber, message);
message.Abandon();
}
if (msg == null)
throw new ApplicationException("Unable to find a message in the deadletter queue with the SequenceNumber: " + msgid);
var strMessage = GetMessage(msg);
var newMsg = new BrokeredMessage(strMessage);
queueClient.Send(newMsg);
msg.Complete();
I have used .net C# code to put messages on the queue and get messages back. I have no problem in accessing the queue and getting messages. Now I want to have the get message calls under Transaction and used explicit transaction option to commit and rollback the messages.
try
{
MQQueueManager queueManager;
MQEnvironment.Hostname = hostName;
MQEnvironment.Channel = channelName;
MQEnvironment.Port = 1414;
MQEnvironment.properties.Add(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES);
queueManager = new MQQueueManager(queueManagerName);
// obtain a read/write queue reference
var queue = queueManager.AccessQueue(queueName, MQC.MQOO_INPUT_AS_Q_DEF + MQC.MQOO_INQUIRE + MQC.MQOO_FAIL_IF_QUIESCING);
IList<string> Messages = new List<string>();
using (var scope = new CommittableTransaction())
{
CommittableTransaction.Current = scope;
var message = new MQMessage();
try
{
var getMessageOptions = new MQGetMessageOptions();
getMessageOptions.Options += MQC.MQGMO_SYNCPOINT ;
int i = queue.CurrentDepth;
queue.Get(message,getMessageOptions);
Console.WriteLine(message.ReadString(message.MessageLength));
scope.Rollback();
}
catch (MQException mqe)
{
if (mqe.ReasonCode == 2033)
{
Console.WriteLine("No more message available");
Console.ReadLine();
scope.Rollback();
}
else
{
Console.WriteLine("MQException caught: {0} - {1}", mqe.ReasonCode, mqe.Message);
Console.ReadLine();
scope.Rollback();
}
}
CommittableTransaction.Current = null;
}
// closing queue
queue.Close();
// disconnecting queue manager
queueManager.Disconnect();
Console.ReadLine();
}
catch (MQException mqe)
{
Console.WriteLine("");
Console.WriteLine("MQException caught: {0} - {1}", mqe.ReasonCode, mqe.Message);
Console.WriteLine(mqe.StackTrace);
Console.ReadLine();
}
The first problem I faced was , related to access to System.Dotnet.XARecovery queue. Even though I had access to the queue to get messages from the queue , the program started to fail because of the access rights on the recovery queue when below line was invoked.
queue.Get(messages),
Then I got the access on the recovery queue and access denied problem was resolved. Now after getting the message from the queue , the messages are not roll backed after scope.RollBack() is called.
I checked in the System.Dotnet.XARecovery queue and dead letter queue and there was not nothing there as well.
Why I am not able to see the rolled back messages in the WMQ message queue.
You have a scope.Commit(); after queue.Get(message); After getting the message you are explicitly calling the Commit. If the Get is successful, the Commit call tells the queue manager to remove the message from queue. So there is no chance of message getting rolled back.
EDIT: GMO_SYNCPOINT option is missing in your code. You need to have something like this
MQGetMessageOptions getMessageOptions = new MQGetMessageOptions();
getMessageOptions.Options += MQC.MQGMO_SYNCPOINT;
queue.Get(message, getMessageOptions);
I figured out the solution of my problem. In my code above if I change the line from
MQEnvironment.properties.Add(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES);
to
MQEnvironment.properties.Add(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES_MANAGED);
then it starts to register the transactions of the local DTC as well as it works fine in rolling back or commit a message on the queue.