For purposes of our integration checking, I want to count the number of messages on an Azure queue. The method looks like this:
internal void VerifyMessagesOnQueue(string queueNameKey, int expectedNumberOfMessages)
{
var azureStorageConnectionKey = ConfigurationManager.AppSettings["AzureStorageConnectionKey"];
var storageAccount = CloudStorageAccount.Parse(azureStorageConnectionKey);
var queueClient = storageAccount.CreateCloudQueueClient();
var queue = queueClient.GetQueueReference(ConfigurationManager.AppSettings[queueNameKey]);
var messages = queue.PeekMessages(int.MaxValue);
messages.Count().Should().Be(expectedNumberOfMessages);
}
Right now I'm using var messages = queue.PeekMessages(int.MaxValue); to try to get all the messages on a queue. It returns an HTML repsonse 400. I have tried var messages = queue.PeekMessages(expectedNumberOfMessages);, but when expectedNumberOfMessages is 0, I also get an HTML response 400.
How can I reliably check the number of messages on an Azure queue without disrupting it (this is why I was using .PeekMessage)?
I want to count the number of messages on an Azure queue
I suggest you could try the following code to achieve your goal. I have created a console project to test.
StorageConnectionString in App.config:
<appSettings>
<add key="StorageConnectionString" value="DefaultEndpointsProtocol=https;AccountName=×××;AccountKey=×××" />
</appSettings>
Code in Program.cs:
static void Main(string[] args)
{
string Queue_Name = "myqueue";
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
Microsoft.Azure.CloudConfigurationManager.GetSetting("StorageConnectionString"));
CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();
CloudQueue queue = queueClient.GetQueueReference(Queue_Name);
queue.FetchAttributes();
var count=queue.ApproximateMessageCount;
Console.WriteLine("message number in queue:"+count);
}
The Result about queue count:
Janley Zhang's answer is correct. To get approximate messages count in a queue, you will need to fetch queue's attribute. Adding a new answer to clarify the following:
Right now I'm using var messages = queue.PeekMessages(int.MaxValue);
to try to get all the messages on a queue. It returns an HTML repsonse
400. I have tried var messages = queue.PeekMessages(expectedNumberOfMessages);, but when
expectedNumberOfMessages is 0, I also get an HTML response 400.
Essentially PeekMessages is used to retrieve the messages from the top of the queue without altering the retrieved messages visibility.
Maximum messages that can be fetched from a queue in a single request is 32 and the minimum is 1. Please check this link (URI Parameters section ) for more details.
In both of scenarios, you're specifying a count that is out of allowed range (1 - 32) and this is why you're getting the 400 error back from the queue.
Related
It really annoys me that we're unable to move messages from a Dead Letter Queue over to the Original Queue for processing when using Azure Servicebus. So, I figured out that I will try to implement this feature myself. We are using Masstransit to publish events. The queuename in ASB will be an events full assembly name.
I've created an REST Endpoint in my application to move messages from the DLQ to the original queue for reprocessing. This is where I'm stuck at the moment.
To get all messages in a DLQ, the user gives me the queuename, and I will format it to contain the DeadLetterQueue. Like this:
myproject.events.usercreatedevent -> myproject.events.usercreatedevent/$DeadLetterQueue
I get all the messages from this queue by using classes from the Nuget package Microsoft.Azure.Servicebus
public async Task RequeueMessagesAsync(string queueName)
{
var msg = new MessageReceiver(BuildConnectionString(), queueName);
var messages = await msg.PeekAsync(50);
foreach (var message in messages)
{
var content = Encoding.UTF8.GetString(message.Body);
var jsonObject = JsonConvert.DeserializeObject<JObject>(content);
var destinationAddress = jsonObject["destinationAddress"].ToString();
var messageContent = jsonObject["message"].ToString();
var messageType = destinationAddress.Split("/").Last();
await _bus.SendAsync(jsonObject, messageType);
}
}
The when calling _bus.SendAsync(object, address) the message ends up in a _skipped queue. I think the reason for this is that the messageHeaders is set to JObject, and not the actual message type. I cannot use reflection to recreated the event either, as we have a lot of microservices and source code of the event it not necessarily available. The code behind the _bus.SendAsync(object, address) looks like this:
public async Task SendAsync(object message, string queueName, CancellationToken cancellationToken = default)
{
ISendEndpoint sender = await GetSenderAsync(queueName);
sender.ConnectSendObserver(new ErrorQueueConfiguration(_addressProvider.GetAddress("error")));
await sender.Send(message, cancellationToken);
}
Can I trick Masstransit to forward this "unknown" type to my Consumer by changing the MessageHeaders somehow? Have anyone successfully moved messages from a DLQ to its original queue?
I have created the code to get the Queue Client data using QueueClient.Receive() with Broken Message
BrokeredMessage deadmessage = client.Receive();
byte[] dataRaw = deadmessage.GetBody<byte[]>();
Due to some corrupted data, I got the exception on second line, while get the body of the broken message. So i was try to get the body of the message on catch block with SteamReader.
Stream stream = deadmessage.GetBody<Stream>();
StreamReader reader = new StreamReader(stream);
I experienced with below exception, Could anyone help me with appropriate fixes?
Exception details :
The message body cannot be read multiple times. To reuse it store the value after reading.
To take multiple attempts to read message body you need to read it as a stream first
serviceBusClient.GetBody<Stream>()
Then you can try to interpret it by different ways. For example it can be serialized directly by following way:
var brokeredMessage = new BrokeredMessage(message);
serviceBusClient.Send(brokeredMessage);
but it's better to serialize it to json first.
var brokeredMessage = new BrokeredMessage(JsonConvert.SerializeObject(message));
serviceBusClient.Send(brokeredMessage);
it's more safe in my view because json serialization ignores namespaces of message type, so you will not break your process when you move class of message to another namespace.
Suppose you are starting to send and read messages serialized in json but some old messages can be still binary serialized. In this case you can use the following logic:
public static T DeserializeMessage<T>(BrokeredMessage brokeredMessage)
{
using (var stream = brokeredMessage.GetBody<Stream>())
using (var streamReader = new StreamReader(stream))
{
string bodyText = streamReader.ReadToEnd();
try
{
return JsonConvert.DeserializeObject<T>(bodyText);
}
catch (JsonReaderException)
{
stream.Position = 0;
var reader = XmlDictionaryReader.CreateBinaryReader(stream, XmlDictionaryReaderQuotas.Max);
var serializer = new DataContractSerializer(typeof(T));
var msgBody = (T)serializer.ReadObject(reader);
return msgBody;
}
}
}
If you need to try to deserialize the message as another type, catch System.Runtime.Serialization.SerializationException on serializer.ReadObject(reader).
As Sean Feldman mentioned that if message is corrupted, then it will be handled by dead-letter queue.
Service Bus queues and topic subscriptions provide a secondary sub-queue, called a dead-letter queue (DLQ). The dead-letter queue does not need to be explicitly created and cannot be deleted or otherwise managed independent of the main entity.
The purpose of the dead-letter queue is to hold messages that cannot be delivered to any receiver, or simply messages that could not be processed.
If you need to know how to create and user Service Bus Queue we can refer to get start with Service Bus queues
To reuse it store the value after reading.
If it be can be read correctly then we can store it messageid and vaule for reuse.
The DLQ is mostly similar to any other queue.
If it is corrupted data, we can get it from the dead-letter queue as mormal queue.
string connectionString = CloudConfigurationManager.GetSetting('Microsoft.ServiceBus.ConnectionString');
QueueClient Client = QueueClient.CreateFromConnectionString(connectionString, deletLetterQueueName);
var message = Client.Receive(TimeSpan.FromSeconds(3));
if (message != null)
{
var ret = message.GetBody<stream>();
message.Complete();
}
I found another reason of the exception. The thing is that when I debugged different problem I got this exception. After some experiments I realized that Visual Studio reads the message behind the scene to show it for example in Watch panel, and when my code tried to get the message it was already read by Visual Studio.
To avoid this it needs to wrap the message to a property with backing field, which will store the value. And then I realized that the exception message abstracly says to make this.
So you should consider that it can be read behind the scene
Is there a way to get the message ID after insert it in a queue Azure ?
CloudStorageAccount storageAccount =
CloudStorageAccount.parse(storageConnectionString);
CloudQueueClient queueClient = storageAccount.createCloudQueueClient();
CloudQueue queue = queueClient.getQueueReference("myqueue");
queue.createIfNotExist();
CloudQueueMessage message = new CloudQueueMessage("Hello, World");
queue.addMessage(message);
// Get message ID here ?
I realize it has been 5 years since this was originally asked; however, it is now possible to achieve this.
CloudQueueMessage message = new CloudQueueMessage("Hello, World");
queue.AddMessage(message);
// here's how you get the id
string id = message.Id;
Only way you could get the message id is by getting the message. So you would have to fetch messages from the queue using GetMessage or GetMessages method. However there's no guarantee that you will get the message you just created as GetMessages can only return up to 32 visible messages from the top of the queue.
Since queue lies on the principle "First In First Out" or FIFO, that is why you can't just get the particular message anytime you want but you have to use the GetMessage and iterate on it.
Is there anyway to get a timestamp as to when a message is placed on my Azure queue? What is the best way?
For example, a partner sends a message to my queue and I want to know the time the partner placed a specific message in the queue.
Thanks
If you're using the .NET API, the property InsertionTime in the CloudQueueMessage you get when fetching a message or peeking the queue will contain;
The time that that message was added to the queue.
For example, in Java code the following would retrieve a message (then get the insertion time) and then peek at the next message (and get its insertion time.) This also shows how access to the queue can be controlled by using a revocable stored access policy string.
public class Dequeue {
public static void main(String[] args) throws InvalidKeyException, URISyntaxException, StorageException
{
String sas = "sv=2012-02-12&sig=XUHWUy2ayrcEjNvZUhcgdPbgKZflwSXxtr0BH87XRII%3D&si=heath";
StorageCredentialsSharedAccessSignature credentials = new StorageCredentialsSharedAccessSignature(sas);
URI baseuri = new URI("http://grassy.queue.core.windows.net");
CloudQueueClient queueClient = new CloudQueueClient(baseuri,credentials);
CloudQueue queue = queueClient.getQueueReference("queue1");
CloudQueueMessage retrievedMessage = queue.retrieveMessage();
System.out.println(retrievedMessage.getMessageContentAsString());
System.out.println(retrievedMessage.getInsertionTime());
queue.deleteMessage(retrievedMessage);
CloudQueueMessage peekedMessage = queue.peekMessage();
System.out.println(peekedMessage.getMessageContentAsString());
System.out.println(peekedMessage.getInsertionTime());
} }
I know there is a way to determine the number of messages (or approximate number) in the Azure Queue (Store Account); however is there a way to query for the number of pending messages on an Azure Service Bus queue?
var nsmgr = Microsoft.ServiceBus.NamespaceManager.CreateFromConnectionString(connectionString);
long count = nsmgr.GetQueue(queueName).MessageCount;
It is called MessagesCountDetails.ActiveMessageCount.
It returns the number of the Active Messages in the Queue. You probably have some Dead letter messages:
var msg = Microsoft.ServiceBus.NamespaceManager.CreateFromConnectionString(Settings.Default.ConnectionString);
numofmessages.Text = msg.GetQueue(QueueName).MessageCountDetails.ActiveMessageCount.ToString();
Correct answer as of 2020+
Use of new packages as follows:
<PackageReference Include="Azure.Messaging.ServiceBus" Version="x.x.x" />
also two namespaces in the same package
using Azure.Messaging.ServiceBus;
using Azure.Messaging.ServiceBus.Administration;
and then you can use the new class ServiceBusAdministrationClient
var administrationClient = new ServiceBusAdministrationClient("connectionString");
var props = await administrationClient.GetQueueRuntimePropertiesAsync("queue");
var messageCount = props.Value.ActiveMessageCount;
have you looked at the Queue Description API? There's a property called MessageCount.
Here's the .NET SDK reference documentation page as well.
Based off what Joseph had as an answer I came up with, but for Topics and Subscriptions.
public async Task<long> GetCounterMessages()
{
var client = new ManagementClient(ServiceBusConnectionString);
var subs = await client.GetSubscriptionRuntimeInfoAsync(TopicName, SubscriptionName);
var countForThisSubscription = subs.MessageCount; //// (Comes back as a Long.)
return countForThisSubscription;
}
I ran into this same problem trying to get the count from the dead letter queue. It looks like the deadletterqueue doesn't allow you to get a count directly, you get it from the MessageCountDetails of the normal Queue.
string connectionString = ConfigurationManager.AppSettings["Microsoft.ServiceBus.Connstr"].ToString();
NamespaceManager nsmgr = Microsoft.ServiceBus.NamespaceManager.CreateFromConnectionString(connectionString);
return nsmgr.GetQueue(QueueName).MessageCountDetails.DeadLetterMessageCount;
Here's a PowerShell example to continually eyeball the queue length as used in Azure Portal Cloud Shell
cd "Azure:\<MySubscription>\"
while (1) {(Get-AzureRmServiceBusQueue -ResourceGroup <myRG> -NamespaceName <myNS> -QueueName <myQueueName>).CountDetails | Select -expand ActiveMessageCount}
I've spent good 2 hours digging through docs to get that and for people using .net core and Microsoft.Azure.ServiceBus nuget package, code looks like that:
var managementClient = new ManagementClient("queue connection string"));
var runtimeInfo = await managementClient.GetQueueRuntimeInfoAsync("queueName");
var messagesInQueueCount = runtimeInfo.MessageCountDetails.ActiveMessageCount;
Aparently you get the information about all Counts(including deadletter, active, etc.) from QueueRuntimeInfo object instead of old QueueDescription object.
As per the recommendation by Microsoft, it is recommended to use Microsoft.Azure.ServiceBus in which you can easily fetch the message count by
var managementClient = new ManagementClient("connection string for queue");
var queue = await managementClient.GetQueueRuntimeInfoAsync("queue name");
var messages = queue.MessageCount;
Also..you can check the pending messages on Azure Management Portal...on the dashboard for service bus queue...under quick glance...you can see the queue length...this is the number of current/pending messages in length at the time of dashboard page load.