I have an Azure Service Bus queue that is not receiving all messages sent to it. I have 2 test console apps (C#) one to send messages, the other to receive that show this. None of these missing messages are going into the DLQ and no exceptions are being thrown.
Below is the pertinent code from each console app.
Send:
var connectionString = "Endpoint=sb://xxxxxx.servicebus.windows.net/;SharedAccessKeyName=client;SharedAccessKey=xxxxxx=;EntityPath=xxxxxx";
var client = QueueClient.CreateFromConnectionString(connectionString);
int nCount = 0;
try
{
while (true)
{
string strBody = "Msg number:" + nCount;
var message = new BrokeredMessage(strBody);
client.Send(message);
Console.WriteLine(DateTime.Now.ToString() + " Received Body: " + strBody);
nCount++;
Thread.Sleep(10000);
}
}
catch (Exception exception)
{
Console.WriteLine(exception.ToString());
}
Receive:
var connectionString = "Endpoint=sb://xxxxxx.servicebus.windows.net/;SharedAccessKeyName=server;SharedAccessKey=xxxxxx;EntityPath=xxxxxx";
var Client = QueueClient.CreateFromConnectionString(connectionString);
Console.WriteLine("Started At: " + DateTime.Now.ToString());
ManualResetEvent CompletedEvent = new ManualResetEvent(false);
OnMessageOptions options = new OnMessageOptions();
options.AutoComplete = true;
options.MaxConcurrentCalls = 1;
Client.OnMessage((message) =>
{
try
{
Console.WriteLine(string.Format(DateTime.Now.ToString() + " Message Body Received: {0}", message.GetBody<string>()));
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}, options);
I don't think the Receive is the problem as if I close the Receive test app only some messages make it into the queue per the Azure portal.
The included image shows the apps running
The left console is send, the right receive. Ignore in the Send console where it accidentally says "Received Body", should have said "Sent".
Also, using Service Bus Explorer I also can not see the missing messages making it into the Queue. I tried creating a new queue and though more messages got into this new queue it still is not near 100%. More like 80%.
Related
I created a simple Azure Service bus (Queue) and a client that is sending message to service bus. Using below code to send message:
using Microsoft.Azure.ServiceBus;
using Microsoft.Extensions.Configuration;
public async Task SendMessageAsync<T>(T message, string queueName)
{
try
{
var queueClient = new QueueClient(_config.GetConnectionString("AzureServiceBus"), queueName);
string messageBody = JsonSerializer.Serialize(message);
var byteMessage = new Message(Encoding.UTF8.GetBytes(messageBody));
queueClient.SendAsync(byteMessage);
Console.WriteLine((message as Employee).FirstName);
}
catch (Exception ex)
{
var c = ex;
}
}
Sending message using:
using SenderApp;
Console.WriteLine("Hello, World!");
QueueService service = new QueueService();
for (int i = 0; i < 100; i++)
{
Employee e = new Employee();
e.FirstName = "1 " + i.ToString();
e.LastName = "2 " + i.ToString();
service.SendMessageAsync<Employee>(e, "employeequeue");
}
When I try to see active messages, There is nothing in the queue:
However I do see some traffic. But the number of message I sent (over 100) is not equal to number of incoming request show (62) at the bottom of the image. I am not sure what is happening to my messages? This defeats the purpose of the queue.
Please guide me why I am not seeing any messages. What is the best way to handle this ?
I am using following nuget packages:
<PackageReference Include="Microsoft.Azure.ServiceBus" Version="5.2.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="6.0.0" />
<PackageReference Include="Microsoft.Extensions.Hosting" Version="6.0.1" />
A message sent to an Azure Service Bus queue will be delivered to the queue unless operation is failing. In that case, an exception will be thrown. Check the following:
Exception handling doesn't swollow exceptions
Await asynchronous send operations to ensure messages are dispatched
Namespace/queue used for sending is what you use to receive
There are no competing consumers, actively receiving messages from the queue.
Validate TCP ports needed for AMQP are not blocked. If those ports are blocked, you could configure your client to use WebSockets.
So I still dont know what caused this issue. But I realized Microsoft.Azure.ServiceBus package was deprecated and later I started using Azure.Messaging.ServiceBus package to connect to service bus and things started to work.
I used following code to send message to queue:
string connectionString = "Endpoint=sb://test.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=f3f+qMYTyVwE18YNl5J6ygJFi30v6J/Smph5HZvyQyE=";
string queueName = "employeequeue";
// since ServiceBusClient implements IAsyncDisposable we create it with "await using"
await using var client = new ServiceBusClient(connectionString);
// create the sender
ServiceBusSender sender = client.CreateSender(queueName);
// create a message that we can send. UTF-8 encoding is used when providing a string.
ServiceBusMessage message = new ServiceBusMessage("Hello world! " + id);
// send the message
await sender.SendMessageAsync(message);
return "Sent";
Used following code to receive message:
string queueName = "employeequeue";
// since ServiceBusClient implements IAsyncDisposable we create it with "await using"
await using var client = new ServiceBusClient(connectionString);
// create a receiver that we can use to receive and settle the message
ServiceBusReceiver receiver = client.CreateReceiver(queueName);
// the received message is a different type as it contains some service set properties
ServiceBusReceivedMessage receivedMessage = await receiver.ReceiveMessageAsync();
string body = receivedMessage.Body.ToString();
// complete the message, thereby deleting it from the service
await receiver.CompleteMessageAsync(receivedMessage);
More info is available # https://github.com/Azure/azure-sdk-for-net/blob/Azure.Messaging.ServiceBus_7.7.0/sdk/servicebus/Azure.Messaging.ServiceBus/README.md
I have an IoT device that is connected to Azure event hub. I'm trying to make this device communicate with azure databricks and azure event hub is placed in between as a middleware. The problem is that after we are able to send messages via ".NET framework", it is never shown in messages received in "python" command line (we should do that as we work separately for each part)
I followed the guidelines .NET framework as sender and python as receiver, and this doesn't work.
I am seeing that there are spikes in the request and message graphs under event hub stream instances, but it just never shows in the receiver
==================================UPDATE==================================
Just deleted the eventhub and recreated and it seems work.
However, messages are received in the form of long strings something like this below:
Received: 662a2a44-4414-4cb5-a9e9-a08d12a417e0
Received: b68ef8f8-305f-4726-84e4-f35b76de30c5
Received: e94dfb73-972c-47b4-baef-1ab41b06be28
Received: 8eda384d-f79d-4cdf-9db3-fe5c2156553b
Received: d154283f-a8c2-4a4c-a7d5-e8d8129b568d
Received: 3e3e190e-f883-416c-a5be-b8cd8547d152
Received: e218c63b-85b3-4f4f-8f04-cb5ffc6d8921
Received: 0adec5ad-e351-4151-ba56-01093e0f383d
Received 8 messages in 0.05406975746154785 seconds
This happens when I read the messages in format below:
print("Received: {}".format(event_data.body_as_str(encoding='UTF-8')))
I just give it a try, and I can repro your issue. And here are something you need to check.
1.In you sender(in c#), you should make sure your message to send is correct. Like below:
static void SendingRandomMessages()
{
var eventHubClient = EventHubClient.CreateFromConnectionString(connectionString, eventHubName);
int i = 0;
while (true)
{
try
{
// make sure the message is correct.
var message = i+":"+Guid.NewGuid().ToString()+":"+DateTime.Now;
Console.WriteLine("{0} > Sending message: {1}", DateTime.Now, message);
var myeventdata = new EventData(Encoding.UTF8.GetBytes(message));
eventHubClient.Send(myeventdata);
i++;
//eventHubClient.Send(new EventData(Encoding.UTF8.GetBytes(message)));
}
catch (Exception exception)
{
Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine("{0} > Exception: {1}", DateTime.Now, exception.Message);
Console.ResetColor();
}
Thread.Sleep(2000);
}
}
2.There seems some delay for the receiver(in python), so I execute the python receiver about 3 times, and I can see the expected output. The screenshot as below:
Update 1022: as we discussed in the comment, there is a solution for fixing just receiving even / odd number event data.
In you sender(in c#), use the code below, which sends event data to partition 0:
static void SendingRandomMessages()
{
var eventHubClient = EventHubClient.CreateFromConnectionString(connectionString, eventHubName);
var myclient = eventHubClient.CreatePartitionedSender("0");
int i = 30;
while (true)
{
var message = i + ":" + Guid.NewGuid().ToString() + ":" + DateTime.Now;
Console.WriteLine("{0} > Sending message: {1}", DateTime.Now, message);
var myeventdata = new EventData(Encoding.UTF8.GetBytes(message));
myclient.Send(myeventdata);
i++;
Thread.Sleep(2000);
}
}
then in your receiver(in python), specify the partition to 0(use this PARTITION = "0"), then you can get all the event data.
I am creating a sample Azure Service Bus application. I created a namespace, topic and Subscription. I have written test messages to the topic and if I go to the subscription in the portal, I see that I have a new message every time I write a new one using the writer application.
But when I go to pull the message, nothing is retrieved. In troubleshooting, I changed the subscription name to an incorrect value and received an error. I changed it back and I get no output and none of the messages are removed when I look in Azure portal. I'm stuck... this seems easy, but it isn't working.
string connectionString = "Endpoint=sb://redacted for obvious reasons";
SubscriptionClient Client = SubscriptionClient.CreateFromConnectionString(connectionString, "NewOrders", "AllOrders");
// Configure the callback options.
OnMessageOptions options = new OnMessageOptions();
options.AutoComplete = false;
options.AutoRenewTimeout = TimeSpan.FromMinutes(1);
Client.OnMessage((message) =>
{
try
{
Console.WriteLine("Body: " + message.GetBody<string>());
message.Complete();
Console.ReadLine();
}
catch (Exception)
{
// Indicates a problem, unlock message in subscription.
message.Abandon();
}
}, options);
It seems that it is not the code issue. I create a demo to retrieve message from topic, it works correctly.
Before I try to retrieve message from topic, I send the message to the topic or make sure that there are messages for subscription. We could check that from the portal
Send message code demo.
private static void SendMessages()
{
topicClient = TopicClient.Create(TopicName);
List<BrokeredMessage> messageList = new List<BrokeredMessage>
{
CreateSampleMessage("1", "First message information"),
CreateSampleMessage("2", "Second message information"),
CreateSampleMessage("3", "Third message information")
};
Console.WriteLine("Sending messages to topic...");
foreach (BrokeredMessage message in messageList)
{
while (true)
{
try
{
topicClient.Send(message);
}
catch (MessagingException e)
{
if (!e.IsTransient)
{
Console.WriteLine(e.Message);
throw;
}
}
Console.WriteLine($"Message sent: Id = {message.MessageId}, Body = {message.GetBody<string>()}");
break;
}
}
topicClient.Close();
}
I also try your mentioned code, it also works correctly for me.
We also could get the demo project from the cloud Project from the template. We also could get more info about How to use Service Bus topics and subscriptions from the document.
I have created a service bus queue in Azure and it works well. And if the message is not getting delivered within default try (10 times), it is correctly moving the message to the dead letter queue.
Now, I would like to resubmit this message from the dead letter queue back to the queue where it originated and see if it works again. I have tried the same using service bus explorer. But it gets moved to the dead letter queue immediately.
Is it possible to do the same, and if so how?
You'd need to send a new message with the same payload. ASB by design doesn't support message resubmission.
We had a batch of around 60k messages, which need to be reprocessed from the dead letter queue. Peeking and send the messages back via Service Bus Explorer took around 6 minutes per 1k messages from my machine. I solved the issue by setting a forward rule for DLQ messages to another queue and from there auto forward it to the original queue. This solution took around 30 seconds for all 60k messages.
Try to remove dead letter reason
resubmittableMessage.Properties.Remove("DeadLetterReason");
resubmittableMessage.Properties.Remove("DeadLetterErrorDescription");
full code
using Microsoft.ServiceBus.Messaging;
using System.Transactions;
namespace ResubmitDeadQueue
{
class Program
{
static void Main(string[] args)
{
var connectionString = "";
var queueName = "";
var queue = QueueClient.CreateFromConnectionString(connectionString, QueueClient.FormatDeadLetterPath(queueName), ReceiveMode.PeekLock);
BrokeredMessage originalMessage
;
var client = QueueClient.CreateFromConnectionString(connectionString, queueName);
do
{
originalMessage = queue.Receive();
if (originalMessage != null)
{
using (var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
{
// Create new message
var resubmittableMessage = originalMessage.Clone();
// Remove dead letter reason and description
resubmittableMessage.Properties.Remove("DeadLetterReason");
resubmittableMessage.Properties.Remove("DeadLetterErrorDescription");
// Resend cloned DLQ message and complete original DLQ message
client.Send(resubmittableMessage);
originalMessage.Complete();
// Complete transaction
scope.Complete();
}
}
} while (originalMessage != null);
}
}
}
Thanks to some other responses here!
We regularly need to resubmit messages. The answer from #Baglay-Vyacheslav helped a lot. I've pasted some updated C# code that works with the latest Azure.Messaging.ServiceBus Nuget Package.
Makes it much quicker/easier to process DLQ on both queues/topics/subscribers.
using Azure.Messaging.ServiceBus;
using System.Collections.Generic;
using System.Threading.Tasks;
using NLog;
namespace ServiceBus.Tools
{
class TransferDeadLetterMessages
{
// https://github.com/Azure/azure-sdk-for-net/blob/Azure.Messaging.ServiceBus_7.2.1/sdk/servicebus/Azure.Messaging.ServiceBus/README.md
private static Logger logger = LogManager.GetCurrentClassLogger();
private static ServiceBusClient client;
private static ServiceBusSender sender;
public static async Task ProcessTopicAsync(string connectionString, string topicName, string subscriberName, int fetchCount = 10)
{
try
{
client = new ServiceBusClient(connectionString);
sender = client.CreateSender(topicName);
ServiceBusReceiver dlqReceiver = client.CreateReceiver(topicName, subscriberName, new ServiceBusReceiverOptions
{
SubQueue = SubQueue.DeadLetter,
ReceiveMode = ServiceBusReceiveMode.PeekLock
});
await ProcessDeadLetterMessagesAsync($"topic: {topicName} -> subscriber: {subscriberName}", fetchCount, sender, dlqReceiver);
}
catch (Azure.Messaging.ServiceBus.ServiceBusException ex)
{
if (ex.Reason == Azure.Messaging.ServiceBus.ServiceBusFailureReason.MessagingEntityNotFound)
{
logger.Error(ex, $"Topic:Subscriber '{topicName}:{subscriberName}' not found. Check that the name provided is correct.");
}
else
{
throw;
}
}
finally
{
await sender.CloseAsync();
await client.DisposeAsync();
}
}
public static async Task ProcessQueueAsync(string connectionString, string queueName, int fetchCount = 10)
{
try
{
client = new ServiceBusClient(connectionString);
sender = client.CreateSender(queueName);
ServiceBusReceiver dlqReceiver = client.CreateReceiver(queueName, new ServiceBusReceiverOptions
{
SubQueue = SubQueue.DeadLetter,
ReceiveMode = ServiceBusReceiveMode.PeekLock
});
await ProcessDeadLetterMessagesAsync($"queue: {queueName}", fetchCount, sender, dlqReceiver);
}
catch (Azure.Messaging.ServiceBus.ServiceBusException ex)
{
if (ex.Reason == Azure.Messaging.ServiceBus.ServiceBusFailureReason.MessagingEntityNotFound)
{
logger.Error(ex, $"Queue '{queueName}' not found. Check that the name provided is correct.");
}
else
{
throw;
}
}
finally
{
await sender.CloseAsync();
await client.DisposeAsync();
}
}
private static async Task ProcessDeadLetterMessagesAsync(string source, int fetchCount, ServiceBusSender sender, ServiceBusReceiver dlqReceiver)
{
var wait = new System.TimeSpan(0, 0, 10);
logger.Info($"fetching messages ({wait.TotalSeconds} seconds retrieval timeout)");
logger.Info(source);
IReadOnlyList<ServiceBusReceivedMessage> dlqMessages = await dlqReceiver.ReceiveMessagesAsync(fetchCount, wait);
logger.Info($"dl-count: {dlqMessages.Count}");
int i = 1;
foreach (var dlqMessage in dlqMessages)
{
logger.Info($"start processing message {i}");
logger.Info($"dl-message-dead-letter-message-id: {dlqMessage.MessageId}");
logger.Info($"dl-message-dead-letter-reason: {dlqMessage.DeadLetterReason}");
logger.Info($"dl-message-dead-letter-error-description: {dlqMessage.DeadLetterErrorDescription}");
ServiceBusMessage resubmittableMessage = new ServiceBusMessage(dlqMessage);
await sender.SendMessageAsync(resubmittableMessage);
await dlqReceiver.CompleteMessageAsync(dlqMessage);
logger.Info($"finished processing message {i}");
logger.Info("--------------------------------------------------------------------------------------");
i++;
}
await dlqReceiver.CloseAsync();
logger.Info($"finished");
}
}
}
It may be "duplicate message detection" as Peter Berggreen indicated or more likely if you are directly moving the BrokeredMessage from the dead letter queue to the live queue then the DeliveryCount would still be at maximum and it would return to the dead letter queue.
Pull the BrokeredMessage off the dead letter queue, get the content using GetBody(), create in new BrokeredMessage with that data and send it to the queue. You can do this in a safe manor, by using peek to get the message content off the dead letter queue and then send the new message to the live queue before removing the message from the dead letter queue. That way you won't lose any crucial data if for some reason it fails to write to the live queue.
With a new BrokeredMessage you should not have an issue with "duplicate message detection" and the DeliveryCount will be reset to zero.
The Service Bus Explorer tool always creates a clone of the original message when you repair and resubmit a message from the deadletter queue. It could not be any different as by default Service Bus messaging does not provide any message repair and resubmit mechanism. I suggest you to investigate why your message gets ends up in the deadletter queue as well as its clone when you resubmit it. Hope this helps!
It sounds like it could be related to ASB's "duplicate message detection" functionality.
When you resubmit a message in ServiceBus Explorer it will clone the message and thereby the new message will have the same Id as the original message in the deadletter queue.
If you have enabled "Requires Duplicate Detection" on the queue/topic and you try to resubmit the message within the "Duplicate Detection History Time Window", then the message will immediately be moved to the deadletter queue again.
If you want to use Service Bus Explorer to resubmit deadletter messages, then I think that you will have to disable "Requires Duplicate Detection" on the queue/topic.
I'm using a Service Bus queue with Sessions enabled and I'm sending 5 messages with the same SessionId. My receiving code uses AcceptMessageSessionAsync to get a session lock so that it will receive all the messages for that session. It then uses session.ReceiveBatchAsync to try and get all the messages for the session. However, it only seems to get the first message, then when another attempt is made, it gets all the others. You should be able to see that there is a gap of almost a minute between the two batches even though all these messages were sent at once:
Session started:AE8DC914-8693-4110-8BAE-244E42A302D5
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_1_08:03:03.36523
Session started:AE8DC914-8693-4110-8BAE-244E42A302D5
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_2_08:03:04.22964
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_3_08:03:04.29515
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_4_08:03:04.33959
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_5_08:03:04.39587
My code to process these is a function in a WebJob:
[NoAutomaticTrigger]
public static async Task MessageHandlingLoop(TextWriter log, CancellationToken cancellationToken)
{
var connectionString = ConfigurationManager.ConnectionStrings["ServiceBusListen"].ConnectionString;
var client = QueueClient.CreateFromConnectionString(connectionString, "myqueue");
while (!cancellationToken.IsCancellationRequested)
{
MessageSession session = null;
try
{
session = await client.AcceptMessageSessionAsync(TimeSpan.FromMinutes(1));
log.WriteLine("Session started:" + session.SessionId);
foreach (var msg in await session.ReceiveBatchAsync(100, TimeSpan.FromSeconds(5)))
{
log.WriteLine("Message received:" + msg.MessageId);
msg.Complete();
}
}
catch (TimeoutException)
{
log.WriteLine("Timeout occurred");
await Task.Delay(5000, cancellationToken);
}
catch (Exception ex)
{
log.WriteLine("Error:" + ex);
}
}
}
This is called from my WebJob Main using:
JobHost host = new JobHost();
host.Start();
var task = host.CallAsync(typeof(Functions).GetMethod("MessageHandlingLoop"));
task.Wait();
host.Stop();
Why don't I get all my messages in the first call of ReceiveBatchAsync?
This was answered in the MSDN forum by Hillary Caituiro Monge: https://social.msdn.microsoft.com/Forums/azure/en-US/9a84f319-7bc6-4ff8-b142-4fc1d5f1e2fa/service-bus-session-receivebatchasync-only-receiving-1-message?forum=servbus
Service Bus does not guarantee you will receive the message count you
specify in receive batch even if your queue has them or more. Having
say that, you can change your code to try to get the 100 messages in
the first call, buy remember that your application should not assume
that as a guaranteed behavior.
Below this line of code varclient =
QueueClient.CreateFromConnectionString(connectionString, "myqueue");
add client.PrefetchCount = 100;
The reason that you are getting only 1 message at all times in the
first call is due to that when you accept a session it may be also
getting 1 prefetched message with it. Then when you do receive batch,
the SB client will give you that 1 message.
Unfortunately I found that setting the PrefetchCount didn't have an affect, but the reason given for only receiving one message seemed likely so I accepted it as the answer.