I have configured function apps in Azure to persist log entries from the function app insights logs to an event hub which then also persists these log entries to ELK. I recently noticed that some log entries were missing and I followed this example to create a console app that would replay the log events. The console app I created based on the examples works and no exceptions are thrown, however I never find the log entries when querying ELK, they don't appear to be persisted there the way I would expect. Below is some example code of what my console app is doing.
private const string connectionString = "CONNECTION_STRING";
private const string eventHubName = "EVENT_HUB_NAME";
static EventHubBufferedProducerClient producerClient;
static async Task Main()
{
string eventDataString = "{"message": "Here's a log entry"}";*/
var eventData = JsonConvert.DeserializeObject<EventData>(eventDataString);
eventData.MessageId = "MessageIDLog1";
eventData.ContentType = "application/json";
eventData.EventBody = new BinaryData(Encoding.UTF8.GetBytes("{message": "Here's a log entry}"));
producerClient = new EventHubBufferedProducerClient(connectionString, eventHubName);
producerClient.SendEventBatchFailedAsync += args =>
{
Console.WriteLine($"Publishing failed for { args.EventBatch.Count } events. Error: { args.Exception.Message }");
return Task.CompletedTask;
};
producerClient.SendEventBatchSucceededAsync += args =>
{
Console.WriteLine($"{ args.EventBatch.Count } events were published to partition: { args.PartitionId }.");
return Task.CompletedTask;
};
try
{
await producerClient.EnqueueEventAsync(eventData);
}
finally
{
await producerClient.DisposeAsync();
}
}
I'm hoping someone out there has some experience with this and could point me in the right direction as I've been on this for a while and I'm not sure what I'm doing incorrectly.
How to read the message from the service bus dead-letter queue? I'm able to read message-id and a sequence number of the message, but I need the actual message. Can someone help me with this? Is it possible to read the actual message?
Before reading message form deadletter queue, you should check what was the reason of fail? If some service was not available then Create WebJob and try below code and process messages.
public void GetDeadLetterMessagesAsync(string connectionString, string queueName)
{
var queueClient = QueueClient.CreateFromConnectionString(connectionString, QueueClient.FormatDeadLetterPath(queueName));
while (true)
{
BrokeredMessage bmessgage = queueClient.Receive();
if (bmessgage != null)
{
string msg = new StreamReader(bmessgage.GetBody<Stream>(), Encoding.UTF8).ReadToEnd();
//Custom business logic to prcess your message
bmessgage.Complete();
}
else
{
break;
}
}
}
If message having an issue then you should read and display message on UI so back office team can correct message otherwise it will fail again.
I have an IoT device that is connected to Azure event hub. I'm trying to make this device communicate with azure databricks and azure event hub is placed in between as a middleware. The problem is that after we are able to send messages via ".NET framework", it is never shown in messages received in "python" command line (we should do that as we work separately for each part)
I followed the guidelines .NET framework as sender and python as receiver, and this doesn't work.
I am seeing that there are spikes in the request and message graphs under event hub stream instances, but it just never shows in the receiver
==================================UPDATE==================================
Just deleted the eventhub and recreated and it seems work.
However, messages are received in the form of long strings something like this below:
Received: 662a2a44-4414-4cb5-a9e9-a08d12a417e0
Received: b68ef8f8-305f-4726-84e4-f35b76de30c5
Received: e94dfb73-972c-47b4-baef-1ab41b06be28
Received: 8eda384d-f79d-4cdf-9db3-fe5c2156553b
Received: d154283f-a8c2-4a4c-a7d5-e8d8129b568d
Received: 3e3e190e-f883-416c-a5be-b8cd8547d152
Received: e218c63b-85b3-4f4f-8f04-cb5ffc6d8921
Received: 0adec5ad-e351-4151-ba56-01093e0f383d
Received 8 messages in 0.05406975746154785 seconds
This happens when I read the messages in format below:
print("Received: {}".format(event_data.body_as_str(encoding='UTF-8')))
I just give it a try, and I can repro your issue. And here are something you need to check.
1.In you sender(in c#), you should make sure your message to send is correct. Like below:
static void SendingRandomMessages()
{
var eventHubClient = EventHubClient.CreateFromConnectionString(connectionString, eventHubName);
int i = 0;
while (true)
{
try
{
// make sure the message is correct.
var message = i+":"+Guid.NewGuid().ToString()+":"+DateTime.Now;
Console.WriteLine("{0} > Sending message: {1}", DateTime.Now, message);
var myeventdata = new EventData(Encoding.UTF8.GetBytes(message));
eventHubClient.Send(myeventdata);
i++;
//eventHubClient.Send(new EventData(Encoding.UTF8.GetBytes(message)));
}
catch (Exception exception)
{
Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine("{0} > Exception: {1}", DateTime.Now, exception.Message);
Console.ResetColor();
}
Thread.Sleep(2000);
}
}
2.There seems some delay for the receiver(in python), so I execute the python receiver about 3 times, and I can see the expected output. The screenshot as below:
Update 1022: as we discussed in the comment, there is a solution for fixing just receiving even / odd number event data.
In you sender(in c#), use the code below, which sends event data to partition 0:
static void SendingRandomMessages()
{
var eventHubClient = EventHubClient.CreateFromConnectionString(connectionString, eventHubName);
var myclient = eventHubClient.CreatePartitionedSender("0");
int i = 30;
while (true)
{
var message = i + ":" + Guid.NewGuid().ToString() + ":" + DateTime.Now;
Console.WriteLine("{0} > Sending message: {1}", DateTime.Now, message);
var myeventdata = new EventData(Encoding.UTF8.GetBytes(message));
myclient.Send(myeventdata);
i++;
Thread.Sleep(2000);
}
}
then in your receiver(in python), specify the partition to 0(use this PARTITION = "0"), then you can get all the event data.
I have created a service bus queue in Azure and it works well. And if the message is not getting delivered within default try (10 times), it is correctly moving the message to the dead letter queue.
Now, I would like to resubmit this message from the dead letter queue back to the queue where it originated and see if it works again. I have tried the same using service bus explorer. But it gets moved to the dead letter queue immediately.
Is it possible to do the same, and if so how?
You'd need to send a new message with the same payload. ASB by design doesn't support message resubmission.
We had a batch of around 60k messages, which need to be reprocessed from the dead letter queue. Peeking and send the messages back via Service Bus Explorer took around 6 minutes per 1k messages from my machine. I solved the issue by setting a forward rule for DLQ messages to another queue and from there auto forward it to the original queue. This solution took around 30 seconds for all 60k messages.
Try to remove dead letter reason
resubmittableMessage.Properties.Remove("DeadLetterReason");
resubmittableMessage.Properties.Remove("DeadLetterErrorDescription");
full code
using Microsoft.ServiceBus.Messaging;
using System.Transactions;
namespace ResubmitDeadQueue
{
class Program
{
static void Main(string[] args)
{
var connectionString = "";
var queueName = "";
var queue = QueueClient.CreateFromConnectionString(connectionString, QueueClient.FormatDeadLetterPath(queueName), ReceiveMode.PeekLock);
BrokeredMessage originalMessage
;
var client = QueueClient.CreateFromConnectionString(connectionString, queueName);
do
{
originalMessage = queue.Receive();
if (originalMessage != null)
{
using (var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
{
// Create new message
var resubmittableMessage = originalMessage.Clone();
// Remove dead letter reason and description
resubmittableMessage.Properties.Remove("DeadLetterReason");
resubmittableMessage.Properties.Remove("DeadLetterErrorDescription");
// Resend cloned DLQ message and complete original DLQ message
client.Send(resubmittableMessage);
originalMessage.Complete();
// Complete transaction
scope.Complete();
}
}
} while (originalMessage != null);
}
}
}
Thanks to some other responses here!
We regularly need to resubmit messages. The answer from #Baglay-Vyacheslav helped a lot. I've pasted some updated C# code that works with the latest Azure.Messaging.ServiceBus Nuget Package.
Makes it much quicker/easier to process DLQ on both queues/topics/subscribers.
using Azure.Messaging.ServiceBus;
using System.Collections.Generic;
using System.Threading.Tasks;
using NLog;
namespace ServiceBus.Tools
{
class TransferDeadLetterMessages
{
// https://github.com/Azure/azure-sdk-for-net/blob/Azure.Messaging.ServiceBus_7.2.1/sdk/servicebus/Azure.Messaging.ServiceBus/README.md
private static Logger logger = LogManager.GetCurrentClassLogger();
private static ServiceBusClient client;
private static ServiceBusSender sender;
public static async Task ProcessTopicAsync(string connectionString, string topicName, string subscriberName, int fetchCount = 10)
{
try
{
client = new ServiceBusClient(connectionString);
sender = client.CreateSender(topicName);
ServiceBusReceiver dlqReceiver = client.CreateReceiver(topicName, subscriberName, new ServiceBusReceiverOptions
{
SubQueue = SubQueue.DeadLetter,
ReceiveMode = ServiceBusReceiveMode.PeekLock
});
await ProcessDeadLetterMessagesAsync($"topic: {topicName} -> subscriber: {subscriberName}", fetchCount, sender, dlqReceiver);
}
catch (Azure.Messaging.ServiceBus.ServiceBusException ex)
{
if (ex.Reason == Azure.Messaging.ServiceBus.ServiceBusFailureReason.MessagingEntityNotFound)
{
logger.Error(ex, $"Topic:Subscriber '{topicName}:{subscriberName}' not found. Check that the name provided is correct.");
}
else
{
throw;
}
}
finally
{
await sender.CloseAsync();
await client.DisposeAsync();
}
}
public static async Task ProcessQueueAsync(string connectionString, string queueName, int fetchCount = 10)
{
try
{
client = new ServiceBusClient(connectionString);
sender = client.CreateSender(queueName);
ServiceBusReceiver dlqReceiver = client.CreateReceiver(queueName, new ServiceBusReceiverOptions
{
SubQueue = SubQueue.DeadLetter,
ReceiveMode = ServiceBusReceiveMode.PeekLock
});
await ProcessDeadLetterMessagesAsync($"queue: {queueName}", fetchCount, sender, dlqReceiver);
}
catch (Azure.Messaging.ServiceBus.ServiceBusException ex)
{
if (ex.Reason == Azure.Messaging.ServiceBus.ServiceBusFailureReason.MessagingEntityNotFound)
{
logger.Error(ex, $"Queue '{queueName}' not found. Check that the name provided is correct.");
}
else
{
throw;
}
}
finally
{
await sender.CloseAsync();
await client.DisposeAsync();
}
}
private static async Task ProcessDeadLetterMessagesAsync(string source, int fetchCount, ServiceBusSender sender, ServiceBusReceiver dlqReceiver)
{
var wait = new System.TimeSpan(0, 0, 10);
logger.Info($"fetching messages ({wait.TotalSeconds} seconds retrieval timeout)");
logger.Info(source);
IReadOnlyList<ServiceBusReceivedMessage> dlqMessages = await dlqReceiver.ReceiveMessagesAsync(fetchCount, wait);
logger.Info($"dl-count: {dlqMessages.Count}");
int i = 1;
foreach (var dlqMessage in dlqMessages)
{
logger.Info($"start processing message {i}");
logger.Info($"dl-message-dead-letter-message-id: {dlqMessage.MessageId}");
logger.Info($"dl-message-dead-letter-reason: {dlqMessage.DeadLetterReason}");
logger.Info($"dl-message-dead-letter-error-description: {dlqMessage.DeadLetterErrorDescription}");
ServiceBusMessage resubmittableMessage = new ServiceBusMessage(dlqMessage);
await sender.SendMessageAsync(resubmittableMessage);
await dlqReceiver.CompleteMessageAsync(dlqMessage);
logger.Info($"finished processing message {i}");
logger.Info("--------------------------------------------------------------------------------------");
i++;
}
await dlqReceiver.CloseAsync();
logger.Info($"finished");
}
}
}
It may be "duplicate message detection" as Peter Berggreen indicated or more likely if you are directly moving the BrokeredMessage from the dead letter queue to the live queue then the DeliveryCount would still be at maximum and it would return to the dead letter queue.
Pull the BrokeredMessage off the dead letter queue, get the content using GetBody(), create in new BrokeredMessage with that data and send it to the queue. You can do this in a safe manor, by using peek to get the message content off the dead letter queue and then send the new message to the live queue before removing the message from the dead letter queue. That way you won't lose any crucial data if for some reason it fails to write to the live queue.
With a new BrokeredMessage you should not have an issue with "duplicate message detection" and the DeliveryCount will be reset to zero.
The Service Bus Explorer tool always creates a clone of the original message when you repair and resubmit a message from the deadletter queue. It could not be any different as by default Service Bus messaging does not provide any message repair and resubmit mechanism. I suggest you to investigate why your message gets ends up in the deadletter queue as well as its clone when you resubmit it. Hope this helps!
It sounds like it could be related to ASB's "duplicate message detection" functionality.
When you resubmit a message in ServiceBus Explorer it will clone the message and thereby the new message will have the same Id as the original message in the deadletter queue.
If you have enabled "Requires Duplicate Detection" on the queue/topic and you try to resubmit the message within the "Duplicate Detection History Time Window", then the message will immediately be moved to the deadletter queue again.
If you want to use Service Bus Explorer to resubmit deadletter messages, then I think that you will have to disable "Requires Duplicate Detection" on the queue/topic.
I have an Azure Service Bus queue that is not receiving all messages sent to it. I have 2 test console apps (C#) one to send messages, the other to receive that show this. None of these missing messages are going into the DLQ and no exceptions are being thrown.
Below is the pertinent code from each console app.
Send:
var connectionString = "Endpoint=sb://xxxxxx.servicebus.windows.net/;SharedAccessKeyName=client;SharedAccessKey=xxxxxx=;EntityPath=xxxxxx";
var client = QueueClient.CreateFromConnectionString(connectionString);
int nCount = 0;
try
{
while (true)
{
string strBody = "Msg number:" + nCount;
var message = new BrokeredMessage(strBody);
client.Send(message);
Console.WriteLine(DateTime.Now.ToString() + " Received Body: " + strBody);
nCount++;
Thread.Sleep(10000);
}
}
catch (Exception exception)
{
Console.WriteLine(exception.ToString());
}
Receive:
var connectionString = "Endpoint=sb://xxxxxx.servicebus.windows.net/;SharedAccessKeyName=server;SharedAccessKey=xxxxxx;EntityPath=xxxxxx";
var Client = QueueClient.CreateFromConnectionString(connectionString);
Console.WriteLine("Started At: " + DateTime.Now.ToString());
ManualResetEvent CompletedEvent = new ManualResetEvent(false);
OnMessageOptions options = new OnMessageOptions();
options.AutoComplete = true;
options.MaxConcurrentCalls = 1;
Client.OnMessage((message) =>
{
try
{
Console.WriteLine(string.Format(DateTime.Now.ToString() + " Message Body Received: {0}", message.GetBody<string>()));
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}, options);
I don't think the Receive is the problem as if I close the Receive test app only some messages make it into the queue per the Azure portal.
The included image shows the apps running
The left console is send, the right receive. Ignore in the Send console where it accidentally says "Received Body", should have said "Sent".
Also, using Service Bus Explorer I also can not see the missing messages making it into the Queue. I tried creating a new queue and though more messages got into this new queue it still is not near 100%. More like 80%.