The last two days I have started getting MessagingEntityNotFoundException when my code calls Microsoft.Azure.ServiceBus.Core.MessageReceiver.PeekAsync on dead letter queue message receiver. The code have not been changed before the exception started to occur, and have been running without problems for the last two months.
The full exception message is
Microsoft.Azure.ServiceBus.MessagingEntityNotFoundException: A request handler with id '56216' was not found in the entity management node $management.
The code runs in a time triggered Azure Function.
The code for setting up the message receivers:
using Microsoft.Azure.ServiceBus;
using Microsoft.Azure.ServiceBus.Core;
using Microsoft.Azure.ServiceBus.Management;
public class DeadLetterQueueMonitor {
private static List<MessageReceiver> _messageReceivers;
private async Task<List<MessageReceiver>> GetMessageReceivers(ILogger logger)
{
if (_messageReceivers == null)
{
_messageReceivers = new List<MessageReceiver>();
var managementClient = new ManagementClient(ConnectionStrings.ServiceBusConnectionString);
var topics = await managementClient.GetTopicsAsync();
foreach (var topic in topics)
{
var subscriptions = await managementClient.GetSubscriptionsAsync(topic.Path);
_messageReceivers.AddRange(subscriptions.Select(subscription =>
{
string entityPath = EntityNameHelper.FormatDeadLetterPath(EntityNameHelper.FormatSubscriptionPath(topic.Path, subscription.SubscriptionName));
return new MessageReceiver(ConnectionStrings.ServiceBusConnectionString, entityPath);
}));
}
}
_messageReceivers = _messageReceivers
.Select(r => r.IsClosedOrClosing ? new MessageReceiver(ConnectionStrings.ServiceBusConnectionString, r.Path) : r)
.ToList();
return _messageReceivers;
}
}
Any suggestions for what might be wrong?
EDIT 25/2-2019 (as response to comments):
I am using the following Nuget packages:
Microsoft.Azure.Amqp 2.3.0
Microsoft.Azure.ServiceBus 3.1.0
Microsoft.Azure.Services.AppAuthentication 1.0.3
I have added a little more context for the method that creates the MessageReceivers.
Related
I have configured function apps in Azure to persist log entries from the function app insights logs to an event hub which then also persists these log entries to ELK. I recently noticed that some log entries were missing and I followed this example to create a console app that would replay the log events. The console app I created based on the examples works and no exceptions are thrown, however I never find the log entries when querying ELK, they don't appear to be persisted there the way I would expect. Below is some example code of what my console app is doing.
private const string connectionString = "CONNECTION_STRING";
private const string eventHubName = "EVENT_HUB_NAME";
static EventHubBufferedProducerClient producerClient;
static async Task Main()
{
string eventDataString = "{"message": "Here's a log entry"}";*/
var eventData = JsonConvert.DeserializeObject<EventData>(eventDataString);
eventData.MessageId = "MessageIDLog1";
eventData.ContentType = "application/json";
eventData.EventBody = new BinaryData(Encoding.UTF8.GetBytes("{message": "Here's a log entry}"));
producerClient = new EventHubBufferedProducerClient(connectionString, eventHubName);
producerClient.SendEventBatchFailedAsync += args =>
{
Console.WriteLine($"Publishing failed for { args.EventBatch.Count } events. Error: { args.Exception.Message }");
return Task.CompletedTask;
};
producerClient.SendEventBatchSucceededAsync += args =>
{
Console.WriteLine($"{ args.EventBatch.Count } events were published to partition: { args.PartitionId }.");
return Task.CompletedTask;
};
try
{
await producerClient.EnqueueEventAsync(eventData);
}
finally
{
await producerClient.DisposeAsync();
}
}
I'm hoping someone out there has some experience with this and could point me in the right direction as I've been on this for a while and I'm not sure what I'm doing incorrectly.
We use an Azure Service Bus to post all of our requests from our Xamarin mobile app. The Azure Service Bus is bound to an Azure Function which is triggered each time a requests hits the Azure Service Bus.
We have found that we are getting errors from this Azure Function when we send data above a certain size. We can send up to 800 records without a problem but when we send >=850 records we get the following error:
[Error] Exception while executing function:
Functions.ServiceBusQueueTrigger. mscorlib: Exception has been thrown
by the target of an invocation. mscorlib: One or more errors occurred.
A task was canceled.
The service that is being invoked is an ASP.NET Web API RESTful service that saves the data records into a database. This doesn't generate any errors at all.
Here is my Azure Function code.
#r "JWT.dll"
#r "Common.dll"
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using Microsoft.ServiceBus.Messaging;
public static void Run(BrokeredMessage message, TraceWriter log)
{
log.Info($"C# ServiceBus queue trigger function processed message: {message.MessageId}");
if (message != null)
{
Common.Entities.MessageObjectEntity messageObject = message?.GetBody<Common.Entities.MessageObjectEntity>();
string msgType = messageObject?.MessageType;
var msgContent = messageObject?.MessageContent;
log.Info($"Message type: {msgType}");
double timestamp = (DateTime.UtcNow - new DateTime(1970, 1, 1)).TotalSeconds;
string subscriber = "MYSUBSCRIBER";
string privatekey = "MYPRIVATEKEY";
Dictionary<string, object> payload = new Dictionary<string, object>()
{
{"iat", timestamp},
{"subscriber", subscriber}
};
string token = JWT.JsonWebToken.Encode(payload, privatekey, JWT.JwtHashAlgorithm.HS256);
using (var client = new HttpClient())
{
string url = $"http://myexamplewebservices.azurewebsites.net/api/routingtasks?formname={msgType}";
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue(subscriber, token);
HttpContent content = new StringContent((string)msgContent, Encoding.UTF8, "application/json");
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var response = client.PostAsync(new Uri(url), content);
if (response == null)
{
log.Info("Null response returned from request.");
}
else
{
if (response.Result.IsSuccessStatusCode && response.Result.StatusCode == System.Net.HttpStatusCode.OK)
{
log.Info("Successful response returned from request.");
}
else
{
log.Info($"Unsuccessful response returned from request: {response.Result.StatusCode}.");
}
}
}
log.Info("Completing message.");
}
}
This code has been working for several years and works across all our other apps / web sites.
Any ideas why we're getting errors wehen we post large amounts of data to our Azure Service Bus / Azure Function?
It may caused by "new httpclient", there is a limit to how quickly system can open new sockets so if you exhaust the connection pool, you may get some errors. You can refer to this link: https://aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/
And could you please share some more error message ?
I can see that you are creating httpclient connection on each request which possibly be causing this issue. Httpclient creates a socket connection underneath it and has hard limit on it. Even when you dispose it it remains there for couple of mins that can't be used. A good practice is to create single static httpclient connection and reuse it. I am attaching some documents for you to go through.
AzFunction Static HttpClient , Http Client Working , Improper instantiation
I am making a small App that should list the number of items in my Azure queues.
When I use FetchAttributesAsync and ApproximateMessageCount in a Console App, I get the expected result in ApproximateMessageCount after a call to FetchAttributesAsync (or FetchAttributes).
When I use the same in a Universal Windows app, ApproximateMessageCount remains stuck at null after a call to FetchAttributesAsync (FetchAttributes is not available there).
Console code:
CloudStorageAccount _account;
if (CloudStorageAccount.TryParse(_connectionstring, out _account))
{
var queueClient = _account.CreateCloudQueueClient();
Console.WriteLine(" {0}", _account.QueueEndpoint);
Console.WriteLine(" ----------------------------------------------");
var queues = (await queueClient.ListQueuesSegmentedAsync(null)).Results;
foreach (CloudQueue q in queues)
{
await q.FetchAttributesAsync();
Console.WriteLine($" {q.Name,-40} {q.ApproximateMessageCount,5}");
}
}
Universal App code:
IEnumerable<CloudQueue> queues;
CloudStorageAccount _account;
CloudQueueClient queueClient;
CloudStorageAccount.TryParse(connectionstring, out _account);
queueClient = _account.CreateCloudQueueClient();
queues = (await queueClient.ListQueuesSegmentedAsync(null)).Results;
foreach (CloudQueue q in queues)
{
await q.FetchAttributesAsync();
var count = q.ApproximateMessageCount;
// count is always null here!!!
}
I have tried all kinds of alternatives, like Wait()'s and such on the awaitables. Whatever I try, the ApproximateMessageCount stays a null with dertermination :-(.
Am I missing something?
I think you have discovered a bug in the storage client library. I looked up the code on Github and essentially instead of reading the value for Approximate Message Count header, the code is reading the value for Lease Status header.
In QueueHttpResponseParsers.cs class:
public static string GetApproximateMessageCount(HttpResponseMessage response)
{
return response.Headers.GetHeaderSingleValueOrDefault(Constants.HeaderConstants.LeaseStatus);
}
This method should have been:
public static string GetApproximateMessageCount(HttpResponseMessage response)
{
return response.Headers.GetHeaderSingleValueOrDefault(Constants.HeaderConstants.ApproximateMessagesCount);
}
I have submitted a bug for this: https://github.com/Azure/azure-storage-net/issues/155.
I send the message to azure event hub. But I am not able to download the message from event hub.
enter code here
string eventHubConnectionString = "<connection string>";
string eventHubName = "<event Hub name>";
string storageAccountName = "<event hub storage>";
string storageAccountKey = "<storage Key>";
string storageConnectionString = string.Format("DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}",storageAccountName, storageAccountKey);
EventProcessorHost eventProcessorHost = new EventProcessorHost("message", eventHubName, EventHubConsumerGroup.DefaultGroupName, eventHubConnectionString, storageConnectionString);
eventProcessorHost.RegisterEventProcessorAsync<SimpleEventProcessor>().Wait();
IEventProcessor:
enter code here
class SimpleEventProcessor : IEventProcessor
{
Stopwatch checkpointStopWatch;
async Task IEventProcessor.CloseAsync(PartitionContext context, CloseReason reason)
{
Console.WriteLine(string.Format("Processor Shuting Down. Partition '{0}', Reason: '{1}'.", context.Lease.PartitionId, reason.ToString()));
if (reason == CloseReason.Shutdown)
{
await context.CheckpointAsync();
}
}
Task IEventProcessor.OpenAsync(PartitionContext context)
{
Console.WriteLine(string.Format("SimpleEventProcessor initialize. Partition: '{0}', Offset: '{1}'", context.Lease.PartitionId, context.Lease.Offset));
this.checkpointStopWatch = new Stopwatch();
this.checkpointStopWatch.Start();
return Task.FromResult<object>(null);
}
async Task IEventProcessor.ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> messages)
{
foreach (EventData eventData in messages)
{
string data = Encoding.UTF8.GetString(eventData.GetBytes());
Console.WriteLine(string.Format("Message received. Partition: '{0}', Data: '{1}'",
context.Lease.PartitionId, data));
}
//Call checkpoint every 5 minutes, so that worker can resume processing from the 5 minutes back if it restarts.
if (this.checkpointStopWatch.Elapsed > TimeSpan.FromMinutes(5))
{
await context.CheckpointAsync();
lock (this)
{
this.checkpointStopWatch.Reset();
}
}
}
}
It show the following error
Aggregate exception handling. One or more error occurred.
Message details: No such host is known
what is EventProcessor host name?
It show error at this line: eventProcessorHost.RegisterEventProcessorAsync().Wait();
it is not calling IEventprocessor. Is it have any other method for consuming message from event hub?
You can trace your exception and look for inner exception while debugging so this should give you a clue whats the real reason.. I had this stupid exception too and it was because when you use the eventHubName variable with EventProcessorHost it should be in lowercase, (containing only letters/numbers and '-' which has to be followed by letter or number which means that '--' is not supported. eventHubName should also begin with a letter)
Even if the event hub name is "myEventHub123" your variable has to be like:
string eventHubName = "myeventhub123";
Hope this will help someone..
I have had success building a event processor using the sample code located here.
It is hard to tell from your sample code what the error is because it could be related to a typo in your connection string/eventhub/storage account name since it is not provided (you did right, don't post your connection string with sensitive data).
The difference between the way the example loads the event hub information from the connection string and the way the code you provided is how the information is provided through the Evenhub Client. Try update the way you build your EventProcessorHost like the example below:
EventHubClient eventHubClient = EventHubClient.CreateFromConnectionString(eventHubConnectionString, this.eventHubName);
// Get the default Consumer Group
defaultConsumerGroup = eventHubClient.GetDefaultConsumerGroup();
string blobConnectionString = ConfigurationManager.AppSettings["AzureStorageConnectionString"]; // Required for checkpoint/state
eventProcessorHost = new EventProcessorHost("singleworker", eventHubClient.Path, defaultConsumerGroup.GroupName, this.eventHubConnectionString, blobConnectionString);
eventProcessorHost.RegisterEventProcessorAsync<SimpleEventProcessor>().Wait();
I am trying to build a simple Web API REST service in Azure with a service bus queue worker on the back end. I can send a single message from the Web API to the worker just fine. However, I was trying to send more messages just to see how everything works. So, I created a simple controller that looks like this:
for (int i = 0; i < 100; i++)
{
var msg = new BrokeredMessage("Ping");
BioConnector.QueueConnector.OrdersQueueClient.Send(msg);
}
When I call the controller, I am only getting about 1/2 or so of the messages being received by the worker. The rest seem to be dropped.
I had issues with getting only about half the messages using the sample code posted here, so I wrote my own test code. I've tried it with > 100 queue messages and have always had 100% send/rec'd parity. Perhaps you had a similar issue with the code.
Create a new C# console project.
Add a reference to the Microsoft.ServiceBus assembly located in C:\Program Files\Microsoft SDKs\Windows Azure.NET SDK\2012-06\ref\Microsoft.ServiceBus.dll.
In the app.config, change it to this with your own values provided:
<appSettings>
<add key="Microsoft.ServiceBus.ConnectionString" value="Endpoint=sb://blahblah.servicebus.windows.net/;SharedSecretIssuer=owner;SharedSecretValue=pDk0b....=" />
</appSettings>
Add these using directives:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Xml;
using System.Xml.Linq;
using Microsoft.ServiceBus;
using Microsoft.ServiceBus.Messaging;
using System.Configuration;
using System.Threading;
Change the code method to the following:
class Program
{
static void Main(string[] args)
{
string connectionString = ConfigurationSettings.AppSettings["Microsoft.ServiceBus.ConnectionString"];
var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
QueueDescription queueDesc = new QueueDescription("TestQueue");
if (!namespaceManager.QueueExists(queueDesc.Path))
{
namespaceManager.CreateQueue(queueDesc);
}
QueueClient topicClient = QueueClient.CreateFromConnectionString(connectionString, queueDesc.Path);
int sentMsgCount = 0;
int recdMsgCount = 0;
for (int i = 0; i < 100; i++)
{
BrokeredMessage msg = new BrokeredMessage("Test message " + i);
topicClient.Send(msg);
sentMsgCount++;
Console.WriteLine("Sent Message: " + msg);
}
QueueClient subClient = QueueClient.CreateFromConnectionString(connectionString, queueDesc.Path);
bool moreMessages = true;
while (moreMessages)
{
BrokeredMessage recdMsg = subClient.Receive(TimeSpan.FromSeconds(3));
if (recdMsg != null)
{
Console.WriteLine("Received Message: " + recdMsg);
recdMsgCount++;
recdMsg.Complete();
}
else
{
moreMessages = false;
}
}
Console.WriteLine("# of sent msgs: " + sentMsgCount + ", # of rec'd msgs: " + recdMsgCount);
Console.Read();
}
}
This was a weird problem. Through a random walk through "trying things" I ended up changing the string name of the queue and then everything started working again. I didn't change anything but the name of the queue - no changes to any configuration parameters at all.
It appears to be something buggy with that particular queue on Azure.
Azure Service Bus provides durable messaging so you will not loose any messages. Some items to investigate further:
1) Is there another instance of the worker role that is pulling off messages from that queue
2) are you using peek-lock as the receive mode as that will be the only way to guarantee at least once delivery. Receive and delete mode does not have the guarantee
3) are the messages going into dead-lettered queue due to either message expiry or exceeding max delivery count, I.e. They are received but not completed several times
4) if none of the above apply then raise a support ticket and the Azure product team can investigate the symptoms because as I mentioned this is a durable messaging system so no messages will be "lost".
I had the WindowsAzure.ServiceBus NuGet package in my project and used QueueClient.Send() to send messages and faced the same messages lost issue.
My solution to completely solve the issue:
On the send side, I had to use REST API to send messages.
On the receive side, this is how I extracted the message body:
using (var stream = brokeredMessage.GetBody<Stream>())
{
using (var streamReader = new StreamReader(stream, Encoding.UTF8))
{
var msg = streamReader.ReadToEnd();
// Convert the JSON message to an object
// var obj = JsonConvert.DeserializeObject<ObjectType>(msg);
}
}