We are using an azure function that is triggered by a service bus queue.
Our function's host.json is as follows:
{
"version": "2.0",
"extensions": {
"serviceBus": {
"prefetchCount": 0,
"autoCompleteMessages": true,
"maxAutoLockRenewalDuration": "00:05:00",
"maxConcurrentCalls": 16,
"maxConcurrentSessions": 2000,
"maxMessages": 1000,
"sessionIdleTimeout": "00:01:00",
"enableCrossEntityTransactions": false
}
},
"logging": {
"applicationInsights": {
"samplingExcludedTypes": "Request",
"samplingSettings": {
"isEnabled": true
}
},
"logLevel": {
"Function.Delegation.User": "Information",
"Function": "Error"
}
}
}
We also have an exponential retry in case messages fail, with infinite retry as we cannot afford to lose any data:
[FunctionName("Delegation")]
[ExponentialBackoffRetry(-1, "00:00:01", "00:01:00")]
public async Task Run([ServiceBusTrigger(DelegationConstants.PlotQueueName, Connection = "ServiceBusConnectionString", IsSessionsEnabled = true)] string myQueueItem, MessageReceiver messageReceiver, string lockToken, string messageId, ILogger log)
Our service bus namespace has a single queue that is triggering this. The queue has sessions and partitions enabled and we currently have a 1GB size across 16 partitions (16GB). Our partition key is a physical device identifier (an IMEI on a mobile device if you are familiar), so it has a very broad range of values (around 55k in total in our estate).
Our service bus message is created with a session id of the partition key (IMEI):
var serviceBusMessage = new ServiceBusMessage(JsonConvert.SerializeObject(delegationMessage))
{
SessionId = delegationMessage.PartitionKey
};
Each of our function invocations take ~200-300ms:
Messages 'per device' must be processed FIFO, hence the sessions. At any one time, we could need up to (and in future possibly more than) 1000 messages per second across many devices, each at 200-300ms.
We have about reached the maximum optimisation of our function code itself, so I can't make that any quicker unfortunately.
Given all the above information, our service bus queue is still increasing, indicating that perhaps the service bus + function setup we have right now is not sufficient to cope with our throughput. Is there anything I am missing here or have I reached the limit of concurrency with this configuration? Our function is running on a P1V3 and sits at ~45% cpu and ~25% memory
Related
I have an azure function in the premium plan and I have deployed a queue trigger function inside the docker image.
Here is the reference -> https://learn.microsoft.com/en-us/azure/azure-functions/functions-create-function-linux-custom-image?tabs=in-process%2Cbash%2Cazure-cli&pivots=programming-language-javascript
host.json
{
"version": "2.0",
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[3.*, 4.0.0)"
},
"functionTimeout": "00:30:00",
"extensions": {
"queues": {
"batchSize": 1,
"newBatchThreshold": 0,
"maxDequeueCount": 1
}
}
}
It was working perfectly after some days it suddenly stopped processing message. When i went into azure portal and checked the storage queue, got to see that there are more 100+ message in the queue and function is not picking up any message.
I just restarted the function and it starts working.
I went through this -> https://github.com/Azure/azure-functions-host/issues/3721#issuecomment-441186710
There someone was trying to set maxPollingInterval up to 2 mins. I just check this doc it is mentioned that the default value of this property is 1 min then setting it to 2 mins doesn’t make any sense.
This is an acutely weird behaviour that is why this is happening and it happened to me a couple of times.
The queue is receiving 20 messages maximum in a day, hence the frequency of messages is very low. Those 20 messages can be pushed into the storage queue at once or every hour depending on the requirement.
Any help would be appreciated.
Thanks in advance.
I'm trying to get the payload of Azure IoT Hub telemetry to a Function. I tried using this documentation, but I must be missing something. While I see data coming through, my function is not executed. I tried to put a Service Bus in between, so I created a Message Route in my IoT Hub and used that according to the same documentation, but for Service Bus instead of IoT Hub. I see the messages from a simulated device in the IoT Hub and the Service Bus, but somehow, the function is not executed. I also have no idea how to debug this problem, why the function is not executed. Any help with debugging tips or documentation tips would be much appreciated.
I added the Service Bus parameters in host.json:
...
"serviceBus": {
"prefetchCount": 100,
"messageHandlerOptions": {
"autoComplete": true,
"maxConcurrentCalls": 32,
"maxAutoRenewDuration": "00:05:00"
},
"sessionHandlerOptions": {
"autoComplete": false,
"messageWaitTimeout": "00:00:30",
"maxAutoRenewDuration": "00:55:00",
"maxConcurrentSessions": 16
},
"batchOptions": {
"maxMessageCount": 1000,
"operationTimeout": "00:01:00",
"autoComplete": true
}
}
...
And set the right trigger binding in functions.json:
{
"scriptFile": "__init__.py",
"bindings": [
{
"name": "msg",
"type": "serviceBusTrigger",
"direction": "in",
"queueName": "[MyQueueName]",
"connection": "Endpoint=sb://[MyServiceBusName].servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=[MyServiceBusSAS]"
}
]
}
So what the instructions lack to tell you, what I did wrong, is that you should give your connection an arbitrary name, say queue_conn_str. Then on the Azure Portal, you go to your function app, and set there the application setting with the actual connection string (Endpoint...), with the same name (queue_conn_str).
With that you can actually connect your IoT hub directly to your function, no need for an Event Hub or Service Bus in between.
I cannot get a new servicebus queue trigger to run even after reading many posts on this error. I have not added any logic to the solution yet, simply trying to get the generated shell to run correctly. It is built in Core 3.1, using the QueueTrigger template in VS2019.
This trigger is in a solution with another Azure Function, but that one it not a Queue Trigger (it's a Kafka topic trigger). I have the queue trigger set as the startup project.
I continue to get these errors in spite of many change attempts to resolve this:
Microsoft.Azure.WebJobs.Host:
Error indexing method 'Function1'.
Microsoft.WindowsAzure.Storage: No valid combination of account information found.
I first tried using dev storage ("AzureWebJobsStorage": "UseDevelopmentStorage=true",) then created an actual storage account. Neither worked. I see the storage emulator start when I attempt to run the trigger.
I read this question but am not clear on what I need to do, if anything, related to blob storage:
Queue trigger needs blob storage
Installed Packages:
Microsoft.NET.Sdk.Functions (3.0.13)
Azure.Messaging.ServiceBus (7.2.1)
Microsoft.Azure.WebJobs.Exetensions.Storage (3.0.10)
Microsoft.Azure.WebJobs.Extensions.ServiceBus (4.3.0)
This is the trigger's signature:
public static class Function1
{
[FunctionName("Function1")]
public static void Run([QueueTrigger("queueNameHere", Connection = "AzureWebJobsServiceBus")]string myQueueItem, ILogger log) {
This is my local.settings.json file. Note, I removed the "EndpointSuffix=core.windows.net/;" suffix since many posts said this is a fix. It is ok that both of my key values end in "=", correct?
{
"Values": {
"AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=XXX;AccountKey=XXX",
"FUNCTIONS_WORKER_RUNTIME": "dotnet",
"AzureWebJobsDashboard": "UseDevelopmentStorage=true",
"AzureWebJobsServiceBus": "Endpoint=sb://MySvcBusNamespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXX"
},
"IsEncrypted": false
}
host.json file:
{
"version": "2.0"
}
I tried with this in host.json but have removed it:
"extensions": {
"serviceBus": {
"prefetchCount": 100,
"messageHandlerOptions": {
"autoComplete": true,
"maxConcurrentCalls": 32,
"maxAutoRenewDuration": "00:05:00"
},
"sessionHandlerOptions": {
"autoComplete": false,
"messageWaitTimeout": "00:00:30",
"maxAutoRenewDuration": "00:55:00",
"maxConcurrentSessions": 16
},
"batchOptions": {
"maxMessageCount": 1000,
"operationTimeout": "00:01:00",
"autoComplete": true
}
}
}
The problem is that you are using Storage queue trigger binding public static void Run([QueueTrigger("queueNameHere"
You need to use the ServiceBusTrigger instead, see example below
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus-trigger?tabs=csharp#example
Also possibly duplicate of Microsoft.WindowsAzure.Storage: No valid combination of account information found - Tried to edit conn string, but did not work
I have below confiugration for my Azure Function in host.json & I am sending 30000 msgs in multiple batches to EvenHub but I see that my azure function doesn't pick more than 64 which is a default value. Any particular reason that it doesn't pick the maxbtachsize value.
Event Hub configuration is with Partition Count 25 for single event Hub and Single TU as 15
"version": "2.0",
"extensions": {
"eventHubs": {
"batchCheckpointFrequency": 5,
"eventProcessorOptions": {
"maxBatchSize": 256,
"prefetchCount": 512
}
}
}
For dotnet core, I had this problem that application did not take the maxBatchSize or other params mentioned in host.json. I could not do it from host.json. But it is done using dependency injection (IOptions)
public override void Configure(IFunctionsHostBuilder builder)
{
builder.Services.PostConfigure<EventHubOptions>(o =>
{
o.EventProcessorOptions.MaxBatchSize = 256;
o.EventProcessorOptions.PrefetchCount = 512;
});
}
Now, it works for me! It is drawing 256 events if available.
You should not expect events to be delivered exactly in batch sizes as configured by maxBatchSize. maxBatchSize defines the maximum batch size so events can come in 1 to maxBatchSize batch sizes.
If a backlog is building due to single host not able to process much then Azure Functions should scale out by adding more hosts. When new hosts become active, they should take over some of the partitions thereby increasing the read and process rate. See scaling section here for more details -https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-hubs
"enableReceiverRuntimeMetric": true,
Gives you maxBatchSize count if events are available.
"eventHubs": {
"batchCheckpointFrequency": 5,
"eventProcessorOptions": {
"enableReceiverRuntimeMetric": true,
"invokeProcessorAfterReceiveTimeout": false,
"maxBatchSize": 256,
"prefetchCount": 512,
"receiveTimeout": "00:05:00"
}
}
As shown in my Stack Overflow question Azure webjob not appearing to respect MaxDequeueCount property, I'm troubleshooting a problem where despite my setting of the MaxDequeueCount property to 1, some items are dequeued many times before they're poisoned (in fact some items may never be poisoned at all and just dequeue, fail and are retried and fail endlessly).
The Webjobs SDK handles the retries and poisoning of queue triggered messages automatically and I'm looking for logs that contain details of that handling.
For example, I can see that my function has detected a new queue item by viewing the webjob's log through the SCM at https://myappengine.scm.azurewebsites.net/vfs/data/jobs/continuous/StuffProcessor/job_log.txt (By the way if I've enabled detailed logging to Azure storage on the web app can I get this same info in a Blob?).
[02/22/2017 01:47:22 > ec8d0f: INFO] Executing: 'StuffProcessor.ProcessQueueMessage' - Reason: 'New queue message detected on 'stuff-processor'.'
[02/22/2017 01:47:26 > ec8d0f: INFO] Executed: 'StuffProcessor.ProcessQueueMessage' (Succeeded)
[02/22/2017 01:47:26 > ec8d0f: INFO] Executing: 'StuffProcessor.ProcessQueueMessage' - Reason: 'New queue message detected on 'stuff-processor'.'
I can also get some information on an item's dequeue count by viewing the logs in the azure-jobs-host-archive container once I've enabled detailed logging to Azure storage on the web app:
{
"Type": "FunctionCompleted",
"EndTime": "2017-02-22T00:07:40.8133081+00:00",
"Failure": {
"ExceptionType": "Microsoft.Azure.WebJobs.Host.FunctionInvocationException",
"ExceptionDetails": "Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: ItemProcessor.ProcessQueueMessage ---> MyApp.Exceptions.MySpecialAppExceptionType: Exception of type 'MyApp.Exceptions.MySpecialAppExceptionType' was thrown.
},
"ParameterLogs": {},
"FunctionInstanceId": "1ffac7b0-1290-4343-8ee1-2af0d39ae2c9",
"Function": {
"Id": "MyApp.Processors.ItemProcessor.ProcessQueueMessage",
"FullName": "MyApp.Processors.ItemProcessor.ProcessQueueMessage",
"ShortName": "ItemProcessor.ProcessQueueMessage",
"Parameters": [
{
"Type": "QueueTrigger",
"AccountName": "MyStorageAccount",
"QueueName": "stuff-processor",
"Name": "sourceFeedItemQueueItem"
},
{
"Type": "BindingData",
"Name": "dequeueCount"
},
{
"Type": "ParameterDescriptor",
"Name": "logger"
}
]
},
"Arguments": {
"sourceFeedItemQueueItem": "{\"SourceFeedUpdateID\":437530,\"PodcastFeedID\":\"2d48D2sf2\"}",
"dequeueCount": "96",
"logger": null
},
"Reason": "AutomaticTrigger",
"ReasonDetails": "New queue message detected on 'stuff-processor'.",
"StartTime": "2017-02-22T00:07:40.6017341+00:00",
"OutputBlob": {
"ContainerName": "azure-webjobs-hosts",
"BlobName": "output-logs/1ffd3c7b012c043438ed12af0d39ae2c9.txt"
},
"ParameterLogBlob": {
"ContainerName": "azure-webjobs-hosts",
"BlobName": "output-logs/1cf2c1b012sa0d3438ee12daf0d39ae2c9.params.txt"
},
"LogLevel": "Info",
"HostInstanceId": "d1825bdb-d92a-4657-81a4-36253e01ea5e",
"HostDisplayName": "ItemProcessor",
"SharedQueueName": "azure-webjobs-host-490daea03c70316f8aa2509438afe8ef",
"InstanceQueueName": "azure-webjobs-host-d18252sdbd92a4657d1a436253e01ea5e",
"Heartbeat": {
"SharedContainerName": "azure-webjobs-hosts",
"SharedDirectoryName": "heartbeats/490baea03cfdfd0416f8aa25aqr438afe8ef",
"InstanceBlobName": "zd1825bdbdsdgga465781a436q53e01ea5e",
"ExpirationInSeconds": 45
},
"WebJobRunIdentifier": {
"WebSiteName": "myappengine",
"JobType": "Continuous",
"JobName": "ItemProcessor",
"RunId": ""
}
}
What I can't find though are logs which show detail for a particular queue item where processing fails due to an exception and is placed in the poison queue. I'm looking for these in an attempt to further troubleshoot the apparent ignoring of the MaxDequeueCount property. Is this logged?
UPDATE: I found the post Azure WebJobs with Storage Queue and Exceptions and it contained the following screenshot:
That screenshot shows the standard "New queue message detected..." messages (which I see both on Azure and running locally in the emulator) and it also shows "Message has reached MaxDequeueCount of X...Moving message to queue 'xyz-poison'", which I only see locally in the emulator. Does the Azure-based runtime not show this info for some reason? I never see the poisoning-related messages like this when running locally in the console window or via the webjobs dashboard or when running on Azure.
Are you using Azure Storage version 8.* ? It contains a breaking change.
When a queue message object gets inserted into a message queue in Azure Storage 8, the Message Id of the original queue message object is overwritten with the unique message id of the newly inserted message.
This means that it looses its reference to the poison message and can no longer remove it.
https://github.com/Azure/azure-webjobs-sdk/issues/985
As for logging, 2 options:
Create your own QueueProcessorFactory and QueueProcessor - you can override the method that handles poison messages to produce more debug output. There you can also fix the issue altogether if you'd like to stay on Azure Storage 8.*.
Register a trace monitor. https://github.com/Azure/azure-webjobs-sdk-extensions/wiki/Error-Monitoring#tracemonitor
var traceMonitor = new TraceMonitor()
.Filter(p => true, "Trace Handler")
.Subscribe(TraceHandler.Process);
config.Tracing.Tracers.Add(traceMonitor);
In your previous post, you have the triggered function as follows
public void ProcessQueueMessage([QueueTrigger("azurewejobtestingqueue")] string item, TextWriter logger)
{
if ( item == "exception" )
{
throw new Exception();
}
}
Anything you write to the logger should appear in the logs on the WebJobs dashboard. My suggestions are:
Change 'if (item == "exception")' to 'if (item.ToLower().StartsWith("exception")'. Then append a number to the exception message when testing. Log the text of the queue message to the TextWriter so you can be sure that you're looking at the same message you think you are inspecting.
You can get the message's dequeue count as a parameter to a function. See here. Check out the FailAlways method in the example. Log this to the TextWriter as well.
Try running this locally. All output will be written to the console window. See if you get the same behavior.
I'm really interested in finding what's causing this behavior, so make sure you let us know if you crack it!