I have a Azure queue trigger app. It tries to process message asap but when I have 1000s of message in the queue I want to limit number queue messages it process per second. Is there a way to setup limit?
My goal is to slow down the rate at which my function processes messages.
{
"generatedBy": "Microsoft.NET.Sdk.Functions-1.0.24",
"configurationSource": "attributes",
"bindings": [
{
"type": "queueTrigger",
"queueName": "fred",
"connection": "",
"name": "myQueueItem"
}
],
"disabled": false,
"scriptFile": "../bin/run.dll",
"entryPoint": "Fred.Run"
}
You cannot limit to "X requests per second" as this depends on your processing logic. However, you can configure the batch size and then also to how many instances your Function will scale out.
See here: https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-queue-trigger?tabs=csharp#concurrency
https://learn.microsoft.com/en-us/azure/azure-functions/functions-app-settings#website_max_dynamic_application_scale_out
Related
Here is the structure, a data-drift-detected event in ML Workspace sends the event into event grid which triggers a function in Azure Function App. I want it to run only once after the data drift detection. However, I got this:
image
It runs every ~20s for a few times ://
Here is my host.json:
{
"version": "2.0",
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[3.*, 4.0.0)"
}
}
and function.json:
{
"scriptFile": "__init__.py",
"bindings": [
{
"type": "eventGridTrigger",
"name": "event",
"direction": "in"
}
]
}
Tried changing default options in "singleton" field in host.json, but nothing changed.
Do you have any idea?
When you create an event grid trigger there you will have retry policies where you can change it 1.
Event grid trigger waits of response if it doesnt get it triggers again until it gets a response, so change it to 1. To only trigger once.
So if event grid doesnt get response it triggers again with some interval of time
If not you are sending responses, so its triggering try not to send more responses to your end point.
References taken from:
Azure Event Grid delivery and retry - Azure Event Grid | Microsoft Learn
Building on an earlier question. The following code is an httptrigger that listed to a gis layer edits and updates. It logs into the queue the url payload. I do not want the payload loaded but a specific repetitive message so that it is overwritten everytime for I do not want to dequeue every now and then. How can I go about this?
import logging
import azure.functions as func
def main(req: func.HttpRequest,msg: func.Out[str]) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
input_msg = req.params.get('message')
logging.info(input_msg)
msg.set(req.get_body())
return func.HttpResponse(
"This is a test.",
status_code=200
)
**function.json**
{
"scriptFile": "__init__.py",
"bindings": [
{
"authLevel": "anonymous",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": [
"get",
"post"
]
},
{
"type": "http",
"direction": "out",
"name": "$return"
},
{
"type": "queue",
"direction": "out",
"name": "msg",
"queueName": "outqueue1",
"connection": "AzureStorageQueuesConnectionString"
}
]
}
I do not want the payload loaded but a specific repetitive message so
that it is overwritten everytime for I do not want to dequeue every
now and then.
No, when you put in the same message, It will not overwritten. It just queued in the queue storage.
If you want to process the message in queue, just use queueclient or use queuetrigger of azure function.(queuetrigger of function is based on queueclient, they are basically same.)
This is the API reference of queue:
https://learn.microsoft.com/en-us/python/api/azure-storage-queue/azure.storage.queue?view=azure-python
You can use it to process message in the queue with python code.
And this is the queuetrigger of azure function:(This is already integrated and can be used directly)
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-queue-trigger?tabs=python
I have been trying to put a message in azure service bus topic. My azure function test says that the message has been accepted(202). But there is no message on the subscription side, Could you please help me in this POC. Here is my code snippet. It's a sample that got generated from VS code, I am using serviceBusTrigger.
const { ServiceBusClient, ReceiveMode } = require("#azure/service-bus");
module.exports = async function(context, mySbMsg) {
context.log('JavaScript ServiceBus topic trigger function processed message', mySbMsg);
context.done();
};
Is there any way that I can check if the service bus topic is working as expected?
I do not see a connection string and a binding associated with the code, Inorder to save the message to the queue, you need to have the Queue connection strings in the settings file. Follow the docs,
{
"bindings": [
{
"schedule": "0/15 * * * * *",
"name": "myTimer",
"runsOnStartup": true,
"type": "timerTrigger",
"direction": "in"
},
{
"name": "outputSbQueue",
"type": "serviceBus",
"queueName": "testqueue",
"connection": "MyServiceBusConnection",
"direction": "out"
}
],
"disabled": false
}
Okay, So the thing was, I tried calling the function after pushing messages onto the topic. But the purpose of ServiceBusTriger is reverse. The function with this trigger processes the message when there is message on queue. There is no need to Call the function separately. Yeah, so basics, just create the function with your available topic and subscription, and then try putting message onto the topic. You can see the message in the log of the function.
I have an Azure ServiceBus Trigger, (Functions App with the following function)
module.exports = function(context, message) {
context.log('Message ', message);
context.done();
};
which has the following configuration,
{
"bindings": [
{
"name": "message",
"type": "serviceBusTrigger",
"direction": "in",
"topicName": "topictest",
"subscriptionName": "sbsub1",
"connection": "testsb_RootManageSharedAccessKey_SERVICEBUS",
"accessRights": "Manage"
}
],
"disabled": false
}
Do you guess what is the start time for this function? It is taking 28+ seconds. It is not just the first instance, any new instance it spins it takes the almost a very long time (greater than 1 second). Few sample timings (28, 43, 93 seconds). No time consistency. Horrible platform to work with unless I make some blender mistakes in my code.
There are no external libraries, not too many other files to process any business logic. It is a very simple function.
What am I doing wrong here? Does implementing it under a different language help to improve performance?
This is my functions.json:
{
"bindings": [
{
"type": "serviceBusTrigger",
"name": "message",
"connection": "ServiceBusConnectionString",
"direction": "in",
"topicName": "samples-topic",
"subscriptionName": "samples",
"accessRights": "manage"
}
]
}
Is it possible to subscribe to more than one topic for the same function?
Cheers.
Multiple topics are not supported, but you can achieve the same results by creating subscriptions to forward messages from the topics you want to use, to a target topic you can use from your function.
Another option is to have multiple functions, but the first recommendation would require less maintenance.