I have a piece of code that pushes a message to a service bus queue every time a new article is added on my web app. This then gets picked up by a ServiceBusTrigger with SendGrid output in my functions app that sends me an email that a new article has been added by someone.
This doesn't happen often at all and the only reason i decided to make it behave this way is to get my feet wet with some of the awesome Azure services.
My question is - since i don't really care about reeciving these notification email in real time... how can I reduce the frequency with which the trigger is checking the queue?
In my functions app's host.json I've already minimized the maxConcurrentCalls to 1 (default is 16).
"serviceBus": {
"maxConcurrentCalls": 1,
"prefetchCount": 100,
"autoRenewTimeout": "00:05:00"
}
Is there a way to also set it so that my trigger only checks the queue every 30 minutes or something like that?
No. Message retrieval is managed by Scaling Controller, which you don't have much influence on, apart from host.json parameters that you have already seen.
To implement your scenario, you would need to switch to Timer trigger running every 30 minutes and retrieve messages manually from Service Bus, arguably losing many benefits of Azure Functions.
Update: You can now integrate your Service Bus to Azure Event Grid and then use Event Grid triggered Function. Unfortunately, as of today it only works for Premium Service Bus namespace, so you'd most probably have to wait until they expand the feature to lower tiers.
Related
I have an Azure Function, with a queue trigger. The function must proces one queue message each time, one by one, sequentially. This is important, since the function uses an external OAuth API, with various limitions on requesting new access and refresh tokens.
In order to process the queue sequentially, I've the following settings:
host.json
"queues": {
"batchSize": 1,
"newBatchThreshold": 0
}
Application settings
FUNCTIONS_WORKER_PROCESS_COUNT = 1
WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT = 1
Despite these settings, it happens sometimes that the function gets triggered multiple times. Not in my tests, where I dump a lot of messages in the queue, but when the functions is live.
I've dived into the Application Insights Log, and I found that the function gets restarted a lot during the day, and when it restarts, it sometimes start the functions twice.
For example:
Two instances are started. I'm guessing this causes my double triggers from the queue.
How can I make sure the function is triggered once?
I believe the only way to make it even more secure that only one message will be executed at the same time is to use sessions. Give every message the same sessionId to guarantee your desired behavior. Be aware that some things like deadlettering will behave differently if you switch to sessions.
You can also use the [Singleton] attribute if that fits your function app. It should also work, but can incur additional costs when you use the consumption plan. So you need to validate how it affects costs given your workload.
See also my other answer here: azure servicebus maxConcurrentCalls totally ignored
I'm working on creating an events system with Azure ServiceBus, I find events generally hits reliably at the scheduled time I had them set to run - so if event 'pop' is supposed to run at 12:30pm it generally would be delivered at that time to my reciever.
I wanted to know is there a guarantee that events are always fired within the scheduled time or is that more of a suggested time and the system can get clogged and backlogged causing longer queues to form?
There are quite a few differences between messages (which are handled with Service Bus) and events, as you can see in the article Choose between Azure messaging services - Event Grid, Event Hubs, and Service Bus.
An event is a lightweight notification of a condition or a state change. The publisher of the event has no expectation about how the event is handled. The consumer of the event decides what to do with the notification. Events can be discrete units or part of a series.
[...]
A message is raw data produced by a service to be consumed or stored elsewhere. The message contains the data that triggered the message pipeline.
It sounds like you need a reliable way to have a timer trigger execute on a specific time. Service Bus is not the correct service for that, since "the message enquing time does not mean that the message will be sent at that time. It will get enqueued, but the actual sending time depends on the queue's workload and its state." (see BrokeredMessage.ScheduledEnqueueTimeUtc Property).
For handling the triggering in a reliable way, you could use services like Logic Apps (if you want to create it low-code/no-code) or Azure Functions (for the Serverless solution with code).
If you're actually looking for events, consider Event Grid.
My logic app gets triggered with a Recurrence which sends a message to the Service Bus Queue.
What I am trying to do is send a different property (or body) to my service bus queue based on the time of day that the Recurrence gets triggered. Right now I am triggering the Recurrence 8 times a day.
I could do this by creating 8 logic apps, however this seems unpractical because the logic apps would be pretty much identical except for the recurrence time and the property being sent to the service bus. And if I have to scale this up to more than 8 times a day, that will be annoying having to create a new logic app each time.
Any ideas on how to accomplish this?
You can use utcNow() with a custom format string that returns maybe only the hour component which you can use to determine when in the day you are.
If the exact local time is important, you can also use convertFromUtc() to get the zone local time first.
I have data going from my system to an azure iot. I timestamp the data packet when I send it.Then I have an azure function that is triggered by the iothub. In the azure function I get the message and get the timestamp and record how long it took the data to get to the function. I also have another program running on my system that listens for data on the iothub and records that time too.
So most of the time, the time in the azure function is in millisecs, but sometimes, I see a large time for the azure function to be triggered(I conclude it is this because the program that reads from the iot hub shows that the data reached the iot hub quickly and there was no delay).
Would anybody know the reasons for why azure function might be triggering late
Is this the same question that was asked here? https://github.com/Azure/Azure-Functions/issues/711
I'll copy/paste my answer for others to see:
Based on what I see in the logs and your description, I think the latency can be explained as being caused by a cold-start of your function app process. If a function app goes idle for approximately 20 minutes, then it is unloaded from memory and any subsequent trigger will initiate a cold start.
Basically, the following sequence of events takes place:
The function app goes idle and is unloaded (this happened about 5 minutes before the trigger you mentioned).
You send the new event.
The event eventually gets noticed by our scale controller, which polls for events on a 10 second interval.
Our scale controller initiates a cold-start of your function app. This can add a few more seconds depending on the content of your function app (it was about 6 seconds in this case).
So unfortunately this is a known behavior with the consumption plan. You can read up on this issue here: https://blogs.msdn.microsoft.com/appserviceteam/2018/02/07/understanding-serverless-cold-start/. The blog post also discusses some ways you can work around this if it's problematic for your scenario.
I have an azure function listening to a service bus queue trigger using the dynamic consumption plan. Based on this documentation of the host.json config...
https://github.com/Azure/azure-webjobs-sdk-script/wiki/host.json
... you can set the following values
"serviceBus": {
// The maximum number of concurrent calls to the callback the message
// pump should initiate. The default is 16.
"maxConcurrentCalls": 16,
// The default PrefetchCount that will be used by the underlying MessageReceiver.
"prefetchCount": 100
},
Is there any documentation on setting the above for use with functions - particularly using a consumption plan.
The service bus performance best practices documentation suggests:
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-performance-improvements
When using the default lock expiration of 60 seconds, a good value for >SubscriptionClient.PrefetchCount is 20 times the maximum processing rates of >all receivers of the factory. For example, a factory creates 3 receivers, and >each receiver can process up to 10 messages per second. The prefetch count >should not exceed 20*3*10 = 600. By default, QueueClient.PrefetchCount is set >to 0, which means that no additional messages are fetched from the service.
Can somebody please shed some light on how these are/should be used within functions?
Thanks!
Looking at the ASB code for Azure WebJobs (base for Functions) it looks like there's a single receiver that is created. Hence the settings that you see that take into consideration a single receiver created.
ASB performance documentation is describing a scenario where you create your own message pumps and control number of factories and receivers.