I have created a Azure service bus topic subscription which receives a message form another http trigger function app and insert information into a database.
Every message I receive has an ID and based on the ID, I decide to either add a new record or updating existing one.
Problem is happening when 2 messages with the same ID are received at the same time and ends up creating 2 database records.
Is there any way to configure topic subscription function to run only 1 instance at a time? I don't have control on the function app which send
Is there any way to configure topic subscription function to run only 1 instance at a time?
You can configure everything like number of instances to be running, number of messages to be processing, number of calls and sessions, etc., in host.json file. All the attributes of host.json related to Azure Functions Service bus trigger bindings are available in this MS Doc reference.
For your requirements such as processing 1 message at a time, you can define the following attributes to 1 in the host.json file:
"maxConcurrentSessions": 1,
"maxMessageBatchSize": 1,
"maxConcurrentCalls": 1,
"messageHandlerOptions": {
"maxConcurrentCalls": 1
},
Normally, Azure Functions will process the messages in multiple and in parallel. So, the attributes like maxConcurrentSessions, maxConcurrentCalls plays the major role to define number of sessions and calls to be processed for every instance.
"batchOptions": {
"maxMessageCount": 1
}
Above complete configuration of host.json will do processing of 2nd message only after the execution of 1st message in the Service bus topic subscription azure function.
Related
Problem: I have a Azure Topic Topic1 with Subscriptions Subscription-A, Subscription-B, Subscription-C
Each Subscription is listen to by an Azure function and it uses the message to perform unique action like charging customer or Updating another subsystem etc.
If the execution in one of the functions results in error , I would like to requeue the message into that specific subscription for execution after a delay say 1 min. not right way. [ Currently, If the function results in error, message is retried right away couple of times before pushing to dead letter. Looks like default behavior by Azure service bus and functions]
I can not resend a message to Topic1 as that message would be read by all the 3 subscriptions. I don't want this as the other message were processed successfully by the respective subscription/functions. This requeued message should be filtered out by other subscriptions on this topic.
Any ideas or suggestion would be of great help.
from Microsoft EventHub Java SDK examples (https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-java-get-started-send), these are the steps that needs to be taken to be able to consume messages from an Even-Hub via the java SDK :
1.Create a storage account
2.Create a new class called EventProcessorSample. Replace the placeholders with the values used when you created the event hub and storage account:
3.
String consumerGroupName = "$Default";
String namespaceName = "----NamespaceName----";
String eventHubName = "----EventHubName----";
String sasKeyName = "----SharedAccessSignatureKeyName----";
String sasKey = "----SharedAccessSignatureKey----";
String storageConnectionString = "----AzureStorageConnectionString----";
String storageContainerName = "----StorageContainerName----";
String hostNamePrefix = "----HostNamePrefix----";
ConnectionStringBuilder eventHubConnectionString = new ConnectionStringBuilder()
.setNamespaceName(namespaceName)
.setEventHubName(eventHubName)
.setSasKeyName(sasKeyName)
.setSasKey(sasKey);
There are several things i don't understand about this flow -
A. Why is a storage account required? Why does it needs to be created only when creating a consumer and not when creating the event hub itself?
B. What is 'hostNamePrefix' and why is it required?
C. More of a generalaztion of A, but i am failing to understand why is this flow so complicated and needs so much configuration. Event Hub is the default and only way of exporting metrics/monitoring data from Azure which is a pretty straightforward flow - Azure -> Event Hub -> Java Application. Am i missing a simpler way or a simpler client option?
All your questions are around consuming events from Event hub.
Why is a storage account required?
Read the event only once: Whenever your application will read event from event hub, you need to store the offset(identifier for the amount of event already read) value somewhere. The storing of this information is known as 'CheckPointing' and this information will be stored in Storage Account.
Read the events from starting everytime your app connects to it: In this case, your application will keep on reading the event from very beginning whenever it will start.
So, the storage account is required to store the offset value while consuming the events from event hub in case if you want to read event only once.
Why does it needs to be created only when creating a consumer and not
when creating the event hub itself?
As it depends upon the scenario, whether you want to read your events only once or every time your app starts and that's why storage account is not required while creating event hub.
What is 'hostNamePrefix' and why is it required?
As the name states 'hostNamePrefix', this is name for your host. The host means the application which is consuming the events. And it's a good practice to make use of GUID as a hostNamePrefix. HostNamePrefix is required by the event hub to manage the connection with the host. In case, if you have 32 partitions, and you have deployed 4 instances of your same application then 8 partition each will be assigned to your 4 different instances and that's where the host name helps the event hub to manage the information about the connection of the respective partitions to their host.
I will suggest you to read this article on event hub for clear picture of the event processor host.
I'm more used to Service Bus Queue but have challenges when using it with Azure Function App.
We have Azure Function App which reads data from Service Bus Queue through ServiceBugTrigger. Per this link, Azure Function App manage Queue message PeekLock internally (at the queue trigger and function execution end), we do not require to Complete() message at the end of the process.
My queue message lock duration is set to 3min (which is enough for my execution, I would say more than my requirement). I also applied other required parameters to treat message well like,
"serviceBus": {
"maxAutoRenewDuration": "00:05:00",
"maxConcurrentCalls": 10,
"prefetchCount": 0
}
I am getting LOCK DURATION EXPIRED error frequently with this implementation. Really no idea what's happening here, Any clue?
I am used to Service Bus Queue and aware with each parameter function. Also, have configured each parameter per requirement.
This happens when your maxAutoRenewDuration is more than the lock duration at servicebus side.
You should check the lock duration specified at Service bus queue side. Ensure it is greater or equal to maxAutoRenewDuration specified in your azure function
You can update it from portal or service bus explorer
I have this setup on Azure.
1 Azure Service Bus
1 Sql Azure Database
1 Dynamic App Service Plan
1 Azure function
I'm writing messages in the service bus, my function is triggered when a message is received and writes to the database.
I have a huge number of messages to process and I have this exception:
The request limit for the database is 90 and has been reached
I dig here on SO and in the docs and I found this answer from Paul Battum: https://stackoverflow.com/a/50769314/1026105
You can use the configuration settings in host.json to control the level of concurrency your functions execute at per instance and the max scaleout setting to control how many instances you scale out to. This will let you control the total amount of load put on your database.
What is the strategy to limit the function, since it can be limited on:
the level of concurrency your functions execute at per instance
the number of instances
Thanks guys!
Look in to using the Durable Extensions for Azure Functions where you can control the number of orchestrator and activity functions. You will need to change your design a little but will then get far better control over concurrency.
{
"version": "2.0",
"durableTask": {
"HubName": "MyFunctionHub",
"maxConcurrentActivityFunctions": 10,
"maxConcurrentOrchestratorFunctions": 10
},
"functionTimeout": "00:10:00"
}
I have a Azure Function scenario as follows:
1) If product = 123, then use Service Bus Topic1
2) If product = 456, then use Service Bus Topic2
I think there are 2 options to solve this:
OPTION 1: Have same Azure Function deploy 2 times (2 diff names) and each having different input/output mapping
OPTION 2: Have only 1 Azure Function but in Application Setting specify the input/output mapping. **From my understanding Application Setting is Key Value. Is this correct? if not, how can I specify complex value in this parameter **.
What is the best way to have this?
What I am thinking about is deploy same Azure Functions 2 times with different settings as follows:
Azure Function 1 with Application Settings as "productid" = 123 and "sbTopic" = topic1
Azure Function 2 with Application Settings as "productid" = 456 and "sbTopic" = topic2
I am wondering if there is a better way such that same Azure Function can run for any of my input/output mapping. If so, where and how do I specify my input (productid) and output (sbTopic) mapping?
EDIT 1: this is CosmosDB Trigger. Whenever we get products in Cosmos DB, i want to send to correct SB Topic
EDIT 2: I have something similar as follows:
Cosmos DB Trigger --> Azure Function --> Service Bus Topic for id=123
I am debating if I should have as follows
Cosmos DB Trigger --> Azure Function1 --> Service Bus Topic for id=123
Cosmos DB Trigger --> Azure Function2 --> Service Bus Topic for id=456
Cosmos DB Trigger --> Azure Function3 --> Service Bus Topic for id=789
which means I would have 3 AF duplicated
etc
or
Cosmos DB Trigger --> 1 Azure Function. Specify mappings (product id and sb topic) in App Settings and --> Add logic in AF such that: if id=123 send message to topic1 ; if id=456 send message to topic 2 etc.
You should have your Pub/Sub model driven by Topic based on the Subscription Rules. The following screen snippet shows an example of this model:
In the above Pub/Sub model, the Publisher fire a Topic with the application properties for additional details such as productid, type, etc. The Topic on the Service Bus can have more multiple Subscriptions entities for specific filters (Rules).
The Subscriber (in this example, the ServiceBusTrigger Function) can be triggered on the TopicName and SubscriptionName configured in the app settings.
In other words, the Subscription entity (Rules) can decided which subscriber will consume the event message. Also advantage of this Pub/Sub model is for continuous deployment, where each environment such as dev, stage, QA, pre-production, production has the same model (topicName, subscriptionName, Rules, etc.) and only connection string will specify which azure subscription (environment) is a current one.
The option suggested by #RomanKiss makes sense, and that's a "canonic" use of Service Bus topics & subscriptions.
But if you need to do what you are asking for, you can achieve that with a single Azure Function and imperative bindings.
First, do NOT declare your output Service Bus binding in function.json.
Implement your Function like this:
public static void Run([CosmosDBTrigger("...", "...")] Product product, Binder binder)
{
var topicName = product.Id == "123" ? "topic1" : "topic2";
var attribute = new ServiceBusAttribute(topicName);
var collector = binder.Bind<ICollector<string>>(attribute);
collector.Add("Your message");
}
By the way, if you deploy your function two times, you will have to put them on different lease collections, otherwise they will compete to each other, not do the processing twice. And you'll have to take care of NOT sending the message to Topic 1 when product is 123 and vice versa.