Azure Functions + EventHub: why batch latency grows up constantly? - node.js

Question

I have next chart:
As you can see my batch latency grows up and count of outgoing messages grows down.
Inside of the function I do append to a blob storage. But blob metrics says everything is ok.
What could be causing the ever-increasing latency?
Function implementation:
const parsedEvents = eventHubMessages.map((event) => {
try {
return JSON.parse(event);
} catch (error) {
context.log(`Error: cannot parse next event: ${event}`);
return {};
}
});
for (const event of parsedEvents) {
const { id } = event;
const data = {
data: 'data',
};
const filename = `${id}.log`;
await blob.append(filename, JSON.stringify(data));
}
Blob append is a instance of a class and looks like:
class AzureStorage {
constructor(config) {
this.config = config;
this.blobServiceClient = BlobServiceClient.fromConnectionString(this.config.storageConnectionString);
this.containerClient = this.blobServiceClient.getContainerClient(this.config.containerName);
}
async append(filename, data) {
const client = this.containerClient.getAppendBlobClient(filename);
await client.createIfNotExists();
await client.appendBlock(data, data.length);
}
}
Another one chart:
Update:
So, my problem was in the blob storage. I did client.createIfNotExists(); and this is the root of the problem. I rewrite my code next way:
I call client.appendBlock
I catch it and if there is an error, then I do client.create(); and then client.appendBlock one more time.

Answer 1

Thanks #JesseSquire for your helpful suggestion. Adding few more troubleshooting points that helps to find the root cause of latency issues in Azure Functions integrated with Event Hubs.
Also check if versioning is enabled on the Storage account which may slows down considerably.
Make sure you can scale out to at least the number of partitions your Event Hub has by checking your Function scaling setup.
Use Logging/Application Insights feature to measure the execution time for blob append in order to check bottlenecks in your code.
Telemetry Logs helps you to find your function performance data & metrics for avoiding the integrated services like Event Hub batch events latency issues, runtime exceptions, etc.
Dedicated storage account is better because of checkpointing, event hub-triggered functions could experience a large volume of storage transactions.
Refer to the MS Doc of Azure Functions Performance for Event Hubs.