I have created a Blob triggered function app in Python. My requirement is to run a separate instance for each blob upload (for parallel processing), but it's not happening. Even I have modified the host.json as below as per the below link:
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-blob
{
"version": "2.0",
"extensions": {
"blobs": {
"maxDegreeOfParallelism": "4"
}
}
}
Still, the same instance is running and processing files one by one. Am I missing something here?
I'm afraid we can't implement this requirement. As far as I know, we can just set the function app to scale out to maximum n(in your case is 4) instances, but we can't scale out instances manaually.
When you modify the configuration to allow the function app to scale out for multiple instances, it can just scale out automatically when lots of requests coming. If there are only 4 request, only one instance will be started in most cases.
Here is another post I did research in the past which is similar problem with this case for your reference.
Related
I am currently trying to list all instances of an activity function and the orchestrator function using azure function core tools. The application synchronizes data from different sources into a centralized location.
The setup is as follows:
TimerTrigger -> Durable Orchestrator -> Multiple Activity Functions
In my concrete example, it is like this:
Start Synchronization -> Orchestrate Synchronizations -> Synchronize Source
So we start the synchronization process which starts the orchestrator. The orchestrator then starts multiple different synchronizations, one for each source. The problem though is that I cannot seem to get the azure function core tools to list me all instances of the functions I am interested in.
Unfortunately, I would really prefer not to have to use the REST api to query for this information. The setup really complicates things with IP restrictions and managed identity authentication. I think I can correct the setup to get things to work from my network + user, if really needed, but I think that will take way longer than required.
I have tried running the following command:
func durable get-instances
in a directory with a file called host.json with the following contents:
{
"version": "2.0",
"AzureWebJobsStorage":"DefaultEndpointsProtocol=https;AccountName=Name;AccountKey=Key;EndpointSuffix=core.windows.net",
}
I have also tried where the contents of the file are as follows:
{
"version": "2.0",
"extensions": {
"durableTask": {
"storageProvider": {
"connectionStringName": "DefaultEndpointsProtocol=https;AccountName=Name;AccountKey=Key;EndpointSuffix=core.windows.net"
}
}
}
}
I have tried calling the func durable get-instances with and without the --connection-string-setting parameter, with the values 'AzureWebJobsStorage' and 'extensions:durableTask:storageProvider:connectionStringName', but nothing seems to work. I keep getting the error No storage connection string found.
I know that the connection string is correct. I have pulled it directly from the storage account 'Access keys' blade.
Is there anything I am missing? What am I doing wrong?
Thanks to #juunas, I got it to work. I edited the host.json file to have the following content:
{
"version": "2.0"
}
and created another file called local.settings.json with the following contents:
{
"IsEncrypted": false,
"Values": {
"AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=Name;AccountKey=Key;EndpointSuffix=core.windows.net",
}
}
Running func durable get-instances now works and returns a continuation token, but an empty list. I was not expecting that, but I can now start exploring and understanding here what is going on.
I have a webapp runnin on Azure. The web site is built in asp net core 3, and is running in a Docker container.
There is a background worker doing a few things such as database cleanup and sending emails built into the application.
My question is how I should best handle this if I need to scale out the application. That is if I create multiple instances of it, whats the best way to make sure the background worker is only running on one of the instances.. And if another instance is removed that another takes over the job.
I realize one solution to this is to break the application apart and run the backgroundworker separately as an Azure function. But I would prefer to avoid this for cost (it's a hobby project) and complexity reasons.
So I'm intrested if there are more ways of solving this which keeps things in one docker container.
Is there for example an environment variable that I can query to get the current instance name and a list of all instances (then I can just say that the first instance in alphabetical order is the "primary" instance). And check this every so often to know if the current instance is the primary instance.
Sidenote: Azure Functions don't cost extra if you reuse the App Service Plan of your website. And complexitywise they are probably less complex than what you are currently thinking about. But if your main goal is to run everything in a single container, you can achieve that as well:
You can use the WebJobs SDK to basically run the "event handler side" of Azure Functions, including the coordination of the required work. Use the singleton attribute if you need additional limitation of concurrency. Infrastructure-wise, WebJobs require a storage account how they manage scale.
You can run WebJobs in the same process as the rest of your ASP.NET Core Application. Some code to get you started if you want to go that route:
var builder = Host.CreateDefaultBuilder(args);
builder.ConfigureWebHostDefaults(webBuilder =>
{
// webBuilder.UseStartup<Startup>();
webBuilder.Configure(app =>
{
app.UseRouting();
app.UseEndpoints(endpoints =>
{
endpoints.MapGet("/", async context =>
{
await context.Response.WriteAsync("Hello World!");
});
});
});
});
builder.ConfigureWebJobs(b =>
{
b.AddAzureStorageCoreServices();
b.AddAzureStorage();
b.AddTimers();
});
IHost host = builder.Build();
using (host)
{
await host.RunAsync();
}
I'm looking to use the Change Feed Processor SDK to monitor for changes to an Azure Cosmos DB collection, however, I have not seen clear documentation about whether the host can be run as an Azure Web Job. Can it? And if yes, are there any known issues or limitations versus running it as a Console App?
There are a good number of blog posts about using the CFP SDK, however, most of them vaguely mention running the host on a VM, and none of them or any examples running the host as an azure web job.
Even if it's possible, as a side question is, if such a host is deployed as a continuous web job and I select the "Scale" setting of the web job to Multi Instance, what are the approaches or recommendations to make the extra instances run with a different instance name, which the CFP SDK requires?
According to my research,Cosmos db trigger could be implemented in the WebJob SDK.
static async Task Main()
{
var builder = new HostBuilder();
builder.ConfigureWebJobs(b =>
{
b.AddAzureStorageCoreServices();
b.AddCosmosDB(a =>
{
a.ConnectionMode = ConnectionMode.Gateway;
a.Protocol = Protocol.Https;
a.LeaseOptions.LeasePrefix = "prefix1";
});
});
var host = builder.Build();
using (host)
{
await host.RunAsync();
}
}
But it seems only Nuget for c# sdk could be used,no clues for other languages.So,you could refer to the Compare Functions and WebJobs to balance your needs and cost.
The Cosmos DB Trigger for Azure Functions it's actually, a WebJobs extension: https://github.com/Azure/azure-webjobs-sdk-extensions/tree/dev/src/WebJobs.Extensions.CosmosDB
And it uses the Change Feed Processor.
Functions run over WebJob technology. So to answer the question, yes, you can run Change Feed Processor on WebJobs, just make sure that:
Your App Service is set to Always On
If you plan to use multiple instances, make sure to set the InstanceName accordingly and not a static/fixed value. Probably something that identifies the WebJob instance.
I have an Azure Function that connects to a database, but it's failing with the following message -
The type or namespace name 'Npgsql' could not be found (are you missing a using directive or an assembly reference?)
In project.json, I have the following declaration -
{
"frameworks": {
"net46":{
"dependencies": {
"Npgsql": "3.2.2",
"System.Runtime.Serialization.Formatters": "4.3.0"
}
}
}
}
The strange thing is, I have the exact same Function (deployed from Octopus) running on another Function App (a lower environment) without issue.
Is anyone able to explain why my dependancies being ignored in one Function App but not the other?
So this seems that this is happening due to the way that the Function App is deployed. I'm using Octopus, deploying via the "Deploy an Azure Web App" process step, and it seems that all that's happeing is the old files are being deleted, and the updated files copied across.
The trouble is, as far as I can tell, the dependancies in project.json are not resolved unless you edit the file via the Portal and then save it.
I'm not sure if this can be classed as a bug in either Octopus or Azure (or just really lazy implementation Function Apps), but it's very frustrating. Essentially it makes remote deployment of a Function App that includes changes to function.json impossible.
I have an XML with 25,000 objects that I want to transition to the cloud instead of keeping it local. Buuuuuut I have no idea how to get the XML document into the service, preferably I want it in the SQL database that comes with the service. I know how to upload from the app but with 25,000 objects the service times out. I am sure there is something I just cant find the documentation on.
private async void bw_Worker(object sender, DoWorkEventArgs e)
{
foreach(card x in cardsList)
{
await App.MobileService.GetTable<card>().InsertAsync(x);
}
}
It gets just past 7500 then times out. I am running it in a background worker. I couldn't find any such limits on that so I just imagined its a process that takes too long to complete.