I have a console application in C# that search for specific records in DB and outputs them to the console. For the console application to find and list all the records it usually takes 30 minutes. If I want to automate that console application to run at specific time and save the records to csv file what feature from Azure platform I can use? Is Azure function suitable for this task or there are some other Azure features I can use?
I tried to investigate Azure Function, however it looks the functions are not suitable for long running task that takes longer than 5 minutes.
A Premium function can run for longer than 5 minutes (a consumption function maxes out at 10 minutes btw).
This is a fudge but you could also take the console app wrap it in a container, run that container in an Azure container instance and start it with a logic app. You only pay for the runtime of the container.
I'd probably ask why it takes 30 minutes to find all the records, is there a better way of running the query?
You could use Azure Durable Functions, but your algorithm should implement Continuation Token logic. Means, that you have some token (which could be JSON object with some properties), and that token allows you to start/continue your algorithms from any point of time.
For example - you are searching for specific records in DB, and memorize current processed DB primary key of the row. And you store it in your continuation token, and that token is stored in Azure Storage Container Blob, or in Table. When your function will be stopped (b/t it will run more than 10 minutes), and will be started again - you will check value of that token, and start search from that point.
Related
I want to use Azure to handle long running tasks that can’t be handled solely by a web server as they exceed the 2 min HTTP limit (and would put unnecessary load on it regardless). In this case, it’s the generation of a PDF report that can take some time (between 2-5 mins). I’ve seen examples of solutions for this using other technologies (Celery, RabbitMQ, AWS Lamda, etc.) but not much using what's available on Azure (Functions and Storage in this case).
Details of my proposed solution are as follows (a diagram is here)
API (that has 3 endpoints):
Generate report – post a message to Azure Queue Storage
Get report generation status – query Azure Table Storage for status
Get report – retrieve PDF from Azure Blob Storage
Azure Queue Storage
Receives a message from the API containing parameters of the requested report
Azure Function
Triggered when a message is added to Azure Queue Storage
Creates report generation status record in Azure Table storage, set to ‘Generating’
Generate a report based on parameters contained in the message
Stores output PDF in Azure Blob Storage
Updates report generation status record in Azure Table storage to ‘Completed’
Azure Table Storage
Contains a table of report generation requests and associated status
Azure Blob Storage
Stores PDF reports
Other points
The app isn’t built yet – so there is no base case I’m comparing against (e.g. Celery/RabbitMQ)
The time it takes to run the report isn’t super important (i.e. I’m not concerned about Azure Function cold starts)
There’s no requirement for immediate notification of completion using something like Webhooks – the client will poll the API every so often using the get report generation endpoint.
There won't be much usage of the app, so having an always active server to handle tasks (vs Azure Function) seems to be a waste of money.
If I find that report generation takes longer than 10 mins, I can split it up into more than one Azure Function (to avoid consumption plan hard limit of 10 mins execution time per function)
My question is essential whether or not this is a good approach (to me it seems good, and relatively cost-effective, I’m just not sure if there’s something I’m missing).
This can be simplified using Durable Functions. Most of the job is already handled by the framework and you also can query an endpoint to check for the completion status.
https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview?tabs=csharp
I have the following Logic app:
This logic app is triggered when my connection goes above 120, it runs a powershell script which reduces the number of connection. The problem that I am facing is once it runs and the connections go back down from 120 or above the logic app is triggered again because the alert is being triggered, this generally happens minutes from each other. Is there a way I can tweak this logic app to make sure it wont trigger again for maybe 10 minutes after it has been triggered, to stop my powershell script from running twice?
You could have persistent value - stored in any of one of cloud services - let me take azure blob for instance.
The immediate running instance can persist the current running time in the Azure Blob storage.
So next instance is triggered - checks for the last run time from the blob - if it is less than 10 minutes. Your logic would be to skip the execution of the PowerShell.
The overall logic will look like below :
Note :
The logic app doesn't have a concept of persistent storage built-in. You can use AzureSQL, CosmosDB,Sharepoint, Azure Storage etc. using their inbuilt connector to achieve this persisting storage functionality.
I have a web api application deployed to App Service.
I need to be able to set arbitrary number of scheduled jobs (http calls to my web api) with arbitrary due date times (from a few hours to a few months).
The point is that i need to be able to set/edit/cancel them on the fly programatically based on different state of my app and i need to maintain thousands of them.
Is there some recommended way to do it?
I would persist the jobs into either SQL database or Table Storage (I would call it 'ScheduledJobs').
Then have an Azure Function that will query the ScheduledJobs storage at some interval, say every hour to pickup jobs that should be processed at that point in time.
The jobs that are due for processing can then be written to a queue (I would name it 'jobs-queue').
Then have another Azure Function that will pickup jobs from 'jobs-queue'. This Azure Function has the business logic of how to process each job.
From my experience, an Azure Function can only run up to 10 minutes. The time it will take to process each job should not be longer than this period.
Hope this gives you some idea.
I have been running my web jobs for a few months now and the history includes hundreds of thousands of instances when some of them ran, mainly TimerTriggers. When I go in the portal to the "Functions" view of web jogs logs I have noticed that my app service plan shoots up to 100% CPU while I am sat on that page. The page constantly says "Indexing...."
When I close the "Functions" view down the CPU goes straight back down to a few percent, it's normal range.
I assume it must be down to the fact that it has been running for so long and the number of records to search through is so vast. I cannot see any option to archive or remove old records of when jobs ran.
Is there a way I can reduce the history of the jobs? Or is there another explanation?
I'm not familiar with Azure Web Jobs, but I am familiar with Azure Functions which is built on top of Web Jobs, so this might work.
In Azure Functions, each execution is stored in Azure Storage Table. There, you can see all of the parameters that were passed in, as well as the result. I could go into the Storage Table and truncate the records I do not need, so you might be able to do the same with Web Jobs.
Here is how to access this information:
Table Storage in markheath.net/post/three-ways-view-error-logs-azure-functions
Based on your description, I checked my webjob and found the related logs for azure webjobs dashboard as follows:
For Invocation Log Recently executed functions, you could find them as follows:
Note: The list records for Invocation Log are under azure-webjobs-dashboard\functions\recent\flat, the detailed invocation logs are under azure-webjobs-dashboard\functions\instances.
For an Azure Search service, I am looking to get updates reflected closest to real-time. I can execute it via REST API using a logic app with recurrence.
If I invoke the logic app very frequently(ever 3 seconds). Is there a catch to this approach?
can the indexer get blocked because of too frequent calls?
is there any cost-implication of this constant invocation (either on logic app or on azure search)
I am trying to see if I can avoid building logic to determine various scenarios when the indexer needs to be called. (can get complex)
If you're OK with indexer running every 5 minutes, you don't need to invoke it at all - it can run on a schedule.
If you need to invoke indexer more frequently, you can run it once every 3 minutes on a free tier service, and as often as you want on any paid tier service. If an indexer is already running when you run it, the call will fail with a 409 status.
However, if you need to run indexer very frequently, usually it's in response to new data arriving and you have that data in hand. In that case, it may be more efficient to directly index that data using the push-style indexing API. See Add, update, or delete documents for REST API reference, or you can use Azure Search .NET SDK