Similar to an Azure worker role, does amazon web services or Heroku offer a service that would be useful for a worker process or recurring worker role. For example, an hourly job that would download a file from a url and parse it into a database or insert it into a dynamo db.
Ideally it would have a status dashboard web interface that would let you see the status of jobs, errors in jobs, etc...
Heroku has a concept of 'worker dynos' to which you can assign to process background jobs, or the other tasks you had asked about.
More information:
https://devcenter.heroku.com/articles/background-jobs-queueing
Related
I have this .NET long running API process/function that usually runs 30 mins in one execution that is hosted in AKS. This API is usually executed from the users coming from the front end of the app.
Due to concurrent executions from users, this is causing exhaustion of the app so I'm planning to implement a some sort of a queueing mechanism with the help of a scheduler(s).
What possibly is applicable Azure service that can execute my API in AKS on a scheduled basis (let's say every minute) and possibly check the database for some flagging values.
I need a way to check the table for some flagging value if there a currently running process or its been completed so it can process the next one, otherwise ignore the call until current on is complete.
I was looking into Azure Web Apps, Web Jobs or Batch Jobs but kinda confused which is applicable with my case.
Please advise thank you in advance.
There are a couple of options here.
Hangfire
Hangfire is an open-source library that can run background jobs in queues. In your case, you can enqueue each request from the client in a queue. Then Hangfire server will process them one by one (even with retry if the job fails). Hangfire supports SQL Server or Redis. You can query the storage to see the status of the queued jobs.
Hangfire can also run scheduled jobs, which will take care of that only one job run at a time.
Azure Service Bus
A more expensive option is to use Azure Service Bus for your queueing capability. For scheduled jobs, you can use AKS CronJobs but you will
implement the check yourself to see if there is a job already running.
Overall, I would recommend Hangfire, which can meet your requirements and is cheaper.
I've an API (python-flask app) running on an app service in azure and want to implement a queuing system using Azure Service Bus such that requests from API are sent to a simple FIFO queue managed/ran by the service bus. Another resource in Azure will be pulling from this queue and running the jobs based on the contents of the json/payload contained in the message in the queue element.
When this element has been processed by the other resource I want to encode the job status/metadata (e.g., "finished" along with metadata such as the location where resulting data was stored). I read about such a system that makes use of the lightweight database offered by Redis, however, I'm wondering if something like this lightweight database/cache system of job status/ids/metadata is available through Azure Service Bus? I'm aware that Redis can be run standalone on a VM in Azure, however, if this can all be managed via the service bus that would be ideal. I couldn't find specifics on this being offered within Azure Service Bus and due to how this job metadata is later being accessed I cannot just push metadata messages to a new queue.
Does anyone have any insight on this or potential alternatives? If Redis can be run alongside flask within the same App Service then that would be ideal, but again I wasn't able to find anything explicit on this and it doesn't seem possible to simultaneously run a flask server/app and Redis server at the same time on an App Service.
Thanks.
I'm wondering if something like this lightweight database/cache system
of job status/ids/metadata is available through Azure Service Bus?
Azure Service Bus is a fully managed enterprise message broker, Azure Redis is a NoSQL database with steroids. It also offers queue mechanism and some other data structures.
it doesn't seem possible to simultaneously run a flask server/app and
Redis server at the same time on an App Service.
You can, but inside containers.
Please check if this can help you: https://stackoverflow.com/a/39008342/1384539
I have created subscription(s) for some of the existing Azure Service Bus topic(s).
Now i need to create Azure Web Jobs that will process for any new Message arriving at the Topic end. This whole process will be used as part of incremental load.
Please provide some input/reference for implementing the above work flow.
Thanks! in advance
You can have Azure Functions instead of Web Jobs to process the messages, which comes with ServiceBusTrigger. You could also go for Web Jobs to process the messages. Refer the following links for more details
https://code.msdn.microsoft.com/Processing-Service-Bus-84db27b4
https://social.technet.microsoft.com/wiki/contents/articles/31981.azure-webjobs-servicebustrigger.aspx
You should provide the Topic Subscription path in place of Queue path to receive message from Subscriptions.
We have a requirement for generating reports based on a schedule. These schedules can be configured by the users for a report. Initially We were planning to use Quartz in a worker role and Create jobs based on the user input.
I'm not sure whether the Azure Scheduler can be used for this because I feel any jobs that could run at the application level could be configured in Azure but not the user jobs.
Please validate and let me know if there are any resources which I could look at.
How about using Azure Web Jobs?
(https://learn.microsoft.com/en-us/azure/app-service/web-sites-create-web-jobs)
Can you please explain more about the nature of reports you're running?
I am writing an Azure hosted MVC website for a gym booking system. I need to be able to maintain membership expiry, suspensions as well as gym class attendence (i.e. logging to the database if a session has been missed). Each of these tasks requires a "c# service function" to be run that will go through the database, perform some checks and update records as and when required.
I need this to run pretty regularly to ensure that missed sessions are logged asap. Should I be developing this as an Azure WebJob and running it continuously? Should i be doing it in another manner? If I could get some suggestions on routes to take that would be massively appreciated.
Thanks
You have a few options: Web Jobs, Scheduler, and Worker Roles.
Web Jobs are a nice addon to an existing azure web app and have the benefit of no additional cost. Web Jobs use Scheduler under the covers if you choose to schedule the Web Job to run at an interval other than continuously. Here is a nice answer that describes the differences between the two.
Worker Roles would be the next logical step up from a Web Job. Worker Roles are dedicated Cloud Service VMs that can provide more dedicated power and offer greater scaling capabilities. Worker Roles can also do much more than just run jobs.
For the application you have described, if you are already running on Azure App Services (Web App) it sounds like a continuously running Web Job would be the correct choice.