I'm completely new to the Windows Azure and Windows Workflow scope of things.
But basically, what I'm trying to implement is the Cloud web-app that's going to be responsible for pushing down tile updates/badge/toast notifications to my Winodws 8 application.
The code to run to send down the tile notification etc is fine, but needs to be executed every hour or so.
I decided the most straight forward approach was to make an MVC application that would have a WebAPI, this WebAPI will be responsible for receiving the ChannelURI from the ModernApplication that sends it to it, and will be stored on SQL Azure.
There will then be a class that has a static method which does the logic for gathering the new data and generating a new Tile/Badge/Toast.
I've created a simple Activity workflow, that has a Sequence with a DoWhile(true) activity. Inside the body of this DoWhile, contains a Sequence which has InvokeMethod and Delay, the InvokeMethod will call my class that contains the static method. The delay is set to one hour.
So that seems to be all okay. I then start this Activity via the Application_Start in Global.asax with the following line:
this.ActivityInvoker = new WorkflowInvoker(new NotificationActivity());
this.ActivityInvoker.InvokeAsync();
So I just tested it with that and it seems to be running my custom static method at the set interval.
That's all good, but now I have three questions in relation to this way of handling it:
Is this the correct/best approach to do this? If not, what are some other ways I should look into.
If a new instance is spun up on Azure, how do I ensure that the running Workflow for both instances won't step on each other's foot? i.e. how do I make sure that the InvokeMethod won't run two times, I only want it to run once an hour regardless of how many instances there are.
How do I ensure that if the instances crash/go-down that the state of it is maintained?
Any help, guidance, etc is much appreciated.
A couple of good questions that I would love to answer, however trying to do some on a forum like this would be difficult. But let's give it a crack. To start with at least.
1) There is nothing wrong with your approach for implementing a scheduled task. I can think of a few other ways of doing it. Like running a simple Worker Role with a Do{Thread.Sleep(); ...} simple, but effective. There are more complex / elegant ways too including using external libraries and frameworks for scheduling tasks in Azure.
2) You would need to implement some sort of Singleton type pattern in your workflow / job processing engine. You could for instance acquire a lease on a 1Kb blob record when your job starts, and not allow another instance to start etc.
For more detailed answers I suggest we take this offline and have a Skype call and discuss in detail your requirements. You know how to get hold of me via email :) look forward to it.
Related
I am a complete newbie for Azure and Azure Functions but my team plans to move to Azure soon. Now I'm researching how I could use Azure Functions to basically do what I would normally do in a .Net console application.
My question is, can Azure Functions handle quite a bit of code processing?
Our team uses several console apps that effectively pick up a pipe delimited file, do some business logic, update a database with the data, and log everything along the way. From what I've been reading so far I typically see that Azure Functions are used for little pieces of code. How little do they mean? Is it best practice to have a bunch of Azure Functions to replace a console app EX: have one function that does the reading of a file and create a list of objects, another function to loop through those items and add business logic, and then another to write the data to a database or can I use one Azure Function to do all of that?
Direct answer is yes - you can run bigger pieces of code as Azure Function - this is not a problem as long as you meet their limitations. You can even have dependency injecton. For chained scenarios, you can use Durable Functions. However, Microsoft do not recommend long running functions, cause of unexpected timeouts. See best practices for azure functions.
Because of that, I would consider alternatives:
If all what you need is run console app in Azure you can use WebJobs. Here is example how to deploy console app directly to azure via VisualStudio
For more complex logic you can use .NET Core Worker Service which behaves as Windows Service, and could be deployed to azure as App Service.
If you need long-running jobs but with scheduled runs only I had really great experience with Hangfire which can be hosted in Azure as well.
This is really hard to answer because we don't know what kind of console app you have over there. I usually try to use the same SOLID principles used to any class on my functions too. And whenever you need to coordenate actions or if you need to run things in parallel you always use Durable Functions Framework too.
The only concern is related to execution time. Your function cans get pretty expensive if you're running on consumption plan and do know pay attention to it. I recommend you the reading of the following gread article:
https://dev.to/azure/is-serverless-really-as-cheap-as-everyone-claims-4i9n
You can do all of that in one function.
If you need on-the-fly data processing, you can safely use Azure Functions even if it takes reading files or database communication.
What you need to be careful at and configure, though, is the timeout. Their scalability is an interesting topic as well.
If you need to host an application, you need a machine or a part of the storage space of a machine in Azure to do that.
I have just used MassTransit in my project, .Net core2.0. It is great but there are some concerns:
That is different between Publish vs Send. In my scenario, I have one email service to send email to out side. Other services will pass request to email service via RabbitMQ. So, in this case we should use "Publish" or "Send".
With Send, we need to pass the full URL of endpoint. There is any best practice to manage endpoint? Because if we have 10 commands, we need to manage 10 endpoints. Is it right?
Relate to event(Publish), if one service is deployed on multiple instances, when one event is published to queue. It will be processed one time or will be processed many times on each instance.
Could you please share me one unit test for consumer? Because with harness test, it seems we just ensure message was queued.
Masstransit is ready for .Net Core 2.1?
Many thanks,
There are way too many questions for one post tbh, on SO it is better
to ask more specific questions, one by one. Some of your questions already have answers on SO.
The difference between publishing events and sending commands is similar to what you expect. We actually cover some of it in the documentation.
You can handle as many message types as you want in one receive endpoint, but you need to be aware of the consequences. The best practice is to have one endpoint per command type or at least one endpoint for related commands. The risk here is that an important command might get stuck waiting in the queue until other, less important commands will be processed.
If you publish events, each endpoint (queue) will get a copy of it. If you have several instances of one endpoint, only one of those instances will get it. It is valid also for sending commands, but it will be only one endpoint that gets a message and only one of the instances will process it.
Although there is no documentation for MT testing just yet, you can look at this test to see how it is done.
MassTransit is compiled for .NET 4.6 and .NET Standard 2.0. There is nothing specifically different in .NET Core 2.1 that would have any effect on MassTransit.
I am working on creating a Web Job on Azure and the purpose is to handle some work load and perform server background tasks on my website.
My website has several Web API methods which are used by my website but I also want the Web Job to perform the same tasks as the Web API methods after they are finished.
My question is, should I get the Web Job to call this web API (if possible) or should I move the Web API code to a Class and have the Web API and also Web Job both call this class?
I just wondered what was normal practice here.
Thanks
I would suggest you put the common logic in a dll and have them both share that library instead of trying to get the webjob to call the webapi.
I think that will be the simple way to get what you want (plus it will help you keep things separated so they can be shared - instead of putting too much logic in your webapi).
I think it's players choice here. Both will run on the same instance(s) in Azure if you choose to deploy them that way. You can either reuse by dogfooding your API or reuse by sharing a class via a .dll. We started off mixed but eventually went with the dogfooding the API as the amount of Webjobs we are using got bigger/more complex. Here are a couple of reasons.
No coupling to the libraries/code used by the API
Easier to move the Web Job into its own solution(s) only dependent on the API and any other libraries we pick for it
Almost free API testing (Dogfooding via our own Client to the API)
We already have logging and other concerns wired up in the API (more reuse)
Both work though in reality, it really comes down to managing the dependencies and the size of app/solution is you are building.
I have a Node.js app with a small set of users that is currently architected with a single web process. I'm thinking about adding an after save trigger that will get called when a record is added to one of my tables. When that after save trigger is executed, I want to perform a large number of IO operations to external APIs. The number of IO operations depends on the number of elements in an array column on the record. Thus, I could be performing a large number of asynchronous operations after each record is saved in this particular table.
I thought about moving this work to a background job as suggested in Worker Dynos, Background Jobs and Queueing. The article gives as a rule of thumb that tasks that take longer than 500 ms be moved to background job. However, after working through the example using RabbitMQ (Asynchronous Web-Worker Model Using RabbitMQ in Node), I'm not convinced that it's worth the time to set everything up.
So, my questions are:
For an app with a limited amount of concurrent users, is it ok to leave a long-running function in a web process?
If I eventually decide to send this work to a background job it doesn't seem like it would be that hard to change my after save trigger. Am I missing something?
Is there a way to do this that is easier than implementing a message queue?
For an app with a limited amount of concurrent users, is it ok to leave a long-running function in a web process?
this is more a question of preference, than anything.
in general i say no - it's not ok... but that's based on experience in building rabbitmq services that run in heroku workers, and not seeing this as a difficult thing to do.
with a little practice, you may find that this is the simpler solution, as I have (it allows simpler code, and more robust code, as it splits the web away from the background processor - allowing each to run without knowing about each other directly)
If I eventually decide to send this work to a background job it doesn't seem like it would be that hard to change my after save trigger. Am I missing something?
are you missing something? not really
as long as you write your current in-the-web-process code in a well structured and modular fashion, moving it to a background process is not usually a big deal
most of the panic that people get from having to move code into the background, comes from having their code tightly coupled to the HTTP request / response process (i know from personal experience how painful it can be)
Is there a way to do this that is easier than implementing a message queue?
there are many options for distributed computing and background processing. i personally like RabbitMQ and the messaging patterns that it uses.
i would suggest giving it a try and seeing if it's something that can work well for you.
other options include redis with pub/sub libraries on top of it, using direct HTTP API calls to another web server, or just using a timer in your background process to check database tables on a given frequency and having the code run based on the data it finds.
p.s. you may find my RabbitMQ For Developers course of interest, if you are wanting to dig deeper into RMQ w/ node: http://rabbitmq4devs.com
Can anyone explain any reason why and when should I use PublishOnBackgroundThread instead of PublishOnUIThread.
I cannot find any use cases for usage PublishOnBackgroundThread and I am not sure what method should I use?
It really depends on the type of the message you're publishing.
If you're using the EventAggregator to surface a message from a low laying service back the UI then PublishOnUIThread makes the most sense since you'll be updating the UI when handling the message. The same applies when you're using it to communicate between view models.
Conversely sometimes it get used for view models to publish events that an underlying service is listening to (rather than the view model depending on that service).
That service may perform some expensive work which makes sense to happen on a background thread. Personally I'd have gone in the background service to push that work onto a background thread but different people want different options.
Ultimately the method was included for completeness.