Given: One Worker role + several Quartz.net jobs. Quartz jobs are host-agnostic and are executed in Worker role.
A worker role can be scaled to multiple instances.
The Goal is: have the ability to define what job to run in what instance at runtime (or define it with configuration only, no code changes). For example:
MyRole Instance 1: Job1, Job2, Job3
MyRole Instance 2: Job4,
MyRole Instance 3: Job4,
MyRole Instance 4: Job4,
MyRole Instance 5: Job5, Job6
In my example Job4 receives a lot of load. So I'd like it to run on more instances. And I also want it to be scalable at runtime (or at least via configuration, w/o code changes).
AFAIK it is not possible to have azure configuration per instance (only per role itself). Search online on similar issues haven't given any results.
Question: Did anybody have similar situation? What would be the best approach? Any other design advice is very appreciated. Thanks.
This is more of an architectural problem then one specific to Azure. The most common solution is to set up a "broker" that each instance/process reaches out to and asks for its individual workload. The challenge, regardless of the platform you are deploying the solution is how to a) identify the broker and b) ensure the "state" information being managed by the broker is persisted in case the broker process dies.
The most common approach to address these concerns in Azure is the use of a blob with a lease on it that allows the broker to be 'self elected' (the first process to get the lease is the broker), and stores both the address of the broker and the broker's state (metadata stored within the blob). You then put the logic for assigning jobs into this broker in a way that best suits your task distribution needs.
Related
I have a web role with multiple instances. I'd like to send an email to my customers with their statistics every morning at 7AM.
My problem is the following: if I use a Cron job to do the work mails will be sent multiple times.
How can I configure my instances in order to send only one email to each of my customers?
Per my experience, I think you can try to use a unique instance id to be ensure there is a single instance works for emailing as cron job.
Here is a simple code.
import com.microsoft.windowsazure.serviceruntime.RoleEnvironment;
String instanceId = RoleEnvironment.getCurrentRoleInstance().getId();
if("<instace-id-for-running-cronjob>".equals(instanceId)) {
// Coding for cron job
.....
}
As reference, please see the description of the function RoleInstance.getId below.
Returns the ID of this instance.
The returned ID is unique to the application domain of the role's instance. If an instance is terminated and has been configured to restart automatically, the restarted instance will have the same ID as the terminated instance.
More details for using the class above from Azure SDK for Java, please see the list of classes below.
RoleEnvironment
RoleInstance
Hope it helps.
Typically the way to solve this problem with Multi-instance Cloud Services would be to use Master Election pattern. What you would do is at 7:00 AM (or whenever your CRON job will fire), all your instances will try to acquire lease on a blob. However only one instance will be successful in acquiring the lease while other instances will fail with PreConditionFailed (419) error (make sure you catch this exception!). The instance that acquires the lease is essentially a Master and that instance will send out email.
Obviously the problem with this approach is that what if this Master instance fails to deliver the messages. But I guess that's another question :).
I would like to have a Node.js app running in multiple instances in Cloud Foundry.
The app has - among others - a scheduler ("node-schedule": "^0.2.7") which writes a status to a mongo-db once every minute. Now with multiple instances of the app, I get multiple entries - for each instance one - in the database.
I'm looking for something to synchronize instances to ensure just one instance "gets the lock" to write to the database once every minute. I already found suggestions to use singleton frameworks or middleware but they are request-centric, not "from the inside" of the app - the scheduler.
Is there a solution for this?
Each application instance will have an environment variable set, CF-INSTANCE-INDEX, to expose the instance index. You could make only the instance with index '0' responsible for scheduling logs entries.
http://docs.run.pivotal.io/devguide/deploy-apps/environment-variable.html#CF-INSTANCE-INDEX
I am looking for a way to have a "Singleton" module over multiple worker role instances.
I would like to have a parallel execution model with Queues and multiple worker roles in Azure.
The idea is that would like to have a "master" instance, that is let's say checking for new data, and is scheduling it by adding it to a queue, processing all messages from a special queue, that is not processed by nobody else, and has mounted blob storage as a virtual drive, with read/write access.
I will always have only one "master instance". When that master instance goes down for some reason, another instance from the one already instantiated should very quickly be "elected" for a master instance (couple of seconds). This should happen before the broken instance is replaced by a new one by the Azure environment (about 15 min).
So it will be some kind of self-organizing, dynamic environment.
I was thinking of having some locking, based on a storage or table data. the opportunity to set lock timeouts and some kind of "watchdog" timer if we can talk with microprocessor terminology.
There is general approach to what you seek to achieve.
First, your master instance. You could do your check based on instance ID. It is fairly easy. You need RoleEnvironment.CurrentRoleInstance to get the "Current instance", now compare the Id property with what you get out of RoleEnvironment.CurrentRoleInstance.Role.Instances first member ordered by Id. Something like:
var instance = RoleEnvironment.CurrentRoleInstance;
if(instance.Id.Equals(instance.Role.Instances.OrderBy(ins => ins.Id).First().Id))
{
// you are in the single master
}
Now you need to elect master upon "Healing"/recycling.
You need to get the RoleEnvironment's Changed event. Check if it is TopologyChange (just check whether it is topology change, you don't need the exact change in topology). And if it is Topology Change - elect the next master based on the above algorithm. Check out this great blog post on how to exactly perform events hooking and change detection.
Forgot to add.
If you like locks - blob lease is the best way to acquire / check locks. However working with just the RoleEnvironment events and the simple master election based on Instance ID, I don't think you'll need that complicated locking mechanism. Besides - everything lives in the Queue until it is successfully processed. So if the master dies before it processes something, the "next master" will process it.
in my service deployment i have two roles.. a web role and a worker role.. in my on_start() method of webrole im enumerating the instances of the worker roles and creating a tcp connection on some internal end point. but very often it fails because the instances of the worker role havent started yet.
so the question is that can i know wether the instances have started or can i wait for instances of the worker role to start in some way?
herez the code
public override bool OnStart()
{
// For information on handling configuration changes
// see the MSDN topic at http://go.microsoft.com/fwlink/?LinkId=166357.
ConnectionStatics.ConnectRouterToWorkers();
Messaging.KeepReadingMessages.Start();
return base.OnStart();
}
I'd recommend building retry logic into your loop so when its unable to establish the connection, it just sleep and retries it again later. Not only will this address your startup issue, but it will help you address changes to the service topology as instances are added/removed by the fabric controller (which can happen for numerous reasons).
Taken a step further, you might be able to leverage the RoleEnvironmentChanging and RoleEnvironmentChanged events to provide notification of when an instances is added/dropped. But I haven't leveraged this personally and can't say with any certain how these methods may or may not reflect the "ready state" of particular instances.
Another option would be to have the worker roles put a message in a queue when they are started up. Then you could just check the queue and wait for it to post a message there.
I have two instances of a worker role.
I want to run a sub-task (on a Thread Pool thread) only on one of the Worker Role instances.
My initial idea was to do something like this:
ThreadPool.QueueUserWorkItem((o) =>
{
if (RoleEnvironment.CurrentRoleInstance.Id == RoleEnvironment.Roles[RoleEnvironment.CurrentRoleInstance.Role.Name].Instances.First().Id)
{
emailWorker.Start();
}
});
However, the above code relies on Role.Instances collection always returning the instances in the same order. Is this the case? or can the items be returned in any order?
Is there another approved way of running a task on one role instance only?
Joe, the solution you are looking for typically rely on:
either acquiring on lease (similar to a lock, but with an expiration) on a specific blob using the Blob Storage as a synchronization point between your role instances.
or queuing / dequeuing a message from the Queue Storage, which is usually the suggested pattern to delay long running operations such as sending an email.
Either ways, you need to go through the Azure Storage to make it work. I suggest to have a look at Lokad.Cloud, as we have designed this open-source framework precisely to handle this sort of situations.
If they need to be doing different things, then it sounds to me like you don't have 2 instances of a single worker role. In reality you have 2 different worker roles.
Especially when looking at the scalability of your application, processes need to be able to run on more than one instance. What happens when that task that you only want to run on one role gets large enough that it needs to scale to 2 or more role instances?
One of the benefits of developing for Azure is that you get scalability automatically if you design your app properly. If makes you work extra to get something that's not scalable, which is what you're trying to do.
What triggers this task to be started? If you use a message on Queue Storage (as suggested by Joannes) then only one worker role will pick up the message and process it and it doesn't matter which instance of your worker role does that.
So for now if you've got one worker role that's doing the sub task and another worker role that's doing everything else, just add 2 worker roles to your Azure solution. However, even if you do that, the worker role that processes the sub task should be written in such a way that if you ever scale it to run more than a single instance that it will run properly. In that case, you might as well stick with the single worker role and code for processing messages off the Queue to start your sub task.