Is it possible to define the order in which Azure roles should start?
I have a cache worker role which I'm obviously using for cache. I'm also using it as my session state provider.
When I start my solution in visual studio and it opens in a browser I get an error message saying the cache doesn't exist. If I make another request it works fine. This seems to be because the web role starts a fraction before the cache role it is trying to use.
It isn't really an issue in a live environment because Azure wont route requests to the application until all of the roles are ready however it's a bit annoying when I'm running locally.
Thanks
You can't specify role startup order. You can, however, keep your web role instances out of the load balancer until you determine the environment is up and running sufficiently.
In your OnStart(), you could put some code that tries getting something from the cache (which will likely fail if the web role instance comes up prior to the cache role). Or maybe ping the cache role instances (I'll leave that up to you, to determine best way of seeing that the cache role instances are up). Just keep retrying periodically until you have success, then return from OnStart(). At this point, the load balancer will start directing traffic to the role instances, and you should be in good shape.
Note: While your web role instances are unavailable, you'll see an http error since the site will be temporarily unavailable during startup, but you shouldn't see the cache error message.
More details around OnStart: here.
Related
I have an Azure service that has been running for a long period of time. It builds a word or powerpoint document based on arguments in the request and returns a uri to the build document. This is access via a visualforce page, when you click a button, it calls the service and displays a link to the document that has just been built. Simple.
All of a sudden, I get an apparently random 503 Service Unavailable error. Sometimes I click the button, no problem. Other times a 503 error. Each time the button triggers exactly the same request. Does anyone know why this might be happening?
Apparently doing the same thing over and over again and expecting a different result, is not insanity!
Thanks for taking the time to read this.
Looking at the monitoring on my service told me the processor was never exceeding 6% of usage, so it couldn't be a lack of resource causing these intermittent 503 errors. It's bizarre and I'm afraid I have no explanation for it, but simply redeploying the cloud service to Azure appears to have done the trick. It now works perfectly. The solution has not changed, so I can only imagine that whatever 'reboot' is necessary after deployment, has rectified whatever the problem was. All I can suggest is that you try the same thing if you are getting intermittent 503 errors.
For me the error went away when I set up auto-scaling. I think failover requests were getting routed to my second VM, and the second VM took some time to spin up because it wasn't ready for the activity. Auto-scaling shut down my second VM and the error no longer appears (I'm assuming it will spin up if/when I get enough traffic to use it).
Hope this also helps someone.
I get this error whenever I create an Azure Function with a storage account in the South Central US. If I use a storage account in a different region the function works.
Try a storage account in a different region than the one you are currently using to see if it resolves your issue.
503 error is simply shows that your application pool was inaccessible, it was intermittent because your application pool is restarting because the lack of resource (processor, memory, etc).
Scale up your instance (Cloud Services or VM) to get more resource for the application pool.
The company I work for is looking to develop a few apps against the cloud.
An ASP.NET Web Api application hosted in an Azure web role.
A Windows Server type application hosted in an Azure worker role.
We are completely new to web or cloud development and would like to know the following:
When being served to the consumer, is the same instance of the application being served to all, is it one per request or are multiple roles being created and served to consumers?
When being served to the consumer, is the same instance of the application being served to all?
That depends on how many instances you've asked Azure to run your application on. If you've only deployed to 1 instance, then it will of course be the same instance that responds to all requests. But if you deploy to multiple instances, requests will be load-balanced, which means you have no guarantee that multiple requests from the same user will be handled by the same instance.
When you're asking this question, it could be because you might be tempted to store local data on the machine running the instance. However, this is not a good idea. Windows Azure can at any time tear down your instance and start your application on a completely different machine. They call this "healing", because it usually happens because Windows Azure tries to be helpful and avoid any potential problem that could mean downtime for your instance. But it also happens if your machine for some reason locks up or something else bad happens. This process of healing means that anything that's not part of your deployment package will be lost. So for example, if you're logging to a file on the disk, this log will be lost if Azure "heals" your instance.
is it one per request or are multiple roles being created and served to consumers?
I'm not completely sure what you mean here, so I'll take a guess and risk interpreting your question wrongly. My guess is that you're asking if there will be one instance per user request. No, there will only be the number of instances that you have decided. Remember that you have to pay per instance that's running, so it's only fair that the number of instances running is dictated by you.
When you have your application packaged and ready to be deployed to Windows Azure, you can decide how many instances of each role you want to have running. You set this number in the deployment package, so that when your package is deployed, Azure will automatically start the requested number of instances. However, you can change the number of running instances of each role after deployment and on-the-fly. This makes it possible for you to scale with more instances within minutes.
I hope this helps and that I understood your questions correctly. :-)
Azure Web and Worker roles in an Azure Cloud Service are deployed on at least one instance (managed VM). Azure allows you to size (memory, CPU) and scale (number of instances). Azure actually lets you scale dynamically, i.e. add more instances on demand. You pay by the hour for the size & number of instances deployed.
For example, a Cloud service can have a single instance of a worker role (background processing) and multiple instances of the Web role. Multiple instances are handled behind a load balancer and the client is unaware of what instance they are using (all instances are created equal).
An Azure role instance is a VM with some specific payload.
For example, in your service you declare you want three instances of "Frontend" web role and two instances of "Backend" worker role. When Azure deploys your service it starts five VMs and three of them will run "Frontend" payload and have IIS started and two of them will run "Backend" payload and have no IIS started.
Now until you ask Azure to change that configuration it remains persistent no matter what requests come and what load occurs. You have five VMs with 3+2 configurations. To change the configuration you need some action on your part.
There're two way to have the configuration changed. You can use Management Portal or an external program to change the "instance count" in either or both roles. You can also add auto-scaling code that will gather metrics and make Management API requests to alter the "instance count". Either way when "instance count" goes up Azure starts more VMs with the same payload and when it goes down it stops some of the VMs.
I have a web service hosted on azure as a web role.
In this web role I override the Run() method and perform some db operations in the following order to basically act as a worker role.
go to blob storage to pull a small set of data.
Cache this data for future use.
Kill some blobs that are too old.
Thread.Sleep(900000);
Basically the operation repeats every 15 minutes in the background.
It runs fine when I run on DevFabric but when deployed the azure portal get stuck in a loop of stabilizing role and then preparing node.
Never actually starting up either instance.
I have enabled diagnostics and it isn't showing me anything to suggest there is a problem.
I'm at a loss for why this could be happening.
Sounds like an error is being thrown in the OnStart. Do you have any way of doing try/catch around the whole function and dumping the error into an EventViewer? From there you would be able to remote into the instance and investigate the error
Most likely your configuration deployed to cloud is different from the one running in an emulator (SQL Azure firewall permissions, pointers to Local Dev storage instead of ATS, etc). Also, make sure that your diagnostics is pointing to a real Azure account instead of local Dev storage.
I would suggest moving this code to Run(). In OnStart(), your role instance isn't yet visible to the load balancer, and since you're introducing a very long (ok, infinite delay) into OnStart(), this is likely why you're seeing the messages about the role trying to stabilize (more info on these messages are here.)
I don't typically like to answer my own question when someone else has made an effort to help me however I feel that the approach I used to solve this should be documented.
I enabled intellitrace when deploying to Azure and I was able to see all the exceptions being thrown and investigate the cause of the exceptions.
Intellisense was critical in solving my deployment issues. I would recommend it to anyone seeing an inconsistency between deploying devfabric and deploying to azure.
In short, is there a RoleEnvironment event that I can handle in code when any other role in my deployment is rebooted or taken offline for patching?
I've got an application in production that has both web roles for an web front end and web roles running WCF services as an application layer (business logic, data access etc). The web layer communicates with the WCF layer over an internal endpoint as we don't want to expose the services at this point in time. So this means it is not possible to use the load balancer to call my service layer through a single url.
So I have to load balance requests to the WCF web roles manually. This has caused problems in the past when a machine has been recycled by the fabric controller for patching.
I'm handling the RoleEnvironment.Changing and RoleEnvironment.Changed events to adjust the list of backend web roles I am communicating with, which works well in testing when I make a configuration change to increase or decrease the number of instances in my deployment. But if I reboot a role through the portal, this does not fire the RoleEnvironment events.
Thanks,
Rob
RoleEnvironment.Changing will be fired "before a change to the service configuration" (my emphasis). In this case no configuration change is occurring, your service is still configured to have exactly the same number of instances. AFAIK there is no way to know when your deployment is taken offline, and clearly their are instances where notice cannot be given in advance (e.g. hardware failure). Therefore you have to code for communication failure, intercept the error, and try another role instance.
I do not believe you can intercept RoleEnvironment changes from a different Role easily.
I would suggest that you have RoleEnvironment changes trapped in the Role where they occur, handle them by throwing a message/record onto some persisted storage and let your Web-roles check that storage either on a regular schedule or every-time when you communicate to the WCF-roles.
Basically, if you're doing your own internal load-balancing, you need a mechanism for registration/tear-down of your instances so that you can manage your wcf workers
You can use the Azure storage queues to post a message when a role is going down and have a worker role that listens on that queue and adjusts things accordingly.
I have read the following from Azure in Action book:
"In Windows Azure, the state server, or out-of-process session state provider,
isn’t supported"
Can anyone tell me why this is not supported. They do not explain in the book. I would have thought I could run the state server and all web roles would be able to use this to read session data.
Thanks for replies
Windows Azure has the AppFabric Cache, which is well-suited for session storage. This went live about 2 weeks ago. You can see details in my StackOverflow answers here and here.
I can't give you an absolute answer on the lack of state server, other than the fact that until recently, Web Roles ran under Hosted Web Core instead of Full IIS and that might have had an impact on offering state server. Additionally, since all of your web role instances are equivalent, how would you specify which is the state server? And what happens if Windows Azure recycles that instance? I think this would be a big headache.
With the AppFabric Cache solution, this is cache-as-a-service, completely independent of your role instances, and managed for you. You simply get an endpoint, attach to the cache, and execute puts and gets on name/value pairs. Through the Azure portal, there's a Client Configuration button you click, and a magic chunk of xml gets generated for you - you place that in web.config, and you now have a custom session state provider that stores/retrieves session data to/from cache.
EDIT: On using SQL Azure for session state: SQL Azure doesn't have a SQL Agent, so you wouldn't have a background job periodically clearing the cache. There's an MSDN blog post showing how to get SQL Azure working as a session state provider (and using a worker role background process to perform session cleanup). However, I'd still recommend the AppFabric Cache solution.