We have an app we deploy to Azure. It involves deploying several cloud services with a mix of web roles and worker roles. Some of the worker roles pick items up off a queue and process them. We also have some scheduled jobs that run periodically (backing up Azure table storage, etc).
Every time we deploy, we have to watch the Staging environment in the portal and manually stop the roles from starting. We have to do this because we don't want the Staging and Production slots both processing information at the same time (e.g. pulling from the same queue but processing it differently, or both running the same scheduled job simultaneously, etc).
The only way I've found to have a deployment go into Staging in a stopped state is to leave the last deployment there also stopped. Downside is you're charged for those instances, even when they're not running.
So, how do you deploy to an empty staging slot in Azure without the deployment starting up?
EDIT: We kick off the builds through Visual Studio or Visual Studio Online (i.e. TFS). Don't usually use powershell.
There is no way to create a deployment but not have it start. What you can do instead is have a setting in your csdef/cscfg that your code would read during OnStart.
For example, you would have a setting called "ShouldRun" set to False. In OnStart you would have a loop that checked that setting and exits the loop if ShouldRun==True. After you deploy you would then go to the portal and change that setting to True whenever you are ready for it to start processing. Once the loop exits then the OnStart method will finish which will cause Azure to call your Run method and bring your instances to the Ready state.
In addition you could add a Changed event handler to stop processing messages when the setting was changed to False. This would let you first stop your production deployment and then start your staging deployment.
For me, you need to separate even your queue's and configs. Another option, you can create a powershell script to stop your cloud service after publish to it.
http://msdn.microsoft.com/en-us/library/dn495211.aspx
Related
I have a continuous WebJob deployed via the Kudu API to /api/continuouswebjobs/{job name} as outlined here. The App Service I'm deploying to has just a single instance running.
On occasion it's looking like the previous version of a WebJob is not removed following a deployment and it runs concurrently with the new version. This can continue to occur and I've seen up to nine processes running concurrently.
Application names redacted but they are identical for each process:
Stopping the WebJob via the Azure Portal causes the process under w3wp (scm) to stop but not the other "detached" processes.
The company I am working with has seen this with different WebJobs across different App Service Plans. We've also confirmed this outside of Azure with our own logs, where we see multiple versions of the same application running.
Update 1
I couldn't reproduce the issue using David's steps from the comments. I did however revert back to our old release process which stopped and started the App Service pre and post release. I've since realised this wasn't needed as WebJobs continue to run under scm even when the App Service is stopped.
However...
Post-deploy while the WebJob is at state "Starting":
Post-deploy while the WebJob is at state "Running":
I've created a dummy site named "dummy739047" on the same App Service Plan as per the Kudu docs if a further investigation is needed.
I have an azure wep app, with a couple of web jobs running inside it. I noticed one of these webjobs was running old code. It is also impossible to remove it, it keeps running no matter what I do.
I have tried:
* Redeploying the code.
* Stopping the webjob
* Stopping the web app
* Deleting the web job.
Even after deletion, the webjob is still running, and consuming from its queue.
There is no web app anywhere that is running a webjob with that old code anymore.
I have a suspicion that the webjob is running inside a deployment slot, but all deployment slots were deleted, so I do not know how to confirm this.
On the kudu dashboard, the webjob is not present in the process explorer.
How do I get rid of this rogue webjob?
You could try to get specific job by name and check job runs history via WebJobs API, which could help you check whether job is indeed deleted/stopped. If the WebJob is deleted from current website, please make sure whether any other WebJob running on your other websites are consuming the queue. In addition please make sure whether Azure Functions are being used. Of cause, the tasks outside of Azure could consuming queue is also possible. If you worry someone is performing your storage maliciously, as you said, you could try to regenerate your storage account keys, but you may need to re-sync the access keys with your applications/services that are dependent on the storage account.
Right. So this article: https://azure.microsoft.com/en-us/documentation/articles/web-sites-create-web-jobs/ mentions that you can "You can run programs or scripts in WebJobs in your App Service web app in three ways: on demand, continuously, or on a schedule. There is no additional cost to use WebJobs."
Which is great, the free-alternative is a Scheduler Job Collection
with a job, but you're limited to running it every hour. So being able to run the webjob as part of the webapp, and on a higher frequency is what we need.
However, I'm really struggling to find any way of automating this process. Using the Azure portal to add a web job works fine - but the "automation script" generation tool doesn't generate a json file which includes anything about the webjob - so we'd always have to manually create it.
There are examples of custom templates for automating the creation of webjobs - but they all create said webjob as part of a Scheduler Job Collection, where we are limited to the hourly execution.
To summarise: I'm looking for a way of automating the creation of a webjob, linked to a web-app (such that it doesn't incur extra costs).
Any help would be much appreciated.
WebJobs are deployed by folder convention (as described here), so deploying a WebJob is no different from deploying a Web App. It's simply a matter of getting the files in the right place.
Specifically, triggered WebJobs (manual or scheduled) go under site\wwwroot\app_data\jobs\triggered\{job name} and continuous WebJobs go under site\wwwroot\app_data\jobs\continuous\{job name}.
I have a continuous webjob that needs to run two webapps in two locations on a TimerTrigger. When I deploy the webjob from Visual Studio to both locations everything works well and both webjobs run at the same time.
Now I'm ready to start deploying this with Octopus-Deploy. I have successfully created a plan with two steps that does that and puts the assemblies in the correct location under the web apps (app_data\jobs\continuous\{jobname}) in Azure. The problem is that only one webjob executes its job at a time even though both webjobs have a status of Running. If I stop and start the one that's executing, the other webjob starts executing its job while the one I turned off/on has a status of running, but doesn't ever execute its job. Also, if I redeploy just one of them from visual studio, they both execute their jobs at both locations again.
I'm not doing anything with Singletons and have actually tried turning it off by using a 'settings.job' file with {is_singleton: false}. Is there something Octopus is doing with the package that makes Azure think the webjob is a singleton?
My guess is that the issue is caused by using the same storage account and host id for both of the apps that you deployed. When you do that, the WebJobs SDK views it as a single Web App that has been scaled out to two instances, and makes sure the timer is only run on one of them.
The simplest solution is to use a different storage account for each app.
I have a website with a bunch of webjobs. The webjobs are continuous, but use Quartz.net to schedule internally. I'm using deployment slots to deploy my site to a staging site, which I then swap into production.
All works well, but I want to stop my webjobs from ever scaling out with my web app (i.e. not participate in auto-scale).
Now, I know I can create a settings.job file and set { "is_singleton": true } ... BUT ... in my testing, that breaks my deployments to my staging site - what happens when I deploy is that on my staging slot they all become stopped (presumably because my settings.job file prevents them from running). If I remove the settings.job file and deploy to my staging site again, this doesn't happen - they remain running.
How do I stop my webjobs from scaling out with auto-scale, without breaking the deployment slot swapping strategy?
Thanks
I re-tested my deployment with a settings.job file with is_singleton set, and it worked correctly - the webjobs deployed successfully and were running after the deployment had finished.
I'm not sure why this wasn't the case before - either something changed with Kudo, or perhaps I was confusing things by deploying my webjobs inconsistently, but in any case, I can report this is no longer an issue :)