I have a website with a bunch of webjobs. The webjobs are continuous, but use Quartz.net to schedule internally. I'm using deployment slots to deploy my site to a staging site, which I then swap into production.
All works well, but I want to stop my webjobs from ever scaling out with my web app (i.e. not participate in auto-scale).
Now, I know I can create a settings.job file and set { "is_singleton": true } ... BUT ... in my testing, that breaks my deployments to my staging site - what happens when I deploy is that on my staging slot they all become stopped (presumably because my settings.job file prevents them from running). If I remove the settings.job file and deploy to my staging site again, this doesn't happen - they remain running.
How do I stop my webjobs from scaling out with auto-scale, without breaking the deployment slot swapping strategy?
Thanks
I re-tested my deployment with a settings.job file with is_singleton set, and it worked correctly - the webjobs deployed successfully and were running after the deployment had finished.
I'm not sure why this wasn't the case before - either something changed with Kudo, or perhaps I was confusing things by deploying my webjobs inconsistently, but in any case, I can report this is no longer an issue :)
Related
I have a web site running in azure app service.
And it has also web jobs deployed.
Now I am creating a new slot for my app service, and deploying the web site code to the new slot.
When I swap the slots, the web jobs still run in the production slot or would they be swap to the new slot?
Thanks!
Under the covers, Webjobs live in your App Service's site \ wwwroot \ App_data \ jobs \ continuous | triggered folder. A slot swap is a set of actions that are performed that ultimately end in a routing rule switch within Azure.
Your webjobs will participate in a slot swap for the App Service.
However, there are some interesting corner cases. For example, let's assume that your staging and production application settings are identical and you have triggers defined for something like a ServiceBusTrigger. Unless you stop your staging slot Webjobs, your staging Webjob instance triggers will continue to poll your production resources, meaning you can be executing both your production and slot Webjob code.
Webjob too will be swapped for sure. There's a config setting to stop all webjobs within a deployment slot (WEBJOBS_STOPPED = 1), but there isn't something to stop just the specific ones within a slot, as far as I know.
In azure you can have slots that you can deploy your web app to.
For example you can have Production and Staging slots.
You can deploy to Staging then test if everything is OK and if so swap Staging with Production slot.
I believe that behind the scenes those slots are just 2 sites in IIS.
My idea is to just swap bindings and rename the websites so the Staging slot/website will get http://prod-superapp.com binding and the Production slot/website will get http://stg-superapp.com binding then rename Production website to Staging and Staging to production.
The last step is needed so that when we deploy the next time we deploy to the Staging slot again.
Can you think of anything better than this? Do you find any problems with that approach that I am currently not aware of? I know that there can't be two site with the same binding running at the same time so that scheme will have some downtime while the whole process completes.
Let's say I have a webapp in Azure that auto deploys a continuous webjob. This webapp also has a production and staging slot, and the site deploys to staging, after which a swap happens (manually or automatic) to move the latest into the production slot.
Further, let's assume that the production webjob is running and processing and generally doing it's job before the swap begins. When the deployment to staging happens, the latest webjob does not start because of the WEBJOBS_STOPPED = 1 configuration setting that is honored.
However, after the swap completes, does the webjob that was previously in production and now is in staging continue running? If not, how does it stop? When does it stop? When does the WEBJOBS_STOPPED flag begin to be honored when a running webjob is moved from a production slot to a staging slot?
I can't have the two webjobs running at the same time and it's unclear what the behavior around a swap is... Continuous singletons are easy using the is_singleton property in the webjob settings, but "vertical" webjobs running at the same time around a swap doesn't appear to be clearly understood or documented.
The key is to make WEBJOBS_STOPPED=1 a slot setting on Staging, so that it is always set for the Staging and never for Production. This will guarantee that you don't have both running at the same time (other than potentially for a few seconds during swap transition).
See this post for some deeper information on how swapping works.
I have an Azure website with a production and staging slot with multiple instances.
I run several Azure webjobs on the site, some of which are triggered others that are continuous.
I understand triggered webjobs will only run once on one instance.
My current setup is:
Production site has no deployment other than swapping from staging
Staging website is setup to continuous deployment from bitbucket
Webjobs are deployed from within VS to production slot using "publish to Azure" (since AFAIK there is no support for continuous deployment of webjobs that are schedule based)
The VS solution for the website is different from the VS solution containing webjobs
I noticed when I swap production and staging, webjobs are also swapped!
I realized this when I had deployed a new webjob to production and subsequently did a website deployment followed by a swap, the newly deployed webjob was no longer in the production slot!
My questions:
is it possible to prevent webjobs from swapping between production and staging (webjobs are really entirely different pieces of code than the website)
What exactly is swapped for webjobs? Binaries? Schedule? Both ?
how do i prevent all webjobs from running on the staging slot? My webjobs do not need to be highly available and i can test them offline easily.
If i VS deploy webjobs to the staging slot and do a swap, then the current staging slot will be missing the deployed webjob and I'll lose it when i do my next website update, so where should i be deploying my webjobs?
There is a related thread but it assumes the webjob is deployed simultaneously with a website which is not what I have currently.
I really like websites and webjobs, but it seems the story related to continuous independent deployment of webjobs and websites is broken.
Would really appreciate advice!
Thanks
Azure websites deployment slots are not an easy concept to understand, together with WebJobs it gets a little bit more difficult.
I suggest reading the following post to get a better understanding on deployment slots - http://blog.amitapple.com/post/2014/11/azure-websites-slots/ (including the comments section for useful information)
To get a better understanding on how WebJobs work and deployed see this post - http://blog.amitapple.com/post/74215124623/deploy-azure-webjobs/
It is important to understand that:
When swapping deployment slots you actually swap the content/files of the website between the 2 slots.
WebJobs are part of the website content.
So when you swap the slot you actually swap the website files including the WebJob.
Note it is a bad practice to deploy a WebJob directly and not as part of your website files/repository, this is probably the cause of issues you are having.
To prevent WebJobs from running on the staging slot you can add an app setting called WEBJOBS_STOPPED and set it to 1 (in the azure portal). (source). Make sure this app setting is sticky to the slot otherwise it'll propagate to the production slot.
The by far easiest solution to this problem is to have a separate Web App for your Web Job. Just place the Worker Web App on the same Azure Service Plan and you will have two separate Web Apps running on the same machine(s).
web-app
web-app-worker
Here's Amit's answer to #3 using PowerShell,
Function Disable-AzureSlotJobs($SiteName, $Slot = 'staging')
{
$site = Get-AzureWebsite -Name $SiteName -Slot $Slot
$site.AppSettings.WEBJOBS_STOPPED = "1"
Set-AzureWebsite -Name $SiteName -Slot $Slot -AppSettings $site.AppSettings
Set-AzureWebsite -Name $SiteName -SlotStickyAppSettingNames #("WEBJOBS_STOPPED")
}
'MySite1', 'MySite2', 'MySite3' | % { Disable-AzureSlotJobs $_ }
We have an app we deploy to Azure. It involves deploying several cloud services with a mix of web roles and worker roles. Some of the worker roles pick items up off a queue and process them. We also have some scheduled jobs that run periodically (backing up Azure table storage, etc).
Every time we deploy, we have to watch the Staging environment in the portal and manually stop the roles from starting. We have to do this because we don't want the Staging and Production slots both processing information at the same time (e.g. pulling from the same queue but processing it differently, or both running the same scheduled job simultaneously, etc).
The only way I've found to have a deployment go into Staging in a stopped state is to leave the last deployment there also stopped. Downside is you're charged for those instances, even when they're not running.
So, how do you deploy to an empty staging slot in Azure without the deployment starting up?
EDIT: We kick off the builds through Visual Studio or Visual Studio Online (i.e. TFS). Don't usually use powershell.
There is no way to create a deployment but not have it start. What you can do instead is have a setting in your csdef/cscfg that your code would read during OnStart.
For example, you would have a setting called "ShouldRun" set to False. In OnStart you would have a loop that checked that setting and exits the loop if ShouldRun==True. After you deploy you would then go to the portal and change that setting to True whenever you are ready for it to start processing. Once the loop exits then the OnStart method will finish which will cause Azure to call your Run method and bring your instances to the Ready state.
In addition you could add a Changed event handler to stop processing messages when the setting was changed to False. This would let you first stop your production deployment and then start your staging deployment.
For me, you need to separate even your queue's and configs. Another option, you can create a powershell script to stop your cloud service after publish to it.
http://msdn.microsoft.com/en-us/library/dn495211.aspx