I have a continuous WebJob deployed via the Kudu API to /api/continuouswebjobs/{job name} as outlined here. The App Service I'm deploying to has just a single instance running.
On occasion it's looking like the previous version of a WebJob is not removed following a deployment and it runs concurrently with the new version. This can continue to occur and I've seen up to nine processes running concurrently.
Application names redacted but they are identical for each process:
Stopping the WebJob via the Azure Portal causes the process under w3wp (scm) to stop but not the other "detached" processes.
The company I am working with has seen this with different WebJobs across different App Service Plans. We've also confirmed this outside of Azure with our own logs, where we see multiple versions of the same application running.
Update 1
I couldn't reproduce the issue using David's steps from the comments. I did however revert back to our old release process which stopped and started the App Service pre and post release. I've since realised this wasn't needed as WebJobs continue to run under scm even when the App Service is stopped.
However...
Post-deploy while the WebJob is at state "Starting":
Post-deploy while the WebJob is at state "Running":
I've created a dummy site named "dummy739047" on the same App Service Plan as per the Kudu docs if a further investigation is needed.
Related
I have an Azure WebJob that is running on a pretty simple App Service Standard: 1 Small (plan).
Now, there was a problem with my WebJob (which has 5 functions running) - I wanted to stop 4 of the 5 running functions. I couldn't see how, so I thought I'd just stop the entire WebJob instead.
So I ended up just stopping the App Service.
Great, that was stopped, but the WebJob was still running and I didn't notice that until a few hours later.
I later noticed that the WebJob blade has its own Start/Stop button.
I thought the WebJobs were tied directly to the App Service.
What is the connection between the two? How does one affect that other?
As far as I know, the web app and web jobs has different process in the azure.
You could find the Process explorer in the web portal as below:
So the stop in the web app will just closed the app's process(6584 as the image shows).
It will not stop the webjob's process(4648).
These two processes has no connection.
If you want to stop both web app process and webjobs process, I suggest you could try to use this way Full stopping a Web App.
In our current setup there are 2 slots for an Azure App service, the question that I have is, when the slot is stopped would it also shut down the webjob that is deployed on that slot or would the web job continue to run?
It is possible to run an Azure WebJob even when the Azure App Service Web App is not running.
Though you can force the WebJob not to run or stop when the Web App is running using WEBJOBS_STOPPED setting in your Web App's "Application Settings"
Take a look at this article about this subject for more details:
https://blogs.msdn.microsoft.com/benjaminperkins/2017/03/01/failed-to-run-webjob/
The following picture shows a running WebJob when the web app is not running:
https://github.com/projectkudu/kudu/wiki/Full-stopping-a-Web-App
I have actually deleted some running jobs, but even though my jobs were shown as deleted in the UI, they are silently running and consuming the memory. I observed this from availability and Performance > Memory Analysis dashboard under Memory Drill Down > Memory Usage (App) > Physical Memory Usage section. Shocked to see the jobs still running after they are deleted. Luckily this solution helped me kill all the processes.
I have a continuous webjob that needs to run two webapps in two locations on a TimerTrigger. When I deploy the webjob from Visual Studio to both locations everything works well and both webjobs run at the same time.
Now I'm ready to start deploying this with Octopus-Deploy. I have successfully created a plan with two steps that does that and puts the assemblies in the correct location under the web apps (app_data\jobs\continuous\{jobname}) in Azure. The problem is that only one webjob executes its job at a time even though both webjobs have a status of Running. If I stop and start the one that's executing, the other webjob starts executing its job while the one I turned off/on has a status of running, but doesn't ever execute its job. Also, if I redeploy just one of them from visual studio, they both execute their jobs at both locations again.
I'm not doing anything with Singletons and have actually tried turning it off by using a 'settings.job' file with {is_singleton: false}. Is there something Octopus is doing with the package that makes Azure think the webjob is a singleton?
My guess is that the issue is caused by using the same storage account and host id for both of the apps that you deployed. When you do that, the WebJobs SDK views it as a single Web App that has been scaled out to two instances, and makes sure the timer is only run on one of them.
The simplest solution is to use a different storage account for each app.
We have a bug in our WebJob running in our live environment, I have identified the bug and fixed it, this I can verify in our Dev-environment. I published my WebJob as a "Azure WebJob" to our live environment but the bug i still present. To add to the confusion the bug now just occurs sometimes. So for some reason the old code is running somewhere sometimes.
Can someone please help me understand this?
I had a similar problem. We deploy using a stage environment in Azure and it turned out that the "old" WebJobs (running code with an old version of the entity framework model) where still running on the queue. These jobs where then fetching messages and consuming them. To add to the problem the exception was consumed in a try catch and the status of the WebJob was success.
Check if you have a stage environment (add -stage to the Webapp name) and if so go in to the Azure management portal and stop them.
Note, it is not enough to stop the Webapp, you must stop the WebJobs directly. This is done (in the new portal) under Settings->WebJobs and then right-clicking on the webjobs name selecting stop.
I spent ages looking into this problem. Turns out I had the web job project running in a console on my PC at work! No matter what I did on Azure the presence of this exe running and using the same storage for the web jobs meant that the old code running on my work PC picked up the jobs before Azure did. Easy fix: just make sure no exes are running outside of Azure!
In our case the web app was published to physical path /site/www instead of the default /site/wwwroot, because of this the Azure web portal interface adds the WebJobs to folder /site/jobs, but the webdeploy via VS or Azure are still trying to publish the webjobs inside the /site/www.
More details at Publishing WebJobs with Azure Pipelines
We have an app we deploy to Azure. It involves deploying several cloud services with a mix of web roles and worker roles. Some of the worker roles pick items up off a queue and process them. We also have some scheduled jobs that run periodically (backing up Azure table storage, etc).
Every time we deploy, we have to watch the Staging environment in the portal and manually stop the roles from starting. We have to do this because we don't want the Staging and Production slots both processing information at the same time (e.g. pulling from the same queue but processing it differently, or both running the same scheduled job simultaneously, etc).
The only way I've found to have a deployment go into Staging in a stopped state is to leave the last deployment there also stopped. Downside is you're charged for those instances, even when they're not running.
So, how do you deploy to an empty staging slot in Azure without the deployment starting up?
EDIT: We kick off the builds through Visual Studio or Visual Studio Online (i.e. TFS). Don't usually use powershell.
There is no way to create a deployment but not have it start. What you can do instead is have a setting in your csdef/cscfg that your code would read during OnStart.
For example, you would have a setting called "ShouldRun" set to False. In OnStart you would have a loop that checked that setting and exits the loop if ShouldRun==True. After you deploy you would then go to the portal and change that setting to True whenever you are ready for it to start processing. Once the loop exits then the OnStart method will finish which will cause Azure to call your Run method and bring your instances to the Ready state.
In addition you could add a Changed event handler to stop processing messages when the setting was changed to False. This would let you first stop your production deployment and then start your staging deployment.
For me, you need to separate even your queue's and configs. Another option, you can create a powershell script to stop your cloud service after publish to it.
http://msdn.microsoft.com/en-us/library/dn495211.aspx