we had a strange issue a couple of times now with our Azure service.
We have a cloud service installed that has a web application running on it.
The service was created sometime around beginning of december and he first deployment was done that time as well. After that we did multiple deployments to the cloud service, but (it happens a couple of times now) sometime the Azure decides to roll back the deployment to initial one - the one that was made 2 months ago. This happened this midnight once again and we see that the files creation date on the "restored" or "rolled back" instance is 12/5/2013, which seems to be the date when we did an initial deployment.
A question:
1) Why does that happen?
2) How can we determine what caused this rollback?
3) How can we prevent the rollback?
or
4) How can we make a "snapshot" of the cloud service so when the Rollback happens, it actually rolls back to the latest stable image?
Thanks,
Denis
How are you doing "After that we did multiple deployments to the cloud service"? Are you doing this via WebDeploy or via RDP to the Azure VM?
PaaS cloud service VMs are stateless. The code that is running your website will frequently be rebuilt from the original .cspkg that was uploaded. See http://blogs.msdn.com/b/kwill/archive/2012/10/05/windows-azure-disk-partition-preservation.aspx for a bit more info.
If you want to make changes to your webrole then you need to upload a new cspkg. See http://msdn.microsoft.com/en-us/library/windowsazure/hh472157.aspx for more information.
If you are deploying via WebDeploy you should know that these changes are only intended for development/testing cycle and that the changes are only temporary. See http://msdn.microsoft.com/en-us/library/windowsazure/ff683672.aspx for more information, in particular the "For development and testing purposes only" section.
Related
I'm working on a project that utilises Azure DevOps for building our .net core application suite.
Last couple of days we've noticed the builds are being queued for hours, we've paid for additional build time on the Azure Hosted Build Agents but there's just such a wait for the builds.
It seems to be random as sometimes the builds happen right away, we don't have any Azure support plan so I thought i'd ask the question if anyone is experiencing similar issues.
This is most likely due to a temporary outage that occurred recently and is now showing as resolved: https://status.dev.azure.com/_event/231083118
You'll need to establish first that you're running into a concurrency limit - fortunately, there are now analytics for that in preview in Azure DevOps:
First, enable the feature (bottom of the screen shot):
Then, go to Project Settings -> Agent Pools -> Azure Pipelines -> Analytics
It would seem to me that if you're not crossing the "Concurrency" line when this is occurring, you could open a ticket with Microsoft. If you are crossing the line, you would need to determine whether to purchase more parallel jobs, or to self-host.
I am new to Microsoft Azure and working on a MVP for a client. I am able to successfully deploy my ASP.NET MVC web app to a Web Role. The issue I'm facing is that Azure Web Role seems to recycle itself and revert to the deployment package. This leads to rollback of manual changes done (if any) after the deployment (like I allow user to upload profile images and am storing them in file system).
Is this the expected behavior? If not, when can I look from debugging purposes?
Looking for any help possible.
Is this the expected behavior? If not, when can I look from debugging
purposes?
It is expected behavior. Azure Fabric Controller is responsible for managing the Virtual Machines in which your WebRole runs. It takes care of patching the VM and do other maintenance related work on that VM. It is also responsible for taking out faulty VMs and replace them with brand new VMs. When the replacement happens, Fabric Controller will create the new VM based on the last deployment package.
You should not be making any changes manually after VM hosting WebRole has been created, these are Stateless VMs. Any changes you make to VM will be rolled back if Fabric Controller decides to recycle the VM. Please note that these changes do persist if the VM is simply rebooted.
Sometimes in our website which is deployed on Azure web roles, issue comes related to small bugs in javascript and HTML. We go to all instances of webroles and fix these JS and HTML file on machines.
But I was looking into some automated way of doing this, downloading the files to patch from some central location and replace the files in all azure web roles. I am using ASP .net MVC for website.
It is possible to redeploy the website with the patch in the package but we don't want to wait for long deployment time. Please let me know if it is possible via some internal WEB API which replaces the content on all azure web roles.
There are 2 ways to deploy a new webrole:
redeploy
inplace update
The first one is the slowest, meaning new VM's are booted.
With inplace upgrade (https://azure.microsoft.com/en-us/documentation/articles/cloud-services-update-azure-service/)
The new application package is mounted on a new drive (usually F: instead of E:) and the IIS website is swapped to the new drive.
You can try this by going to the old portal and upload a new application package. In just a few seconds/minutes the update is done.
After digging many things on stackoverflow, I crafted my own solution which is creating a topic and subscribing to the topic in code when website starts. When I want to patch the web app then I send a message to Topic to start patching then each machine in the web roles will get notification from topic and start patching themselves. Patching itself is very easy, which is going to a web storage and downloading files from there and replacing files in approot.
When azure maintenance happens this patching may go away, so for this situation I made patching work started at start up of website too.
Cloud service deployment packages tend to be slow since they are basically a recipe on how to build and configure your deployment. The deployment not only puts the recipe out in Azure (so it can be used again if it needs to move your machine), but also follows the recipe to build out a VM for your Cloud Service (WebRoles/WorkerRoles are platform as a service so you don't have to worry about the OS and infrastructure level like you would if you were using the Virtual Machine Azure product but they do still run in VMs on physical hardware).
What you are looking to do is something that will update the recipe (your cloud service package) and your deployment after it is out and running already ... there is no simple way to do that in Cloud Services.
However, yes you could create a startup script that could pull the site files from blob storage or some other centralized location - this would compare to how applications (fiddler for example) look for updates then know how to update and replace themselves. For that sort of feature you will likely need to run code as an elevated user - one nice thing about startup scripts are they can run as an elevated user - so they can do about anything you need done on a machine (but will require you to restart the instance for them to run). Basically you would need to write some code that will allow your site to update itself. This link may help: https://azure.microsoft.com/en-us/documentation/articles/cloud-services-startup-tasks/
If you have the ability to migrate to WebApps and WebJobs, I would recommend looking into that since that compute product solves your problem really well.
Here is a useful answer of the differences between WebApps and Cloud Services: What is the difference between an Azure Web Site and an Azure Web Role
We have a bug in our WebJob running in our live environment, I have identified the bug and fixed it, this I can verify in our Dev-environment. I published my WebJob as a "Azure WebJob" to our live environment but the bug i still present. To add to the confusion the bug now just occurs sometimes. So for some reason the old code is running somewhere sometimes.
Can someone please help me understand this?
I had a similar problem. We deploy using a stage environment in Azure and it turned out that the "old" WebJobs (running code with an old version of the entity framework model) where still running on the queue. These jobs where then fetching messages and consuming them. To add to the problem the exception was consumed in a try catch and the status of the WebJob was success.
Check if you have a stage environment (add -stage to the Webapp name) and if so go in to the Azure management portal and stop them.
Note, it is not enough to stop the Webapp, you must stop the WebJobs directly. This is done (in the new portal) under Settings->WebJobs and then right-clicking on the webjobs name selecting stop.
I spent ages looking into this problem. Turns out I had the web job project running in a console on my PC at work! No matter what I did on Azure the presence of this exe running and using the same storage for the web jobs meant that the old code running on my work PC picked up the jobs before Azure did. Easy fix: just make sure no exes are running outside of Azure!
In our case the web app was published to physical path /site/www instead of the default /site/wwwroot, because of this the Azure web portal interface adds the WebJobs to folder /site/jobs, but the webdeploy via VS or Azure are still trying to publish the webjobs inside the /site/www.
More details at Publishing WebJobs with Azure Pipelines
Just getting used to VS2012 publishing of Cloud Services. At present I have a one instance webrole which contains a MVC3 application. I can publish it to Azure without issue, and it creates the Cloud Service>Web Role>VMs. Fine. Takes a little while.
However when I do a little code change how can I migrate just this code change without replacing all the VMs that implement the WebRole etc.
It seems that Code and infrastructure are inseparable, or have I misunderstood. Is there a way to just update the code bit?
Thanks.
When you roll out an update, you upload an entire package containing not only your code files, but also the configuration for the VM, such as # of instances, ports to open on the firewall, local resources to allocate, etc. These configuration settings are part of the code package - so there is more going on than just updating code files.
However, there are a couple of methods you can use to have more granular control over updates.
Use Web Deploy. One thing to keep in mind, is that any automatic service updates will restore your website to the last fully-deployed package, which may not be as up-to-date. You would only want to use this in staging, then do a full package update for production rollout.
Use an Azure Web Site instead, which allows continuous integration with your source control provider, and direct updates to the code.
Use an Iaas VM instead. These are basically the same as running your own custom server in the Azure cloud, and you have full control over the OS. However, you also have full responsibility for keeping the OS updated and secure.
You can also enable RDP to your Azure Web Role VM's. You will find all your code files there and IIS, but I wouldn't recommend updating your code this way for the same reasons listed in #1.
The code and infrastructure, in a cloud service, are actually separate. All you upload is a deployment package containing just your code and supporting libraries / files. You don't upload a vhd. Azure provides that for you, spinning up a vhd, and then accessing your code on a file folder on that vhd. Same process happens each time you scale out to more instances.
when you make a code change, you build a new deployment package and deploy that. If you do it as an in-place update (vs delete+redeploy), each role is updated on each instance (when you have multiple instances of a role, they're not all updated at the same time). You can even specify that you only want a single role within the deployment to be updated (helpful if, say, you have a worker role in addition to your web role, and want to leave all the worker role instances running).
when the code update happens, the VMs aren't replaced, but they are recycled, and when they start back up, they are running the updated code.
You can use WebDeploy with Cloud Services in production across multiple servers using the AzureWebFarm project (disclaimer: I maintain it).
Alternatively, you can also use the excellent Octopus Deploy deployment technology in conjunction with the AzureWebFarm.OctopusDeploy project (disclaimer: I maintain this one too).
To be honest though, if you just have a simple web app then I wouldn't both with cloud services - I'd just use Web Sites. Feel free to check out my blog post to see the situations which might force you to use cloud services though.
If you enable WebDeploy on the cloud service, you can use web deploy to publish the MVC application.
See http://msdn.microsoft.com/en-us/library/windowsazure/ff683672.aspx for details.
All of the above answers are correct and if you are trying to change your code for a production service you definitely want to do an in place upgrade as described. However, frequently during the dev/test phase or troubleshooting I want to make one small change and test it out quickly. To do this check out http://blogs.msdn.com/b/kwill/archive/2013/09/05/how-to-modify-a-running-azure-service.aspx which describes how to modify the code via RDP to the Azure VM.