I am trying to deploy a Cloud Service with 1 Web Role to Azure.
When I do so, I get this message:
Your role instances have recycled a number of times during an update or upgrade operation. This indicates that the new version of your service or the configuration settings you provided when configuring the service prevent the role instances from running. Verify your code does not throw unhandled exceptions and that your configuration settings are correct and then start another update or upgrade operation.
The project runs just fine locally, and I'm having a hard time figuring out how to start debugging this issue. Are there any common problems that cause this message or steps to figure out what is causing it?
See https://learn.microsoft.com/en-us/archive/blogs/kwill/windows-azure-paas-compute-diagnostics-data. This will walk through all of the diagnostic data available as well as how to troubleshoot the most common issues.
We also had this annoying problem and in our case:
We use local storage, but it wasn't defined in service definition (or Worker Role's properties)
Our worker role project has reference to a service project which has reference to data layer project. But, the worker role project doesn't have reference to the data layer project. As soon as we added reference to data layer project in worker role project, it deploys successfully.
Problem #1 can be easily noticed if you first run the project in your local machine. Exception will be thrown.
Problem #2, however, is more difficult, mainly because it runs just fine in local machine. After 5 days of trouble shooting, we finally found the problem. So, check all references and try to add sub-reference projects, those that are referenced by other references.
We had similar problem, and it was due to some DDLs failed to load. (due to different version from the one MS have deployed to the VM)
Try to set CopyLocal to "true" for all the References in the project, and re-deploy.
I would either remote desktop to the cloud instance and review the Windows Event Logs for exceptions or redeploy with IntelliTrace Enabled. If you choose the later, you can download the IntelliTrace logs from Visual Studio and debug
http://msdn.microsoft.com/en-us/library/windowsazure/ff683671.aspx
One way to find out the actual error is to click on the " 1 instance" at the top of Dashboard after trying to deploy your web role. It will tell you the status of the role instance. The status should include more information about the type of error which blocks your deployment.
It depends on what your case is. For me, the status claimed that I had an unhandled Security exception. After some investigation, it turned out that under my role's OnStart(), I tried to create a event source. However, Azure service doesn't have the permission to create an event source.
For more possible issues, check http://blogs.msdn.com/b/kwill/archive/2013/09/06/troubleshooting-scenario-3-role-stuck-in-busy.aspx
For me, the issue was with my SQL Azure DB firewall rules. My Azure SQL Database servers are not set to "Allow Access to Azure services", so I have to explicitly list IPs that are allowed.
I discovered this after wrapping my code in a try/catch that swallowed all exceptions, refactoring my OnStart() and RunAsync() methods, and setting all my references to Copy Local = True. None of that worked, then I saw that I had this line in my RunAsync() method:
log4net.Config.XmlConfigurator.Configure();
I am using the AdoNetAdapter for log4net and connecting to an Azure SQL DB for logging, so that led me to check the firewall rules.
For me, I had some differing version of nuget packages in my various projects. Once I consolidated everything to the same version(s), it worked fine.
With the release of Windows Azure SDK version 2.2 for Visual Studio 2012 and 2013, Now you can Remote Debug Cloud Resources within Visual Studio.
Once your cloud service is published and running live in the cloud, you can simply set a breakpoint in your local source code. This may help you in digging out what's going wrong!
Related
I'm running an Azure function in Azure, the function gets triggered by a file being uploaded to blob storage container. The function detects the new blob (file) but then outputs the following message - Did not find any initialized language workers.
Setup:
Azure function using Python 3.6.8
Running on linux machine
Built and deployed using azure devops (for ci/cd capability)
Blob Trigger Function
I have run the code locally using the same blob storage container, the same configuration values and the local instance of the azure function works as expected.
The functions core purpose is to read in the .xml file uploaded into blob storage container and parse and transform the data in the xml to be stored as Json in cosmos db.
I expect the process to complete like on my local instance with my documents in cosmos db, but it looks like the function doesn't actually get to process anything due to the following error:
Did not find any initialized language workers
Troy Witthoeft's answer was almost certainly the right one at the time the question was asked, but this error message is very general. I've had this error recently on runtime 3.0.14287.0. I saw the error on many attempted invocations over about 1 hour, but before and after that everything worked fine with no intervention.
I worked with an Azure support engineer who gave some pointers that could be generally useful:
Python versions: if you have function runtime version ~3 set under the Configuration blade, then the platform may choose any of python versions 3.6, 3.7, or 3.8 to run your code. So you should test your code against all three of these versions. Or, as per that link's suggestion, create the function app using the --runtime-version switch to specify a specific python version.
Consumption plans: this error may be related to a consumption-priced app having idled off and taking a little longer to warm back up again. This depends, of course, on the usage pattern of the app. (I infer (but the Engineer didn't say this) that perhaps if the Azure datacenter my app is in happens to be quite busy when my app wants to restart, it might just have to wait for some resources to become available.). You could address this either by paying for an always-on function app, or by rigging some kind of heartbeat process to stop the app idling for too long. (Easiest with a HTTP trigger: probably just ping it?)
The Engineer was able to see a lower-level error message generated by the Azure platform, that wasn't available to me in Application Insights: ARM authentication token validation failed. This was raised in Microsoft.Azure.WebJobs.Script.WebHost.Security.Authentication.ArmAuthenticationHandler.HandleAuthenticate() at /src/azure-functions-host/src/WebJobs.Script.WebHost/Security/Authentication/Arm/ArmAuthenticationHandler.cs. There was a long stack trace with innermost exception being: System.Security.Cryptography.CryptographicException : Padding is invalid and cannot be removed.. Neither of us were able to make complete sense of this and I'm not clear whether the responsibility for this error lies within the HandleAuthenticate() call, or outside (invalid input token from... where?).
The last of these points may be some obscure bug within the Azure Functions Host codebase, or some other platform problem, or totally misleading and unrelated.
Same error but different technology, environment, and root cause.
Technology Net 5, target system windows. In my case, I was using dependency injection to add a few services, I was getting one parameter from the environment variables inside the .ConfigureServices() section, but when I deployed I forget to add the variable to the application settings in azure, because of that I was getting this weird error.
This is due to SDK version, I would suggest to deploy fresh function App in Azure and deploy your code there. 2 things to check :
Make sure your local function app SDK version matches with Azure function app.
Check python version both side.
This error is most likely github issue #4384. This bug was identified, and a fix was released mid-june 2020. Apps running on version 3.0.14063 or greater should be fine. List of versions is here.
You can use azure application insights to check your version. KUSTO Query the logs. The exception table, azure SDK column has your version.
If you are on the dedicated App Service plan, you may be able to "pull" the latest version from Microsoft by deleting and redeploying your app. If you are on consumption plan, then you may need to wait for this bugfix to rollout to all servers.
Took me a while to find the cause as well, but it was related to me installing a version of protobuf explicitly which conflicted with what was used by Azure Functions. Fair, there was a warning about that in the docs. How I found it: went to <your app name>.scm.azurewebsites.net/api/logstream and looked for any errors I could find.
I had a site running asp.net 5 beta4, and decided to upgrade to beta5. The site runs locally fine. I pushed the changes to master and it was picked up from bitbucket and deployed successfully.
When I try to hit the site in azure, I get a 500 Internal Server Error. I've tried a number of things, but can't seem to track down the root cause of the failure. I'm looking for suggestions as I'm hitting a wall. From what I've tried below it seems like some fundamental initialization is failing.
Here's what I've tried:
Enabling customerrors="off". I added a web.config to the wwwroot folder with system.web/customErrors mode="Off". I've verified that the web.config is populated correctly in the deployed wwwroot and had the appsettings containing the dnxversion etc merged correctly.
Customizing the custom error page, adding runtimeinfo. I have the following set in my Startup.cs:
app.UseErrorHandler("/Home/Error");. I also have set the error page to display the exception. This doesn't seem to be hit.
Attached to the remote process to debug. Visual studio eventually freezes, so haven't gotten anywhere with this.
Enabled application insights. This registers events when I debug locally, but doesn't capture anything from the azure instance.
Enabled application logs and request failure tracing. The detailed errors show a 500.0, without much detailed information.
Imgur
Imgur
I've also verified through the console that the runtime is set correctly to beta5.
Update:
I set the ASPNET_ENV to Development and it loaded with appsettings loaded via the azure portal. Setting ASPNET_ENV to something else isn't working. I also removed any custom code from startup.cs pertaining to the non-development environments, with no help. I'm still looking for a means of capturing the original error.
Assuming you are targeting DNX451 and not dnxcore50, there is a good chance Azure it still trying to run it against the beta4 runtime instead of beta5. If that's the case, you won't get a decent error message.
Try adding an environment variable in Azure "SCM_DNX_VERSION" and set it to 1.0.0-beta5. It looks like kudu was recently upgraded to support beta5 https://github.com/projectkudu/kudu/commit/55175a017779bf493ff8e6ce87b96dd1451f7d7b, so you might want to try to redeploy from bitbucket in the case that the Kudu team has already deployed this change.
For a little more detail, you can check out my previous answer (although it is very dated and references the old "K" names) here:
Deploying ASP.NET vNext beta 2 on Azure with Kudu
Every time you update to a new beta, you will have to update your SCM_DNX_VERSION environment variable.
Locally the project works perfectly.
It is executing and there are proper entries in database.
But when I am deploying it to cloud at staging it is giving me this error as object reference not set to an instance.
My project requires 2 instances of each web and worker role but since I have a limited edition to free instances I am using only 2. one each. Can that be a problem ?
I am using SDK 1.8
It is ASP.Net project.
Can anyone suggest me how to resolve this issue ?
I had a similar issue where I was getting the following when publishing from Visual Studio 2013 to my Azure Cloud Service with a web role and a worker role. This occurred after upgrading the project from Azure SDK v2.4 to v2.5:
2:41:59 PM - Applying Diagnostics extension.
2:42:01 PM - Object reference not set to an instance of an object.
Even after working with a Microsoft Developer I could not get it to show me any kind of stack trace to indicate where the null was coming from. He just happened to notice something funny in my cloud configuration under the "Roles" folder that looked strange.
The problem was that the diagnostics.wadcfgx is needed for an SDK 2.5 project, whereas before that version the diagnostics.wadcfg is needed. (Note the "x" in the file extension.) Apparently the worker role did not have this file automatically created when the project was upgraded.
In order to fix this particular issue, just right click on the role (in the roles folder under cloud configuration) and select "Add Diagnostic Configuration", then Build the solution and attempt to publish. These diagnostics files are ONLY needed when publishing directly from Visual Studio.
I haved same error today, after upgrading Azure SDK to version 2.5
I'm add configuration diagnostics to each role in service and the error disappeared
Had this with MS Azure Tool 2.9. Turned out that it needed the following in diagnostics.wadcfgx :
<PrivateConfig xmlns="http://schemas.microsoft.com/ServiceHosting/2010/10/DiagnosticsConfiguration">
<StorageAccount endpoint="" />
</PrivateConfig>
Hope it helps someone!
I ran into the same issue when upgrading to 2.5. It created the .wadcfgx files and they looked sensible but deployment via Visual Studio would still fail with the same error.
Applying Diagnostics extension.
Object reference not set to an instance of an object.
Package deployment would still work. I was able to get past it via the following steps.
Delete the generated .wadcfgx files
Have Visual Studio regenerate them via the context menu option on the role.
Doing a clean build and redeploying.
I'd love to know what the actual error was and how to get more details. The messaging is pretty terrible.
I've been having a generally wretched time deploying my worker roles to Azure. I'll publish my worker role once from Visual Studio and everything will work fine. I'll publish the worker role again later and the deployment fails. The instance goes into a 'recycling loop'. I spend hours trying to figure out what I broke. I tried Intellitrace but it always fails with a 'cannot download intellitrace logs' error message. Then eventually I'll delete the deployment from inside Azure Management Portal and try again and the same code that's been failing to deploy for hours will magically work.
This doesn't happen all the time, and some projects seem to 'fix' themselves and stop demonstrating this behavior all together. But what seems to be happening is that a publish from Visual Studio will fail unless I go manually delete the existing deployment.
I know this maybe a little vague but I've really got nothing to go on here. Intellitrace never works and I can't Remote Desktop into the role to poke around because it recycles so fast (which may also be why Intellitrace isn't working).
Does anyone have any idea what might be going on here?
I did more research and I think I may know what's going on. Apparently Visual Studio tries to upgrade your worker roles in place when you deploy. If that fails, for some reason such as you changing service configuration between deployments, it just complains that there's something wrong with your role and that your instance is recycling.
In deployment options there is an option called 'If deployment can't be updated, do a full deployment' which will delete the existing deployment and deploy from scratch if the existing deployment can't be updated. I'm not sure why this isn't checked by default instead of 'fail mysteriously'.
I'm currently working on an Azure project that works 100% locally with emulator resources. I'm now trying to deploy a worker role, but I'm running into an issue that I'm not sure how to troubleshoot.
Upon deploying the worker role in my Azure portal, the two instances continually loop through "recycling".
I can try to RDP into the role, but I only have about a minute to look around before the connection closes, I'm assuming due to the recycling.
After some searching it doesn't seem like this is a super common problem. Is there something trivial I'm overlooking that could be causing this issue? How would you go about troubleshooting this? Thank you for your time :)
In case of missing Reference you can troubleshoot this issue by:
Unzip your CSPKG file and then again unzip .CSSX file (just rename CSSX to zip) and match that everything references and static content is all there.. This way you can match what is on VM. Also in 2 minute windows when you RDP, try to look for Application event log for exception and get it because that would be the key to find the root cause.
IF you could see the exception in event log and look for the exception, you sure can find where it was generated. You can also use Intellitrace which might require you to redeploy the app.
Also there are ways were copying WinDBG and locking to the specific process you can debug it. I am not sure how much you would want to try but just copy the WinDBG to VM and use it would be enough (not sure how much experience you have with WinDBG though and how much time you would want to spent.)
Also been pestered by this role recycle issue numerous times. Here is the sequence of steps to debug persistent role recycles:
Debugging Azure Role Recycles
Enable Remote Access to your role - RDP login
Check eventvwr.msc (Windows Logs -> Application, App & Service Logs->Windows Azure)
Review the Azure text file logs across both C:\logs and c:\resources
Review custom logs in the Volume E: or F: for any custom role startup logging
Run AzureTools and attach to startup processes (download WinDBG, use Utils->Attach Debugger, select process - WaWorkerHost/WaIISHost, etc), use G to continue and watch debugger output for assemblies failing to load.
Installing Azure Debugging Tools via Powershell
PS> md c:\tools; Import-Module bitstransfer; Start-BitsTransfer http://dsazure.blob.core.windows.net/azuretools/AzureTools.exe c:\tools\AzureTools.exe; c:\tools\AzureTools.exe
If all items above fail - try using other tools in the AzureTools treasure trove - such as fusion logging, etc, this approach above will work!
WinDBG Sample Output - Failing to Locate Assembly (WaIISHost)
The most likely cause is that you have a missing assembly. One tactic to catch this is to wrap any startup processing in a master try/catch that manual logs the error to Azure storage.
If you added any referrences, check to make sure they're set to copylocal=true and that any external assets that were included in your service package were also set to be included.
From Avkash above:
Yes. this mean some issue in your Worker Role code is causing your Worker Role Host Process to crash.. If you look your fault stack you must see the function or the link from your code which generate this fault. IF you need help open a free Azure Support incident to Windows Azure Support team and they will help you.
Just a suggestion: Also Check the installable(if any)and any other references you use are 64bit.Azure VMs have 64bit OS. Once i was stuck up with this kind of problem due to 32/64 bit issues.
Are your worker roles exiting their work loop? A local recycle is very fast and you might not notice it, but spin-up time in the cloud can be long.
If the issue is caused by a startup batch file, I have stopped the loop by editing the batch file on the instance to include "exit /b 0" at the beginning. This will tell Azure that the startup was successful and you then have all the time you need to diagnose issues without the VM getting killed.