We have several Azure app services live and I've ran into a consistent problem when restarting any of them.
We've right-click -> Published from Visual Studio upto an existing instance of an App Service which brings it online and functional immediately, we can update and republish new code without a problem, but restarting to app service from Azure Portal doesn't launch dotnet.exe.
When restarting or stopping/starting, the thread count instantly drops to 0 and continues to do nothing.
I have to manually publish from VS again to reboot dotnet.exe.
I can see that dotnet.exe isn't an active process via Kudu -> Process Explorer, and manually publishing up to the App Service and then refreshing the Process Explorer list again shows that dotnet.exe is now an active process and the App Service begins to function as expected.
Is there a way to prevent this, or at the very least debug why it's happening?
As it turns out this is being caused by silent failures, outside of error handling I'd put in place. IIS would spin up the associated processes but dotnet would throw and error and "silently" die.
I use quotes because I'd failed to utilise Azure's indepth logging, which hid the error message from plain view until switched on.
For some basic debugging steps:
Open up your Kudu console (Advanced Tools -> Go -> Debug Console) and navigate to LogFiles, open up eventlog.xml, the latest events are listed at the bottom.
Run dotnet myDll.dll against your debug folder locally, where myDll.dll is the main dll in your project. This will typically throw the error causing dotnet to exit out, which wasn't obvious at design time.
Ensure that your server-side appsettings.json (or the associated appsettings.[development/release].json contains the connection strings you're attempting to access.
The errors that were occuring we'ren't due to design-time bugs, it generally fell over when trying to access run-time resources (such as appsettings).
Related
I just stopped an Application Pool in IIS. When trying to start it, IIS complains that,
The service cannot accept control messages at this time. (Exception from HRESULT: 0x80080425).
What gives? Whence did this error come?
Looking at the Event Viewer > System shows these warnings:
A worker process '1456' serving application pool 'MyAppPool' failed to stop a listener channel for protocol 'http' in the allotted time. The data field contains the error number.
A process serving application pool 'MyAppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '10592'. The data field contains the error number.
A process serving application pool 'MyAppPool' exceeded time limits during shut down. The process id was '10516'.
This resolved itself after about 5-minutes, at which point we tried to restart the website, and received:
The World Wide Web Publish Service (W3SVC) is stopped. Web sites cannot be started unless the World Wide Web Publishing Service (W3SVC) is running.
So, we started the W3SVC service, and then we could start our website.
This helped me: just wait about a minute or two.
Wait a few minutes, then retry your operation.
Ref: https://msdn.microsoft.com/en-us/library/ms833805.aspx
The error message could result due to the following reason:
The service associated with Credential Manager does not start.
Some files associated with the application have gone corrupt.
Please follow the steps mentioned below to resolve the issue:
Method 1:
Click on the “Start”
In the text box that reads “Search Program and Files” type “Services”
Right click on “Services” and select “Run as Administrator”
In the Services Window, look for Credential Manager Service and “Stop” it.
Restart the computer and “Start” the Credential Manager Service and set it to “Automatic”.
Restart the computer and it should work fine.
Method 2:
1. Run System File Checker. Refer to the link mentioned below for additional information:
http://support.microsoft.com/kb/929833
In my case, the VS debugger was attached to the w3wp process. After detaching the debugger, I was able to restart the Application Pool
I stopped the IIS Worker Process (in task manager), and then started the IIS again.
It worked.
I killed related w3wp.exe (on a friends' advise) at task manager and it worked.
Note: Use at your own risk. Be careful picking which one to kill.
Restarting the machine worked for me but not every time.
If you are really stuck on this then follow below steps
Open Task Manager
A window will open. Click on Details tab.
Search for the process name you wanted to restart/stop.
Select process, right click on it, select End task option.
A confirmation dialog box will appear. Click on End process button.
Now try to restart your service from Services.msc window.
I forgot I had mine attached to Visual Studio debugger. Be sure to disconnect from there, and then wait a moment. Otherwise killing the process viewing the PID from the Worker Processes functionality of IIS manager will work too.
Restarting the IIS windows service (World Wide Web Publishing Service) and then starting the application pool has worked for me. However, as the top answer suggests it may have just been the waiting that caused it to subsequently work.
I had this issue recently,
Problem statement:
Mine was a windows service that I run locally by attaching VS debugger. When I stop debugging and try to restart/stop the service (under services.msc) I used to get the mentioned error.
Solution:
Open up Task manager.
Search for the service (based on the exe name and not service name, for those that are different).
Kill the service.
On doing the above the service is stopped.
Being impatient, I created a new App Pool with the same settings and used that.
I kept having this problem whenever I tried to start an app pool more than once. Rather than rebooting, I simply run the Application Information Service. (Note: This service is set to run manually on my system, which may be the reason for the problem.) From its description, it seems obvious that it is somehow involved:
Facilitates the running of interactive applications with additional administrative privileges. If this service is stopped, users will be unable to launch applications with the additional administrative privileges they may require to perform desired user tasks.
Presumably, IIS manager (as well as most other processes running as an administrator) does not maintain admin privileges throughout the life of the process, but instead request admin rights from the Application Information service on a case-by-case basis.
Source: social.technech.microsoft.com
Visual Studio 2013 update 3. I have deployed several Webjobs to Azure, both manually (upload a bundle via the portal) and also published directly from Visual Studio. The webjobs I am testing with are set to run "Continuous" and have code that runs inside a while(true) {} loop; so the process never ends. The webjobs are functioning properly and I see their runtime output in the Webjobs dashboard log output screen.
I've enabled remote debugging on the website. When I select Debug, Attach to process, the webjob process names never show on the list. All I see are the remote debugger processes and w3wp.exe.
Ideas?
Instead of attaching to SITENAME.azurewebsites.net, try to attach to SITENAME.scm.azurewebsites.net. Use same credentials as for the regular debugging connection. WebJobs run under a different context (Kudu), hence the different endpoint.
I am trying to deploy a Cloud Service with 1 Web Role to Azure.
When I do so, I get this message:
Your role instances have recycled a number of times during an update or upgrade operation. This indicates that the new version of your service or the configuration settings you provided when configuring the service prevent the role instances from running. Verify your code does not throw unhandled exceptions and that your configuration settings are correct and then start another update or upgrade operation.
The project runs just fine locally, and I'm having a hard time figuring out how to start debugging this issue. Are there any common problems that cause this message or steps to figure out what is causing it?
See https://learn.microsoft.com/en-us/archive/blogs/kwill/windows-azure-paas-compute-diagnostics-data. This will walk through all of the diagnostic data available as well as how to troubleshoot the most common issues.
We also had this annoying problem and in our case:
We use local storage, but it wasn't defined in service definition (or Worker Role's properties)
Our worker role project has reference to a service project which has reference to data layer project. But, the worker role project doesn't have reference to the data layer project. As soon as we added reference to data layer project in worker role project, it deploys successfully.
Problem #1 can be easily noticed if you first run the project in your local machine. Exception will be thrown.
Problem #2, however, is more difficult, mainly because it runs just fine in local machine. After 5 days of trouble shooting, we finally found the problem. So, check all references and try to add sub-reference projects, those that are referenced by other references.
We had similar problem, and it was due to some DDLs failed to load. (due to different version from the one MS have deployed to the VM)
Try to set CopyLocal to "true" for all the References in the project, and re-deploy.
I would either remote desktop to the cloud instance and review the Windows Event Logs for exceptions or redeploy with IntelliTrace Enabled. If you choose the later, you can download the IntelliTrace logs from Visual Studio and debug
http://msdn.microsoft.com/en-us/library/windowsazure/ff683671.aspx
One way to find out the actual error is to click on the " 1 instance" at the top of Dashboard after trying to deploy your web role. It will tell you the status of the role instance. The status should include more information about the type of error which blocks your deployment.
It depends on what your case is. For me, the status claimed that I had an unhandled Security exception. After some investigation, it turned out that under my role's OnStart(), I tried to create a event source. However, Azure service doesn't have the permission to create an event source.
For more possible issues, check http://blogs.msdn.com/b/kwill/archive/2013/09/06/troubleshooting-scenario-3-role-stuck-in-busy.aspx
For me, the issue was with my SQL Azure DB firewall rules. My Azure SQL Database servers are not set to "Allow Access to Azure services", so I have to explicitly list IPs that are allowed.
I discovered this after wrapping my code in a try/catch that swallowed all exceptions, refactoring my OnStart() and RunAsync() methods, and setting all my references to Copy Local = True. None of that worked, then I saw that I had this line in my RunAsync() method:
log4net.Config.XmlConfigurator.Configure();
I am using the AdoNetAdapter for log4net and connecting to an Azure SQL DB for logging, so that led me to check the firewall rules.
For me, I had some differing version of nuget packages in my various projects. Once I consolidated everything to the same version(s), it worked fine.
With the release of Windows Azure SDK version 2.2 for Visual Studio 2012 and 2013, Now you can Remote Debug Cloud Resources within Visual Studio.
Once your cloud service is published and running live in the cloud, you can simply set a breakpoint in your local source code. This may help you in digging out what's going wrong!
I am getting the following error:
Error: The process cannot access the file 'C:\DWASFiles\Sites\mywebsitename\VirtualDirectory0\site\wwwroot\newrelic\NewRelic.Agent.Core.dll' because it is being used by another process.
In the Running deployment command... log file when attempting to deploy an Azure website from Github.
Would appreciate any pointers as to what could be causing this.
UPDATE: Turns out this is also failing when publishing directly from VS.NET with the following:
1>C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\Web\Microsoft.Web.Publishing.targets(4196,5): Warning : An error was encountered when processing operation 'Create File' on 'NewRelic.Agent.Core.dll'.
1>Retrying operation 'Update' on object filePath (mywebsitename\newrelic\NewRelic.Agent.Core.dll). Attempt 1 of 2.
1>C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\Web\Microsoft.Web.Publishing.targets(4196,5): Error : Web deployment task failed. ((06/07/2013 23:54:58) An error occurred when the request was processed on the remote computer.)
This was working before and I am not sure why it would have stopped.
NewRelic recommend stopping the website to unload the file and allow the deployment to go through.
As an alternative, you can set COR_ENABLE_PROFILING to 0 in your app settings on the configure tab to temporarily disable the profiling, which should then allow you to continue with the deployment while leaving the website operational throughout.
Instead of stopping the website you can temporarily turn off New Relic monitoring via the Configure tab on manage.windowsazure.com:
Configure > developer analytics > select "OFF" > Save
Deploy
Configure > developer analytics > select "ADD-ON" > Choose Add-on from dropdown > Save
Worked for me, both with a regular deployment from VS and an automatic build from VSO.
This is a known issue with the New Relic .NET agent for Azure Websites when performing an upgrade of the agent. The workaround is to stop the website to release the dll, finish the deployment and then restart the instance.
https://newrelic.com/docs/dotnet/azure-web-sites#h2-1
Not really a solution but more of a work-around, in the publish dialog view a preview of the changes and uncheck the NewRelic.Agent.Core.dll file so that it doesn't get published.
None of these answers work for me anymore. I have an Azure Basic tier website plan, which hosts multiple actual websites.
If I don't stop the website, I get the error mentioned above (newrelic.agent.core.dll is in use)...
If I do stop the website (or all of them), I get an error saying that the publishing endpoint isn't available.
If I go to the configure tab and disable the AddOn, we still get the error mentioned above (newrelic.agent.core.dll is in use)...
Pretty much we just republish over and over again with different permutations of the above until if works. It took me hours the other day, took me 10 minutes today.
If you are using webdeploy, then you can configure your webdeploy settings so that it ignores the file. However, if you do that, you will manually have to deploy any updates to the new relic agent.
I had a similar issue with the new relic log file being locked, and solved it by:
Moving the new relic log file to a subdirectory of the web root (e.g. \newreliclogs)
Adding 2 lines to my powershell script that configured the skip directive to ignore that whole directory. e.g. (where destBaseOptions is of type Microsoft.Web.Deployment.DeploymentBaseOptions
$skipDirective = new-object Microsoft.Web.Deployment.DeploymentSkipDirective("NewRelicLog","objectName=dirPath,absolutePath=.*\newreliclogs$")
$destBaseOptions.SkipDirectives.Add($skipDirective)
Depending on how you are using webdeploy, the configuration is achieved slightly differently, I used the following links to help me piece it together:
https://technet.microsoft.com/en-us/library/dd569089(WS.10).aspx
https://msdn.microsoft.com/en-us/library/microsoft.web.deployment.deploymentskipdirective(v=vs.90).aspx
https://msdn.microsoft.com/en-us/library/dd543313(v=vs.90).aspx
http://blogs.iis.net/jamescoo/archive/2009/11/03/msdeploy-api-scenarios.aspx
http://forums.iis.net/p/1192163/2031814.aspx#2031813
And I used the powershell script from the Octopus Deploy Library at https://library.octopusdeploy.com/#!/step-template/actiontemplate-web-deploy-publish-website-(msdeploy).
I now have my Windows Azure environment set up so that I can access my Worker Role with Remote Desktop. However, I'm not sure how to proceed at the moment. After much digging I found a web site that was offline but in Google's cache there was mention of attaching to the Worker Role running in the Azure Cloud from the Visual Studio debugger. But I only have Visual Developer (not studio) 2010 and I have searched all over and as far as I can see there is no such option to attach to a remote server. I am able to publish my project to the Azure Cloud without error and I have a "healthy" instance of my Worker Role showing as active and running.
I did connect with RDP through the Azure Management portal. The login worked fine and up came the remote desktop window. I searched through much of what I could find and was unable to find my Worker Role. I must have the wrong impression of RDP, because I had hoped to see the Worker Role's main display form when I logged in, just like I do when I debug it locally in the Cloud Emulator. But instead all I saw was a blank desktop with some base level server inspection and management routines. I even checked the Event Viewer for Application related messages and saw none.
So now I'm stuck wondering if my Worker Role is actually running or not, despite the seemingly positive status messages from the Management Portal, and I still want to attach to my Worker Role for debugging through Visual Developer, if it's possible, but I am unable to figure out how.
Anyone with experience in this area that can give me some solid tips on what to do next, please respond.
UPDATE: I believe my worker role may be running because I opened a command window and did a Netstat and saw it listening on the correct port. However, that may just be my Worker Role shell class that starts the custom EXE I have it launch as a spawned proces. I still haven't confirmed if my custom EXE is running yet.
UPDATE-2: Just ran TaskList from a command window and the custom EXE is listed.
UPDATE-3: Everything is working as I just ran a remote test of the service so that's not a problem. Still want to know how to attach to the Worker Role from Visual Developer 2010 for remote debugging, and if it's possible to see the custom EXE's display form like I do when doing local debugging in the Cloud Emulator.
-- roschler
There is a set of articles here which goes in length on how to set up for remote debugging in Azure:
http://blogs.u2u.be/peter/post/2011/06/21/Remote-debugging-an-Azure-Worker-role-using-Azure-Connect-Remote-desktop-and-the-remote-debugger.aspx
http://blogs.u2u.be/peter/post/2011/06/24/Remote-debugging-an-Azure-worker-role-using-Azure-Connect-remote-desktop-and-remote-debugger-part-2.aspx
http://blogs.u2u.be/peter/post/2011/06/26/Remote-debugging-a-Windows-Azure-Worker-Role-using-Azure-Connect-Remote-desktop-and-the-remote-debugger-part-3.aspx
The key takeaway is that you don't need to actually install Visual Studio on Azure, you only need to copy the Remote Debugger bits and then use Azure Connect to add your developer machine to the Virtual Network.
You can setup Remote Debugging with Visual Studio 2012
http://code.msdn.microsoft.com/Remote-Debugging-Windows-dedaaec9
When you say:
But instead all I saw was a blank desktop with some base level server inspection and management routines.
this is exactly what you get with an Azure VM. It's a basic OS install, plus the bare minimum of Azure stuff it needs to run and the code you've uploaded. There's no fancy monitoring or health checks available on the machine by default, you're expected to have provided those yourself to have them available without having to RDP into the machine to check on it.
RDP is very good for tracking down certain problems, like checking that a startup task will run, checking which directories items are installed in and just generally being nosey. If you need extra tools to track down a problem, you can just install them while you're connected to the server. For example I have RDPed into a server and installed the Microsoft Debugging Tools, to track down a memory issue.
I suppose you could remote into your VM, install Visual Studio there, and debug the process...
I also suppose it might be possible to enable remote debugging (not sure what's involved there, but such a thing exists, and it works over TCP) and debug from a local instance of Visual Studio.
To my knowledge, neither is commonly done.
Based on other answers, you would be better off writing a log file to a local storage. You can read the file from RDP if you reallyhace to. Keep in mind, debugging on Azure isn't really simple, and rightly so.
What I was thinking though was, maybe you could run the process using the user's credentials. I can't verify at the moment, but you have a better shot of seeing the ui when you rdp.