I have deployed my azure cloud service, but some of my instances stays unhealthy and keep recycling. I see 'Recycling (Role has encountered an error and has stopped. Sites were deployed.' in status column for unhealthy instances.
I enabled diagnostics for the role that does not work fine from configuration pane, but I don't know where to find logs of instance. Would you please tell me where to find?
Thanks,
Since this is a very popular issue, I'd like to offer an in-depth explanation here, thus some of this stuff you may already know:
There are a few popular scenarios when instances recycle:
1) Your app is missing dependent DLLs that do not allow IIS to start the app.
2) Your WorkerRole.cs or WebRole.cs contains an unhandled exception
3) Your startup script contains errors
In order to see properly debug this, you need to make sure that:
1) Azure Diagnostics is enabled (this can be done in Visual Studio via a checkbox) on Role's properties. A storage account is required to host the diagnostic data. It can be the one you publish your solution through.
2) Your WebRole.cs and/or WorkerRole.cs files contain proper try/catch with logging to some external storage. Logging to Trace seems simplest since Azure Diagnostics supports transferring the data out to external storage.
3) You have configured Azure Diagnostics to actually transfer the right data to a diagnostic storage account. Best way is to throw in a diagnostics.wadcfg file along with your solution that contains the needed configuration to transfer EventViewer, Infrastructure and Trace logs.
4) Lastly, you need to configure monitoring Trace logs in the app.config of the application so that the Trace infrastructure actually listens to your Trace calls.
This link contains more in-depth info on all of the information above.
Finally, after having done all of this you can actually inspect the data in the diagnostic storage account to see what is wrong. There are tools to help:
1) RedGate/Cerebrata's Management Studio # http://www.cerebrata.com/products/azure-management-studio/introduction (commercial product)
2) Built-in tools into Visual Studio to help you explore contents of any storage account. (You have to hunt and query the necessary WAD tables and containers manually with it, but it is free)
3) AzureWatch # http://www.paraleap.com (commercial product, geared toward active monitoring but lets you inspect most of the logs on-demand, disclaimer: I'm affiliated with this product)
Hope this helps
Try using Azure Explorer or the like to explore the content of your blob storage. If diagnostics is enables as you said, then Azure persist the configuration in an Azure blob named wad-control-container.
Related
Are the resource logs (which are part of platform logs) from Azure supported in QRadar or do we need to build a custom parser for each of the resource type in the subscription?
I read the DSM documentation of QRadar, and it mentions platform activity logs, but not resource logs. Let’s take an example where we get gateway logs, websocket connection logs, request logs, etc. from our Azure deployment. Are all resource logs supported by QRadar to be taken from event hub and integrate to QRadar (list of supported resource logs by QRadar)?
if I understand your question correctly you are looking to extend existing parsers to QR without having to implement custom properties.
For this IBM has published the "IBM QRadar Content Extension for Azure":
https://exchange.xforce.ibmcloud.com/hub/extension/7a89f51852efa37de0809457ef1006dd
I recommend installing another extension "Microsoft Azure Security Center Connected Assets & Risks Connector" (https://exchange.xforce.ibmcloud.com/hub/extension/0dbfab6a22bca7add7a99fa19fdd426f), which allows you to monitor other risk events via ASC and integrate assets that are not yet parsed into the QR.
And probably the best scenario how to solve issue with Azure log data is to run side-by-side QR + Sentinel and use Azure Sentinel and turn on Data Connectors for Azure specific resources. This keeps you up to date with integration, data parsing and current buildin rules. We have this scenario deployed and it is for selected sources (Exchange, Teams, risk signins, etc.) and we monitor them via buildin rules in Sentinel. Subsequently, we integrate them into the QR see. https://techcommunity.microsoft.com/t5/microsoft-sentinel-blog/azure-sentinel-side-by-side-with-qradar/ba-p/1488333. We finally store the logs in QRadar, but we use Sentinel for Azure-specific rules and then integrate the incidents into QR.
Regards.
We're looking for a lightweight solution that keeps track of our reasons and execution of changes made to azure tenant, there is no approval processes necessary but we would like a tracking system that allows us to easily and quickly catch up on history and existing state.
By default, everything you do to an azure resource is recorded in the azure activity log. You can learn more about it from here. But I would recommend enabling Diagnostic Logging to your default Log Analytics workspace which will be part of your Azure Monitor Logs now. Learn more about Diagnostic Logging from here.
I'm setting up Azure WebApp logging. My concern is that error logs are stored in webapp server level, the size increasing day by day from Elmah. Is there a best approach to maintaining the logs, both storing and automating archiving or deleting?
My web development is based on angular. Any suggestion for aggregating logs, like what kind of logs would be generated?
Yes, by default, logs are not automatically deleted (with the exception of Application Logging (Filesystem)). To automatically delete logs, set the Retention Period (Days) field. You could automate the deletion by leveraging KUDU Virtual File System (VFS) Rest API. For a sample script, checkout this discussion thread for a similar approach:
How can you delete all log files from an Azure WebApp using powershell?
Just to highlight, these are the logging that you could capture on WebApps:
• Detailed Error Logging
• Failed Request Tracing
• Web Server Logging
• Application logging - you can turn on the file system option temporarily for debugging purposes. This option turns off automatically in 12 hours. You can also turn on the blob storage option to select a blob container to write logs to.
For log directory information kindly refer to this document: https://learn.microsoft.com/azure/app-service/troubleshoot-diagnostic-logs
Is Azure diagnostics only implemented through code? Windows has the Event Viewer where various types of information can be accessed. ASP.Net websites have a Trace.axd file at the root that can viewed for trace information.
I was thinking that something similar might exist in Azure. However, based on the following url, Azure Diagnostics appears to require a custom code implementation:
https://azure.microsoft.com/en-us/documentation/articles/cloud-services-dotnet-diagnostics/#overview
Is there an easier, more built-in way to access Azure diagnostics like I described for other systems above? Or does a custom Worker role need to be created to capture and process this information?
Azure Worker Roles have extensive diagnostics that you can configure up.
You get to them via the Role configuration:
Then, through the various tabs, you can configure up specific types of diagnostics and have them periodically transferred to a Table Storage account for later analysis.
You can also enable a transfer of application specific logs, which is handy and something that I use to avoid having to remote into the service to view logs:
(here, I transfer all files under the AppRoot\logs folder to a blob container named wad-processor-logs, and do so every minute.)
If you go through the tabs, you will find that you have the ability to extensively monitor quite a bit of detail, including custom Performance Counters.
Finally, you can also connect to your cloud service via the Server Explorer, and dig into the same information:
Right-click on the instance, and select View Diagnostics Data.
(a recent deployment, so not much to see)
So, yes, you can get access to Event Logs, IIS Logs and custom application logs without writing custom code. Additionally, you can implement custom code to capture additional Performance Counters and other trace logging if you wish.
"Azure diagnostics" is a bit vague since there are a variety of services in Azure, each with potentially different diagnostic experiences. The article you linked to talks about Cloud Services, but are you restricted to using Cloud Services?
Another popular option is Azure App Service, which allows you many more options for capturing logs, including streaming them, etc. Here is an article which goes into more details: https://azure.microsoft.com/en-us/documentation/articles/web-sites-enable-diagnostic-log/
We need to enable 25+ performance counters in windows azure web roles. I'm thinking of RDP'ing and enable them one by one. But this could take long time and also not guranteed for if we scale up.
Could someone please help me whether it's possible to automate this process? Preferably Powershell but other solutions are ok as well.
There is no way to enable performance counters by RDP'ing into Windows Azure machines, because performance counters are published by Windows regardless.
However, what I think you're asking for is to capture the 25+ performance counters into Azure Diagnostics store?
If that is the case, you will need to:
1) Enable Azure Diagnostics on your Web Roles. This must be done before deployment. It is a best practice and mostly everyone does it (I sure wish Microsoft would have just done it for every Role w/o explicit configuration setting, but
2) There are multiple ways to instrument capture of performance counters into diagnostics store:
a) using diagnostics.wadcfg file http://msdn.microsoft.com/en-us/library/gg604918.aspx (you will need to redeploy your app with that file)
b) using powershell (although I've never done it myself) http://michaelwasham.com/2011/09/19/windows-azure-diagnostics-and-powershell-performance-counters/ or http://www.davidaiken.com/2011/10/18/how-to-easily-enable-windows-azure-diagnostics-remotely/
c) using in-code instrumentation (you'll need to re-upload your app everytime you change which counters you want enabled) http://www.codeproject.com/Articles/303686/Windows-Azure-Diagnostics-Performance-Counters-In (I dont recommend in-code configuration, because it is too brittle)
d) using 3rd party tools like Cerebrata Diagnostics Manager or AzureWatch
e) using Azure Service Management API in conjunction with Azure Diagnostics API to get at the individual instance configuration and update it (this is how the third party tools & powershell do it)
In cases of using powershell, management API directly, or a tool like Cerebrata, you configuration will "stick" for the life of the deployment. Once you re-upload a new version of the app, the configuration will be lost.
Using diagnostics.wadcfg, in-code instrumentation, or AzureWatch, your configuration will persist throughout the re-uploads of the app
HTH