Is Azure diagnostics only implemented through code? Windows has the Event Viewer where various types of information can be accessed. ASP.Net websites have a Trace.axd file at the root that can viewed for trace information.
I was thinking that something similar might exist in Azure. However, based on the following url, Azure Diagnostics appears to require a custom code implementation:
https://azure.microsoft.com/en-us/documentation/articles/cloud-services-dotnet-diagnostics/#overview
Is there an easier, more built-in way to access Azure diagnostics like I described for other systems above? Or does a custom Worker role need to be created to capture and process this information?
Azure Worker Roles have extensive diagnostics that you can configure up.
You get to them via the Role configuration:
Then, through the various tabs, you can configure up specific types of diagnostics and have them periodically transferred to a Table Storage account for later analysis.
You can also enable a transfer of application specific logs, which is handy and something that I use to avoid having to remote into the service to view logs:
(here, I transfer all files under the AppRoot\logs folder to a blob container named wad-processor-logs, and do so every minute.)
If you go through the tabs, you will find that you have the ability to extensively monitor quite a bit of detail, including custom Performance Counters.
Finally, you can also connect to your cloud service via the Server Explorer, and dig into the same information:
Right-click on the instance, and select View Diagnostics Data.
(a recent deployment, so not much to see)
So, yes, you can get access to Event Logs, IIS Logs and custom application logs without writing custom code. Additionally, you can implement custom code to capture additional Performance Counters and other trace logging if you wish.
"Azure diagnostics" is a bit vague since there are a variety of services in Azure, each with potentially different diagnostic experiences. The article you linked to talks about Cloud Services, but are you restricted to using Cloud Services?
Another popular option is Azure App Service, which allows you many more options for capturing logs, including streaming them, etc. Here is an article which goes into more details: https://azure.microsoft.com/en-us/documentation/articles/web-sites-enable-diagnostic-log/
Related
There's an awful lot of disjointed documentation on monitoring network/resources in Azure. What I'm looking for is which pieces are needed to get information from VMs, NVA firewalls, azure load balancers, and other network resources and network connectivity into a single pain of glass in Azure. Only concerned about Azure, not on-prem for now.
I've come across azure monitor, log analytics work spaces, event hub, vm extensions, network watcher, insights, etc...but I'm not sure which are required and which are not. One doc leads to the next and I end up with 30 tabs open. I'll also need to be able to push logs to other security devices such as a SIEM.
Does anyone know of a deployment guide that wraps this all up in a more logical fashion? Does anyone have any feedback on which pieces from azure (not 3rd parties) are required at a minimum to accomplish a single pane of glass to view my Azure environment holistically?
General overview of observability in Azure
Likely, the thing you're looking for is Azure Monitor. It's an umbrella term for everything observability related inside Azure.
To store Metrics and Logs you need Log Analytics: it can query data with kusto query language, visualize results, define Alerts on queries.
Alerts is quite a complex beast, as it is spread across the entire cloud. Two types that I use the most:
log-analytics alert (which I mentioned above)
Alerts tab, which is available at every Azure component view. for example, open resource group, and scroll down to Monitoring section
Each component also has a subset of built-in metrics. Likely, you noticed that many azure components on the Overview view display some charts. For example, Azure Storage Account displays Total egress, Total ingress, and other line-charts. When you click on these charts you can customize them. These metrics and charts are free to use.
Microsoft also has all-in-one observability solution for Azure Functions and Web Apps: Application Insights
Dashboards allows to join multiple charts into a single view and share it with others.
If you care about security, Azure proposes Azure Security Center
Deployment/management strategy
I suggest to start with:
Create Log Analytics Workspace, which is the storage for metrics and logs. The azure docs article explains how to design it: how many instances to use, how to rate-limit ingestion (it might be expensive if goes out of control), how to access it and so on.
To get Azure components logs, look for Diagnostic Settings tab at a component page at Azure portal, but not all components has it (sic!). I suggest
sending the most critical data to Log Analytics workspace to store them in a queryable format for 30 days (it's in free tier). This is needed for investigating current issues with your infrastructure
if you might need logs later than 30 days - send them to Storage Account
you mentioned SIEM integration - route required events to Event Hub and then process the stream according to your requirements
So, if you need long-term storage - you need to create Azure Storage Account.
If you need real-time analysis - you need to build a pipeline based on Azure Event Hub.
If you have Azure Functions and Web Apps - add Application Insights. According to my experience, I would suggest starting with a separate instance per each Azure Function resource or Service.
Create Alerts for each component separately. If you do it through UI - open component page at the portal and look for Alerts tab there. If you're automating the process (please do so as soon as possible), do not expect easy trip: I used ARM templates and terraform - in both cases, there are dozens of barely documented features.
Join related components core-metrics into Dashboards and share it with the team. This guide is a good starting point. Note, when you share the dashboard, it's also persisted as an azure resource in the subscription.
I have several virtual machines and virtual machine scale sets in Azure for which I want to collect Windows Security event logs. I attempted to add these events to the Log Analytics workspace used by Sentinel through the portal.
This produces the following error message.
'Security' event log cannot be collected by this intelligence pack
because Audit Success and Audit Failure event types are not currently
supported.
It's a hard requirement for me that Sentinel has access these Security logs. I've been trying to figure out what my options are, and I haven't found a good one yet.
The prescribed approach appears to be setting up a Data Connector in Sentinel for the Security Events. I hit a couple of interesting things attempting this.
Virtual machine scale sets support is limited. No actions are
available at this moment.
It looks like I can't connect virtual machine scale sets, which is a big problem. Additionally, I can't even select the tier of the security events (see below) from this context.
So it looks like I have to use Azure Security Center. From within Azure Security Center the only way I can add these Security Events is to turn on Auto-Provisioning and install the Microsoft Monitoring agent (MMA) on every VM, something I don't want to do. I'm also concerned about costs using ASC.
Are there any other options? Am I going about this the wrong way?
The Security event log is automatically added behind the scenes when adding the monitoring agent on the VM.
In regards to the VMSS, I am not sure what your options are there.
We currently have a window service which send some notification emails to users after doing some processing on database(SQL database). Runs once in day.
We want to move this on azure cloud. One alternate is to put it on Azure VM as is. but I am finding some other best possible solution for that.
I study about recurring and on demand Web jobs but I am not sure is this is best solution.
Also is there any possibility to update configuration of service code in App.config without re-deploy the code of service on cloud. I means we can manage configuration from Azure portal.
Thanks in advance.
Update 11/4/2016
Since this was written, there are 2 additional features available in Azure that are both excellent choices depending on what functionality you need:
Azure Functions (which was based on the WebJobs described below): Serverless code that can be trigger/invoked in various ways, and has scaling support.
Azure Service Fabric: Microservice platform, with support for actor model, stateful and stateless services.
You've got 3 basic options:
Windows service running on VM
WebJob
Cloud service
There's a lot of information out there on the tradeoffs between these choices, but here's a brief summary.
VM - Advantages: you can move your service basically as it is without having to change much or any of your code. They also have the easiest connectivity with other resources in Azure (blob storage, virtual networks, etc). The disadvantage is you're giving up all the of PaaS advantages and are still stuck managing your own VM infrastructure
WebJob - Advantages: Multiple invocation options (queues, blobs, manually, queue receive loops, continuous while-loop style, etc), scheduled (would cover your case). Easy to deploy (can go with website, as a console app, automatically through Kudu), has some built in logging in Azure portal - and yes, to answer your question, you can alter the configuration in the portal itself for connection strings and app settings.
Disadvantages - you'll need to update code, you don't have access to underlying resources (if you need that), and more of something to keep in mind than a disadvantage - it uses the same resources as the webapp it's deployed with.
Web Jobs are the newest of the options, but at the same time appear to have active development going on to increase the functionality and usefulness.
Cloud Service - like a managed VM, has some deployment options, access to underlying VM if needed. Would require some code changes from your existing service.
There's nothing you've mentioned in your use case that makes me think a Web Job shouldn't be first thing you try.
(Edit: Troy Hunt has a great and relatively recent blog post illustrating most of the points I've mentioned about Web Jobs above: http://www.troyhunt.com/2015/01/azure-webjobs-are-awesome-and-you.html)
I have deployed my azure cloud service, but some of my instances stays unhealthy and keep recycling. I see 'Recycling (Role has encountered an error and has stopped. Sites were deployed.' in status column for unhealthy instances.
I enabled diagnostics for the role that does not work fine from configuration pane, but I don't know where to find logs of instance. Would you please tell me where to find?
Thanks,
Since this is a very popular issue, I'd like to offer an in-depth explanation here, thus some of this stuff you may already know:
There are a few popular scenarios when instances recycle:
1) Your app is missing dependent DLLs that do not allow IIS to start the app.
2) Your WorkerRole.cs or WebRole.cs contains an unhandled exception
3) Your startup script contains errors
In order to see properly debug this, you need to make sure that:
1) Azure Diagnostics is enabled (this can be done in Visual Studio via a checkbox) on Role's properties. A storage account is required to host the diagnostic data. It can be the one you publish your solution through.
2) Your WebRole.cs and/or WorkerRole.cs files contain proper try/catch with logging to some external storage. Logging to Trace seems simplest since Azure Diagnostics supports transferring the data out to external storage.
3) You have configured Azure Diagnostics to actually transfer the right data to a diagnostic storage account. Best way is to throw in a diagnostics.wadcfg file along with your solution that contains the needed configuration to transfer EventViewer, Infrastructure and Trace logs.
4) Lastly, you need to configure monitoring Trace logs in the app.config of the application so that the Trace infrastructure actually listens to your Trace calls.
This link contains more in-depth info on all of the information above.
Finally, after having done all of this you can actually inspect the data in the diagnostic storage account to see what is wrong. There are tools to help:
1) RedGate/Cerebrata's Management Studio # http://www.cerebrata.com/products/azure-management-studio/introduction (commercial product)
2) Built-in tools into Visual Studio to help you explore contents of any storage account. (You have to hunt and query the necessary WAD tables and containers manually with it, but it is free)
3) AzureWatch # http://www.paraleap.com (commercial product, geared toward active monitoring but lets you inspect most of the logs on-demand, disclaimer: I'm affiliated with this product)
Hope this helps
Try using Azure Explorer or the like to explore the content of your blob storage. If diagnostics is enables as you said, then Azure persist the configuration in an Azure blob named wad-control-container.
I would like to fetch window event logs from Azure webrole's instance. when i connect to RDP of instance then by Event viewer i am able to see Window Event Logs of "Application" & "Error" type. Is there any way to directly access that logs using API or something else or i need to use Diagnostic to enable that log data to transfer in to storage and from there i can access?
IF you want to directly access the Azure VM Event logs, the best option is to use use Azure Diagnostics and Azure Cmdlets to access Event Log details. In my understand this one is very easy to setup and once you have access to Event logs, you can download and save it to your local machine. The method is described as below:
http://michaelwasham.com/2011/09/20/windows-event-logs-with-windows-azure-diagnostics-and-powershell/
There is another simple method is that you can create an ASP page and use Event Log API to simply access the event log directly on your web role and customize it the way you would want to see. You can find several examples on net on how to do it i.e. this one. This is a very simple way to get what you are looking for and the only drawback to this approach is that the ASP page will be available on website, unless you find some ways to protect it.
Although you can use any of the above method, setting up Windows Azure Diagnostics to collect Event log from the machine and send to Windows Azure Storage, is best and preferred method. The steps are described here, in case you don't know:
https://msdn.microsoft.com/en-us/library/windows/desktop/bb427443(v=vs.85).aspx
Any other method to collect these logs (using Azure Connect and Remoting etc) would be complex and troublesome.
Should be possible using http://technet.microsoft.com/en-us/library/cc766438.aspx
Although the port will be blocked by default, you will need to change the firewall settings.
See this article for the port numbers (search event log) http://support.microsoft.com/kb/832017/en