We are looking into building-out some reporting dashboards for use by product owners/etc that might not be familiar with Application Insights or Azure Monitor. We would like to display a list of historical alerts by type as well as some basic aggregations. We see where we can display this information (somewhat) from within Azure Monitor, but we do not see how we can actually surface this information on a dashboard within Azure.
I have found this API call, which seems to retrieve the data, but we are not looking to develop a custom solution.
I hope Serverless360 will be of great fit to your requirements where you will be able to display the historical alerts by type as well as with some advanced aggregations. You can easily set up the powerful dashboards and which can be easily handled by product owners (doesn't necessarily to be familiar with Azure Monitor) to get a high level of insights.
Below is a sample of how a dashboard would look like
Related
There's an awful lot of disjointed documentation on monitoring network/resources in Azure. What I'm looking for is which pieces are needed to get information from VMs, NVA firewalls, azure load balancers, and other network resources and network connectivity into a single pain of glass in Azure. Only concerned about Azure, not on-prem for now.
I've come across azure monitor, log analytics work spaces, event hub, vm extensions, network watcher, insights, etc...but I'm not sure which are required and which are not. One doc leads to the next and I end up with 30 tabs open. I'll also need to be able to push logs to other security devices such as a SIEM.
Does anyone know of a deployment guide that wraps this all up in a more logical fashion? Does anyone have any feedback on which pieces from azure (not 3rd parties) are required at a minimum to accomplish a single pane of glass to view my Azure environment holistically?
General overview of observability in Azure
Likely, the thing you're looking for is Azure Monitor. It's an umbrella term for everything observability related inside Azure.
To store Metrics and Logs you need Log Analytics: it can query data with kusto query language, visualize results, define Alerts on queries.
Alerts is quite a complex beast, as it is spread across the entire cloud. Two types that I use the most:
log-analytics alert (which I mentioned above)
Alerts tab, which is available at every Azure component view. for example, open resource group, and scroll down to Monitoring section
Each component also has a subset of built-in metrics. Likely, you noticed that many azure components on the Overview view display some charts. For example, Azure Storage Account displays Total egress, Total ingress, and other line-charts. When you click on these charts you can customize them. These metrics and charts are free to use.
Microsoft also has all-in-one observability solution for Azure Functions and Web Apps: Application Insights
Dashboards allows to join multiple charts into a single view and share it with others.
If you care about security, Azure proposes Azure Security Center
Deployment/management strategy
I suggest to start with:
Create Log Analytics Workspace, which is the storage for metrics and logs. The azure docs article explains how to design it: how many instances to use, how to rate-limit ingestion (it might be expensive if goes out of control), how to access it and so on.
To get Azure components logs, look for Diagnostic Settings tab at a component page at Azure portal, but not all components has it (sic!). I suggest
sending the most critical data to Log Analytics workspace to store them in a queryable format for 30 days (it's in free tier). This is needed for investigating current issues with your infrastructure
if you might need logs later than 30 days - send them to Storage Account
you mentioned SIEM integration - route required events to Event Hub and then process the stream according to your requirements
So, if you need long-term storage - you need to create Azure Storage Account.
If you need real-time analysis - you need to build a pipeline based on Azure Event Hub.
If you have Azure Functions and Web Apps - add Application Insights. According to my experience, I would suggest starting with a separate instance per each Azure Function resource or Service.
Create Alerts for each component separately. If you do it through UI - open component page at the portal and look for Alerts tab there. If you're automating the process (please do so as soon as possible), do not expect easy trip: I used ARM templates and terraform - in both cases, there are dozens of barely documented features.
Join related components core-metrics into Dashboards and share it with the team. This guide is a good starting point. Note, when you share the dashboard, it's also persisted as an azure resource in the subscription.
I have below two questions can someone help on them.
1.Is there a script or a way to create custom alert format for azure alerts?
2.Is there a way to pin all the azure VM status to dashboard?
Regarding #1, the feature to customize or configure alert email format is currently not supported. If interested, I suggest you to raise your feedback / feature request here in UserVoice / feedback forum. Responsible product / feature team would triage / start checking feasibility and would prioritize the feedback.
Regarding #2, If 'status' is meant as 'PowerState' (i.e., status of VM whether it is running, deallocated, etc.) or if it's meant as 'StatusCode' (i.e., ok, etc.) or if it's meant as 'ProvisioningState' (i.e., succeeded, etc.) then I don't think we have straight-forward way for it so that we can ingest that particular data directly to dashboard but said that, you may just leverage 'Heartbeat' Log Analytics Kusto table at first place and create a custom view as dashboard using view designer but as views in Azure Monitor are being phased out and replaced with workbooks so I suggest leveraging these workbooks now.
If not, you may leverage a new feature called as Azure Monitor for VMs which basically helps to analyze the performance and health of your Windows and Linux VMs, and monitor their processes and dependencies on other resources and external processes. Here again, you can create interactive reports Azure Monitor for VMs with VM insights workbooks.
Hope these inputs helps!
Dynamics CRM 365 Online has Telemetry to monitor application performance and usage. My understanding is it stores data in Azure by sending via Script and on this Alerts can be set in Azure.
There is a CRM solution Organization Insight which is used for counters like API faults etc. my understanding is that this solution stores data in CRM and not Azure.
Please help if this solution generated data can also be used for settings alerts in azure and if yes, how? Thanks.
Over a period of time different products, SaaS offerings, platform tools, O365 capability were introduced/evolving in and around the Telemetry space. Some are strong in broader horizontal features vs deeper vertical strengths. Upto you to decide based on functionality matrix.
Dynamics CRM using Application Insights - this is just leveraging Azure Application Insights to track Availability, Performance, Usage of our CRM online using AppInsights features. It gives you the power to utilize client side telemetry like user activities/metrics/exceptions/browser, Server side agent for on-premise logs similar to Event viewer/IIS logs, data export, Alerting mechanism & web hooks, custom query, visualizations inside Azure portal resources blade (slice/dice), predictive analysis, etc.
CRM in-house Organization Insights - this is inbuilt telemetry solution from Product team, not yet fully delivered (still with bugs & limited functionality for data consumption itself), but we can expect more features, options & utilities to explore more in future. These are missing today & thats what you are asking. You can download data, call MS to give you the exported data, but still Alerts & monitoring wont fit using only available odata support for charts. This covers a lot of useful Admin stuffs, like natively gives you all plugins, WF, API calls, user interactions, mailbox, storage data, etc within CRM.
Activity logging - this is in preview and MS saying it’s more than what we have in Audit today, gives you extra than CRM audit but less than what Org Insights providing, even less than AppInsights features. But I expect more functionalities here like Alerts, Monitoring, web hooks as this is across O365 platforms & available for Global admins.
We run a software application on azure for one of our customers. The customer want to see the performance of the systems. This consist of two parts. One is the metric information of the servers and they also want to see some information I want to provide by custom logging.
My plan is to give the customer access to the portal and only allow him access to the metric information and the custom tables.
It seems to me that by assigning a role to the customer I should be able to block all the other possibilities.
Does someone can me tell which actions I have to allow/forbid to achieve this? Or were I can find the information for this?
Solution #1
Instead of giving Read access to the virtual machine which may breaks security policy, I'd recommend to go with Azure Log Analytics (ref: https://learn.microsoft.com/en-us/azure/log-analytics/log-analytics-overview
) workspace. That said, you will need to create a workspace which collects and stores server metrics (ref: https://learn.microsoft.com/en-us/azure/log-analytics/log-analytics-quick-collect-windows-computer) and other custom metrics.
Your customer will be given access to the workspace only which he can see all metrics in a dashboard. If there is a need for log filtering, you can use Log Analytics query language (ref: https://learn.microsoft.com/en-us/azure/log-analytics/log-analytics-log-search-transition)
Log Analytics is a paid service. You are given free up to 10 workspaces per subscription. The workspace is considered an Azure resource so the limit follows by subscription limit, which means you can create up to 800 workspaces per a resource group. A subscription can allow 800 * 800 (for reference if you would like to do capacity planning for your workspace-based solution). For Log Analytics pricing, read here (https://azure.microsoft.com/en-us/pricing/details/log-analytics/).
Log Analytics is a good choice as its value proportion is to offer your customer intuitive dashboard to monitor their virtual machine performance, and to offer Near Real Time monitoring. And this solution is a cloud native compatibility.
There is a management solution which offers a bundle of VM capacity and performance monitoring which you can try now https://learn.microsoft.com/en-us/azure/log-analytics/log-analytics-capacity
Solution #2
Log Analytics might not be your choice because it might add more Azure service and operational cost. If you need a cheaper cost, you would need to collect your virtual machine by Performance Counter which is a built-in feature in Windows OS. With Performance Counter you can export to Excel file, or visualize into Power BI or some custom chart.
Other Solutions
You can utilize Azure Monitor and API to get data, For example, this API https://learn.microsoft.com/en-us/rest/api/monitor/metricdefinitions/list. You would certainly need to visualize or format in some intuitive way to satisfy your customer. It can be a custom front-end web, or Power BI or even Excel with chart.
You can just query to Azure Blob Storage and use Stream Analytics combining with Power BI to visualize your data (https://thuansoldier.net/?p=7187).
There is not a single solution. This really depends on your existing resource capacity, financial stuff or so on.
My problem is a simple one. I'm working on a project that uses the service bus from Microsoft Azure to send messages asynchronously between different modules on different virtual machines. And a lot of messages are sent through this bus, so we want to have some indicators about it's performance and other usage information. Why? Because when everything is working, users are happy. When the system is slow, we want to show the user some interesting graphs, statistics, meters and other gadgets to give them an indication if there's a problem within Azure or with something else. And to do this, I need data about the usage of the Azure service bus.
So, which Azure API's are available to display what kind of (diagnostic) information about the service bus?
(Users should have no access to Azure itself! They should just see some performance data to re-assure them Azure is working fine. Or else I could look at it and discover some problem with it, fix it and then make users happy again.)
To elaborate what I'm looking for, the Azure website has some nice chart when you click the Monitor of the Azure bus showing you overviews of the number of incoming messages, the number of errors and their types, size information and the number of succesful operations, all based on a specified period. It would be nice if I could receive this data within my project.
The entity metrics API will give you the exact data the portal is using:
http://msdn.microsoft.com/en-us/library/windowsazure/dn163589.aspx
Here's a Subscribe! episode I recorded with Rajat on the topic http://channel9.msdn.com/Blogs/Subscribe/Service-Bus-Namespace-Management-and-Analytics
I've spent quite some time to make the entity metrics API work, so I decided to share the results.
Here is a full C# code example of how to consume those API:
github repository.
It's a small library which wraps the HTTP request into strongly typed .NET classes. You can also grab it from NuGet.
Finally, here is my blog post with the walkthrough.