Here is some data about the monitoring metrics Spark exposes.
I don't want to pull from the rest API. Is there a way of telling the driver to send specific metrics that I found valuable to elastic search (So I can then display them on Kibana)?
Here are all the metrics available but I couldn't understand how can I push, (not pull as for example - Prometheus) those metrics.
Related
I am working on Azure Monitor Dashboards.
I need to check the Health Status of my App Service.
If i run use option Metrics (2nd Image) add Health Status metrics and create chart
vs
If i run query on AzureMetrics Table will both return same result? I mean HOW both options are different from each other?
Both use the same source. The difference is that using the "Metrics" blade you can create charts withouth having to write queries using Kusto and anyone with basic knowledge can quickly create charts.
When using the "Logs" blade you have to write a query using Kusto to get the desired results and format the chart manually but you have more control in what and how data is displayed.
If i run query on AzureMetrics Table will both return same result? I
mean HOW both options are different from each other?
The difference between logs and metrics is,
Metrics reveal a service or application's tendencies and proclivities,
while logs focus on specific events. The goal of logs is to save as
much information—mostly technical informations—as possible about a
single event. Log data can be used to investigate occurrences and
assist with root-cause analysis of problems or defects, as well as an
increasing number of other applications.
For more information please refer the below links:-
MSFT TECHCOMMUNITY|Difference between Log Analytics and Monitor
Blogs|Azure Monitor and Azure Log Analytics & Logs or Metrics.
We have legacy applications that currently write out various run time metrics (SQL calls run time, api / http request run times etc) to local SQL DB.
format:( source, event, data, executionduration)
We are moving away from storing those in local SQL DB, and are now publishing those same metrics to azure event hub.
Looking for a good place to store those metrics for the purpose of monitoring the health of the application. Simple solution would be to store in some DB and build custom application to visualize the data in custom ways.
We are also considering using Azure Monitor for this purpose via data collector API (https://learn.microsoft.com/en-us/azure/azure-monitor/platform/data-collector-api)
QUESTION: Are there any issues with azure monitor that would prevent us from achieving this type of health monitoring?
Details
each event is small (few hundred characters)
expecting ~ 10 million events per day
retention of 1-2 days is enough
ability to aggregate old events per source per event is important (to have historical run time information)
Thank you
You can do some simple graphs and with the Log Analytics query language, you can do just about any form of data analytics you need.
Here's a pretty good article on Monitor Visualizations.
learn.microsoft.com/en-us/azure/azure-monitor/log-query/charts
I am using AWS RDS SQL server and I need to do enhanced level monitoring via Cloudwatch. By default there are some basic monitoring available but I want use custom metrics as well.
In my scenario I need to create an alarm whenever we get more number of deadlock in SQL server. We are able to fetch the details of deadlock via script and I need to prepare custom metrics for the same.
Can any one help on this or kindly suggest any alternate solution?
In Azure's Portal you can view instance specific metrics per site if you go to a resource, select Metrics per instance (Apps), select the tab Site Metrics and then click an individual instance (starting with RD00... in the screenshot below):
I'd like to get this data (per instance, including the instance name RD00...) using some REST API call. I've looked at Azure's Resource Manager and their Metrics API, but couldn't find a way to get this data.
Is this possible, and, if so, how/where can I get this data?
I've looked at Azure's Resource Manager and their Metrics API, but couldn't find a way to get this data.
Based on the supported metrics with Azure Monitor of websites, Azure Metrics API only supports total and average type metrics for Azure Web App. We can't get per instance metrics by Azure Metrics API.
If you turn on the Web server logging in Azure portal, you will get the detail request data from /LogFiles/http/RawLogs/ folder using FTP. You could download the log and generate the metrics according the log.
Following is a record of raw logs. The ARRAffinity property will specify which instance is used to handle user request.
2017-04-27 08:51:32 AMOR-WEBAPP-TESTMSBUILD GET /home/index X-ARR-LOG-ID=bbdf4e53-3b96-4884-829c-cf82554abcc7 80 - 167.220.255.28 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/51.0.2704.79+Safari/537.36+Edge/14.14393 ARRAffinity=8f8ac6c076f7a9e2132f2eea1ff0fc61836fde1fef8c5525da0e81359003c9e8;+_ga=GA1.3.411824075.1493282866;+_gat=1 - amor-webapp-testmsbuild.azurewebsites.net 200 0 0 2607 1145 10095
ARRAffinity=8f8ac6c076f7a9e2132f2eea1ff0fc61836fde1fef8c5525da0e81359003c9e8
I have Event logs of Application Insights where events are logged and stored as json in text files stored in a blob storage. I need to find those jsons where a customProperty meets a criteria. The number of hit jsons are very less (around 10 or 20), however the data logged is very large. Any suggestions how this can be accomplished efficiently?
I have read in Microsoft documentation that HDInsights understand blob storage and is efficient. Is this relevant in my scenario? If so, could someone provide some starting points.
HDInsight, being a Hadoop-compliant implementation, is a good technology for logs analysis. It is being stated on the AppInsighs page about telemetry as well.
"On larger scales, consider HDInsight - Hadoop clusters in the cloud. HDInsight provides a variety of technologies for managing and analyzing big data."
On the same page, you may find the information about continuous export of AppInsights telemetry into the Azure Blobs storage.
Next step could be to use HDInsight for the analysis of that, but it will need you to implement some kind of algorithm.
For uploading the data to the HDInsight from Azure Blob you may see that link (and this for querying).
For understanding on the logs processing pipeline, which is a common task for Hadoop/HDInsight, some walkthroughs and manuals are available, for example this. But you will need to adjust this algorithm to your scenario.
In case of Application Insights there is another option. New analytic tool Application Insights Analytics has been introduced.
https://blogs.msdn.microsoft.com/bharry/2016/03/28/introducing-application-analytics
This tool also alows you to work with all logged data using the specific language:
requests
| where timestamp >= ago(24h)
| summarize count() by client_CountryOrRegion
| order by count_ desc
You can export data you need.