I'm using azure functions which is integrated with application insights. While exporting the data from application insights , I'm able to see only 1898 records. How do I get complete logging information data exported to excel without creating azure data explorer cluster (I don't have permissions to create it)
You can achieve this by running a kql query in the logs of Application Insights from Function App
From the Azure Portal,
Go to Function App -> ApplicationInsights -> Logs and
Execute the below Kql query:
union
availabilityResults,
requests,
exceptions,
pageViews,
traces,
customEvents,
dependencies
| where timestamp > datetime("2022-10-14T12:02:57.123Z") and timestamp < datetime("2022-10-15T12:02:57.123Z")
| order by timestamp desc
Log information from Application Insights:
Refer: How to use Union Operator
Related
successfully configured data-bricks cluster with Azure log analytics and i could see the following tables and respective log messages in log analytics but i'm not sure how to create a reports of out as i know KQL.
Can you please help to write sample query to get the following metrics by using the tables.
SparkListenerEvent_CL
SparkLoggingEvent_CL
SparkMetrics_CL
how many messages processed per sec?
no of inputs message and output message for sec?
Messages Latency ?
input vs out put message rate
no. of message processing rate per sec
++++++++++++++++++++++++
Azure Databricks is a fast, powerful Apache Spark based analytics service that makes it easy to rapidly develop and deploy big data analytics and artificial intelligence solutions.
Connecting Azure Databricks with Log Analytics allows monitoring and tracing each layer within Spark workloads, including the performance and resource usage on the host as well as Spark metrices.
When you enable diagnostic setting for databricks, it will create 3 tables SparkMetrics, SparkLoggingEvent and SparkListenerEvent for logging spark related events.
Spark metrics
Spark metrics are automatically collected into the SparkMetric_CL Log Analytics custom log. We can use simple KQL query to work on them as shown below.
SparkMetric_CL
| where name_s contains "executor.cpuTime"
| extend sname = split(name_s, ".")
| extend executor=strcat(sname[0], ".", sname[1])
| project TimeGenerated, cpuTime=count_d / 100000
Structured Streaming metrics
Streaming job metrics are automatically collected into the SparkListenerEvent_CL Log Analytics custom log. Similar to the one mentioned above we can use simple KQL query to work on them as shown in the example below.
SparkListenerEvent_CL
| where Event_s contains "queryprogressevent"
| extend sname=strcat(progress_name_s,"-","triggerexecution")
| summarize percentile(progress_duration_Ms_triggerExecution_d, 90) by bin (TimeGenerated, 1m), sname
| order by TimeGenerated asc null last
Spark logs
Spark logs are available in the Databricks UI and can be delivered to a storage account. However, Log Analytics is a much more convenient log store since it indexes the logs at high scale and supports a powerful query language. Spark logs are automatically collected into the SparkLoggingEvent_CL Log Analytics custom log. Following are the example of simple KQL query.
SparkLoggingEvent_CL
| project Message
| limit 50
There are also some prebuilt query names for retrieving Spark metrics in this Access prebuilt queries section of the Microsoft document it will help you solve the remaining problems. When you open log analytics workspace and go to logs, it will open a query dialog. There you can search this prebuild queries.
I would suggest to read these Stream processing with Azure Databricks
and Use Kusto queries documents for more information.
We have a large web application. We need to store user analytics about which ads has users seen. What accordions has been clicked etc. As far as I can see log analytics and application insights is more targeted towards logging and instrumentation (correct me if I am wrong). Does Azure provide any services for this kind of functionality, or would you have to store it your self to a SQL server/Table storage/CosmosDB and make a job that calculates the different statistics needed?
If "which ads has users seen" is actually a page view / request operation, then you can choose application insights.
If you're using app insights SDK, then you can track the events for these ads by using methods like TrackEvent. This event data is useful for analytics in application insights.
For analytics, you can use log queries which is available for azure monitor / application insights(to use log query, nav to azure portal -> your application insights -> in the left panel, select "Logs(Analytics)" under Monitoring). With log query, you can build your own query for any different statistics, or generate some charts.
Please let me know if you have more concerns.
I'm working on a Azure project which has several Azure functions app. One function app is linked to one Application Insights. So we have as much as functions app as Applications Insights.
These functions app will talk between them, and some processes can pass by multiple functions app. On the Azure function, I add logs with a custom EntityID and this EntityID is the same between all function apps.
Now my question is : How I can retrieve all the logs with the same EntityID on all the Applications Insights ?
Like :
traces
| where operation_Name == "TestLogManagement"
| where customDimensions.EventId == 888
but on several Applications Insights.
I know that we can regroup multiple Applications Insights by a Tag but no way to retrieve my logs after I created it.
app('app1').traces
| union app('app2').traces
| where * contains "42"
| summarize count() by appId
Please have a look here:
Unify multiple Azure Monitor Application Insights resources
You can also do this in workbooks in azure monitor, so you don't need to encode app names in the query, you can just write the query you want and pick multiple instances of either AI resources or LA workspaces
We've logs(W3CIISLogs) on Log analytics workspace for websites hosted on VMs. Similarly we have app insights enabled for websites hosted on App service. Now we want to access telemetry data of both type of websites thru single interface, either via app insights or via Log analytics. Just wondering if it's possible and what's the best way.
With Azure Monitor you can now query not only across multiple Log Analytics workspaces, but also data from a specific Application Insights app in the same resource group, another resource group, or another subscription. This provides you with a system-wide view of your data. You can only perform these types of queries in Log Analytics.
Querying across Log Analytics workspaces and from Application Insights - reference another workspace in your query, use the workspace identifier and for an app from Application Insights, use the app identifier.
Cross-resource query limits:
The number of Application Insights resources that you can include in
a single query is limited to 100.
Cross-resource query is not supported in View Designer. You can Author a query in Log
Analytics and pin it to Azure dashboard and visualize a log search.
Cross-resource query in log alerts is supported in the new
scheduledQueryRules API. By default, Azure Monitor uses the legacy
Log Analytics Alert API for creating new log alert rules from Azure
portal, unless you switch from legacy Log Alerts API. After the
switch, the new API becomes the default for new alert rules in Azure
portal and it lets you create cross-resource query log alerts rules.
You can create cross-resource query log alert rules without making
the switch by using the ARM template for scheduledQueryRules API –
but this alert rule is manageable though scheduledQueryRules API and
not from Azure portal.
Documentation Reference - Cross-Resource Log queries in Azure Monitor
Hope the above information helps.
Our service writes a lot of custom events and metrics to App Insights.
Using the AI portal we can make complex queries and see nice charts, but I would like to access the data from outside the portal.
The Azure Application Insights REST API page (https://dev.applicationinsights.io) claims that those API can be used to perform such task, but I am unable to make them work - again I want to query custom events and metrics, not the standard ones.
Does anyone have any examples?
Here is for example one our queries:
customEvents
| where name startswith "Monitor.Xxxxxxx"
| summarize count() by bin(timestamp, 1min)
| order by timestamp desc
It turned out I was using the wrong AppId/Key; once I plugged the correct ones I am able to use the API Explorer.