We have a large web application. We need to store user analytics about which ads has users seen. What accordions has been clicked etc. As far as I can see log analytics and application insights is more targeted towards logging and instrumentation (correct me if I am wrong). Does Azure provide any services for this kind of functionality, or would you have to store it your self to a SQL server/Table storage/CosmosDB and make a job that calculates the different statistics needed?
If "which ads has users seen" is actually a page view / request operation, then you can choose application insights.
If you're using app insights SDK, then you can track the events for these ads by using methods like TrackEvent. This event data is useful for analytics in application insights.
For analytics, you can use log queries which is available for azure monitor / application insights(to use log query, nav to azure portal -> your application insights -> in the left panel, select "Logs(Analytics)" under Monitoring). With log query, you can build your own query for any different statistics, or generate some charts.
Please let me know if you have more concerns.
Related
I have a hard time here understanding what a log analytics workspace is. As I have a requirement to monitor 7 application insigths out of many. Send emails if some of them throws exceptions. I can see in alerts you can only select a single application insight. And I dont want to create 7 alerts. So my plan was to create a analytics workspace. I havent found anyway to bind application insights to a workspace. Is that possible? I can see you in alerts can choose a analytics workspace as source, but what does that mean? Sounds like you some how can group data in that specific workspace?
I can see when I create a new Application Insight service i have the option to choose a workspace, but what with existing?
So as per your requirement, you can create the Workspace-based Application Insights resources, or connect the existing classical AI to azure log analytics. And of course, these 6 AI should connect to the same azure log analytics.
After you have the the Workspace-based Application Insights resources, the logs are logged into both the AI and the azure log analytics. So when you create an alert rule, select the azure log analytics, and create a custom log search alert rule. That can totally meet your requirement.
And you should also understand the table scheme between AI and Azure log Analytics. For example, in AI, the exceptions are logged in exceptions table; but if it's connected to azure log analytics, then in log analytics, the exceptions are in AppSystemEvents table.
There's an awful lot of disjointed documentation on monitoring network/resources in Azure. What I'm looking for is which pieces are needed to get information from VMs, NVA firewalls, azure load balancers, and other network resources and network connectivity into a single pain of glass in Azure. Only concerned about Azure, not on-prem for now.
I've come across azure monitor, log analytics work spaces, event hub, vm extensions, network watcher, insights, etc...but I'm not sure which are required and which are not. One doc leads to the next and I end up with 30 tabs open. I'll also need to be able to push logs to other security devices such as a SIEM.
Does anyone know of a deployment guide that wraps this all up in a more logical fashion? Does anyone have any feedback on which pieces from azure (not 3rd parties) are required at a minimum to accomplish a single pane of glass to view my Azure environment holistically?
General overview of observability in Azure
Likely, the thing you're looking for is Azure Monitor. It's an umbrella term for everything observability related inside Azure.
To store Metrics and Logs you need Log Analytics: it can query data with kusto query language, visualize results, define Alerts on queries.
Alerts is quite a complex beast, as it is spread across the entire cloud. Two types that I use the most:
log-analytics alert (which I mentioned above)
Alerts tab, which is available at every Azure component view. for example, open resource group, and scroll down to Monitoring section
Each component also has a subset of built-in metrics. Likely, you noticed that many azure components on the Overview view display some charts. For example, Azure Storage Account displays Total egress, Total ingress, and other line-charts. When you click on these charts you can customize them. These metrics and charts are free to use.
Microsoft also has all-in-one observability solution for Azure Functions and Web Apps: Application Insights
Dashboards allows to join multiple charts into a single view and share it with others.
If you care about security, Azure proposes Azure Security Center
Deployment/management strategy
I suggest to start with:
Create Log Analytics Workspace, which is the storage for metrics and logs. The azure docs article explains how to design it: how many instances to use, how to rate-limit ingestion (it might be expensive if goes out of control), how to access it and so on.
To get Azure components logs, look for Diagnostic Settings tab at a component page at Azure portal, but not all components has it (sic!). I suggest
sending the most critical data to Log Analytics workspace to store them in a queryable format for 30 days (it's in free tier). This is needed for investigating current issues with your infrastructure
if you might need logs later than 30 days - send them to Storage Account
you mentioned SIEM integration - route required events to Event Hub and then process the stream according to your requirements
So, if you need long-term storage - you need to create Azure Storage Account.
If you need real-time analysis - you need to build a pipeline based on Azure Event Hub.
If you have Azure Functions and Web Apps - add Application Insights. According to my experience, I would suggest starting with a separate instance per each Azure Function resource or Service.
Create Alerts for each component separately. If you do it through UI - open component page at the portal and look for Alerts tab there. If you're automating the process (please do so as soon as possible), do not expect easy trip: I used ARM templates and terraform - in both cases, there are dozens of barely documented features.
Join related components core-metrics into Dashboards and share it with the team. This guide is a good starting point. Note, when you share the dashboard, it's also persisted as an azure resource in the subscription.
We've logs(W3CIISLogs) on Log analytics workspace for websites hosted on VMs. Similarly we have app insights enabled for websites hosted on App service. Now we want to access telemetry data of both type of websites thru single interface, either via app insights or via Log analytics. Just wondering if it's possible and what's the best way.
With Azure Monitor you can now query not only across multiple Log Analytics workspaces, but also data from a specific Application Insights app in the same resource group, another resource group, or another subscription. This provides you with a system-wide view of your data. You can only perform these types of queries in Log Analytics.
Querying across Log Analytics workspaces and from Application Insights - reference another workspace in your query, use the workspace identifier and for an app from Application Insights, use the app identifier.
Cross-resource query limits:
The number of Application Insights resources that you can include in
a single query is limited to 100.
Cross-resource query is not supported in View Designer. You can Author a query in Log
Analytics and pin it to Azure dashboard and visualize a log search.
Cross-resource query in log alerts is supported in the new
scheduledQueryRules API. By default, Azure Monitor uses the legacy
Log Analytics Alert API for creating new log alert rules from Azure
portal, unless you switch from legacy Log Alerts API. After the
switch, the new API becomes the default for new alert rules in Azure
portal and it lets you create cross-resource query log alerts rules.
You can create cross-resource query log alert rules without making
the switch by using the ARM template for scheduledQueryRules API –
but this alert rule is manageable though scheduledQueryRules API and
not from Azure portal.
Documentation Reference - Cross-Resource Log queries in Azure Monitor
Hope the above information helps.
Following is the proposed transition in our application:
Web Application is deployed in on-premises IIS (Web Server 1).
Web Application has one functionality (for example, Generate Invoice for selected customer).
For each new request of Generate Invoice, the web application is writing message to the Azure Service Bus Queue.
Azure function gets triggered for each new message in Azure Service Bus Queue.
Azure function triggers Web API (deployed on-premises).
Web API generates Invoice for the customer and stores in the local file storage.
As of now, we have everything setup on-premises, and instead of Service Bus and Azure function, we directly consume Web API. With this type of infrastructure in place, we are currently logging all events in an MongoDB collection, and providing single consolidated view to the user. So they can identify what happened to the Generate Invoice request, and at which level and with which error it got failed (in case of failures).
With the new proposed architecture, we are in process of identifying ways for logging and tracing here, and display consolidated view to the users.
The only option, I can think of is to log all events in Azure Cosmos DB from everywhere (i.e., Website, Service bus, function, Web API), and then provide consolidated view.
Can anyone suggest if the suggested approach looks OK? Or if anyone has some better solution?
Application Insights monitors the availability, performance, and usage of your web applications whether they're hosted in the cloud or on-premises. It leverages the powerful data analysis platform in Azure Monitor to provide you with deep insights into your application's operations and diagnose errors without waiting for a user to report them.
Workbooks combine data visualizations, Analytics queries, and text into interactive documents. You can use workbooks to group together common usage information, consolidate information from a particular incident, or report back to your team on your application's usage.
For more details, you could refer to this article.
I have the following resources
One Mobile/API app
One MVC app
Three Logic apps
One Azure function deployment with 5 functions
I want to have a single tracking number (correlation ID) to track across all instances at the same time. I'm looking at the Contoso Insurance sample, but I'm rebuilding it by hand (not using Azure Deploy scripts).
I've read the deployment code, but I'm not sure if I can merge app insight logs together, or if it's a hack of some sort.
Observations
When I right click on visual studio, I can only associate to Application insights instances that aren't already connected to a *app (web | mobile | api).
However, in the configuration, I can give Application insights a direct GUID which might allow me to achieve the goal of one App Insights activity log for the entire process
Question
Is it possible to have one app insights log among all Mobile/API/Logic/MVC sites?
Is there a way to have (or should I have) one standard app insights instance per web app, then a special dedicated shared app insights instance for my code to call into and log?
What is contoso insurance doing with Azure App Insights?
Jeff from Logic Apps team here -- So the answer is yes - but there are some caveats. We are working to make the experience seamless and automatic, but for now it will require the following. First as a heads up:
First, for Logic Apps we have what's called the client tracking ID -- this is a header you can set on an incoming HTTP Request or Service Bus message to track and correlate events across actions. It will be sent to all steps (functions, connectors, etc.) with the x-ms-client-tracking-id header.
Logic Apps emits all logs to Azure Monitor - which unfortunately today only has a sink into Event Hubs, Storage, and Log Analytics -- not App Insights.
With all of that in-mind, here's the architecture we see many following:
Have your web apps just emit to App Insights directly. Use some correlation ID as needed. When firing any Logic Apps, pass in the x-ms-client-tracking-id header so you can correlate events.
Log your events to App Insights in the Function app. This blog details some of how to do that, and it is also being worked on for a better experience soon.
In your logic app - either write a Function to consume events off of Azure monitor and push to App Insights, or write a function that is an App Insight "logger" that you can call in your workflow to also get the data into App Insights.
This is how Contoso Insurance is leveraging App Insights as far as I understand. We are working across all teams (App Insights, Azure Monitor, Azure Functions, Logic Apps) to make this super-simple and integrated in the coming weeks/months, but for now achievable with above. Feel free to reach out for any ?s