Application Insights Log4Net Appender - telemetry sampling and reliability - log4net

We use log4net and DB / File appenders currently to capture log4net logging. In Production, while we mostly have WARN and higher level logging alone, there are occasions INFO level logging is turned on to troubleshoot end user issues.
These appenders are reliable and we haven't had experience where logs were lost so our support could not troubleshoot issues.
We are setting up Azure hosting for our application and Application Insights is adding a lot of value with its standard telemetry collection. We are now evaluating redirecting log4net logs into Application Insights and gradually make AI the log aggregator.
MS has released a log4net AI appender - https://www.nuget.org/packages/Microsoft.ApplicationInsights.Log4NetAppender
I did not find enough information on how sampling can affect the logs. For e.g. if some INFO logs are lost to sampling, it is not ideal but we might still be ok. If WARN, ERROR or FATAL logs are lost, that will just make AI unreliable for our support to use and they would just fall back to the traditional logs in DB or files.
Sampling on automatic telemetry (like requests, dependencies) is ok and understood, but is there a mechanism to ensure log4net logs are reliably made available in Application Insights?

According to this issue, the Log4Net appender should be sampled in the same way as other telemetry is sampled.
So you can just turn off the sampling(the default sampling is Adaptive sampling) by using code.
For example, if it's an .net core application, you can use the code below as per this doc:
public void ConfigureServices(IServiceCollection services)
{
// ...
var aiOptions = new Microsoft.ApplicationInsights.AspNetCore.Extensions.ApplicationInsightsServiceOptions();
aiOptions.EnableAdaptiveSampling = false;
services.AddApplicationInsightsTelemetry(aiOptions);
//...
}

Related

Trying to add heartbeat for my console application

A discussion on github mentions that heartbeat is automatically sent when Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsTelemetryModule& is added to the applicationinsights.config file. I want to create an alert in case the app hangs or crashes. Where can I track the heart beat on the AI instance? Also, currently its not sending anything . I am using .net framework 4.7.2. Am I configuring it wrong? I am not able to track the heartbeat on application insights instance. Where can I track it? Can someone provide a snippet for config file?
Currently, there is an issue involving this telemetry module AppServicesHeartbeatTelemetryModule .
Can someone provide a snippet for config file?
There is given a temporary workaround for WorkerServices.
Add application insights to worker service using services.AddApplicationInsightsTelemetryWorkerService().
Not only this DiagnosticsTelemetryModule adding to DI and also we need to configure it's heartbeat interval/module using the below snippet:
services.TryAddSingleton<ITelemetryModule, DiagnosticsTelemetryModule>();
services.ConfigureTelemetryModule<DiagnosticsTelemetryModule>((mod,opt) => mod.HeartbeatInterval = TimeSpan.FromSeconds(30));
services.AddApplicationInsightsTelemetryWorkerService();
The TelemetryConfiguration.Active is not recommended because Worker service is new SDK, its not touching .active or any other static singletons.
Where can I track it?
After Deploying, Look for "HeartbeatState" in customMetrics.
Note:
Disabling DiagnosticsTelemetryModule will cause the following settings to be ignored: EnableHeartbeat, EnableAzureInstanceMetadataTelemetryModule, EnableAppServicesHeartbeatTelemetryModule.
Enable Azure Monitor from the Portal.
References:
Configuring or removing default TelemetryModules (HeartBeat Telemetry Issue)
Enabling Hearbeat in App Services for .NET and .NET Core

Azure function reduce log on Application insight

When my system is running some time I got the connection error so I want to remove it from my Application Insign
It is possible If I want to remove the exception and trace come from EventProcessorHost error. You can see my insign log as below.
The only way is that you can use app insights Purge api to delete logs from Exceptions table and Traces table.
But the limitation is that you cannot specify such detailed filters, like the messages are from EventProcessorHost etc.
And the delete operation will be competed in 7 days in background, you should know these limitaions when using this api.
If the question was "how do i not collect these in the future", I believe the information you are looking for is here:
https://learn.microsoft.com/en-us/azure/azure-functions/functions-monitoring?tabs=cmd#configure-categories-and-log-levels
summary:
Log configuration in host.json
The host.json file configures how much logging a function app sends to Application Insights. For each category, you indicate the minimum log level to send
there are a lot of samples in the link above to turn on and off things of various levels, sources, sampling, batching and is probably too much to paste here and keep up to date

How does ILogger logs to Azure Application Insights?

In an Azure Function, when you enable telemetry to Application Insight and fire a (for example) logger.LogInformation call (where logger is an ILogger instance), does it send it to the Application Insight instance asynchronously (ie non-blocking), synchronously (blocking), or through a local log that gets drained asynchronously?
Generally, the logger would be hooked up to turn log calls into the various trackMessage or related calls in the Application Insights SDK. those messages get batched up in the AI side, and then sent after a threshold count of messages has been met, or after a certain amount of time has elapsed. the calls into application insights are all non-blocking, and will not throw exceptions (you don't want telemetry to negatively affect your real app!)
the c# sdks that azure functions would use would be here: https://github.com/Microsoft/ApplicationInsights-dotnet/
I said generally at the top, because all this depends on how the SDK is configured, and that would be up to the Azure functions underlying code. The GitHub with their info is here: https://github.com/Azure/Azure-Functions, and they have a specific wiki set up with AI info as well, here: https://github.com/Azure/Azure-Functions/wiki/App-Insights
This appears the be the relevant code for specifically how data is sent to Application Insights:
https://github.com/Microsoft/ApplicationInsights-dotnet/tree/develop/src/Microsoft.ApplicationInsights/Channel
The ILogger wraps a TelemetryClient, which sends data to an ITelemetryChannel.
The InMemoryTelemetryChannel contains the logic for how data is pooled and sent to Application Insights. As John mentioned, the channel uses a "buffer" for storing data that hasn't been sent. The buffer is flushed and the data sent asynchronously to Azure Portal when either the buffer is full or at a specific time internal (30 seconds).

Azure - App Availability percentage is Zero

our Api app is in UAT on Azure with service plan (Standard 3 large). What should we do if App Availability is Zero. It is getting slow response or timeout issue. When i restart the application it is up to normal. (We are using Parallel Language programming.(Async/Await)
How to find the route cause from it for slowness issue.
Ensure that Always On feature is enabled.
Such problems may be caused by application level issues, such as:
network requests taking a long time
application code or database queries being inefficient
application using high memory/CPU
application crashing due to an exception
You could enable web server diagnostics to fetch more details on the issue.
Detailed Error Logging - Detailed error information for HTTP status codes that indicate a failure (status code 400 or greater). This may contain information that can help determine why the server returned the error code.
Failed Request Tracing - Detailed information on failed requests, including a trace of the IIS components used to process the request and the time taken in each component. This can be useful if you are attempting to improve web app performance or isolate what is causing a specific HTTP error.
Web Server Logging - Information about HTTP transactions using the W3C extended log file format. This is useful when determining overall web app metrics, such as the number of requests handled or how many requests are from a specific IP address.
Also, Azure Application Insights collects telemetry from your application to help analyze its operation and performance. You can use this information to identify problems that may be occurring or to identify improvements to the application that would most impact users. This tutorial takes you through the process of analyzing the performance of both the server components of your application and the perspective of the client: https://learn.microsoft.com/en-us/azure/application-insights/app-insights-tutorial-performance
Ref: https://learn.microsoft.com/en-us/azure/app-service/app-service-web-troubleshoot-performance-degradation

TelemetryClient produces inconsistent results in Application Insights

I tried tracking custom metrics with and without flushing it. However, the metrics only intermittently shows up in Application Insights under the "Custom" section. First question: Is it required to run "flush()" after every single "TrackMetric(metric)" call in order for the telemetry to be sent to Application Insights? Second: Why is there this intermittent behavior? I'm only writing one metric at a time, so it's not as if I'm overloading Application Insights with thousands of separate calls. Here is my code (This is from a simple Console App):
public class Program
{
public static void Main(string[] args)
{
var telemetryClient = new TelemetryClient()
{
Context = { InstrumentationKey = "{{hidden instrumentation key}}" }
};
var metric = new MetricTelemetry
{
Name = "ImsWithContextMetric2",
Sum = 42.0
};
telemetryClient.TrackMetric(metric);
telemetryClient.Flush();
}
}
I'm also getting this strange behavior in Application Insights in which the custom metric I add shows up under a "Unavailable/deprecated Metrics" section. And a metric that I didn't even add called "Process CPU (all cores)" pops up under the "Custom" section. Any ideas why this strange behavior would occur?:
Is it required to run "flush()" after every single "TrackMetric(metric)" call in order for the telemetry to be sent to Application Insights?
Since you are using a Console Application to send events to Application Insights, which might be short-lived, it is definitely a good practice to call .Flush() every once in a while. The SDK uses the InMemoryChannel to send telemetry and sends it in batches using from an in-memory queue. So it is very important to call the .Flush() so that the data is forcefully pushed. A good practice might be to add a bit of wait after the event:
telemetryClient.Flush();
Thread.Sleep(1000);
More reading: Flushing data, Ensure you don't lose telemetry
However, the metrics only intermittently shows up in Application Insights under the "Custom" section. Why is there this intermittent behavior? I'm only writing one metric at a time, so it's not as if I'm overloading Application Insights with thousands of separate calls.
Sometimes there is a delay in metrics showing up in the Azure Portal. It can be up to a few minutes too. But if you have set it up correctly, you aren't exceeding the throttling limit, and adaptive sampling is disabled, then there is no reason for which telemetry should be intermittent. However if you still feel something is wrong, start a fiddler trace (make sure you are capturing from non-browser sessions) and check if a call is going out to dc.services.visualstudio.com. Make sure the response is 200 OK and if the items were accepted by the server.
I'm also getting this strange behavior in Application Insights in which the custom metric I add shows up under a "Unavailable/deprecated Metrics" section.
What version of the SDK are you using? I just tried out the same scenario and the custom metrics are showing up correctly.
And a metric that I didn't even add called "Process CPU (all cores)" pops up under the "Custom" section.
"Process CPU" is a performance counter which is used to track CPU utilization. I believe the SDK will only be able to track these counters if the app is running under IIS or on Azure. It probably got added internally when you created your Application Insights resource. You can ignore it since it won't have data to chart.
Hope this helps!

Resources