I have created the following TelemetryFilter:
public class TelemetryFilter : ITelemetryProcessor
{
private ITelemetryProcessor Next { get; set; }
public TelemetryFilter(ITelemetryProcessor next)
{
Next = next;
}
public void Process(ITelemetry item)
{
var dependency = item as DependencyTelemetry;
if (dependency != null && dependency.Success == true) return;
Next.Process(item);
}
}
And added TelemetryFilter to TelemetruyProcessors in ApplicationInsights.config. It works when I run the application on my machine but when it is deployed to test and production environments, dependencies are getting collected by Azure AppInsights. When I see them in Azure Portal they have the property Call status: true. Is Call status refers to dependency.Success? What's the best way to filter out all successful calls to decrease our AppInsights data ingress and lower our Azure bill?
Filter out all successful dependencies:
you can initialize the filter in code. In a suitable initialization class,
AppStart in Global.asax.cs, insert your processor into the chain:
var builder = TelemetryConfiguration.Active.DefaultTelemetrySink.TelemetryProcessorChainBuilder;
builder.Use((next) => new SuccessfulDependencyFilter(next));
// If you have more processors:
builder.Use((next) => new AnotherProcessor(next));
builder.Build();
Refer for filtering sampling
& for request filtering
To Reduce Application Insights cost
You need to optimize Telemetry with Application Insights check here
Check here for some more methods to reduce Application insights cost
I found that ApplicationInsights.config file wasn't set to be copied into the output folder by the build process. That's why it didn't work.
Related
Previously I have Azure Web App (.net core) and It successfully track the SQL Server and Service Bus dependency into Application Insights. It is not working some how with Azure Functions.
Environment
dotnet 6
dotnet-isolated mode
log level default set to "Information".
Azure Environment using Consumption plan for Azure Functions.
Application Insights key is configured.
I have Azure API management at front and backend is Azure Function and that call SQL Server and Service Bus.
Api Management Service to Azure function dependency successfully resolved but Azure Function to other component is not working.
I know I am posting my own answer. Also there are chance that in future there may be some good solution or it get integrated the way it is in in-process mode.
By then follow steps.
Add Package
Microsoft.ApplicationInsights.WorkerService
In program.cs in configuring host.
services.AddApplicationInsightsTelemetryWorkerService();
More info at
https://learn.microsoft.com/en-us/azure/azure-monitor/app/worker-service
The only way I've managed to solve this issue so far was by setting up custom Middleware
.ConfigureFunctionsWorkerDefaults(config =>
{
config.UseMiddleware<AiContextMiddleware>();
})
In the IServiceCollection you need to setup simply
.AddApplicationInsightsTelemetryWorkerService()
public class AiContextMiddleware : IFunctionsWorkerMiddleware
{
private readonly TelemetryClient _client;
private readonly string _hostname;
public AiContextMiddleware(TelemetryClient client)
{
_client = client;
_hostname = Environment.GetEnvironmentVariable("AI_CLOUD_ROLE_NAME");
}
public async Task Invoke(FunctionContext context, FunctionExecutionDelegate next)
{
var operationId = ExtractOperationId(context.TraceContext.TraceParent);
// Let's create and start RequestTelemetry.
var requestTelemetry = new RequestTelemetry
{
Name = context.FunctionDefinition.Name,
Id = context.InvocationId,
Properties =
{
{ "ai.cloud.role", _hostname},
{ "AzureFunctions_FunctionName", context.FunctionDefinition.Name },
{ "AzureFunctions_InvocationId", context.InvocationId },
{ "AzureFunctions_OperationId", operationId }
},
Context =
{
Operation =
{
Id = operationId,
ParentId = context.InvocationId,
Name = context.FunctionDefinition.Name
},
GlobalProperties =
{
{ "ai.cloud.role", _hostname},
{ "AzureFunctions_FunctionName", context.FunctionDefinition.Name },
{ "AzureFunctions_InvocationId", context.InvocationId },
{ "AzureFunctions_OperationId", operationId }
}
}
};
var operation = _client.StartOperation(requestTelemetry);
try
{
await next(context);
}
catch (Exception e)
{
requestTelemetry.Success = false;
_client.TrackException(e);
throw;
}
finally
{
_client.StopOperation(operation);
}
}
private static string ExtractOperationId(string traceParent)
=> string.IsNullOrEmpty(traceParent) ? string.Empty : traceParent.Split("-")[1];
}
It's definitely not a perfect solution as you then get two starting logs, but as end result, you get all logs traces + dependencies correlated to an operation.
I've solved this issue in the first place like that, now I'm revisiting whether there are any better ways to solve this.
Let me know too whether you managed to solve this issue on your side.
We are using smart detectors in Azure App Insights to generate some alerts when there are anomalies in our app. However, there are some failures that are intentional in our code, where we throw a 403. Is there a way to modify these "Smart Alerts" in Application Insights, so that these known failures can be excluded in its detection logic? We have a specific exception type that is relevant for these expected failures that we can easily use to exclude these in the anomaly detection if there is a way to do that, but I can't find an option on the UI to do this.
Thanks for any pointers.
You cannot do that directly from the Azure portal but you need to implement a Telemetry Processor which can help you override telemetry properties set.
If request flag as failed with a response code = 403. But if you want to treat it as a success, you can provide a telemetry initializer that sets the success property.
Define your initializer
C#
using System;
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.DataContracts;
using Microsoft.ApplicationInsights.Extensibility;
namespace MvcWebRole.Telemetry
{
/*
* Custom TelemetryInitializer that overrides the default SDK
* behavior of treating response codes >= 400 as failed requests
*
*/
public class MyTelemetryInitializer : ITelemetryInitializer
{
public void Initialize(ITelemetry telemetry)
{
var requestTelemetry = telemetry as RequestTelemetry;
// Is this a TrackRequest() ?
if (requestTelemetry == null) return;
int code;
bool parsed = Int32.TryParse(requestTelemetry.ResponseCode, out code);
if (!parsed) return;
if (code >= 400 && code < 500)
{
// If we set the Success property, the SDK won't change it:
requestTelemetry.Success = true;
// Allow us to filter these requests in the portal:
requestTelemetry.Properties["Overridden400s"] = "true";
}
// else leave the SDK to set the Success property
}
}
}
In ApplicationInsights.config:
XMLCopy
<ApplicationInsights>
<TelemetryInitializers>
<!-- Fully qualified type name, assembly name: -->
<Add Type="MvcWebRole.Telemetry.MyTelemetryInitializer, MvcWebRole"/>
...
</TelemetryInitializers>
</ApplicationInsights>
For more information, you can refer to this Document.
Is it possible to tell Application Insights to use a different InstrumentationKey depending on the request URL?
Our application works with different clients and we want to separate the logs for them in different instances of Application Insights.
Url format: https://webapi.com/v1/{client_name}/bla/bla
It would be great to setup configuration to select InstrumentationKey by client_name from request.
If the goal is to send different telemetry items to different instrumentation key, the correct way to achieve that is by modifying the individual item with a TelemetryInitializer to have the correct ikey.
An initializer like the following:
item.Context.InstrumentationKey = ikey.
This initializer should access HttpContext and decide the ikey dynamically from request route/other params.
Modifying TC.Active is not recommended for this purpose as its a global shared setting.
(This is not a very common use case - but there are teams inside Microsoft who does this for PROD scale apps)
You can do that. If you have a logger, have the ApplicationInsightsKey parameter-ized and pass the Key for the client on every call, or inject it on load if your application is tenant based.
Checkout the Docs here: Separating telemetry from Development, Test, and Production
Microsoft.ApplicationInsights.Extensibility.
TelemetryConfiguration.Active.InstrumentationKey = <App-Insights-Key-for-the-client>
Just change the Application Insights key before logging and it will do the job.
It would be great to setup configuration to select InstrumentationKey
by client_name from request.
You can dynamically select the ikey as per the client_name from the request. First, you need to get the request url, then check the client_name.
To do that, you can add the following code to the Global.asax file:
void Application_BeginRequest(Object source, EventArgs e)
{
var app = (HttpApplication)source;
//get the request url
var uriObject = app.Context.Request.Url.ToString();
if (uriObject.Contains("/client_name_1"))
{
Microsoft.ApplicationInsights.Extensibility.
TelemetryConfiguration.Active.InstrumentationKey = "ikey_1";
}
else if (uriObject.Contains("/client_name_2"))
{
Microsoft.ApplicationInsights.Extensibility.
TelemetryConfiguration.Active.InstrumentationKey = "ikey_2";
}
else
{
Microsoft.ApplicationInsights.Extensibility.
TelemetryConfiguration.Active.InstrumentationKey = "ikey_3";
}
}
The test result:
But I want to say we rarely use 1 more ikeys in one environment. If your goal is to make the data not being cluttered, I suggest you can use only 1 ikey, and then use Kusto query for your purpose.
Thanks to the answers from #cijothomas and #danpop (link) I was able to understand the whole picture.
Step 1: Create custom ITelemetryInitializer (Microsoft Documentation):
public class MyTelemetryInitializer : ITelemetryInitializer
{
public void Initialize(ITelemetry telemetry)
{
var appKey = CallContext.LogicalGetData("ApplicationKey")?.ToString();
switch (appKey)
{
case "App1":
telemetry.Context.InstrumentationKey = "d223527b-f34e-4c47-8aa8-1f21eb0fc349";
return;
default:
telemetry.Context.InstrumentationKey = "f8ceb6cf-4357-4776-a2b6-5bbed8d2561c";
return;
}
}
}
Step 2: Register custom initializer:
<ApplicationInsights xmlns="http://schemas.microsoft.com/ApplicationInsights/2013/Settings">
<TelemetryInitializers>
<Add Type="Application.WebAPI.MyTelemetryInitializer, Application.WebAPI"/>
</TelemetryInitializers>
<!--<InstrumentationKey>f8ceb6cf-4357-4776-a2b6-5bbed8d2561c</InstrumentationKey>-->
</ApplicationInsights>
OR
protected void Application_Start()
{
// ...
TelemetryConfiguration.Active.TelemetryInitializers.Add(new MyTelemetryInitializer());
}
Step 3: Make some adjustments to the logger (source code taken from #danpop answer Logger target configuration):
var config = new LoggingConfiguration();
ConfigurationItemFactory.Default.Targets.RegisterDefinition("ai", typeof());
ApplicationInsightsTarget aiTarget = new ApplicationInsightsTarget();
aiTarget.InstrumentationKey = "your_key";
aiTarget.Name = "ai";
config.AddTarget("ai", aiTarget);
LogManager.Configuration = config;
ILogger configuration exmples: Log4Net, NLog, System.Diagnostics
I'm looking to have our distributed event logging have proper correlation. For our Web Applications this seems to be automatic. Example of correlated logs from one of our App Services API:
However, for our other (non ASP, non WebApp) services were we use Log4Net and the App Insights appender our logs are not correlated. I tried following instructions here: https://learn.microsoft.com/en-us/azure/azure-monitor/app/correlation
Even after adding unique operation_Id attributes to each operation, we're not seeing log correlation (I also tried "Operation Id"). Example of none correlated log entry:
Any help on how to achieve this using log4net would be appreciated.
Cheers!
Across services correlation IDs are mostly propagated through headers. When AI is enabled for Web Application, it reads IDs/Context from the incoming headers and then updates outgoing headers with the appropriate IDs/Context. Within service, operation is tracked with Activity object and every telemetry emitted will be associated with this Activity, thus sharing necessary correlation IDs.
In case of Service Bus / Event Hub communication, the propagation is also supported in the recent versions (IDs/Context propagate as metadata).
If service is not web-based and AI automated correlation propagation is not working, you may need to manually get incoming ID information from some metadata if any exists, restore/initiate Activity, start AI operation with this Activity. When telemetry item is generated in scope of that Activity, it will get proper IDs and will be part of the overarching trace. With that, if telemetry is generated from Log4net trace that was executed in the scope of AI operation context then that telemetry should get right IDs.
Code sample to access correlation from headers:
public class ApplicationInsightsMiddleware : OwinMiddleware
{
// you may create a new TelemetryConfiguration instance, reuse one you already have
// or fetch the instance created by Application Insights SDK.
private readonly TelemetryConfiguration telemetryConfiguration = TelemetryConfiguration.CreateDefault();
private readonly TelemetryClient telemetryClient = new TelemetryClient(telemetryConfiguration);
public ApplicationInsightsMiddleware(OwinMiddleware next) : base(next) {}
public override async Task Invoke(IOwinContext context)
{
// Let's create and start RequestTelemetry.
var requestTelemetry = new RequestTelemetry
{
Name = $"{context.Request.Method} {context.Request.Uri.GetLeftPart(UriPartial.Path)}"
};
// If there is a Request-Id received from the upstream service, set the telemetry context accordingly.
if (context.Request.Headers.ContainsKey("Request-Id"))
{
var requestId = context.Request.Headers.Get("Request-Id");
// Get the operation ID from the Request-Id (if you follow the HTTP Protocol for Correlation).
requestTelemetry.Context.Operation.Id = GetOperationId(requestId);
requestTelemetry.Context.Operation.ParentId = requestId;
}
// StartOperation is a helper method that allows correlation of
// current operations with nested operations/telemetry
// and initializes start time and duration on telemetry items.
var operation = telemetryClient.StartOperation(requestTelemetry);
// Process the request.
try
{
await Next.Invoke(context);
}
catch (Exception e)
{
requestTelemetry.Success = false;
telemetryClient.TrackException(e);
throw;
}
finally
{
// Update status code and success as appropriate.
if (context.Response != null)
{
requestTelemetry.ResponseCode = context.Response.StatusCode.ToString();
requestTelemetry.Success = context.Response.StatusCode >= 200 && context.Response.StatusCode <= 299;
}
else
{
requestTelemetry.Success = false;
}
// Now it's time to stop the operation (and track telemetry).
telemetryClient.StopOperation(operation);
}
}
public static string GetOperationId(string id)
{
// Returns the root ID from the '|' to the first '.' if any.
int rootEnd = id.IndexOf('.');
if (rootEnd < 0)
rootEnd = id.Length;
int rootStart = id[0] == '|' ? 1 : 0;
return id.Substring(rootStart, rootEnd - rootStart);
}
}
Code sample for manual correlated operation tracking in isolation:
async Task BackgroundTask()
{
var operation = telemetryClient.StartOperation<DependencyTelemetry>(taskName);
operation.Telemetry.Type = "Background";
try
{
int progress = 0;
while (progress < 100)
{
// Process the task.
telemetryClient.TrackTrace($"done {progress++}%");
}
// Update status code and success as appropriate.
}
catch (Exception e)
{
telemetryClient.TrackException(e);
// Update status code and success as appropriate.
throw;
}
finally
{
telemetryClient.StopOperation(operation);
}
}
Please note that the most recent version of Application Insights SDK is switching to W3C correlation standard, so header names and expected format would be different as per W3C specification.
I am trying to create a service that will update an external list of Service Endpoints for applications running in my service fabric cluster. (Basically I need to replicate the Azure Load Balancer in my on premises F5 Load Balancer.)
During last month's Service Fabric Q&A, the team pointed me at RegisterServiceNotificationFilterAsync.
I made a stateless service using this method, and deployed it to my development cluster. I then made a new service by running the ASP.NET Core Stateless service template.
I expected that when I deployed the second service, the break point would hit in my first service, indicating that a service had been added. But no breakpoint was hit.
I have found very little in the way of examples for this kind of thing on the internet, so I am asking here hopping that someone else has done this and can tell me where I went wrong.
Here is the code for my service that is trying to catch the application changes:
protected override async Task RunAsync(CancellationToken cancellationToken)
{
var fabricClient = new FabricClient();
long? filterId = null;
try
{
var filterDescription = new ServiceNotificationFilterDescription
{
Name = new Uri("fabric:")
};
fabricClient.ServiceManager.ServiceNotificationFilterMatched += ServiceManager_ServiceNotificationFilterMatched;
filterId = await fabricClient.ServiceManager.RegisterServiceNotificationFilterAsync(filterDescription);
long iterations = 0;
while (true)
{
cancellationToken.ThrowIfCancellationRequested();
ServiceEventSource.Current.ServiceMessage(this.Context, "Working-{0}", ++iterations);
await Task.Delay(TimeSpan.FromSeconds(1), cancellationToken);
}
}
finally
{
if (filterId != null)
await fabricClient.ServiceManager.UnregisterServiceNotificationFilterAsync(filterId.Value);
}
}
private void ServiceManager_ServiceNotificationFilterMatched(object sender, EventArgs e)
{
Debug.WriteLine("Change Occured");
}
If you have any tips on how to get this going, I would love to see them.
You need to set the MatchNamePrefix to true, like this:
var filterDescription = new ServiceNotificationFilterDescription
{
Name = new Uri("fabric:"),
MatchNamePrefix = true
};
otherwise it will only match specific services. In my application I can catch cluster wide events when this parameter is set to true.