We are using smart detectors in Azure App Insights to generate some alerts when there are anomalies in our app. However, there are some failures that are intentional in our code, where we throw a 403. Is there a way to modify these "Smart Alerts" in Application Insights, so that these known failures can be excluded in its detection logic? We have a specific exception type that is relevant for these expected failures that we can easily use to exclude these in the anomaly detection if there is a way to do that, but I can't find an option on the UI to do this.
Thanks for any pointers.
You cannot do that directly from the Azure portal but you need to implement a Telemetry Processor which can help you override telemetry properties set.
If request flag as failed with a response code = 403. But if you want to treat it as a success, you can provide a telemetry initializer that sets the success property.
Define your initializer
C#
using System;
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.DataContracts;
using Microsoft.ApplicationInsights.Extensibility;
namespace MvcWebRole.Telemetry
{
/*
* Custom TelemetryInitializer that overrides the default SDK
* behavior of treating response codes >= 400 as failed requests
*
*/
public class MyTelemetryInitializer : ITelemetryInitializer
{
public void Initialize(ITelemetry telemetry)
{
var requestTelemetry = telemetry as RequestTelemetry;
// Is this a TrackRequest() ?
if (requestTelemetry == null) return;
int code;
bool parsed = Int32.TryParse(requestTelemetry.ResponseCode, out code);
if (!parsed) return;
if (code >= 400 && code < 500)
{
// If we set the Success property, the SDK won't change it:
requestTelemetry.Success = true;
// Allow us to filter these requests in the portal:
requestTelemetry.Properties["Overridden400s"] = "true";
}
// else leave the SDK to set the Success property
}
}
}
In ApplicationInsights.config:
XMLCopy
<ApplicationInsights>
<TelemetryInitializers>
<!-- Fully qualified type name, assembly name: -->
<Add Type="MvcWebRole.Telemetry.MyTelemetryInitializer, MvcWebRole"/>
...
</TelemetryInitializers>
</ApplicationInsights>
For more information, you can refer to this Document.
Related
I have created the following TelemetryFilter:
public class TelemetryFilter : ITelemetryProcessor
{
private ITelemetryProcessor Next { get; set; }
public TelemetryFilter(ITelemetryProcessor next)
{
Next = next;
}
public void Process(ITelemetry item)
{
var dependency = item as DependencyTelemetry;
if (dependency != null && dependency.Success == true) return;
Next.Process(item);
}
}
And added TelemetryFilter to TelemetruyProcessors in ApplicationInsights.config. It works when I run the application on my machine but when it is deployed to test and production environments, dependencies are getting collected by Azure AppInsights. When I see them in Azure Portal they have the property Call status: true. Is Call status refers to dependency.Success? What's the best way to filter out all successful calls to decrease our AppInsights data ingress and lower our Azure bill?
Filter out all successful dependencies:
you can initialize the filter in code. In a suitable initialization class,
AppStart in Global.asax.cs, insert your processor into the chain:
var builder = TelemetryConfiguration.Active.DefaultTelemetrySink.TelemetryProcessorChainBuilder;
builder.Use((next) => new SuccessfulDependencyFilter(next));
// If you have more processors:
builder.Use((next) => new AnotherProcessor(next));
builder.Build();
Refer for filtering sampling
& for request filtering
To Reduce Application Insights cost
You need to optimize Telemetry with Application Insights check here
Check here for some more methods to reduce Application insights cost
I found that ApplicationInsights.config file wasn't set to be copied into the output folder by the build process. That's why it didn't work.
Is it possible to tell Application Insights to use a different InstrumentationKey depending on the request URL?
Our application works with different clients and we want to separate the logs for them in different instances of Application Insights.
Url format: https://webapi.com/v1/{client_name}/bla/bla
It would be great to setup configuration to select InstrumentationKey by client_name from request.
If the goal is to send different telemetry items to different instrumentation key, the correct way to achieve that is by modifying the individual item with a TelemetryInitializer to have the correct ikey.
An initializer like the following:
item.Context.InstrumentationKey = ikey.
This initializer should access HttpContext and decide the ikey dynamically from request route/other params.
Modifying TC.Active is not recommended for this purpose as its a global shared setting.
(This is not a very common use case - but there are teams inside Microsoft who does this for PROD scale apps)
You can do that. If you have a logger, have the ApplicationInsightsKey parameter-ized and pass the Key for the client on every call, or inject it on load if your application is tenant based.
Checkout the Docs here: Separating telemetry from Development, Test, and Production
Microsoft.ApplicationInsights.Extensibility.
TelemetryConfiguration.Active.InstrumentationKey = <App-Insights-Key-for-the-client>
Just change the Application Insights key before logging and it will do the job.
It would be great to setup configuration to select InstrumentationKey
by client_name from request.
You can dynamically select the ikey as per the client_name from the request. First, you need to get the request url, then check the client_name.
To do that, you can add the following code to the Global.asax file:
void Application_BeginRequest(Object source, EventArgs e)
{
var app = (HttpApplication)source;
//get the request url
var uriObject = app.Context.Request.Url.ToString();
if (uriObject.Contains("/client_name_1"))
{
Microsoft.ApplicationInsights.Extensibility.
TelemetryConfiguration.Active.InstrumentationKey = "ikey_1";
}
else if (uriObject.Contains("/client_name_2"))
{
Microsoft.ApplicationInsights.Extensibility.
TelemetryConfiguration.Active.InstrumentationKey = "ikey_2";
}
else
{
Microsoft.ApplicationInsights.Extensibility.
TelemetryConfiguration.Active.InstrumentationKey = "ikey_3";
}
}
The test result:
But I want to say we rarely use 1 more ikeys in one environment. If your goal is to make the data not being cluttered, I suggest you can use only 1 ikey, and then use Kusto query for your purpose.
Thanks to the answers from #cijothomas and #danpop (link) I was able to understand the whole picture.
Step 1: Create custom ITelemetryInitializer (Microsoft Documentation):
public class MyTelemetryInitializer : ITelemetryInitializer
{
public void Initialize(ITelemetry telemetry)
{
var appKey = CallContext.LogicalGetData("ApplicationKey")?.ToString();
switch (appKey)
{
case "App1":
telemetry.Context.InstrumentationKey = "d223527b-f34e-4c47-8aa8-1f21eb0fc349";
return;
default:
telemetry.Context.InstrumentationKey = "f8ceb6cf-4357-4776-a2b6-5bbed8d2561c";
return;
}
}
}
Step 2: Register custom initializer:
<ApplicationInsights xmlns="http://schemas.microsoft.com/ApplicationInsights/2013/Settings">
<TelemetryInitializers>
<Add Type="Application.WebAPI.MyTelemetryInitializer, Application.WebAPI"/>
</TelemetryInitializers>
<!--<InstrumentationKey>f8ceb6cf-4357-4776-a2b6-5bbed8d2561c</InstrumentationKey>-->
</ApplicationInsights>
OR
protected void Application_Start()
{
// ...
TelemetryConfiguration.Active.TelemetryInitializers.Add(new MyTelemetryInitializer());
}
Step 3: Make some adjustments to the logger (source code taken from #danpop answer Logger target configuration):
var config = new LoggingConfiguration();
ConfigurationItemFactory.Default.Targets.RegisterDefinition("ai", typeof());
ApplicationInsightsTarget aiTarget = new ApplicationInsightsTarget();
aiTarget.InstrumentationKey = "your_key";
aiTarget.Name = "ai";
config.AddTarget("ai", aiTarget);
LogManager.Configuration = config;
ILogger configuration exmples: Log4Net, NLog, System.Diagnostics
I'm looking to have our distributed event logging have proper correlation. For our Web Applications this seems to be automatic. Example of correlated logs from one of our App Services API:
However, for our other (non ASP, non WebApp) services were we use Log4Net and the App Insights appender our logs are not correlated. I tried following instructions here: https://learn.microsoft.com/en-us/azure/azure-monitor/app/correlation
Even after adding unique operation_Id attributes to each operation, we're not seeing log correlation (I also tried "Operation Id"). Example of none correlated log entry:
Any help on how to achieve this using log4net would be appreciated.
Cheers!
Across services correlation IDs are mostly propagated through headers. When AI is enabled for Web Application, it reads IDs/Context from the incoming headers and then updates outgoing headers with the appropriate IDs/Context. Within service, operation is tracked with Activity object and every telemetry emitted will be associated with this Activity, thus sharing necessary correlation IDs.
In case of Service Bus / Event Hub communication, the propagation is also supported in the recent versions (IDs/Context propagate as metadata).
If service is not web-based and AI automated correlation propagation is not working, you may need to manually get incoming ID information from some metadata if any exists, restore/initiate Activity, start AI operation with this Activity. When telemetry item is generated in scope of that Activity, it will get proper IDs and will be part of the overarching trace. With that, if telemetry is generated from Log4net trace that was executed in the scope of AI operation context then that telemetry should get right IDs.
Code sample to access correlation from headers:
public class ApplicationInsightsMiddleware : OwinMiddleware
{
// you may create a new TelemetryConfiguration instance, reuse one you already have
// or fetch the instance created by Application Insights SDK.
private readonly TelemetryConfiguration telemetryConfiguration = TelemetryConfiguration.CreateDefault();
private readonly TelemetryClient telemetryClient = new TelemetryClient(telemetryConfiguration);
public ApplicationInsightsMiddleware(OwinMiddleware next) : base(next) {}
public override async Task Invoke(IOwinContext context)
{
// Let's create and start RequestTelemetry.
var requestTelemetry = new RequestTelemetry
{
Name = $"{context.Request.Method} {context.Request.Uri.GetLeftPart(UriPartial.Path)}"
};
// If there is a Request-Id received from the upstream service, set the telemetry context accordingly.
if (context.Request.Headers.ContainsKey("Request-Id"))
{
var requestId = context.Request.Headers.Get("Request-Id");
// Get the operation ID from the Request-Id (if you follow the HTTP Protocol for Correlation).
requestTelemetry.Context.Operation.Id = GetOperationId(requestId);
requestTelemetry.Context.Operation.ParentId = requestId;
}
// StartOperation is a helper method that allows correlation of
// current operations with nested operations/telemetry
// and initializes start time and duration on telemetry items.
var operation = telemetryClient.StartOperation(requestTelemetry);
// Process the request.
try
{
await Next.Invoke(context);
}
catch (Exception e)
{
requestTelemetry.Success = false;
telemetryClient.TrackException(e);
throw;
}
finally
{
// Update status code and success as appropriate.
if (context.Response != null)
{
requestTelemetry.ResponseCode = context.Response.StatusCode.ToString();
requestTelemetry.Success = context.Response.StatusCode >= 200 && context.Response.StatusCode <= 299;
}
else
{
requestTelemetry.Success = false;
}
// Now it's time to stop the operation (and track telemetry).
telemetryClient.StopOperation(operation);
}
}
public static string GetOperationId(string id)
{
// Returns the root ID from the '|' to the first '.' if any.
int rootEnd = id.IndexOf('.');
if (rootEnd < 0)
rootEnd = id.Length;
int rootStart = id[0] == '|' ? 1 : 0;
return id.Substring(rootStart, rootEnd - rootStart);
}
}
Code sample for manual correlated operation tracking in isolation:
async Task BackgroundTask()
{
var operation = telemetryClient.StartOperation<DependencyTelemetry>(taskName);
operation.Telemetry.Type = "Background";
try
{
int progress = 0;
while (progress < 100)
{
// Process the task.
telemetryClient.TrackTrace($"done {progress++}%");
}
// Update status code and success as appropriate.
}
catch (Exception e)
{
telemetryClient.TrackException(e);
// Update status code and success as appropriate.
throw;
}
finally
{
telemetryClient.StopOperation(operation);
}
}
Please note that the most recent version of Application Insights SDK is switching to W3C correlation standard, so header names and expected format would be different as per W3C specification.
I am trying to integrate Azure App Insights with an Azure Function App (HttpTriggered). I want to add my own keys and values in the "customDimensions" object of the requests table. Right now it only shows the following:
On query
requests
| where iKey == "449470fb-****" and id == "5e17e23e-****"
I get this:
LogLevel: Information
Category: Host.Results
FullName: Functions.FTAID
StartTime: 2017-07-14T14:24:10.9410000Z
param__context: ****
HttpMethod: POST
param__req: Method: POST, Uri: ****
Succeeded: True
TriggerReason: This function was programmatically called via the host APIs.
EndTime: 2017-07-14T14:24:11.6080000Z
I want to add more key values such as:
EnvironmentName: Development
ServiceLine: Business
Based on this answer, I implemented the ITelemetryInitializer interface as follows:
public class CustomTelemetry : ITelemetryInitializer
{
public void Initialize(ITelemetry telemetry)
{
var requestTelemetry = telemetry as RequestTelemetry;
if (requestTelemetry == null) return;
requestTelemetry.Context.Properties.Add("EnvironmentName", "Development");
}
}
Here is how the run.csx code for the Azure Function App looks like:
public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, ExecutionContext context, TraceWriter log)
{
// Initialize the App Insights Telemetry
TelemetryConfiguration.Active.InstrumentationKey = System.Environment.GetEnvironmentVariable("APPINSIGHTS_INSTRUMENTATIONKEY", EnvironmentVariableTarget.Process);
TelemetryConfiguration.Active.TelemetryInitializers.Add(new CustomTelemetry());
TelemetryClient telemetry = new TelemetryClient();
var jsonBody = await req.Content.ReadAsStringAsync();
GetIoItemID obj = new GetIoItemID();
JArray output = obj.GetResponseJson(jsonBody, log, telemetry);
var response = req.CreateResponse(HttpStatusCode.OK);
response.Content = new StringContent(output.ToString(), System.Text.Encoding.UTF8, "application/json");
return response;
}
But this did not work...
I believe, since you're creating the TelemetryClient yourself in this example, you don't need to bother with the telemetry initializer, you could just do
var telemetry = new TelemetryClient();
telemetry.Context.Properties["EnvironmentName"] = "Development";
directly, and everything sent by that instance of that telemetry client will have those properties set.
You'd need that telemetry initializer if you don't have control over who's creating the telemetry client and want to touch every item of telemetry created wherever?
I don't know how that TelemetryClient instance gets used downstream in azure functions though, so i'm not entirely positive, though.
Edit: from azure functions post about this, it says:
We’ll be working hard to get Application Insights ready for production
workloads. We’re also listening for any feedback you have. Please file
it on our GitHub. We’ll be adding some new features like better
sampling controls and automatic dependency tracking soon. We hope
you’ll give it a try and start to gain more insight into how your
Functions are behaving. You can read more about how it works at
https://aka.ms/func-ai
and the example from that func-ai link has a couple things:
1) it creates the telemetry client statically up front once (instead of in each call to the function)
private static TelemetryClient telemetry = new TelemetryClient();
private static string key = TelemetryConfiguration.Active.InstrumentationKey = System.Environment.GetEnvironmentVariable("APPINSIGHTS_INSTRUMENTATIONKEY", EnvironmentVariableTarget.Process);
and inside the function it is doing:
telemetry.Context.Operation.Id = context.InvocationId.ToString();
to properly do correlation with events you might create with your telemetry client so you might want to do that too.
2) it appears that the telemetry client you create you can use, but they create their own telemetry client and send data there, so anything you touch in your telemetry client's context isn't seen by azure functions itself.
so, to me that leads me to something you can try:
add a static constructor in your class, and in that static constructor, do the telemetry initializer thing you were doing above. possibly this gets your telemetry initializer added to the context before azure functions starts creating its request and calling your method?
If that doesn't work, you might need to post on their GitHub or email the person listed in the article for more details on how to do this?
I've created and registered custom http module to show maintenance message to user after administrator turns on maintenance mode via configuration change.
When I pass request for html it should return custom html loaded from file, but it returns message: "The service is unavailable." I can't find that string in my entire solution. Custom log message from custom maintenance module is written to log4net logs.
... INFO DdiPlusWeb.Common.MaintenanceResponder - Maintenance mode is on. Request rejected. RequestUrl=...
Seems something is miss configured in IIS on Azure. Something intercepts my 503 response. How to fix it?
Module code
void context_BeginRequest(object sender, EventArgs e)
{
HttpApplication application = (HttpApplication)sender;
HttpContext context = application.Context;
if (AppConfig.Azure.IsMaintenance)
{
MaintenanceResponder responder = new MaintenanceResponder(context, MaintenaceHtmlFileName);
responder.Respond();
}
}
Interesting part of responder code.
private void SetMaintenanceResponse(string message = null)
{
_context.Response.Clear();
_context.Response.StatusCode = 503;
_context.Response.StatusDescription = "Maintenance";
if (string.IsNullOrEmpty(message))
{
_context.Response.Write("503, Site is under maintenance. Please try again a bit later.");
}
else
{
_context.Response.Write(message);
}
_context.Response.Flush();
_context.Response.End();
}
EDIT: I lied. Sorry. Maintenance module returns the same message for requests that expect json or html.
This answer led me to the solution.
I've added another line to SetMaintenanceResponse method.
_context.Response.TrySkipIisCustomErrors = true;
It works now. Here is more about what it exactly means.