Azure AppInsights end to end correlation - azure

I am looking into Azure AppInsights for my telemetry correlation requirement. I have created 3 simple APIs that call one another in succession as below:
First Api -----> Middle Api -----> Another Api
The Api calls are made using Typed HttpClient through a simple service class. All the Api projects have Microsoft.ApplicationInsights.AspNetCore and Microsoft.Extensions.Logging.ApplicationInsights NuGets references added. I have program and service classes for all the APIs as below:
Program.cs
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddControllers();
...
//App Insights
builder.Services.AddSingleton(typeof(ITelemetryChannel),
new ServerTelemetryChannel() { StorageFolder = "/tmp/myfolder" });
builder.Services.AddApplicationInsightsTelemetry();
builder.Services.AddSingleton<IConfiguration>(builder.Configuration);
builder.Services.AddScoped<IWeatherService, DummyWeatherService>();
builder.Services.AddHttpClient<IWeatherService, DummyWeatherService>();
var app = builder.Build();
...
app.Run();
Service
using System.Net.Http.Headers;
using AppInsightsDemo.Api.Models;
namespace AppInsightsDemo.Api.Services;
public class DummyWeatherService : IWeatherService
{
private readonly IConfiguration _configuration;
private readonly HttpClient _httpClient;
public DummyWeatherService(
IConfiguration configuration,
HttpClient httpClient)
{
_configuration = configuration;
_httpClient = httpClient;
_httpClient.BaseAddress = GetMiddleApiBaseUri();
_httpClient.DefaultRequestHeaders.Accept.Clear();
_httpClient.DefaultRequestHeaders.Accept.Add(
new MediaTypeWithQualityHeaderValue("application/json"));
}
private Uri GetAnotherApiBaseUri()
{
var configurationSection = _configuration.GetRequiredSection("Dependencies");
var baseUri = configurationSection.GetValue<string>("MiddleApiUri")
?? throw new ArgumentException("Another Api base uri is empty");
return new Uri(baseUri);
}
public async Task<Weather?> GetWeatherAsync()
{
Weather? weather = null;
var response = await _httpClient.GetAsync("middle");
if (response.IsSuccessStatusCode)
{
weather = await response.Content.ReadAsAsync<Weather>();
}
return weather;
}
}
This is what I end up with in AppInsights sample. The third API event has the same operation id as the first two Api events have but the third event has a different parent id. I expect the third event to have the id of my middle (second) api event (localhost://7290) as its parent id and the three events show up accordingly in a hierarchy.
Can anyone please advise if I am missing some configuration or not using this SDK right? Thank you

This is rather silly of me. I configured the ApplicationInsights connection string for my first api(:7176) and last api(:7206) but missed to configure it for my middle api (:7290) though I have added ApplicationInsights service to all Api projects. It took me a while to figure out the missing connection string. Now I get a nice dependency hierarchy as below:
I guess a connection string validation might come handy. Sorry for the trouble. Thanks.

Related

Azure Functions Dependency Tracking for SQL Server and Service Bus Into Application Insights

Previously I have Azure Web App (.net core) and It successfully track the SQL Server and Service Bus dependency into Application Insights. It is not working some how with Azure Functions.
Environment
dotnet 6
dotnet-isolated mode
log level default set to "Information".
Azure Environment using Consumption plan for Azure Functions.
Application Insights key is configured.
I have Azure API management at front and backend is Azure Function and that call SQL Server and Service Bus.
Api Management Service to Azure function dependency successfully resolved but Azure Function to other component is not working.
I know I am posting my own answer. Also there are chance that in future there may be some good solution or it get integrated the way it is in in-process mode.
By then follow steps.
Add Package
Microsoft.ApplicationInsights.WorkerService
In program.cs in configuring host.
services.AddApplicationInsightsTelemetryWorkerService();
More info at
https://learn.microsoft.com/en-us/azure/azure-monitor/app/worker-service
The only way I've managed to solve this issue so far was by setting up custom Middleware
.ConfigureFunctionsWorkerDefaults(config =>
{
config.UseMiddleware<AiContextMiddleware>();
})
In the IServiceCollection you need to setup simply
.AddApplicationInsightsTelemetryWorkerService()
public class AiContextMiddleware : IFunctionsWorkerMiddleware
{
private readonly TelemetryClient _client;
private readonly string _hostname;
public AiContextMiddleware(TelemetryClient client)
{
_client = client;
_hostname = Environment.GetEnvironmentVariable("AI_CLOUD_ROLE_NAME");
}
public async Task Invoke(FunctionContext context, FunctionExecutionDelegate next)
{
var operationId = ExtractOperationId(context.TraceContext.TraceParent);
// Let's create and start RequestTelemetry.
var requestTelemetry = new RequestTelemetry
{
Name = context.FunctionDefinition.Name,
Id = context.InvocationId,
Properties =
{
{ "ai.cloud.role", _hostname},
{ "AzureFunctions_FunctionName", context.FunctionDefinition.Name },
{ "AzureFunctions_InvocationId", context.InvocationId },
{ "AzureFunctions_OperationId", operationId }
},
Context =
{
Operation =
{
Id = operationId,
ParentId = context.InvocationId,
Name = context.FunctionDefinition.Name
},
GlobalProperties =
{
{ "ai.cloud.role", _hostname},
{ "AzureFunctions_FunctionName", context.FunctionDefinition.Name },
{ "AzureFunctions_InvocationId", context.InvocationId },
{ "AzureFunctions_OperationId", operationId }
}
}
};
var operation = _client.StartOperation(requestTelemetry);
try
{
await next(context);
}
catch (Exception e)
{
requestTelemetry.Success = false;
_client.TrackException(e);
throw;
}
finally
{
_client.StopOperation(operation);
}
}
private static string ExtractOperationId(string traceParent)
=> string.IsNullOrEmpty(traceParent) ? string.Empty : traceParent.Split("-")[1];
}
It's definitely not a perfect solution as you then get two starting logs, but as end result, you get all logs traces + dependencies correlated to an operation.
I've solved this issue in the first place like that, now I'm revisiting whether there are any better ways to solve this.
Let me know too whether you managed to solve this issue on your side.

Azure AppConfiguration (using DefaultAzureCredential on local) returns no keys

When I inject IConfiguration in a function, it does not find any keys that only live in my "Azure App Configuration".
I have a functionApp (V3) that accesses App Configuration using the DefaultAzureCredential. I am running this locally in debug hence the need for a default credential. I also have multiple Tenants so I had to set the VisualStudioTenantId and SharedTokenCacheTenantId on DefaultAzureCredentialOptions. My Visual studio user was also given the role "App Configuration Data Reader" to be able to debug.
When connecting to App configuration I get no errors.
Editedto add: I have setup AppConfiguration to authenticate with AzureAD.
See code below:
public override async void ConfigureAppConfiguration(IFunctionsConfigurationBuilder builder)
{
var credOptions = new DefaultAzureCredentialOptions();
var tenantId = Environment.GetEnvironmentVariable("Tenant_Id");
credOptions.VisualStudioTenantId = tenantId;
credOptions.SharedTokenCacheTenantId = tenantId;
var cred = new DefaultAzureCredential(credOptions);
/*Works but requires SharedTokenCacheTenantId*/
var secretClient = new SecretClient(new Uri(vaultURI), cred);
var secret = await secretClient.GetSecretAsync("<secret name>");
/*Works but where are my keys when I try to access them?*/
builder.ConfigurationBuilder.AddAzureAppConfiguration(options =>
{
options.Connect(new Uri(appConfigURI), cred);
}).Build(); //Should I be building this??
}
In my function
public FunctName(IConfiguration configuration)
{
_configuration = configuration;
}
And when I access the property
var prop = _configuration["PropertyName"];
There is an example function app that uses IFunctionsConfigurationBuilder here https://github.com/Azure/AppConfiguration/blob/main/examples/DotNetCore/AzureFunction/FunctionApp/Startup.cs . I would recommend taking a look and seeing if there are any missing pieces.
The title mentions "using DefaultAzureCredential on local". Does that mean that this works as expected if you use a connection string?
Notice the async void ConfigureAppConfiguration. This caused my ConfigureAppConfiguration to not execute synchronously, causing configure to add my App Configuration before it was populated.

Application Insights for WebAPI application

Is it possible to tell Application Insights to use a different InstrumentationKey depending on the request URL?
Our application works with different clients and we want to separate the logs for them in different instances of Application Insights.
Url format: https://webapi.com/v1/{client_name}/bla/bla
It would be great to setup configuration to select InstrumentationKey by client_name from request.
If the goal is to send different telemetry items to different instrumentation key, the correct way to achieve that is by modifying the individual item with a TelemetryInitializer to have the correct ikey.
An initializer like the following:
item.Context.InstrumentationKey = ikey.
This initializer should access HttpContext and decide the ikey dynamically from request route/other params.
Modifying TC.Active is not recommended for this purpose as its a global shared setting.
(This is not a very common use case - but there are teams inside Microsoft who does this for PROD scale apps)
You can do that. If you have a logger, have the ApplicationInsightsKey parameter-ized and pass the Key for the client on every call, or inject it on load if your application is tenant based.
Checkout the Docs here: Separating telemetry from Development, Test, and Production
Microsoft.ApplicationInsights.Extensibility.
TelemetryConfiguration.Active.InstrumentationKey = <App-Insights-Key-for-the-client>
Just change the Application Insights key before logging and it will do the job.
It would be great to setup configuration to select InstrumentationKey
by client_name from request.
You can dynamically select the ikey as per the client_name from the request. First, you need to get the request url, then check the client_name.
To do that, you can add the following code to the Global.asax file:
void Application_BeginRequest(Object source, EventArgs e)
{
var app = (HttpApplication)source;
//get the request url
var uriObject = app.Context.Request.Url.ToString();
if (uriObject.Contains("/client_name_1"))
{
Microsoft.ApplicationInsights.Extensibility.
TelemetryConfiguration.Active.InstrumentationKey = "ikey_1";
}
else if (uriObject.Contains("/client_name_2"))
{
Microsoft.ApplicationInsights.Extensibility.
TelemetryConfiguration.Active.InstrumentationKey = "ikey_2";
}
else
{
Microsoft.ApplicationInsights.Extensibility.
TelemetryConfiguration.Active.InstrumentationKey = "ikey_3";
}
}
The test result:
But I want to say we rarely use 1 more ikeys in one environment. If your goal is to make the data not being cluttered, I suggest you can use only 1 ikey, and then use Kusto query for your purpose.
Thanks to the answers from #cijothomas and #danpop (link) I was able to understand the whole picture.
Step 1: Create custom ITelemetryInitializer (Microsoft Documentation):
public class MyTelemetryInitializer : ITelemetryInitializer
{
public void Initialize(ITelemetry telemetry)
{
var appKey = CallContext.LogicalGetData("ApplicationKey")?.ToString();
switch (appKey)
{
case "App1":
telemetry.Context.InstrumentationKey = "d223527b-f34e-4c47-8aa8-1f21eb0fc349";
return;
default:
telemetry.Context.InstrumentationKey = "f8ceb6cf-4357-4776-a2b6-5bbed8d2561c";
return;
}
}
}
Step 2: Register custom initializer:
<ApplicationInsights xmlns="http://schemas.microsoft.com/ApplicationInsights/2013/Settings">
<TelemetryInitializers>
<Add Type="Application.WebAPI.MyTelemetryInitializer, Application.WebAPI"/>
</TelemetryInitializers>
<!--<InstrumentationKey>f8ceb6cf-4357-4776-a2b6-5bbed8d2561c</InstrumentationKey>-->
</ApplicationInsights>
OR
protected void Application_Start()
{
// ...
TelemetryConfiguration.Active.TelemetryInitializers.Add(new MyTelemetryInitializer());
}
Step 3: Make some adjustments to the logger (source code taken from #danpop answer Logger target configuration):
var config = new LoggingConfiguration();
ConfigurationItemFactory.Default.Targets.RegisterDefinition("ai", typeof());
ApplicationInsightsTarget aiTarget = new ApplicationInsightsTarget();
aiTarget.InstrumentationKey = "your_key";
aiTarget.Name = "ai";
config.AddTarget("ai", aiTarget);
LogManager.Configuration = config;
ILogger configuration exmples: Log4Net, NLog, System.Diagnostics

Setting up AAD Authentication in a Project with Autofac

I have a web API that has AAD auth (in code because it runs in IaaS not PaaS) it works well, but if I add Autofac configuration to the Startup.cs, the Authentication breaks, (if I put Autofac after Auth inizialization Autofac breaks) which makes me think that the configurations are overwriting eachother.
I have tried to find any documentation on how to use both of them together but I have not been able to find any information. One Uses HttpConfiguration and the other uses the IAppBuilder and I don't know how to combine them for them to work together.
here is my Authentication code:
public void ConfigureAuth(IAppBuilder app)
{
app.SetDefaultSignInAsAuthenticationType(CookieAuthenticationDefaults.AuthenticationType);
app.UseCookieAuthentication(new CookieAuthenticationOptions());
app.Map("/api", inner =>
{
inner.UseWindowsAzureActiveDirectoryBearerAuthentication(new WindowsAzureActiveDirectoryBearerAuthenticationOptions()
{
Tenant = tenant,
TokenValidationParameters = new Tokens.TokenValidationParameters
{
ValidAudience = Audience
}
});
});
}
and here is the Autofac Code
public static void Register(HttpConfiguration configuration)
{
var builder = new ContainerBuilder();
Bootstrapper.Configure(builder);
var container = builder.Build();
configuration.DependencyResolver = new AutofacWebApiDependencyResolver(container);
}
what are the best practices for using these two tools together?
I was not properly setting all the WebAPI autofac references to get all the dependencies I followed this Quick start and then Added my references. Bellow is the new ConfigureAutofac function (the configure auth stayed the same)
private void ConfigureAutofac(IAppBuilder app)
{
//Autofac info from https://autofaccn.readthedocs.io/en/latest/integration/webapi.html#quick-start
var builder = new ContainerBuilder();
// STANDARD WEB API SETUP:
// Get your HttpConfiguration. In OWIN, you'll create one
// rather than using GlobalConfiguration.
var config = new HttpConfiguration();
// Register your Web API controllers.
builder.RegisterApiControllers(Assembly.GetExecutingAssembly()); //Register WebApi Controllers
builder.RegisterType<AutofacManager>().As<IAutofacManager>();
builder.RegisterSource(new AnyConcreteTypeNotAlreadyRegisteredSource());
var container = builder.Build();
config.DependencyResolver = new AutofacWebApiDependencyResolver(container);
GlobalConfiguration.Configuration.DependencyResolver = new AutofacWebApiDependencyResolver((IContainer)container); //Set the WebApi DependencyResolver
// and finally the standard Web API middleware.
app.UseAutofacMiddleware(container);
app.UseAutofacWebApi(config);
app.UseWebApi(config);
}

How to add more entries in customDimensions in to Application Insights Telemetry in Azure Function

I am trying to integrate Azure App Insights with an Azure Function App (HttpTriggered). I want to add my own keys and values in the "customDimensions" object of the requests table. Right now it only shows the following:
On query
requests
| where iKey == "449470fb-****" and id == "5e17e23e-****"
I get this:
LogLevel: Information
Category: Host.Results
FullName: Functions.FTAID
StartTime: 2017-07-14T14:24:10.9410000Z
param__context: ****
HttpMethod: POST
param__req: Method: POST, Uri: ****
Succeeded: True
TriggerReason: This function was programmatically called via the host APIs.
EndTime: 2017-07-14T14:24:11.6080000Z
I want to add more key values such as:
EnvironmentName: Development
ServiceLine: Business
Based on this answer, I implemented the ITelemetryInitializer interface as follows:
public class CustomTelemetry : ITelemetryInitializer
{
public void Initialize(ITelemetry telemetry)
{
var requestTelemetry = telemetry as RequestTelemetry;
if (requestTelemetry == null) return;
requestTelemetry.Context.Properties.Add("EnvironmentName", "Development");
}
}
Here is how the run.csx code for the Azure Function App looks like:
public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, ExecutionContext context, TraceWriter log)
{
// Initialize the App Insights Telemetry
TelemetryConfiguration.Active.InstrumentationKey = System.Environment.GetEnvironmentVariable("APPINSIGHTS_INSTRUMENTATIONKEY", EnvironmentVariableTarget.Process);
TelemetryConfiguration.Active.TelemetryInitializers.Add(new CustomTelemetry());
TelemetryClient telemetry = new TelemetryClient();
var jsonBody = await req.Content.ReadAsStringAsync();
GetIoItemID obj = new GetIoItemID();
JArray output = obj.GetResponseJson(jsonBody, log, telemetry);
var response = req.CreateResponse(HttpStatusCode.OK);
response.Content = new StringContent(output.ToString(), System.Text.Encoding.UTF8, "application/json");
return response;
}
But this did not work...
I believe, since you're creating the TelemetryClient yourself in this example, you don't need to bother with the telemetry initializer, you could just do
var telemetry = new TelemetryClient();
telemetry.Context.Properties["EnvironmentName"] = "Development";
directly, and everything sent by that instance of that telemetry client will have those properties set.
You'd need that telemetry initializer if you don't have control over who's creating the telemetry client and want to touch every item of telemetry created wherever?
I don't know how that TelemetryClient instance gets used downstream in azure functions though, so i'm not entirely positive, though.
Edit: from azure functions post about this, it says:
We’ll be working hard to get Application Insights ready for production
workloads. We’re also listening for any feedback you have. Please file
it on our GitHub. We’ll be adding some new features like better
sampling controls and automatic dependency tracking soon. We hope
you’ll give it a try and start to gain more insight into how your
Functions are behaving. You can read more about how it works at
https://aka.ms/func-ai
and the example from that func-ai link has a couple things:
1) it creates the telemetry client statically up front once (instead of in each call to the function)
private static TelemetryClient telemetry = new TelemetryClient();
private static string key = TelemetryConfiguration.Active.InstrumentationKey = System.Environment.GetEnvironmentVariable("APPINSIGHTS_INSTRUMENTATIONKEY", EnvironmentVariableTarget.Process);
and inside the function it is doing:
telemetry.Context.Operation.Id = context.InvocationId.ToString();
to properly do correlation with events you might create with your telemetry client so you might want to do that too.
2) it appears that the telemetry client you create you can use, but they create their own telemetry client and send data there, so anything you touch in your telemetry client's context isn't seen by azure functions itself.
so, to me that leads me to something you can try:
add a static constructor in your class, and in that static constructor, do the telemetry initializer thing you were doing above. possibly this gets your telemetry initializer added to the context before azure functions starts creating its request and calling your method?
If that doesn't work, you might need to post on their GitHub or email the person listed in the article for more details on how to do this?

Resources