How to extend Dependency tracking for outgoing http requests in Application Insights - azure

I have a .NET core API that performs HTTP connections to other API. I am able to visualize the outgoing HTTP request in Application Insights, under Dependency Event Types, but it has only basic information. I'm looking on how to add more information about the outgoing HTTP call (like the HTTP headers for instance).
I've looked into https://learn.microsoft.com/en-us/azure/azure-monitor/app/api-custom-events-metrics#trackdependency but I didn't find any concrete way of doing this.

As it has been said, the solution proposed by IvanYang is using the recived request instead of the dependency request.
I've built this ITelemetryInstance for that:
public void Initialize(ITelemetry telemetry)
{
var dependecyTelemetry = telemetry as DependencyTelemetry;
if (dependecyTelemetry == null) return;
if (dependecyTelemetry.TryGetOperationDetail("HttpRequest", out object request)
&& request is HttpRequestMessage httpRequest)
{
foreach (var item in httpRequest.Headers)
{
if (!dependecyTelemetry.Properties.ContainsKey(item.Key))
dependecyTelemetry.Properties.Add(item.Key, string.Join(Environment.NewLine, item.Value));
}
}
if (dependecyTelemetry.TryGetOperationDetail("HttpResponse", out object response)
&& response is HttpResponseMessage httpResponse)
{
var responseBody = httpResponse.Content.ReadAsStringAsync().GetAwaiter().GetResult();
if (!string.IsNullOrEmpty(responseBody))
dependecyTelemetry.Properties.Add("ResponseBody", responseBody);
}
}
This will record all the headers sent to the dependency and also the response received

The other solution given doesn't actually work the way you think it should, since it's attaching the header from the incoming HTTP request to the outgoing dependency request, which is misleading. If you want to attach dependency data to dependency logs then you need to wrap the dependency in a custom dependency wrapper, eg here I'm logging the outgoing payload of the dependency so I can see what's being sent by my system:
Activity activity = null;
IOperationHolder<DependencyTelemetry> requestOperation = null;
if ((request.Method == HttpMethod.Post || request.Method == HttpMethod.Put) && _httpContextAccessor?.HttpContext != null)
{
var bodyContent = await request.Content?.ReadAsStringAsync();
if (!string.IsNullOrEmpty(bodyContent))
{
activity = new Activity("Wrapped POST/PUT operation");
activity.SetTag("RequestBody", bodyContent);
requestOperation = _telemetryClient.StartOperation<DependencyTelemetry>(activity);
}
}
// perform dependency function
httpResponseMessage = await base.SendAsync(request, cancellationToken);
if (activity != null && requestOperation != null)
{
_telemetryClient.StopOperation(requestOperation);
}

I think what you're looking for is ITelemetryInitializer, which can add custom property for dependency telemetry.
And for .net core web project, you can refer to this link.
I write a demo as below:
1.Create a custom ITelemetryInitializer class to collect any dependency data:
public class MyTelemetryInitializer: ITelemetryInitializer
{
IHttpContextAccessor httpContextAccessor;
public MyTelemetryInitializer(IHttpContextAccessor httpContextAccessor)
{
this.httpContextAccessor = httpContextAccessor;
}
public void Initialize(ITelemetry telemetry)
{
//only add custom property to dependency type, otherwise just return.
var dependencyTelemetry = telemetry as DependencyTelemetry;
if (dependencyTelemetry == null) return;
if (!dependencyTelemetry.Context.Properties.ContainsKey("custom_dependency_headers_1"))
{
//the comment out code use to check the fields in Headers if you don't know
//var s = httpContextAccessor.HttpContext.Request.Headers;
//foreach (var s2 in s)
//{
// var a1 = s2.Key;
// var a2 = s2.Value;
//}
dependencyTelemetry.Context.Properties["custom_dependency_headers_1"] = httpContextAccessor.HttpContext.Request.Headers["Connection"].ToString();
}
}
}
2.Then in the Startup.cs -> ConfigureServices method:
public void ConfigureServices(IServiceCollection services)
{
//other code
//add this line of code here
services.AddSingleton<ITelemetryInitializer, MyTelemetryInitializer>();
}
3.Test result, check if the custom property is added to azure portal -> Custom Properties:

Related

How to add customDimensions and set operation_parentId for Azure function log

I created a http trigger V1 azure function on net framework 4.8, and used ILogger for logging. The code is like this.
I checked the Application Insight and queried for traces table. This table contains columns named customDimensions and operation_ParentId. May I ask is there anyway to add custom property in customDimensions column, or set a new Guid value for operation_ParentId? I know that I can use TelemetryClient sdk to create a custom telemetry client for logging. Just curious if there is any easy way which doesn't need to create a new telemetry client, because azure function offers bulit-in integration with application insight.
Also, since azure function runtimes automatically tracks requests, is there any way to change the operation_ParentId and customDimensions for requests table as well? Thanks a lot!
To get both the headers and App Insights to get the custom operation Id, two things must be overridden.
The first is an Activity that wraps the HttpClient, which is responsible for controlling the correlation headers and the other is App Insights' dependency tracing.
Although you can disable Actions completely in your HttpClients, you can just remove the one in the client by setting Activity.Current = null to limit side effects.
var operationId = "CR" + Guid.NewGuid().ToString();
var url = "https://www.microsoft.com";
using (var client = new HttpClient())
{
using (var requestMessage =
new HttpRequestMessage(HttpMethod.Get, url))
{
//Makes the headers configurable
Activity.Current = null;
//set correlation header manually
requestMessage.Headers.Add("Request-Id", operationId);
await client.SendAsync(requestMessage);
}
}
The next step is to remove the App Insights default tracking for this request. Again, you can disable dependency tracking completely, or you can filter out the default telemetry for this request. Processors are registered inside the Startup class just like initializers.
services.AddApplicationInsightsTelemetryProcessor<CustomFilter>();
public class CustomFilter : ITelemetryProcessor
{
private ITelemetryProcessor Next { get; set; }
// next will point to the next TelemetryProcessor in the chain.
public CustomFilter(ITelemetryProcessor next)
{
this.Next = next;
}
public void Process(ITelemetry item)
{
// To filter out an item, return without calling the next processor.
if (!OKtoSend(item)) { return; }
this.Next.Process(item);
}
// Example: replace with your own criteria.
private bool OKtoSend(ITelemetry item)
{
var dependency = item as DependencyTelemetry;
if (dependency == null) return true;
if (dependency.Type == "Http"
&& dependency.Data.Contains("microsoft.com")
//This key is just there to help identify the custom tracking
&& !dependency.Context.GlobalProperties.ContainsKey("keep"))
{
return false;
}
return true;
}
}
Finally, you must inject a telemetry client and call TelemetryClient.TrackDependency() in the method that makes the remote call.
var operationId = "CR" + Guid.NewGuid().ToString();
//setup telemetry client
telemetry.Context.Operation.Id = operationId;
if (!telemetry.Context.GlobalProperties.ContainsKey("keep"))
{
telemetry.Context.GlobalProperties.Add("keep", "true");
}
var startTime = DateTime.UtcNow;
var timer = System.Diagnostics.Stopwatch.StartNew();
//continue setting up context if needed
var url = "https:microsoft.com";
using (var client = new HttpClient())
{
//Makes the headers configurable
Activity.Current = null;
using (var requestMessage =
new HttpRequestMessage(HttpMethod.Get, url))
{
//Makes the headers configurable
Activity.Current = null;
//set header manually
requestMessage.Headers.Add("Request-Id", operationId);
await client.SendAsync(requestMessage);
}
}
//send custom telemetry
telemetry.TrackDependency("Http", url, "myCall", startTime, timer.Elapsed, true);
Refer here more information.
Note: The above is possible by disabling the built-in dependency tracking and App Insights and handling it on your own. But the better approach is let .NET Core & App Insights do the tracking.

How to log outgoing HttpClient request body to Application Insights dependencies table?

I have a .Net core Web App API which accept request from front-end and then send HTTP POST request to Azure search to get search results.
I just use the build-in application insights logging log basic info in request and dependencies sources with zero logging code.
Now I want to extend the default Application Insights dependencies table to add the request body to Azure search.
What is the easiest way with minimum code?
As per your requirement, you can try the code below:
public class RequestBodyInitializer : ITelemetryInitializer
{
readonly IHttpContextAccessor httpContextAccessor;
public RequestBodyInitializer(IHttpContextAccessor httpContextAccessor)
{
this.httpContextAccessor = httpContextAccessor;
}
public void Initialize(ITelemetry telemetry)
{
if (telemetry is RequestTelemetry requestTelemetry)
{
if ((httpContextAccessor.HttpContext.Request.Method == HttpMethods.Post ||
httpContextAccessor.HttpContext.Request.Method == HttpMethods.Put) &&
httpContextAccessor.HttpContext.Request.Body.CanRead)
{
const string jsonBody = "JsonBody";
if (requestTelemetry.Properties.ContainsKey(jsonBody))
{
return;
}
//Allows re-usage of the stream
httpContextAccessor.HttpContext.Request.EnableRewind();
var stream = new StreamReader(httpContextAccessor.HttpContext.Request.Body);
var body = stream.ReadToEnd();
//Reset the stream so data is not lost
httpContextAccessor.HttpContext.Request.Body.Position = 0;
requestTelemetry.Properties.Add(jsonBody, body);
}
}
}
and add this to the Startup > ConfigureServices
services.AddSingleton<ITelemetryInitializer, RequestBodyInitializer>();

How to make outgoing request or webhook in Acumatica?

I'm integrating an Asp.NET application with Acumatica that needs to update shipping information (tracking #, carrier, etc.) when it becomes available in Acumatica. Is there a way to have Acumatica call an endpoint on my Asp.NET app when a shipment is created? I've searched through a lot of the docs (available here), but I haven't come across anything to send OUT information from Acumatica to another web service.
Ideally, this outgoing call would send the shipment object in the payload.
This wasn't available when you asked the question but push notifications seem to be exactly what you're looking for:
Help - https://help.acumatica.com/(W(9))/Main?ScreenId=ShowWiki&pageid=d8d2835f-5450-4b83-852e-dbadd76a5af8
Presentation - https://adn.acumatica.com/content/uploads/2018/05/Push-Notifications.pdf
In my answer I suppose that you know how to call some outside service from C# code, and for your is a challenge how to send notification from Acumatica.
I propose you to extend each Persist method in each Acumatica graph, from which you expect to send notification when object is persisted in db. IMHO the best option for this is to override method persist ( btw, it overriding persist method is well described in T300 ). In code of extension class you can do the following:
public void Persist(PersistDelegate baseMethod)
{
baseMethod(); // calling this method will preserve your changes in db
//here should go your code, that will send push/pop/delete etc web request into your asp.net application. Or in other words your web hook.
}
If you don't have Acumatica 2017R2, then you have to create your own extension project and then you can call it from your Acumatica code:
using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
namespace MyApp
{
public static class Utility
{
private static WebRequest CreateRequest(string url, Dictionary headers)
{
if (Uri.IsWellFormedUriString(url, UriKind.Absolute))
{
WebRequest req = WebRequest.Create(url);
if (headers != null)
{
foreach (var header in headers)
{
if (!WebHeaderCollection.IsRestricted(header.Key))
{
req.Headers.Add(header.Key, header.Value);
}
}
}
return req;
}
else
{
throw(new ArgumentException("Invalid URL provided.", "url"));
}
}
public static string MakeRequest(string url, Dictionary headers = null)
{
WebResponse resp = CreateRequest(url, headers).GetResponse();
StreamReader reader = new StreamReader(resp.GetResponseStream());
string response = reader.ReadToEnd();
reader.Close();
resp.Close();
return response;
}
public static byte[] MakeRequestInBytes(string url, Dictionary headers = null)
{
byte[] rb = null;
WebResponse resp = CreateRequest(url, headers).GetResponse();
using (BinaryReader br = new BinaryReader(resp.GetResponseStream()))
{
rb = br.ReadBytes((int)resp.ContentLength);
br.Close();
}
resp.Close();
return rb;
}
}
}
You can then call it like this:
try
{
Utility.MakeRequest(theUrl, anyHeadersYouNeed);
}
catch(System.Net.WebException ex)
{
throw(new PXException("There was an error.", ex));
}

Use OWIN middleware or a delegating MessgaeHandler to log api requests/responses?

In my old non-OWIN APIs, I use a MessageHanlder to log all HttpRequests and HttpResponses. Here is the MessageHandler:
public class MessageHandler : DelegatingHandler
{
private static readonly ILog RequestApiLogger = LogManager.GetLogger("RequestApiPacketLogger");
private static readonly ILog ResponseApiLogger = LogManager.GetLogger("ResponseApiPacketLogger");
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
var correlationId = Guid.NewGuid();
RequestApiLogger.LogHttpRequest(request, correlationId);
return await base.SendAsync(request, cancellationToken).ContinueWith(
task =>
{
var response = task.Result;
response.Headers.Add("http-tracking-id", correlationId.ToString("D"));
ResponseApiLogger.LogHttpResponse(response, correlationId);
return response;
}, cancellationToken);
}
}
However, in my newer projects I could write custom OWIN middleware to do something similar using the OwinContext like this:
//use an alias for the OWIN AppFunc
using AppFunc = Func<IDictionary<string, object>, Task>;
public class LoggingMiddleware
{
private readonly AppFunc _next;
public LoggingMiddleware(AppFunc next)
{
_next = next;
}
public async Task Invoke(IDictionary<string, object> environment)
{
IOwinContext context = new OwinContext(environment);
// Get the identity
var identity = (context.Request.User != null && context.Request.User.Identity.IsAuthenticated)
? context.Request.User.Identity.Name
: "(anonymous)";
// Buffer the request (body is a string, we can use this to log the request later
var requestBody = new StreamReader(context.Request.Body).ReadToEnd();
var requestData = Encoding.UTF8.GetBytes(requestBody);
context.Request.Body = new MemoryStream(requestData);
var apiPacket = new ApiPacket
{
CallerIdentity = identity,
Request = requestBody,
RequestLength = context.Request.Body.Length
};
// Buffer the response
var responseBuffer = new MemoryStream();
var responseStream = context.Response.Body;
context.Response.Body = responseBuffer;
// add the "http-tracking-id" response header so the user can correlate back to this entry
var responseHeaders = (IDictionary<string, string[]>)environment["owin.ResponseHeaders"];
responseHeaders["http-tracking-id"] = new[] { apiPacket.TrackingId.ToString("d") };
await _next.Invoke(environment);
responseBuffer.Seek(0, SeekOrigin.Begin);
var reader = new StreamReader(responseBuffer);
apiPacket.Response = await reader.ReadToEndAsync();
apiPacket.ResponseLength = context.Response.ContentLength ?? 0;
WriteRequestHeaders(context.Request, apiPacket);
WriteResponseHeaders(context.Response, apiPacket);
// You need to do this so that the response we buffered is flushed out to the client application.
responseBuffer.Seek(0, SeekOrigin.Begin);
await responseBuffer.CopyToAsync(responseStream);
//TODO: persist the ApiPacket in the database
}
private static void WriteRequestHeaders(IOwinRequest request, ApiPacket packet)
{
packet.Verb = request.Method;
packet.RequestUri = request.Uri;
packet.RequestHeaders = "{\r\n" + string.Join(Environment.NewLine, request.Headers.Select(kv => "\t" + kv.Key + "=" + string.Join(",", kv.Value))) + "\r\n}";
}
private static void WriteResponseHeaders(IOwinResponse response, ApiPacket packet)
{
packet.StatusCode = response.StatusCode;
packet.ReasonPhrase = response.ReasonPhrase;
packet.ResponseHeaders = "{\r\n" + string.Join(Environment.NewLine, response.Headers.Select(kv => "\t" + kv.Key + "=" + string.Join(",", kv.Value))) + "\r\n}";
}
}
I'm using log4net to write the information to a SQL2012 database. Both ways accomplish my goal. However, I'm looking for a reason to use one method over the other. Should I use custom OWIN middleware OR a MessageHandler, and why? Thanks in advance.
Since you already have the MessageHandler implementations, I would recommend using that until you have a reason otherwise.
However, off the top of my head one valid reason to move logging to an OwinMiddleware would be if you have other OwinMiddleware components that require (or would benefit from) that logging functionality (assuming that you are using WebApi whereby the MessageHandlers will run after all of the OwinMiddleware in the request-pipeline).
Looks like I will be using OWIN middleware. I found that inside the MessageHandler the Principal.IIdentity has not yet been resolved. For example, if I put breakpoints in my message handler, an API controller's constructor, and in the API method, this is what I see (in order).
Using Message Handler
In MessageHandler > Principal.IIdentity not yet resolved.
In API controller's constructor > Principal.IIDentity not yet resolved.
In API controller's GET method, the Principal.IIdentity is finally resolved.
Thus, I can't pull out and log the authorized user's id in the MessageHandler.
However, when using the OWIN middleware, the Principal.IIdentity IS resolved there, so I can write the userId to my log table at that point. This is why I've decided to use the middleware.
Maybe someone can provide some clarity as to when the IIDentity is set in an API project though.

Do the Request filters get run from BasicAppHost?

I know that the services get wired-up by instantiating the BasicAppHost, and the IoC by using the ConfigureContainer property, but where is the right place to add the filters? The test in question never fire the global filter:
[TestFixture]
public class IntegrationTests
{
private readonly ServiceStackHost _appHost;
public IntegrationTests()
{
_appHost = new BasicAppHost(typeof(MyServices).Assembly)
{
ConfigureContainer = container =>
{
//
}
};
_appHost.Plugins.Add(new ValidationFeature());
_appHost.Config = new HostConfig { DebugMode = true };
_appHost.GlobalRequestFilters.Add(ITenantRequestFilter);
_appHost.Init();
}
private void ITenantRequestFilter(IRequest req, IResponse res, object dto)
{
var forTennant = dto as IForTenant;
if (forTennant != null)
RequestContext.Instance.Items.Add("TenantId", forTennant.TenantId);
}
[TestFixtureTearDown]
public void TestFixtureTearDown()
{
_appHost.Dispose();
}
[Test]
public void CanInvokeHelloServiceRequest()
{
var service = _appHost.Container.Resolve<MyServices>();
var response = (HelloResponse)service.Any(new Hello { Name = "World" });
Assert.That(response.Result, Is.EqualTo("Hello, World!"));
}
[Test]
public void CanInvokeFooServiceRequest()
{
var service = _appHost.Container.Resolve<MyServices>();
var lead = new Lead
{
TenantId = "200"
};
var response = service.Post(lead); //Does not fire filter.
}
}
ServiceStack is set at 4.0.40
Updated
After perusing the ServiceStack tests (which I highly recommend BTW) I came across a few example of the AppHost being used AND tested. It looks like the "ConfigureAppHost" property is the right place to configure the filters, e.g.
ConfigureAppHost = host =>
{
host.Plugins.Add(new ValidationFeature());
host.GlobalRequestFilters.Add(ITenantRequestFilter);
},
ConfigureContainer = container =>
{
}
Updated1
And they still don't fire.
Updated2
After a bit of trial and error I think it's safe to say that NO, the filters are not hooked up while using the BasicAppHost. What I have done to solve my problem was to switch these tests to use a class that inherits from AppSelfHostBase, and use the c# servicestack clients to invoke the methods on my service. THIS does cause the global filters to be executed.
Thank you,
Stephen
No the Request and Response filters only fire for Integration Tests where the HTTP Request is executed through the HTTP Request Pipeline. If you need to test the full request pipeline you'd need to use a Self-Hosting Integration test.
Calling a method on a Service just does that, i.e. it's literally just making a C# method call on a autowired Service - there's no intermediate proxy magic intercepting the call in between.

Resources