How does one log the full HTTP request and response when running integration tests of an ASP.NET MVC 3 application? - c#-4.0

Assume I have a ASP.NET MVC 3 controller and action for which I'd wrote integration tests.
namespace Sample.Controllers {
public class MyController : Controller {
[HttpPost]
public ActionResult Edit(MyModel model)
{
// process requests
}
}
}
The tests have the following "pattern".
namespace Sample.Controllers.IntegrationTests {
[TestClass]
public class MyControllerTests
{
[TestMethod]
public void Edit()
{
var request = (HttpWebRequest) WebRequest.Create(new Uri(uri));
request.Method = "POST";
request.Accept = "application/json";
request.AllowAutoRedirect = false;
request.AutomaticDecompression = DecompressionMethods.GZip;
request.UserAgent = "IntegrationTests/1.0";
request.ContentType = "application/json";
var resource = CreateResource();
using (Stream stream = request.GetRequestStream())
{
// serialize resource to stream
}
using (var response = (HttpWebResponse) request.GetResponse())
{
Assert.AreEqual(HttpStatusCode.OK, response.StatusCode);
/* perform more tests */
}
}
public MyModel CreateResource() {
/* create an instance of mymodel */
}
}
}
These tests aim to verify functionally, security and so forth; so mocking and unit tests don't suffice. The tests are executed in an unattended fashion and always uses https (and certs are verified for security reasons), so using a proxy is problematic. They are also executed with limited privileges.
Every now and then tests fail. I'd like to save a copy of the request and response (similar to what fiddler does) and associate it with each test, so that one can retrospectively use it to diagnose problems, if any exists.
Can one save the request and responses (incl. their headers) to disk without side effects that change the outcome of the tests? For example, for large streams, the response stream isn't seekable, so attempting to save the stream to disk before continuing to test the response doesn't seem to work.

Related

How to log outgoing HttpClient request body to Application Insights dependencies table?

I have a .Net core Web App API which accept request from front-end and then send HTTP POST request to Azure search to get search results.
I just use the build-in application insights logging log basic info in request and dependencies sources with zero logging code.
Now I want to extend the default Application Insights dependencies table to add the request body to Azure search.
What is the easiest way with minimum code?
As per your requirement, you can try the code below:
public class RequestBodyInitializer : ITelemetryInitializer
{
readonly IHttpContextAccessor httpContextAccessor;
public RequestBodyInitializer(IHttpContextAccessor httpContextAccessor)
{
this.httpContextAccessor = httpContextAccessor;
}
public void Initialize(ITelemetry telemetry)
{
if (telemetry is RequestTelemetry requestTelemetry)
{
if ((httpContextAccessor.HttpContext.Request.Method == HttpMethods.Post ||
httpContextAccessor.HttpContext.Request.Method == HttpMethods.Put) &&
httpContextAccessor.HttpContext.Request.Body.CanRead)
{
const string jsonBody = "JsonBody";
if (requestTelemetry.Properties.ContainsKey(jsonBody))
{
return;
}
//Allows re-usage of the stream
httpContextAccessor.HttpContext.Request.EnableRewind();
var stream = new StreamReader(httpContextAccessor.HttpContext.Request.Body);
var body = stream.ReadToEnd();
//Reset the stream so data is not lost
httpContextAccessor.HttpContext.Request.Body.Position = 0;
requestTelemetry.Properties.Add(jsonBody, body);
}
}
}
and add this to the Startup > ConfigureServices
services.AddSingleton<ITelemetryInitializer, RequestBodyInitializer>();

Azure Function slow response on first HTTPS call, with Always On ( same with ASP.NET Core Web API )

Need help to understand why first request always takes longer than others. Test case: send binary data via POST request.
This is a typical picture from Azure Application Insights, firing 2 series of 4 requests, within the same minute:
Server side
Simply reading the binary data into byte array.
with Azure Function:
[FunctionName("TestSpeed")]
public static HttpResponseMessage Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = "TestSpeed")]HttpRequestMessage req,
Binder binder,
ILogger log)
{
Stopwatch sw = new Stopwatch();
sw.Start();
byte[] binaryData = req.Content.ReadAsByteArrayAsync().Result;
sw.Stop();
return req.CreateResponse(HttpStatusCode.OK, $"Received {binaryData.Length} bytes. Data Read in: {sw.ElapsedMilliseconds} ms");
}
Or with ASP.NET web app API:
public class MyController : ControllerBase
{
private readonly ILogger<MyController> _logger;
public MyController(ILogger<MyController> logger)
{
_logger = logger;
}
[HttpPost]
public IActionResult PostBinary()
{
_logger.LogInformation(" - TestSpeed");
var sw = new Stopwatch();
sw.Start();
var body = Request.Body.ToByteArray();
sw.Stop();
return Ok($"Received {body.Length} bytes. Data Read in: {sw.ElapsedMilliseconds} ms");
}
}
Client (for testing only)
Using .NET Framework, C# console application...
private static void TestSpeed()
{
Console.WriteLine($"- Test Speed - ");
string requestUrl = "https://*******.azurewebsites.net/api/TestSpeed";
string path = "/Users/temp/Downloads/1mb.zip";
byte[] fileToSend = File.ReadAllBytes(path);
var sw = new Stopwatch();
for (int i = 0; i < 4; i++)
{
sw.Reset();
sw.Start();
var response = SendFile(fileToSend, requestUrl);
sw.Stop();
Console.WriteLine($"{i}: {sw.ElapsedMilliseconds} ms. {response}");
}
}
private static string SendFile(byte[] bytesToSend, string requestUrl)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUrl);
request.Method = "POST";
request.ContentType = "application/octet-stream";
request.ContentLength = bytesToSend.Length;
using (Stream requestStream = request.GetRequestStream())
{
// Send the file as body request.
requestStream.Write(bytesToSend, 0, bytesToSend.Length);
requestStream.Close();
}
try
{
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (var sr = new StreamReader(response.GetResponseStream()))
{
var responseString = sr.ReadToEnd();
return responseString;
}
}
}
catch (Exception e)
{
return "ERROR:" + e.Message;
}
}
Suspects I've tried:
Its not a cold start/warmup thing because the behavior repeats within the same minute.. and I have "Always On" enabled as well.
Compare HTTP and HTTPS - same behavior.
Azure functions vs ASP.NET core web API app - same behavior. The only difference I noticed is that with functions, request content is already fully received on server side before invocation:
ASP.NET web API: 5512 ms. Received 1044397 bytes. Data Read in: 3701 ms
Function App: 5674 ms. Received 1044397 bytes. Data Read in: 36 ms
Sending 1Kb vs 1Mb - same behavior, first call take much more.
Running server on Localhost - similar behavior, but much smaller difference than with distant servers! (looks like network distance matters here... )
Is there some session creation overhead? If so, why is it so huge?
Anything I can do about it?
Because your test interface is in the web program, even if you turn on the always on switch, what happens to the program or whether it can be kept active, you need to raise a support ticket to confirm with the official. From a developer's perspective, it is recommended that you test like this:
After redeploying the web interface, first use the function app to test, and then use the webapi interface to test to compare the test time.
Re-deploy again, first use webapi for testing and then use function app for testing. Compare test time.
No deployment is required. On the basis of the second test, test again after 5 minutes. The order of using function app or webapi does not matter. Look at the test time data.
I think the problem should be on IIS. IIS itself has a recycling mechanism. The application will not be used for a long time or there will be a delay after deployment. It is recommended to confirm with the official.

How to extend Dependency tracking for outgoing http requests in Application Insights

I have a .NET core API that performs HTTP connections to other API. I am able to visualize the outgoing HTTP request in Application Insights, under Dependency Event Types, but it has only basic information. I'm looking on how to add more information about the outgoing HTTP call (like the HTTP headers for instance).
I've looked into https://learn.microsoft.com/en-us/azure/azure-monitor/app/api-custom-events-metrics#trackdependency but I didn't find any concrete way of doing this.
As it has been said, the solution proposed by IvanYang is using the recived request instead of the dependency request.
I've built this ITelemetryInstance for that:
public void Initialize(ITelemetry telemetry)
{
var dependecyTelemetry = telemetry as DependencyTelemetry;
if (dependecyTelemetry == null) return;
if (dependecyTelemetry.TryGetOperationDetail("HttpRequest", out object request)
&& request is HttpRequestMessage httpRequest)
{
foreach (var item in httpRequest.Headers)
{
if (!dependecyTelemetry.Properties.ContainsKey(item.Key))
dependecyTelemetry.Properties.Add(item.Key, string.Join(Environment.NewLine, item.Value));
}
}
if (dependecyTelemetry.TryGetOperationDetail("HttpResponse", out object response)
&& response is HttpResponseMessage httpResponse)
{
var responseBody = httpResponse.Content.ReadAsStringAsync().GetAwaiter().GetResult();
if (!string.IsNullOrEmpty(responseBody))
dependecyTelemetry.Properties.Add("ResponseBody", responseBody);
}
}
This will record all the headers sent to the dependency and also the response received
The other solution given doesn't actually work the way you think it should, since it's attaching the header from the incoming HTTP request to the outgoing dependency request, which is misleading. If you want to attach dependency data to dependency logs then you need to wrap the dependency in a custom dependency wrapper, eg here I'm logging the outgoing payload of the dependency so I can see what's being sent by my system:
Activity activity = null;
IOperationHolder<DependencyTelemetry> requestOperation = null;
if ((request.Method == HttpMethod.Post || request.Method == HttpMethod.Put) && _httpContextAccessor?.HttpContext != null)
{
var bodyContent = await request.Content?.ReadAsStringAsync();
if (!string.IsNullOrEmpty(bodyContent))
{
activity = new Activity("Wrapped POST/PUT operation");
activity.SetTag("RequestBody", bodyContent);
requestOperation = _telemetryClient.StartOperation<DependencyTelemetry>(activity);
}
}
// perform dependency function
httpResponseMessage = await base.SendAsync(request, cancellationToken);
if (activity != null && requestOperation != null)
{
_telemetryClient.StopOperation(requestOperation);
}
I think what you're looking for is ITelemetryInitializer, which can add custom property for dependency telemetry.
And for .net core web project, you can refer to this link.
I write a demo as below:
1.Create a custom ITelemetryInitializer class to collect any dependency data:
public class MyTelemetryInitializer: ITelemetryInitializer
{
IHttpContextAccessor httpContextAccessor;
public MyTelemetryInitializer(IHttpContextAccessor httpContextAccessor)
{
this.httpContextAccessor = httpContextAccessor;
}
public void Initialize(ITelemetry telemetry)
{
//only add custom property to dependency type, otherwise just return.
var dependencyTelemetry = telemetry as DependencyTelemetry;
if (dependencyTelemetry == null) return;
if (!dependencyTelemetry.Context.Properties.ContainsKey("custom_dependency_headers_1"))
{
//the comment out code use to check the fields in Headers if you don't know
//var s = httpContextAccessor.HttpContext.Request.Headers;
//foreach (var s2 in s)
//{
// var a1 = s2.Key;
// var a2 = s2.Value;
//}
dependencyTelemetry.Context.Properties["custom_dependency_headers_1"] = httpContextAccessor.HttpContext.Request.Headers["Connection"].ToString();
}
}
}
2.Then in the Startup.cs -> ConfigureServices method:
public void ConfigureServices(IServiceCollection services)
{
//other code
//add this line of code here
services.AddSingleton<ITelemetryInitializer, MyTelemetryInitializer>();
}
3.Test result, check if the custom property is added to azure portal -> Custom Properties:

Do the Request filters get run from BasicAppHost?

I know that the services get wired-up by instantiating the BasicAppHost, and the IoC by using the ConfigureContainer property, but where is the right place to add the filters? The test in question never fire the global filter:
[TestFixture]
public class IntegrationTests
{
private readonly ServiceStackHost _appHost;
public IntegrationTests()
{
_appHost = new BasicAppHost(typeof(MyServices).Assembly)
{
ConfigureContainer = container =>
{
//
}
};
_appHost.Plugins.Add(new ValidationFeature());
_appHost.Config = new HostConfig { DebugMode = true };
_appHost.GlobalRequestFilters.Add(ITenantRequestFilter);
_appHost.Init();
}
private void ITenantRequestFilter(IRequest req, IResponse res, object dto)
{
var forTennant = dto as IForTenant;
if (forTennant != null)
RequestContext.Instance.Items.Add("TenantId", forTennant.TenantId);
}
[TestFixtureTearDown]
public void TestFixtureTearDown()
{
_appHost.Dispose();
}
[Test]
public void CanInvokeHelloServiceRequest()
{
var service = _appHost.Container.Resolve<MyServices>();
var response = (HelloResponse)service.Any(new Hello { Name = "World" });
Assert.That(response.Result, Is.EqualTo("Hello, World!"));
}
[Test]
public void CanInvokeFooServiceRequest()
{
var service = _appHost.Container.Resolve<MyServices>();
var lead = new Lead
{
TenantId = "200"
};
var response = service.Post(lead); //Does not fire filter.
}
}
ServiceStack is set at 4.0.40
Updated
After perusing the ServiceStack tests (which I highly recommend BTW) I came across a few example of the AppHost being used AND tested. It looks like the "ConfigureAppHost" property is the right place to configure the filters, e.g.
ConfigureAppHost = host =>
{
host.Plugins.Add(new ValidationFeature());
host.GlobalRequestFilters.Add(ITenantRequestFilter);
},
ConfigureContainer = container =>
{
}
Updated1
And they still don't fire.
Updated2
After a bit of trial and error I think it's safe to say that NO, the filters are not hooked up while using the BasicAppHost. What I have done to solve my problem was to switch these tests to use a class that inherits from AppSelfHostBase, and use the c# servicestack clients to invoke the methods on my service. THIS does cause the global filters to be executed.
Thank you,
Stephen
No the Request and Response filters only fire for Integration Tests where the HTTP Request is executed through the HTTP Request Pipeline. If you need to test the full request pipeline you'd need to use a Self-Hosting Integration test.
Calling a method on a Service just does that, i.e. it's literally just making a C# method call on a autowired Service - there's no intermediate proxy magic intercepting the call in between.

Using ServiceStack MiniProfiler to profile all service client calls

Context: I'm writing a service using ServiceStack. This service is calling some other remote services (using the ServiceStack JsonServiceClient).
Requirement: show every call to the remote service as a step in MiniProfiler.
Question: what would be the best way to implement this in a generic way?
The original code in my service looked like the following:
// Registration of the serviceclient in Apphost.cs:
// container.Register<IRestClient>(x => new JsonServiceClient("http://host:8080/"));
var client = ResolveService<IRestClient>();
HelloResponse response;
using (Profiler.Current.Step("RemoteService: Get Hello"))
{
response = client.Get(new Hello { Name = "World!" });
}
// ... do something with response ...
I wanted to get rid of the using (Profiler.Current.Step()) in this part of my code to make it easier to read and write.
// Registration of the serviceclient in Apphost.cs:
// container.Register<IRestClient>(x => new ProfiledRestClient("RemoteService", new JsonServiceClient("http://host:8080/")));
var client = ResolveService<IRestClient>();
HelloResponse response = client.Get(new Hello { Name = "World!" });
// ... do something with response ...
I made a wrapper around the existing client that contains the Profiler.Current.Step() code for every method of the IRestClient interface
mentioning the name of the client, the method and the request(type).
// The implementation of the wrapper:
public class ProfiledRestClient : IRestClient
{
readonly string clientName;
readonly IRestClient wrappedClient;
public ProfiledRestClient(string clientName, IRestClient wrappedClient)
{
this.clientName = clientName;
this.wrappedClient = wrappedClient;
}
public TResponse Get<TResponse>(IReturn<TResponse> request)
{
using (Profiler.Current.Step("{0}: Get {1}".Fmt(clientName, request.GetType().Name)))
{
return wrappedClient.Get(request);
}
}
public TResponse Post<TResponse>(IReturn<TResponse> request)
{
using (Profiler.Current.Step("{0}: Post {1}".Fmt(clientName, request.GetType().Name)))
{
return wrappedClient.Post(request);
}
}
// etc. the same for all other methods of IRestClient interface
}
It is working but it feels a bit dirty. Is there a better way of doing this?
Thank you for your insight.

Resources