I tried tracking custom metrics with and without flushing it. However, the metrics only intermittently shows up in Application Insights under the "Custom" section. First question: Is it required to run "flush()" after every single "TrackMetric(metric)" call in order for the telemetry to be sent to Application Insights? Second: Why is there this intermittent behavior? I'm only writing one metric at a time, so it's not as if I'm overloading Application Insights with thousands of separate calls. Here is my code (This is from a simple Console App):
public class Program
{
public static void Main(string[] args)
{
var telemetryClient = new TelemetryClient()
{
Context = { InstrumentationKey = "{{hidden instrumentation key}}" }
};
var metric = new MetricTelemetry
{
Name = "ImsWithContextMetric2",
Sum = 42.0
};
telemetryClient.TrackMetric(metric);
telemetryClient.Flush();
}
}
I'm also getting this strange behavior in Application Insights in which the custom metric I add shows up under a "Unavailable/deprecated Metrics" section. And a metric that I didn't even add called "Process CPU (all cores)" pops up under the "Custom" section. Any ideas why this strange behavior would occur?:
Is it required to run "flush()" after every single "TrackMetric(metric)" call in order for the telemetry to be sent to Application Insights?
Since you are using a Console Application to send events to Application Insights, which might be short-lived, it is definitely a good practice to call .Flush() every once in a while. The SDK uses the InMemoryChannel to send telemetry and sends it in batches using from an in-memory queue. So it is very important to call the .Flush() so that the data is forcefully pushed. A good practice might be to add a bit of wait after the event:
telemetryClient.Flush();
Thread.Sleep(1000);
More reading: Flushing data, Ensure you don't lose telemetry
However, the metrics only intermittently shows up in Application Insights under the "Custom" section. Why is there this intermittent behavior? I'm only writing one metric at a time, so it's not as if I'm overloading Application Insights with thousands of separate calls.
Sometimes there is a delay in metrics showing up in the Azure Portal. It can be up to a few minutes too. But if you have set it up correctly, you aren't exceeding the throttling limit, and adaptive sampling is disabled, then there is no reason for which telemetry should be intermittent. However if you still feel something is wrong, start a fiddler trace (make sure you are capturing from non-browser sessions) and check if a call is going out to dc.services.visualstudio.com. Make sure the response is 200 OK and if the items were accepted by the server.
I'm also getting this strange behavior in Application Insights in which the custom metric I add shows up under a "Unavailable/deprecated Metrics" section.
What version of the SDK are you using? I just tried out the same scenario and the custom metrics are showing up correctly.
And a metric that I didn't even add called "Process CPU (all cores)" pops up under the "Custom" section.
"Process CPU" is a performance counter which is used to track CPU utilization. I believe the SDK will only be able to track these counters if the app is running under IIS or on Azure. It probably got added internally when you created your Application Insights resource. You can ignore it since it won't have data to chart.
Hope this helps!
Related
I have following controller class and a PUT method in an azure app service (.NET C#)
MyController {
[MyApiRoute("anapi")]
// PUT: anapi/items
[HttpPut("items")]
public async Task<ActionResult<AnObject>> ManageLineItems()
{
//some code here....
}
}
I am trying to measure performance of making a call to this REST URL using azure insights. It shows First line with AJAX call as 7.7 sec and second line with controller method as 3.9 seconds as shown in diagram. My understanding is that first line in azure insights would correspond to call to this URL and second line corresponds to the time the method in controller took to execute. If that is correct, were typically on azure app service would 7.7 sec - 3.9 sec would be spent? This doesn't appear to be time spent by my code.
If my understanding about these two lines in azure insights is incorrect, can any experts here explain me what this means? Also, were could 2.8 seconds which is difference between total execution for route and method would have been spent?
Please let me know if i need to provide more details about the problem.
Application Insights is a full-scale application performance monitoring tool which does much more than only logging. It is Application Performance Management (APM) service for collecting and monitoring the application log data
In the above picture in your question, the first line represents the outgoing dependency form your web service and the following line represents the incoming request. And the duration column you see is the cumulative request duration at each level.
If Service A calls service B then from point of view of Service A , call to Service B is an ‘Outgoing dependency’ and when this request is received at Service B then from its point of view the call is an ‘Incoming request’ from service A. That's the reason App insights captures both of them .
This is called end to end correlation, which works by default when you enable application insight.
I would suggest to read this End to end correlation using Azure application insights for asp.net mvc apps document to understand the entire to understand hierarchical view of the request flow telling which service or API calls happened along with where and at what point exception occurred along with all the logs / traces that happened during that request.
I have an Azure function with ServiceBusTrigger which will post the message content to a webservice behind an Azure API Manager. In some cases the load of the (3rd party) webserver backend is too high and it collapses returning error 500.
I'm looking for a proper way to implement circuit breaker here.
I've considered the following:
Disable the azure function, but it might result in data loss due to multiple messages in memory (serviceBus.prefetchCount)
Implement API Manager with rate-limit policy, but this seems counter productive as it runs fine in most cases
Re-architecting the 3rd party webservice is out of scope :)
Set the queue to ReceiveDisabled, this is the preferred solution, but it results in my InputBinding throwing a huge amount of MessagingEntityDisabledExceptions which I'm (so far) unable to catch and handle myself. I've checked the docs for host.json, ServiceBusTrigger and the Run parameters but was unable to find a useful setting there.
Keep some sort of responsecode resultset and increase retry time, not ideal in a serverless scenario with multiple parallel functions.
Let API manager map 500 errors to 429 and reschedule those later, will probably work but since we send a lot of messages it will hammer the service for some time. In addition it's hard to distinguish between a temporary 500 error or a consecutive one.
Note that this question is not about deciding whether or not to trigger the circuitbreaker, merely to handle the appropriate action afterwards.
Additional info
Azure functionsV2, dotnet core 3.1 run in consumption plan
API Manager runs Basic SKU
Service Bus runs in premium tier
Messagecount: 300.000
We are looking at implementing AppInsights for our non-web application. One of the things that we want to monitor for is processes that may be "hung" for more than N number of seconds or minutes. I have been unable to find something built in that does this. The closest thing I have seen or thought of would be to log 2 custom events for the start and end of a process, and then have an alert for a custom log that queries events with no matching "end" event after N minutes.
Is there another way to monitor for hung processes using AppInsights that I am not seeing? Thanks for any help.
If you choose to use application insights, here is the suggestion just for your reference(but if you have another better solution, you can ignore this):
As per this post, you can leverage heartbeat feature, details as below:
if this application runs more than several seconds, you can leverage heartbeat
feature - it sends metric every N minutes/seconds (configurable) and the absence of such
metric will indicate that application is no longer actively running. However, if
Application Insights thread survives, then heartbeat will still be reported.
You can rely on presense/absense of the telemetry from this app in general as well as
couple custom events as you outlined above - Azure Monitor allows to set an alert on
analytics query, so you'll be able to craft a query that returns nothing in case of
application issues and set an alert on 0 count returned by such a query.
I'm facing an issue with my MVC app service deployed on Azure.
My MVC action method receives requests and depening on parameters in querystring, it performs a redirect to external URLs.
The usual response time is milliseconds but sometimes there are requests that took a real higher response time:
The action method is real simple and there's not so much logic in it so it could be summarized as follows:
public ActionResult performRedirect(string id)
{
System.Diagnostics.Trace.TraceInformation("start");
if (id == "1")
return Redirect("http://URLA");
else if (id == "2")
return Redirect("http://URLB");
else
return Redirect("http://URLC");
}
My application uses ApplicationInsights therefore i performed analysis on that and what i found is that whenever there are "slow" requests there's a sort of delay between the time the request is handled by the action method and the diagnostic tracking "start" (up to 10 seconds!).
My question is: why is this happening? is it because of an increase in the requests to the action that cannot be managed and therefore there's a incoming queue to be emptied? should i increase the performances of the resource (now i'm using a S1 with 2 instances)?
To the performance issue, there is not a specific explanation. It may caused by several reasons, like bandwidth restrictions, source limited, etc.
You could try to troubleshoot with the article. Also some ways to mitigate the issue,
Scale the web app
Use AutoHeal
Restart the web app
Besides, here is a similar issue, you could refer to it.
In an Azure Function, when you enable telemetry to Application Insight and fire a (for example) logger.LogInformation call (where logger is an ILogger instance), does it send it to the Application Insight instance asynchronously (ie non-blocking), synchronously (blocking), or through a local log that gets drained asynchronously?
Generally, the logger would be hooked up to turn log calls into the various trackMessage or related calls in the Application Insights SDK. those messages get batched up in the AI side, and then sent after a threshold count of messages has been met, or after a certain amount of time has elapsed. the calls into application insights are all non-blocking, and will not throw exceptions (you don't want telemetry to negatively affect your real app!)
the c# sdks that azure functions would use would be here: https://github.com/Microsoft/ApplicationInsights-dotnet/
I said generally at the top, because all this depends on how the SDK is configured, and that would be up to the Azure functions underlying code. The GitHub with their info is here: https://github.com/Azure/Azure-Functions, and they have a specific wiki set up with AI info as well, here: https://github.com/Azure/Azure-Functions/wiki/App-Insights
This appears the be the relevant code for specifically how data is sent to Application Insights:
https://github.com/Microsoft/ApplicationInsights-dotnet/tree/develop/src/Microsoft.ApplicationInsights/Channel
The ILogger wraps a TelemetryClient, which sends data to an ITelemetryChannel.
The InMemoryTelemetryChannel contains the logic for how data is pooled and sent to Application Insights. As John mentioned, the channel uses a "buffer" for storing data that hasn't been sent. The buffer is flushed and the data sent asynchronously to Azure Portal when either the buffer is full or at a specific time internal (30 seconds).