I need to log information from all invocations, successful or not, of a function app in Azure. I first tried just using log.LogInformation() and found that messages were not being written from all function invocations. Doing some research I got to understand that in high load scenarios (mine is a high load scenario), sometimes the runtime decides not to log some of the successful invocations. Fair enough.
I then tried using custom events to do logging and capture the info I needed:
TelemetryConfiguration config = TelemetryConfiguration.CreateDefault();
TelemetryClient tc = new TelemetryClient(config);
Dictionary<string, string> props = new Dictionary<string, string>();
props["msgid"] = msgid;
tc.TrackEvent("MsgToBenefitsService", props);
Still no luck, in some runs I did, I saw only 82 rows in app insights from 1000 invocations. I haven't been able to find any documentation saying that Custom Events might not be logged, so I expected that I would see 1000 events logged for 1000 invocations.
Is there anything wrong with the logging code above ? And are there any options to guarantee that I can write information from an invocation to AppInsights ? Or am I stuck with having to explicitly log myself from the function app ?
As background, this function app has a service bus trigger to read messages off a topic. I'm using v3 of the runtime.
Any help would be appreciated.
Thanks.
Please disable sampling in host.json:
"applicationInsights": {
"samplingSettings": {
"isEnabled": false
}
}
applicationInsights.samplingSettings
Sampling in Application Insights
Related
I have an ASP.NET Core 3.1 solution deployed into an Azure Web App hooked up to Application Insights. I can't for the life of me get exceptions and stack traces to log into Application Insights, instead I get a basic request trace with no exception information attached:
I've tried most combinations of setting up logging/application insights telemetry, here are some of the things I've tried:
services.AddApplicationInsightsTelemetry(); in the ConfigureServices() method of Startup.cs
Adding logging.AddApplicationInsights(); to my logging builder in Program.cs
Removing the custom error page exception handler in case that was affecting things
I have the APPINSIGHTS_INSTRUMENTATIONKEY environment variable set on my Web App in Azure.
I'm using the following code to generate exceptions in Application Insights:
[AllowAnonymous]
[Route("autoupdate")]
public async Task<IActionResult> ProfileWebhook()
{
var formData = await this.Request.ReadFormAsync();
var config = TelemetryConfiguration.CreateDefault();
var client = new TelemetryClient(config);
client.TrackException(new Exception(string.Join("~", formData.Keys)));
logger.LogError(new Exception(string.Join("~", formData.Keys)), "Fail");
throw new Exception(string.Join("~", formData.Keys));
}
Nothing is working and I'm going crazy! Any help greatly appreciated.
Usually, Application insights will guarantee that all the kinds of telemetries(like exceptions, trace, event etc.) will be arrived around 5 minutes, please refer this doc: How long does it take for telemetry to be collected?. But there is still a chance that it will take a longer time due to beckend issue(a very small chance).
If you're using visual studio, you can check if the telemetry is sent or not via Application Insights search.
You can also check if you're using a correct IKey, or if you have enabled sampling.
But if it keeps this behavior in your side, you should consider contacting MS support to find the root cause.
Hope it helps.
I have a 3-week-old Function on the v2 SDK and uses the out-box App Insights support, i.e. just set the environment variable and ...magic.
I did read that by default it doesn't send Information level logs to AppInsights and this looks correct because I can't see any of my information tracing but I do see occasional errors I've logged out with my custom message.
But for metrics, its broken. I can see some semblance of my telemetry in the logs, sort of. It's strange.
Look.
Now look at these curious logs.
...see those n/a that match the LogMetric lines in my code?
And when I search explicitly for one of them, nothing is found.
And if I drill down on those n/a logs.
...they're pretty empty.
Is it me or is this all rather fishy?
How do I log metrics from a Function in a way that works?
First, for the metric, do you mean custom metrics? If yes, then you should find it in the customMetrics table in your app insights.
Second, what's the method are you using to send metrics? From your code, what's the implementation of this.Logger?.LogMetric method?
For sending metrics, you can use TrackMetric(just tested with this method, and works) or GetMetric method.
A sample code use TrackMetric(deprecated, but still can use) / GetMetric(recommended) method:
TelemetryConfiguration.Active.InstrumentationKey = "your key";
TelemetryClient client = new TelemetryClient();
client.TrackMetric("my metric", 60);
client.GetMetric("my metric").TrackValue(89);
client.Flush();
In my case, the settings in the host.json file were excluding the metrics from getting to Application Insights. The metrics are being logged at the Information level and I originally had mine configured for the Warning level.
"logging": {
"logLevel": {
"Function.<function name>": "Information", //<--won't see metrics if set to Warning
...
See here for more information on configuring the logging levels.
I am using this code to send events to application insights in a console application
TelemetryConfiguration.Active.InstrumentationKey = "XXXXXXXXX";
TelemetryClient telemetryClient = new TelemetryClient();
for (int i = 0; i < 100; i++)
{
telemetryClient.TrackEvent("Hello World!");
telemetryClient.TrackException(new OutOfMemoryException());
}
telemetryClient.Flush();
Task.Delay(60000).Wait();
Now the problem i am having is that it is not seeming to log all my events , sometimes the visual studio toolbar says 44 , sometimes it is 68 and never 100 .
The type of information i am going to send is important cause i will be monitoring several console applications from this service .
Is there any way to have application insights send every thing to azure and not skip events ? I think i am giving it enough time to send every thing before exiting .
Without full code, its hard to say the configuration used.Couple of things to look for:
Have you enabled sampling? If you really want accurate count of events, then disable sampling (https://learn.microsoft.com/en-us/azure/azure-monitor/app/sampling)
Have you configured channel explicitly? If not, the default will be InMemoryChannel, which does not do any retries for transient issues. Its best to use ServerTelemetryChannel, to protect data loss in the even of network issue or application insights backend transient issues.
var config = new TelemetryConfiguration(); // or active or create default...
var channel = new ServerTelemetryChannel();
channel.initialize(config)
// create client from the config.
TelemetryClient tc= new TelemetryClient(config);
I have an ASP.NET Core a website with a lot of simultaneous users which crashes many times during the day and I scaled up and out but no luck.
I have been told my numerous Azure support staff that the issue is that I'm sending out a lot of database calls although database utilization improved after creating indexes. Can you kindly advise what you think the problem is as I have done my best...
I was told that I have "socket leaks".
Please note:
I don't have any external service calls except to sendgrid
I have not used ConfigureAwait(false)
I'm not using "using" statements or explicitly disposing contexts
This is my connection string If it may help...
Server=tcp:sarahah.database.windows.net,1433;Initial Catalog=SarahahDb;Persist Security Info=False;User ID=********;Password=******;MultipleActiveResultSets=True;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;Max Pool Size=400;
These are some code examples:
In Startup.CS:
services.AddDbContext<ApplicationDbContext>(options =>
options.UseSqlServer(Configuration.GetConnectionString("DefaultConnection")));
Main class:
private readonly ApplicationDbContext _context;
public MessagesController(ApplicationDbContext context, IEmailSender emailSender, UserManager<ApplicationUser> userManager)
{
_context = context;
_emailSender = emailSender;
_userManager = userManager;
}
This an important method code for example:
string UserId = _userManager.GetUserId(User);
var user = await _context.Users.Where(u => u.Id.Equals(UserId)).Include(u => u.Messages).FirstOrDefaultAsync();
// some other code
return View(user.Messages);
Please advise as I have tried my best but this is very embarrassing to me in font of my customers.
Without the error messages that you're seeing, here's a few ideas that you can check.
I'd start with going to your Web App's Overview blade in the Azure Portal. Update the monitoring graph to a time period when you're experiencing problems. Are you CPU bound? Have you exhausted memory? Also, check the HTTP Queue length. If your HTTP queue is really long, it's because your server is choking trying to service the requests and users are experiencing timeout issues.
Next, jump over to your SQL Server's Overview blade in the Azure Portal, and look at the resource utilization chart. Set the time period on the chart to when you're experiencing problems. Have you pegged out your DTUs for your database? If so, it's a sign of poor indexing, poor schema design, or you're just undersized and need to scale up.
Turn on ApplicationInsights if you haven't already. You can use the ApplicationInsights API to insert your own trace statements into your code. Or, you might be able to see exceptions causing the issue without having to do your own tracing.
Check the Kudu logs for your Web Apps.
I agree with Tseng - your usage of EF and .NET Core's DI framework looks correct.
Let us know how the troubleshooting goes and provide additional information on exactly what kind of errors you're seeing. Best of luck!
It looks like a DI issue to me. You are injecting ApplicationDbContext context. Which means the ApplicationDbContext will be resolved from the DI container meaning it will stay open the entire request (transient) as Tseng pointed out. It should be a scoped.
You can inject IServiceScopeFactory scopeFactory in your controller and do something like:
using (var scope = _scopeFactory.CreateScope())
{
var context = scope.ServiceProvider.GetRequiredService<ApplicationDbContext>();
}
Note that if you are using ASP.NET Core 1.1 and want to be sure that all your services are being resolved correctly change your ConfigureService method in the Startup to:
public IServiceProvider ConfigureServices(IServiceCollection services)
{
// Register services
return services.BuildServiceProvider(validateScopes: true);
}
Is there any way to configure triggers without attributes? I cannot know the queue names ahead of time.
Let me explain my scenario here.. I have one service bus queue, and for various reasons (complicated duplicate-suppression business logic), the queue messages have to be processed one at a time, so I have ServiceBusConfiguration.OnMessageOptions.MaxConcurrentCalls set to 1. So processing a message holds up the whole queue until it is finished. Needless to say, this is suboptimal.
This 'one at a time' policy isn't so simple. The messages could be processed in parallel, they just have to be divided into groups (based on a field in message), say A and B. Group A can process its messages one at a time, and group B can process its own one at a time, etc. A and B are processed in parallel, all is good.
So I can create a queue for each group, A, B, C, ... etc. There are about 50 groups, so 50 queues.
I can create a queue for each, but how to make this work with the Azure Webjobs SDK? I don't want to copy-paste a method for each queue with a different ServiceBusTrigger for the SDK to discover, just to enforce one-at-a-time per queue/group, then update the code with another copy-paste whenever another group is needed. Fetching a list of queues at startup and tying to the function is preferable.
I have looked around and I don't see any way to do what I want. The ITypeLocator interface is pretty hard-set to look for attributes. I could probably abuse the INameResolver, but it seems like I'd still have to have a bunch of near-duplicate methods around. Could I somehow create what the SDK is looking for at startup/runtime?
(To be clear, I know how to use INameResolver to get queue name as at How to set Azure WebJob queue name at runtime? but though similar this isn't my problem. I want to setup triggers for multiple queues at startup for the same function to get the one-at-a-time per queue processing, without using the trigger attribute 50 times repeatedly. I figured I'd ask again since the SDK repo is fairly active and it's been a year..).
Or am I going about this all wrong? Being dumb? Missing something? Any advice on this dilemma would be welcome.
The Azure Webjob Host discovers and indexes the functions with the ServiceBusTrigger attribute when it starts. So there is no way to set up the queues to trigger at the runtime.
The simpler solution for you is to create a long time running job and implement it manually:
public class Program
{
private static void Main()
{
var host = new JobHost();
host.CallAsync(typeof(Program).GetMethod("Process"));
host.RunAndBlock();
}
[NoAutomaticTriggerAttribute]
public static async Task Process(TextWriter log, CancellationToken token)
{
var connectionString = "myconnectionstring";
// You can also get the queue name from app settings or azure table ??
var queueNames = new[] {"queueA", "queueA" };
var messagingFactory = MessagingFactory.CreateFromConnectionString(connectionString);
foreach (var queueName in queueNames)
{
var receiver = messagingFactory.CreateMessageReceiver(queueName);
receiver.OnMessage(message =>
{
try
{
// do something
....
// Complete the message
message.Complete();
}
catch (Exception ex)
{
// Log the error
log.WriteLine(ex.ToString());
// Abandon the message so that it can be retry.
message.Abandon();
}
}, new OnMessageOptions() { MaxConcurrentCalls = 1});
}
// await until the job stop or restart
await Task.Delay(Timeout.InfiniteTimeSpan, token);
}
}
Otherwise, if you don't want to deal with multiple queues, you can have a look at azure servicebus topic/subscription and create SqlFilter to send your message to the right subscription.
Another option could be to create your own trigger: The azure webjob SDK provides extensibility points to create your own trigger binding :
Binding Extensions Overview
Good Luck !
Based on my understanding, your needs seems to be building a message batch system in parallel. The #Thomas solution is good, but I think Azure Batch service with Table storage may be better and could be instead of the complex solution of ServiceBus queue + WebJobs with a trigger.
Using Azure Batch with Table storage, you can control the task creation and execute the task in parallel and at scale, even monitor these tasks, please refer to the tutorial to know how to.