I've configured the Azure Diagnostics so that the logs get uploaded to a storage table. I'm using Trace.TraceXxx from my code and all works well.
Now I'm trying to add tracing from the Role OnStart() and OnStop() methods. I know that the tracing works as I see the lines in the Debug window when running in the emulator. But from the cloud deployment, it seems that these trace lines never get uploaded to the table. My guess is that it is somewhat related to TraceSources, as the only trace lines I've in the table come from the w3wp.exe source... Any hint ?
Thanks
Like you said you can add the trace listener using the WaIISHost.exe.config, but besides that you can also add the trace listener in code (you'll need a reference to Microsoft.WindowsAzure.Diagnostics.dll):
public class WebRole : RoleEntryPoint
{
public override void Run()
{
var listener = new Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener();
Trace.Listeners.Add(listener);
...
}
}
Another way of setting up diagnostics is through the configuration file. If you created a VS solution recently, it will automatically create the diagnostics plug-in and configuration for the trace listener. With the config file (diagnostics.wadcfg) there is no code that needs to be written for the different data sources. Here is a link where you can get started and a sample file:
http://msdn.microsoft.com/en-us/library/gg604918.aspx
You cannot include custom performance counters right now and you need to make sure that you don’t try to allocate more than 4GB of buffer to anything (you can leave at 0), or it tends to fail.
Note, the time interval format (e.g PT1M). That is a serialization format, so PTXM is X minutes, while PTXS is X in seconds. You need to mark this as content and copy always in Visual Studio (place at root of project) so it gets packaged.
And here is a link to the three ways to setup diagnostics
http://msdn.microsoft.com/en-us/library/windowsazure/hh411541.aspx
Ranjith
http://www.opstera.com
Related
I have a .Net Framework 4.5.1 app service that has the following code:
Console.WriteLine("🔥 Console.WriteLine");
Trace.WriteLine("🔥 Trace.WriteLine");
Trace.TraceInformation("🔥 Trace.TraceInformation");
Trace.TraceWarning("🔥 Trace.TraceWarning");
Trace.TraceError("🔥 Trace.TraceError");
I would like to be able to see logs that we log into Azure when the app is in production. So I enabled Application Service Logging for my web app:
As you can see, I have the logging level set to verbose because we want to see everything. I also have Application Insights turned on for my app. But when I check the logs, all I see are the logs at the warning level and higher.
I checked in the "logs" tab:
I also checked in my blob storage. I see logs, but they're all on a warning level or higher.
There was one way I got this to work, and that's using the feature that saves logs to a file system. But that feature turns itself off after 12 hours, and the file might get too big anyways, and I'd like to use Azure Insights since it has a nicer interface.
How can I view all my logs my application outputs over a long period of time?
I usually just remove all logging configuration from config files like appsettings.json and use the ApplicationInsightsLoggerProvider type specifier in Startup.cs, like so:
public void ConfigureServices(IServiceCollection services)
{
services.AddLogging(conf =>
{
conf.AddApplicationInsights();
conf.AddFilter<ApplicationInsightsLoggerProvider>("", LogLevel.Information);
conf.AddFilter<ApplicationInsightsLoggerProvider>("Microsoft", LogLevel.Warning);
conf.AddFilter<ApplicationInsightsLoggerProvider>("System", LogLevel.Warning);
});
...
}
It's documented here.
If you have various trace levels in your web app but are only interested in having certain levels of logs sent to the logging endpoint, you can set a filter for the minimum level in your application settings under Configuration. By default, even without the app setting, the minimum trace level is set to Warning.
How to set App setting for AppServiceAppLogs level
The application setting name will be APPSERVICEAPPLOGS_TRACE_LEVEL and the value will be the minimum level (ie. Error, Warning, Verbose, etc.). Refer to TraceLevel for more info.
Links
https://learn.microsoft.com/en-us/azure/app-service/troubleshoot-diagnostic-logs
https://azure.github.io/AppService/2020/08/31/AzMon-AppServiceAppLogs.html (Image Source)
I have an ASP.NET Core 3.1 solution deployed into an Azure Web App hooked up to Application Insights. I can't for the life of me get exceptions and stack traces to log into Application Insights, instead I get a basic request trace with no exception information attached:
I've tried most combinations of setting up logging/application insights telemetry, here are some of the things I've tried:
services.AddApplicationInsightsTelemetry(); in the ConfigureServices() method of Startup.cs
Adding logging.AddApplicationInsights(); to my logging builder in Program.cs
Removing the custom error page exception handler in case that was affecting things
I have the APPINSIGHTS_INSTRUMENTATIONKEY environment variable set on my Web App in Azure.
I'm using the following code to generate exceptions in Application Insights:
[AllowAnonymous]
[Route("autoupdate")]
public async Task<IActionResult> ProfileWebhook()
{
var formData = await this.Request.ReadFormAsync();
var config = TelemetryConfiguration.CreateDefault();
var client = new TelemetryClient(config);
client.TrackException(new Exception(string.Join("~", formData.Keys)));
logger.LogError(new Exception(string.Join("~", formData.Keys)), "Fail");
throw new Exception(string.Join("~", formData.Keys));
}
Nothing is working and I'm going crazy! Any help greatly appreciated.
Usually, Application insights will guarantee that all the kinds of telemetries(like exceptions, trace, event etc.) will be arrived around 5 minutes, please refer this doc: How long does it take for telemetry to be collected?. But there is still a chance that it will take a longer time due to beckend issue(a very small chance).
If you're using visual studio, you can check if the telemetry is sent or not via Application Insights search.
You can also check if you're using a correct IKey, or if you have enabled sampling.
But if it keeps this behavior in your side, you should consider contacting MS support to find the root cause.
Hope it helps.
I have an ASP.NET Core a website with a lot of simultaneous users which crashes many times during the day and I scaled up and out but no luck.
I have been told my numerous Azure support staff that the issue is that I'm sending out a lot of database calls although database utilization improved after creating indexes. Can you kindly advise what you think the problem is as I have done my best...
I was told that I have "socket leaks".
Please note:
I don't have any external service calls except to sendgrid
I have not used ConfigureAwait(false)
I'm not using "using" statements or explicitly disposing contexts
This is my connection string If it may help...
Server=tcp:sarahah.database.windows.net,1433;Initial Catalog=SarahahDb;Persist Security Info=False;User ID=********;Password=******;MultipleActiveResultSets=True;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;Max Pool Size=400;
These are some code examples:
In Startup.CS:
services.AddDbContext<ApplicationDbContext>(options =>
options.UseSqlServer(Configuration.GetConnectionString("DefaultConnection")));
Main class:
private readonly ApplicationDbContext _context;
public MessagesController(ApplicationDbContext context, IEmailSender emailSender, UserManager<ApplicationUser> userManager)
{
_context = context;
_emailSender = emailSender;
_userManager = userManager;
}
This an important method code for example:
string UserId = _userManager.GetUserId(User);
var user = await _context.Users.Where(u => u.Id.Equals(UserId)).Include(u => u.Messages).FirstOrDefaultAsync();
// some other code
return View(user.Messages);
Please advise as I have tried my best but this is very embarrassing to me in font of my customers.
Without the error messages that you're seeing, here's a few ideas that you can check.
I'd start with going to your Web App's Overview blade in the Azure Portal. Update the monitoring graph to a time period when you're experiencing problems. Are you CPU bound? Have you exhausted memory? Also, check the HTTP Queue length. If your HTTP queue is really long, it's because your server is choking trying to service the requests and users are experiencing timeout issues.
Next, jump over to your SQL Server's Overview blade in the Azure Portal, and look at the resource utilization chart. Set the time period on the chart to when you're experiencing problems. Have you pegged out your DTUs for your database? If so, it's a sign of poor indexing, poor schema design, or you're just undersized and need to scale up.
Turn on ApplicationInsights if you haven't already. You can use the ApplicationInsights API to insert your own trace statements into your code. Or, you might be able to see exceptions causing the issue without having to do your own tracing.
Check the Kudu logs for your Web Apps.
I agree with Tseng - your usage of EF and .NET Core's DI framework looks correct.
Let us know how the troubleshooting goes and provide additional information on exactly what kind of errors you're seeing. Best of luck!
It looks like a DI issue to me. You are injecting ApplicationDbContext context. Which means the ApplicationDbContext will be resolved from the DI container meaning it will stay open the entire request (transient) as Tseng pointed out. It should be a scoped.
You can inject IServiceScopeFactory scopeFactory in your controller and do something like:
using (var scope = _scopeFactory.CreateScope())
{
var context = scope.ServiceProvider.GetRequiredService<ApplicationDbContext>();
}
Note that if you are using ASP.NET Core 1.1 and want to be sure that all your services are being resolved correctly change your ConfigureService method in the Startup to:
public IServiceProvider ConfigureServices(IServiceCollection services)
{
// Register services
return services.BuildServiceProvider(validateScopes: true);
}
I like to change the timespan a Q-message is invisible in the scenario where a webjob is triggered upon arrival of a message in the Queue.
I need this in the dev-stage while debugging. Sometimes an old Q-msg re-appears due to a bug but I already cleaned up the blob-storage-item the message was refering to. Hence WebJob crashes because it tries to get that not existing blob-object.
If I could set that the Q-messages are again visible after 1 min, that would be helpfull.
public partial class Functions
{
public static void ProcessBulkFileQ(
[QueueTrigger(AppConst.Q_BULKFILES)] JobInfo JobInfo,
[Blob(AppConst.CONTAINER_BULKFILES + "/{BlobName}", FileAccess.Read)] Stream InputStream,
[Blob(AppConst.CONTAINER_BULKFILES + "/{BlobName}")] CloudBlockBlob BlobToDelete
)
{
In the latest v1.1.0 release of the SDK we've added the ability for you to add custom QueueProcessors to control various aspects of message processing. You can see an example of what you're looking to do here in our tests.
In that example, for failed messages, you can explicitly set the visibility to your own value based on the error details. You can register custom QueueProcessors by creating your own factory and setting it on JobHostConfiguration.Queues.QueueProcessorFactory.
I am running an azure webjobs SDK console application (continuous) with the recommended setup:
public static void ProcessQueueMessage([QueueTrigger("logqueue")] string logMessage, TextWriter logger)
The azure queue I am running against has ~6000 messages in it and I am running the web-job locally, as a console application.
The problem I'm having is that the processing randomly stops after processing between zero and ~30 messages. The console stays open, but no more console messages are displayed.
For example, it might just process 2 messages:
Executing: 'Functions.ProcessQueueMessage' - Reason: 'New queue message detected on 'QueueName'.'
Executed: 'Functions.ProcessQueueMessage' (Succeeded)
Executing: 'Functions.ProcessQueueMessage' - Reason: 'New queue message detected on 'QueueName'.'
Executed: 'Functions.ProcessQueueMessage' (Succeeded)
And then, nothing. There doesn't seem to be anything wrong with my internet connection and I can't trace the issues down to any particular messages.
Has anyone else had issues with this SDK?
Update:
I made sure that I was using the right versions of all of the dependencies by removing the nuget packages and then re-running install-package Microsoft.Axure.Webjobs. I am now using webjobs version 1.1.0 which has pulled in version 4.3 of azure storage.
As recommended by Matthew, I have pulled down the source code for azure webjobs to determine where the process is freezing up. Once the freez-up occurs, I pause execution and checked the running threads for what I believe is the culprit within Microsoft.Azure.WebJobs.Host.CompositeTraceWriter
protected virtual void InvokeTextWriter(TraceEvent traceEvent)
{
if (_innerTextWriter != null)
{
string message = traceEvent.Message;
if (!string.IsNullOrEmpty(message) &&
message.EndsWith("\r\n", StringComparison.OrdinalIgnoreCase))
{
// remove any terminating return+line feed, since we're
// calling WriteLine below
message = message.Substring(0, message.Length - 2);
}
_innerTextWriter.WriteLine(message);
if (traceEvent.Exception != null)
{
_innerTextWriter.WriteLine(traceEvent.Exception.ToDetails());
}
}
}
The line it freezes on is line 66 : _innerTextWriter.WriteLine(message);
_innerTextWriter is an instance of System.IO.TextWriter.SyncTextWriter
Is it possible there is some deadlock issue with this class or the way it is being used?
Some notes:
I am running in the debugger, so in this case I believe the textwriter is forwarding to the console internally
I have my batchsize set to 1 via config.Queues.BatchSize = 1;, not sure if that could matter
I'm currently working on setting up an environment on another computer so that I can see if it is reproducible somewhere other than this machine (surface book).
Update
The issue was me not understanding how the new windows 10 command prompt works. Any time you click on the command window, it goes into "select" mode which completely pauses execution of the process.
Basically: https://superuser.com/questions/419717/windows-command-prompt-freezing-randomly?newreg=ece53f5584254346be68f85d1fd2f18d
You can tell it is in this state because it will prefix the window title with the word "Select":
You have to press enter or click again to get it going once again.
So, two final comments:
1) What an incredibly confusing and un-intuitive behavior for a command window!
2) I hope some admin will come take pity on the shame I have brought upon myself and my family by deleting this question.
To get rid of this strange behavior, you can disable QuickEdit mode:
Strange. When it is in this stuck state, can you try adding a new queue message to the queue and see if that triggers? Are you sure your function isn't hanging internally? What version of the SDK are you using? You might also try upgrading to v1.1.0 which we just released last week. If there are really a bunch of messages in the queue waiting to be processed, I can't think of anything that would cause this. The queue listener in the SDK should chug along, reading batches of messages in parallel and dispatching them to your function. Have you changed any of the JobHostConfiguration.Queues configuration knobs? You haven't force updated the version of the Azure SDK have you to something higher than the WebJobs SDK supports?
Another option if you can't figure this out might be to clone the SDK, build it and debug it locally. The repo is here. The main queue processing loop is here.