We are sending all trace logs to our NLog logger. (with a trace listener)
Nlog is configured to work with DryIoC.
Locally this works perfect, however, in Azure (web app), the first trace message is logged, before we could create our DryIoc container. Even a PreAppStartMethodAttribute does not help, as the trace log has occurred even before the PreAppStartMethodAttribute.
Is there a way to do some initialization tasks before azure logs it's first trace message?
I found a work-around by using an async wrapper that prevents NLog from flushing the data until the container/configuration has been initiliazed.
Related
I have an Azure Function App with multiples functions connected with Application Insights.
For some reason that I don't know sometimes, some requests and traces get lost and it's like they never happen, but I can see the data in our DB and also in others systems.
Here is a new function with just one call, in the azure function dashboard I can see the log:
But in Application Insights, when I try to search for the logs of the trace or the request, there is not info retrived.
This's not happening everytime, but there's not the first time I saw this issue. I can see the logs for others requests but I don't know why sometimes logs are lost.
Azure function info:
Runtime Version: 3
Stack: NodeJS
Have you configured sampling? This can appear as data loss.
You can control it as follows, as per the documentation:
const appInsights = require("applicationinsights");
appInsights.setup("<instrumentation_key>");
appInsights.defaultClient.config.samplingPercentage = 33; // 33% of all telemetry will be sent to Application Insights
appInsights.start();
Sometimes my azure function fails and I have no record of what happened. Function just stops executing.
I think there is major error like StackOverflow, but since there is no record of it I can't be sure.
I created a simple azure function to emulate simple stack overflow:
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req,
ILogger log, ExecutionContext executionContext)
{
RunStackOverflow();
}
private static void RunStackOverflow()
{
RunStackOverflow();
}
When I call this using http trigger, I get 502 error in browser, but there is nothing in logs about this failure. Screenshot: https://www.screencast.com/t/ymWoBey4KX
StackOverflow is just one of the exceptions that can't be caught and can result in function crash. Locally when I run the function in emulator I see stack overflow error in cmd window where function starts. Screenshot: https://www.screencast.com/t/f85U2KmdEBBt
In Azure portal I checked:
function invocations (screenshot: https://www.screencast.com/t/ufB1Zfthz)
function logs (screenshot: https://www.screencast.com/t/A2ix6yuSuJkE)
app insights (screenshot: https://www.screencast.com/t/NyRFLDK23p)
But there is no log entry of this crash anywhere.
I contacted Azure support, but they are not very helpful so far.
Update on Apr 12
Using KUDU I can create memory dump using command like this
c:\devtools\sysinternals\procdump -e -ma -w 12268
This shows me all stack traces for all threads and this is what I need, but only when first chance exception occurs.
The command to trigger memory dump when there is such exception is:
c:\devtools\sysinternals\procdump -accepteula -e -g -ma 8844
but when I run it and then trigger StackOverflow exception here is what is written out to command line:
[11:37:36] Exception: E0434352.CLR
[11:37:36] Exception: C00000FD.STACK_OVERFLOW <--- Stack overflow
[11:37:37] The process has exited.
[11:37:37] Dump count not reached.
Unfortunately there is no memory dump created, so I can't see a stack trace that caused stack overflow.
I also tried:
c:\devtools\sysinternals\procdump -accepteula -e -g -ma -t 13244
-t option triggers memory dump when process exits.
This one actually records a memory dump when Function crashes. Unfortunately this dump doesn't include stack trace for StackOverflow. It seems to get dumped after the thread already crashed.
Update on Apr 21
There are multiple ways to host Azure functions described here:
https://learn.microsoft.com/en-us/azure/azure-functions/functions-scale
The most common and default way is Consumption plan. After a bunch of trial-and-errors I found that Diagnostics Tools (https://www.screencast.com/t/DyT6Jpuqm2uo) which can be used to detect and analyze crashes are not available with Consumption plan. On the other hand they are available with App Service (Basic and plus) and other plans. Azure support told me that currently there are no plans to add it to Consumption plan.
So for now I made a new Azure Function hosted using App Service Plan and I was able to use Diagnostic tools to record crash dumps. After fixing the issues I plan to go back to Consumption plan, so it is a bit of a hack, but it does work for now.
Currently, this level of logs are not supported very well.
You can use Diagnose and solve problems option from azure portal by following this link, but note that some features(like Application Crashes) in this option are still in development.
Steps:
1.In azure portal -> your function app -> click Diagnose and solve problems -> then click the Function App Down or Reporting Errors link. Here is the screenshot:
2.Wait for a while before the report completes generating -> then check the items starts with red exclamation mark(by using your code, the error details are in the Web App Restarted item. But it just shows a common message like app crashes, not stackoverflow):
Thanks in advance.
I want to use Nlog in Azure function V1,I have added Nlog and config file.I can see the ilogger object getting created, but that is not writing to console:(
Only tracewriter provided by Azure function alone working in console log.
Have never tried deploying an Azure Function, but many seems to have issues with deploying NLog.config (or other resource/config-files).
Maybe try putting the NLog-configuration into appsetting.json (Requires explicit load of NLogLoggingConfiguration) :
https://github.com/NLog/NLog.Extensions.Logging/wiki/Json-NLog-Config
The NLog Trace Target (with rawWrite=true) should support Azure Streaming Log.
See also: https://github.com/NLog/NLog.Extensions.Logging/wiki/NLog-cloud-logging-with-Azure-function-or-AWS-lambda
I am desperately trying to debug an error 500 only when I try to update an object from my xamarin.Forms offline DB to Azure. I am using Azure Mobile Client.
I set all the logging to ON in azure, then I downloaded the log. I can see the generic error, but nothing useful.
<failedRequest url="https://MASKED:80/tables/Appel/9A3342A2-0598-4126-B0F6-2999B524B4AE"
siteId="Masked"
appPoolId="Masked"
processId="6096"
verb="PATCH"
remoteUserName=""
userName=""
tokenUserName="IIS APPPOOL\Masked"
authenticationType="anonymous"
activityId="{80000063-0000-EA00-B63F-84710C7967BB}"
failureReason="STATUS_CODE"
statusCode="500"
triggerStatusCode="500"
timeTaken="625"
xmlns:freb="http://schemas.microsoft.com/win/2006/06/iis/freb"
>
The table that failed is the only one I extend with some virtual runtime calculated field of navigation field. But I add the [JsonIgnore] to stop AzureService to create field in the local DB (that work) or send it on the wire to the server. But I always got the 500 error, not exception when debugging the c# Azure backend too.
How I can find the stack trace or the "deep" reason for this 500 error in my backend?
For C# Mobile App backend, you could add the following code in the ConfigureMobileApp method of your Startup.MobileApp.cs file for including error details and return to your client side.
config.IncludeErrorDetailPolicy = IncludeErrorDetailPolicy.Always;
You could just capture the exception in your mobile application or leverage fiddler to capture the network traces when invoking the PATCH operation to retrieve the detailed error message.
Moreover, you are viewing the Failed Request Traces log, you need to check the Application logs. Details you could follow Enable diagnostics logging for web apps in Azure App Service.
Using .ConfigureAndWatch we have configured log4net to reconfigure its logging level without restarting application processes.
A process start we trace the currently configured log level to our trace file.
How can we detect and trace to our trace files the log level changes happening through .ConfigureAndWatch?
I just (re)discovered that ILoggerRepository has a ConfigurationChanged event.
Tracing log level changes from there works fine.
private static void repository_ConfigurationChanged(object sender, System.EventArgs e)
{
var currentTracelLevel = ((log4net.Repository.Hierarchy.Logger)_trace.Logger).EffectiveLevel;
_trace.InfoFormat("----------------------------- log4net level={0}", currentTracelLevel);
}
You can activate the log4net internal debugging, then you will receive messages from the ConfigureAndWatchHandler class when a change occurs, either on the console or in the trace output (see link for more information)
re your comment: In order to pull back the event into your own code and not rely on the console or trace output I'd recommend creating a custom trace appender that loops back into log4net. This way you can filter out messages you want to bring back into your own logs.