I've got an Azure app up and running, but various requests generate a 500 error. There are no other details that come back from the server to let me know exactly what the problem is. No stack trace, no error message. The only thing I get back from the server are the http headers indicating I've got an error.
I've done a little looking around but can't seem to find a way to retrieve the error details that I'm looking for. I've seen some articles that suggest that I enable logging, but I'm not sure 1) how to do that, 2) where those log files would go and 3) how to access said log files. I've seen posts that say to add a whole bunch of code to my application to enable logging, but all I'm looking for is an error message and a stack trace from a 500 error. Do I really have to add a bunch of code to my app to see that information? If not, how can I get at it?
Thanks!
Chris
The best long-term solution is to enable Azure Diagnostics, which I think is what you're referring to. If you want a quick-and-dirty solution, you can log errors out to a file and then RDP into the role instances to view them. This is very similar to what you would do on a server in your own datacenter.
You can create the logs however you like. I've used log4net and RollingFileAppenders with some success. Setting the logfile path to something like "\logs\mylog.txt" will place the logs in the E: drive of the VM. Note you'll still need code somewhere in your app to capture the error and write it to the log - typically the global error handler in Global.asax is a good place for that.
You'll also have to enable RDP access to your role instances. There are many articles detailing how to do that. Here's one.
This is not a generally recommended approach because the logs may disappears when the role recycles or is recreated. It's also a pain in the butt to log to keep an eye on all those different servers.
One other warning - it's possible that the 500 error is due to some failure in your web.config. If that is the case, all the the application-level error logging in the world isn't going to help you. So be sure that your web.config is valid, and also check the Windows Event Logs while you're RDP'd into the server.
500 internal server error is most generally caused by some problem on the server when it was not able to understand incoming requests or there was some problem in configuration. So, try to run the app locally and see if there is some problem. You can record errors in a database in catches/application_error and also can use tracing. Believe me they are very helpful and worth a few extra lines of code.
For tracing have a look here, http://msdn.microsoft.com/en-us/magazine/ff714589.aspx
Related
Apologies Paul, this is a duplicate to the post I put on OpenNTF, however the site will not allow me to log in the last 2 days to follow up, plus the wider audience of Stack might find me someone with an identical issue.
To keep it short.
I have 1 openLog database in a folder structure, logs/xpageslog.nsf
During development, I could log to this database, for example, using Paul Withers XPages OpenLog Logger, to log uncaught exceptions with the following settings:
private String logDbName = "logs\\xpageslog.nsf"; // in OpenLogItem.java from OpenLogClass library
logDbName = "logs/xpages.nsf" // in OpenLogFunctions.ls
xsp.openlog.filepath=log/xpageslog.nsf // in xsp.properties
However, if I then change all the above, to simply go to xpageslog.nsf, in the root of the server (this is a 2nd openLog database) errors still get logged to the first database.
I've tried building, cleaning, re-compiling, all to no avail. It seem's to be that somewhere, or somehow, the references to the original database are not being overwritten.
Any ideas?
It is good practice to use restart task http instead of tell http restart. Both commands have different effects.
As confirmed in comments, this solved the problem.
Some use tell http quit followed by load http, the effect is the same as with restart task http. At the other hand, simple tell http restart does not fully initialize http task, it's kind of soft reset and I recommend not using it.
I'm able to successfully call my functions and make them do what I want them to do. The problem is that it doesn't look like the logs are being saved anywhere and I don't see how I can view them. Which I'll want to do in the event of an error. As a test I have my working function just doing a log.Info as soon as it's called. When testing locally it prints the message to the console. I believe I've enabled everything correctly but let me explain what I've done in case I didn't.
In my app service, under Monitoring -> Diagnostic Logs, I have enabled everything. Application Logging (filesystem) verbose, Application Logging (Blob) verbose (with the storage location set), detailed error messages and failed request tracing turned on.
In my function, I'm using the TraceWriter object that's passed to my run method (I started from a template).
Please note that functions are set to require authentication. If I click on the "Monitor" tab nothing appears. It just says "Loading..." forever and there's no information. Perhaps this is because of the authentication?
I used the Azure Storage Explorer to browse to my blob. The "log" blob exists, and I do see a set of nested directories that lead up to now. However it just contains a 354 byte file that contains a few lines of some random info. This file never seems to update or get larger.
I used FTP to try and browse to where the logs might be, but there's no directory on there that contains any log files.
I also went to KUDU for my function app ({myfunctionapp}.scm.azurewebsites.net/azurejobs/#/functions). While I do see that my function was called successfully, I don't see anything from the call to log.Info anywhere.
I tried using a different logger, and as a test did: System.Diagnostics.Trace.TraceError("test error");
I also don't see this message appearing anywhere.
Am I missing something as far as set up goes? Is the problem the fact that I require authentication? If it's the latter, is there still a way to view logs? I definitely have to have auth enabled. Thanks. And if it helps, below are links to what my settings and the monitor tab look like.
Settings: https://postimg.org/image/u57m2xbl5/
Monitor: https://postimg.org/image/uou10arch/
Authentication should not cause any problems with logging and Log.Info should work out of the box, no setup required.
I highly recommend that you enable AlwaysOn for your dedicated function app. The long loading of the Monitor tab could be because your site is in a 'cold' state, where it takes longer to start up.
If you go to {myfunctionapp}.scm.azurewebsites.net/DebugConsole and navigate to LogFiles/Application/Functions do you see any expected logs there? Also, when you run a function from the portal do you see logs in the log window?
Same thing happened to me if I had Fiddler open....close Fiddler and all is good.
I had an issue on my test server that was throwing an exception, I noticed that it logged nothing to application insights so first tried to debug on my local machine. I managed to replicate the issue, and low and behold in my DEV application insights I have server exceptions logged.
So I go back to my Test Application insights and again replicate the issue but still no errors logged (Even after an hour or so).
However I can see it logs them as "Failed Requests", just not Server Exceptions. Without the server exceptions I can't see the stacktrace/error message.
I am using the Log4Net app insights extension to log these exceptions. It's obviously not a code issue because the same code is being run in both cases. So I looked at configuration issues.
My log4net.config and applicationinsights.config have no transforms from Dev to Test (Or to prod). They stay the same.
My Web.config only has a transform to remove the debug=true as is standard.
I do set some environment properties in Azure (Using Azure Websites), but the only one that is actually application changing is the AppInsightsInstrumentation key to switch it to a different AppInsights. However I know that this key is correct because I'm logging some things, just not others.
Here's the answer (Kinda)
I was logging the exception in a Basecontroller, this worked fine on my local machine and I could debug through.
What happens though is when I push it remotely, .net's out of the box error handling kicks in. Notably the global "HandleErrorAttribute" (Check your FilterConfig.cs). This seems to capture and swallow the exception. There is lots of other talks about this if you wish to override this behaviour, mostly when talking about other logging tools like Elmah.
The reason I only see this in my test environment is because CustomErrors defaults to remoteOnly. So I only see it not logging when it's not on my machine.
Because I'm handling errors myself, I can remove this line from my FilterConfig.cs
filters.Add(new HandleErrorAttribute());
And server exceptions should be logged now (I also handle showing the user a nice error page in my BaseController for now)
I'm working with a production system that has a moderate amount of load. The amount of trace events and AI sends up is way too detailed, and makes it difficult to wade through logs later.
Each request to the server has information such as:
Message='Selected formatter='JsonMediaTypeFormatter', content-type='application/json; charset=utf-8'', Operation=DefaultContentNegotiator.Negotiate
and
Message='Action returned 'RZ.API.Support.Controllers.OperationActionResult`1[System.Collections.Generic.List`1[RZ.Entity.System.ClientMessage]]'', Operation=ReflectedHttpActionDescriptor.ExecuteAsync
There are maybe 30 entries for each request!
I just need the request type:
12/16/2015, 9:17:29 AM - REQUEST
GET /api/v1/user/messages
And the result code - as well as any custom stuff I do along the way.
So basically I want to trim most the traces except the request and the result (and any errors etc).
I have my eye on this bad boy in the AI config:
<Add Type="Microsoft.ApplicationInsights.Web.RequestTrackingTelemetryModule, Microsoft.AI.Web"/>
... but I cannot for the life of me see any doco on how to ask it to reduce the amount of stuff that is sent!
Any help is much appreciated.
Jordan.
P.S. All the extra logging has put us over the 15m a month plan, we had to upgrade!
RequestTrackingTelemetryModule does not do anything like you described. It adds requests, exceptions and dependencies collection. And in you example you are saying you see verbose WebApi traces being forwarded to ApplicationInsights. I assume you actually use Application Insights logging adapter.
Here you can read how WebApi traces can be forwarded to AI Version 1: http://apmtips.com/blog/2014/11/13/collect-asp-dot-net-mvc-web-api-traces-with-application-insights/
Here you can read how WebApi traces can be forwarded to AI Version 2:
http://apmtips.com/blog/2016/01/05/webapi-tracing-powered-by-ai-in-vs2015-update1/
Source code of logging adapters: https://github.com/Microsoft/ApplicationInsights-dotnet-logging
Documentation: https://azure.microsoft.com/en-us/documentation/articles/app-insights-search-diagnostic-logs/#trace
So you have multiple options:
Do not use logging adapters
Change verbosity of WebApi tracing (read http://www.asp.net/web-api/overview/testing-and-debugging/tracing-in-aspnet-web-api). I would prefer this one since you probably want to collect failures.
Remove WebApi tracing (as you did)
To answer my own question.
In my WebApiConfig file, I had:
config.EnableSystemDiagnosticsTracing();
Removing this line drastically cut down the clutter to what I was trying to achieve.
As of version 2.0 of the Application Insights SDKs, you can also limit the data sent by enabling sampling:
https://azure.microsoft.com/en-us/documentation/articles/app-insights-sampling/
if you add
<MaxTelemetryItemsPerSecond>5</MaxTelemetryItemsPerSecond>
to your ApplicationInsights.config, the sdk can limit how much goes out. The article above has a LOT more settings/configuration you can use to get other specific behavior, but the one above is the simplest.
As far as I know there are no configuration options available for the RequestTrackingTelemetryModule. You could just turn it off (by uninstalling the respective NuGet package or commenting the xml) and / or install different / additional telemetry modules.
See app-insights-configuration-with-applicationinsights-config for a list of modules and configuration options.
My application is deployed on IIS 7. I want to check the number of failures as my logic is getting failed at some point and getting errors.Is there any general weblogs in IIS.I can only see system errors in the event logs. Is there any web logs?
Manually trawling the standard W3C logs is ok if you're chasing down requests for certain content types, but they won't tell you an awful lot about why your web application is failing and responding with many 4XX and 5XX status codes. You'll get a status code, but that's about it.
Failed Request Tracing:
Your "go to" diagnostic tool should be the Failed Request Tracing feature that is built into IIS7+.
FRT is one of my favourite features of IIS7/8 for tracking down problems with production sites, especially when debugging apps built on the WebAPI and Ajaxy type stuff.
For more information see:
http://www.iis.net/learn/troubleshoot/using-failed-request-tracing
For example, last week FRT helped me get to the bottom of an issue with a client's hosted site. A particular part of the site (which uses the WebAPI) was failing with a 405 Method Not Allowed status code when making a HTTP DELETE request and despite the DELETE verb being permitted.
Using FRT I was able to generate trace of the failing request which showed me this:
Expanding the "View Trace" entries revealed this error:
The solution for our customer was to disable (it's not used) the WebDAV native module which doesn't permit non-Windows authenticated requests with certain verbs (such as DELETE) to complete. Even if the WebDAV module isn't handling the request it's still in the request pipeline inspecting and validating request headers.
Failed Request Tracing is a really invaluable diagnostic tool, you should learn how to use it.
The HTTPERR Logs:
You should also check the HTTPERR logs located in:
C:\Windows\System32\LogFiles\HTTPERR
If you get 503 - Service Unavailable errors they're a good place to look for clues as to what went wrong if an application pool fails catastrophically, and often.
The is a folder named 'logs' in your 'inetpub' folder where all the logs live. You can look at the Logging tab under IIS in IIS Manager to see the name of the specific log you should check for your site.