We call REST based APIs hosted by Azure Functions and fail to implement a consistent error handling supporting App Insights and wonder what can be done about it:
If we don't handle exceptions of the function, then App Insights
reports a 'failure', but the service returns only the the error code to the caller, but no error content:
Hence, the client receives a 500 and thats it.
If we handle the exception and log it (to AppInsights) then App Insights stops reporting a 'failure' hence monitoring on function level is broken. We can query for the exception, but they are out-of-context (i.e. we can see the exception by a custom query only) and we don't know which function is impacted actually.
How to marry up the two needs:
Let the function fail so that AppInsights reports the failure (and monitor can alert)
Return a bit more meaningful error message to the caller than 500.
Example on how it looks in AppInsights:
Exception is visible on the Exceptions tab, but the underlying operation has not failed
UPDATE:
According to Microsoft, App Insight Failures are exclusive to unhandled exceptions. Still, open whether there is a way to at least pass-through an error message.
If don't handle exceptions of the function, then App Insights reports a 'failure', returns the error code, but no error content. Hence, the client receives a 500 and thats it.
App Insights doesn't return anything, so what do you mean with returns the error code?
If we handle the exception, LOG it (to AppInsights), return simple error message in code 500, then App Insights doesn't log this as 'failure' hence monitoring is not possible.
Can you show how you do the logging? Because as soon as you log an exception using App Insights you should see it under failures.
This should work:
try
{
...
}
catch(Exception e)
{
logger.LogError(e, e.Message);
return httpRequest.CreateResponse(HttpStatusCode.InternalServerError, e.Message);
}
Related
I want to treat 4xx HTTP responses from a function app (e.g. a 400 response after sending a HTTP request to my function app) as failures in application insights. The function app is being called by another service I control so a 4xx response probably means an implementation error and so I'd like to capture that to ultimately run an alert on it (so I can get an email instead of checking into Azure everytime).
If possible, I'd like it to appear here:
If not, are there any alternative approaches that might fit my use case?
Unless an unhandled exception occurs the function runtime will mark the invocation as succesful, whether the status code is actually denoting an error or not. Since this behavior is defined by the runtime there are 2 things you can do: throw an exception in the code of the function and/or remove exception handling so the invocation is marked as not succesful.
Since you ultimately want to create an alert, you better alert on this specific http status code using a "Custom log search" alert
requests
| where toint(resultCode) >= 400
I am using the Azure Device SDK for .NET Core in order to connect my devices to Azure IoT Hub. From time to time the server rejects some messages (like twin updates or telemetry messages) from the devices and responds with status code 400. As a result there are exceptions thrown on client side but due to its asynchronous nature they are swallowed somewhere inside the Azure SDK and never thrown at my code.
How can I actually be notified about these errors so I can handle and display them?
I can also see from the Azure Device SDK code that it uses some kind of logging (EventSource) but this is never enabled in the code:
From Logging.Common.cs:
Log.IsEnabled() // always returns false
Can you point me to some way where I can 1) actually enable logging in the Azure Device SDK and 2) find the content that was actually logged?
Update: Details regarding exception that is swallowed somewhere
// Fired here after I send twin reported properties to server:
AmqpTransportHandler.VerifyResponseMessage:
if (status >= 400)
{
throw new InvalidOperationException("Service rejected the message with status: " + status);
}
// Then becomes caught and re-fired here:
AmqpTransportHandler.SendTwinPatchAsync:
throw AmqpClientHelper.ToIotHubClientContract(exception);
// Then it disappears somewhere in the "dance" of the async tasks
You can capture traces: https://github.com/Azure/azure-iot-sdk-csharp/tree/master/tools/CaptureLogs
Our sample demonstrates best practice regarding exception catching, for example: https://github.com/Azure/azure-iot-sdk-csharp/blob/master/iothub/device/samples/DeviceClientMqttSample/Program.cs
i am encountering a problem i'm not familiar with.
So i'm trying to log exceptions from a test Azure Function but when i throw an exception and return a 400 bad request, application insight registers the log as a successful request.
As i understand it it is probably register the function's successful run but what i don't understand is how i then should log the exception.
So what i have done so far is this.
(I Will be referring to Application Insights as AI from here on out)
I started by created an AI-Resource.
Then i took the instrument key and applied it to the app settings of my function.
After that i installed the AI NUGET to my function bu creating a Projet.json file and then pasting something like this which installed the necessary assemblies and such.
{
"frameworks": {
"net46":{
"dependencies": {
"Microsoft.ApplicationInsights": "2.4.0"
}
}
}
}
after this i initialize the TelemetryClient in the function and try to log and exception in a catch:
Initiation:
string key = TelemetryConfiguration.Active.InstrumentationKey = System.Environment.GetEnvironmentVariable("APPINSIGHTS_INSTRUMENTATIONKEY", EnvironmentVariableTarget.Process);
TelemetryClient telemetry = new TelemetryClient() {
InstrumentationKey = key
};
Catch:
catch (Exception e)
{
Dictionary<string,string> properties = new Dictionary<string,string>();
properties.Add("Function Payload", data.ToString());
properties.Add("Function Exception", e.ToString());
telemetry.TrackException(e, properties);
return req.CreateResponse(HttpStatusCode.BadRequest, e);
}
Test running the function i get:
2018-03-07T14:24:36.171 [Info] Function started (Id=0292b455-314d-4c4c-872a-2b8137a72305)
2018-03-07T14:24:37.092 [Info] Function completed (Success, Id=0292b455-314d-4c4c-872a-2b8137a72305, Duration=931ms)
In Application insights i can can only see bad requests for StatusCode: 500
but 400 Bad requests gets logged as Successful requests.
And also the TrackException functionality doesn't log any of the custom properties...
So what am i missing?
MORE DETAILS ABOUT EXCEPTION LOGGING:
#Mikhail is right that we treat this as a success because the function is a success. We didn't want to use status codes to guess whether there was a successful operation or not, so we look for whether the function threw an exception.
Your exception isn't appearing in that screen because it hasn't been property correlated with this function execution. If you go to App Insights Analytics and query for that ExceptionTelemetry, you should see it.
In order to correlate it with a specific function, you'd need to set the OperationId, which is the same as the function's InvocationId. There is a sample that shows how to do this with Events, Metrics, and Dependencies, but doing it for Exceptions is the same (you can ignore the User.Id assignment): https://learn.microsoft.com/en-us/azure/azure-functions/functions-monitoring#custom-telemetry-in-c-functions
Update: From what you've shown of your function above, you may be able to get away with doing something like:
catch (Exception e)
{
log.Error("Function Payload " + data.ToString());
throw;
}
That would return a 500 (rather than a 400), and Functions would log the trace to Application Insights, and then log the exception as well as the Request failure. If you're not using your TelemetryClient anywhere else, you could remove that from your code.
The server (Azure Function) processed the request without errors, you returned a result from it, so from the point of view of Function App runtime the request was processed successfully. You can see that from the log too:
... Function completed (Success, ...
So, it makes sense that the Function App registers the call as success in Application Insights too. At least, that's how they chose to implement it.
the "for this operation" not showing exceptions implies that the exception that you sent does not have the same operationId as the azure function. operation
id is how application insights "links" related telemetry together.
your "exeption logging" screenshot is not an exception, but a request, so the custom properties logged on your exception won't be there.
if you want your azure function to fail, and show as a failed request, and log an exception, why are you catching the exception and logging it yourself? doesn't catching the exception then cause the azure function to succeed? why not just let the exception trickle out and let the function runtime do that part for you? (doesn't it?)
I have a Xamarin app using the Microsoft.Azure.Devices.Client.DeviceClient to listen for messages from the Azure IoT hub. Occasionally I get errors thrown that I cannot catch, and deal with. One such exception is:
Microsoft.Azure.Devices.Client.Exceptions.UnauthorizedException
This exception is just an example of an exception I am getting, and one that I can reliably recreate by changing system time. Other IoTHubExceptions are sporadic and difficult to recreate and hence the need to try and catch the exception.
Can anyone tell me how I can catch these IoT hub based errors? The code is:
try
{
await _deviceClientInbound.OpenAsync();
// execution never gets passed this line and the exception is eventually thrown, having reached this point
Message receivedMessage = await _deviceClientInbound.ReceiveAsync();
if (receivedMessage == null) continue;
await _deviceClientInbound.CompleteAsync(receivedMessage);
}
catch (Exception e)
{
// Exception is never caught in here ...
// How can I catch the IoTHubException based exceptions
}
If you are getting unauthorized exception . Then please check your iot hub connection string used in device client to confirm that the shared access polity used with connection string have all the privileges or not, that is needed at your end.
I believe this has to do with the various refactorings done for the supported platforms and you will likely have to catch all exceptions, process the ones you want, and rethrow ones you don't. Depending on platform you might be able to include:
using Microsoft.Azure.Devices.Client.Exceptions;
The sources are here if you feel like spelunking:
https://github.com/Azure/azure-iot-sdk-csharp/search?utf8=%E2%9C%93&q=using+Microsoft.Azure.Devices.Client.Exceptions
I want to do something when/if an insert operation on Azure Table Storage fails. Assume that I want to return false from the below code when I receive an error. _table is of type CloudTable and the code below works.
public bool InsertEntity(TableEntity entity)
{
var insertOperation = TableOperation.Insert(entity);
var result = _table.Execute(insertOperation);
return (result.HttpStatusCode == (int)System.Net.HttpStatusCode.OK);
}
I get the result 203 when the operation succeeds. But there are other possible results like "200 OK".
How can I write a piece of code that will allow me to understand from the status code that something went wrong?
Using the .NET SDK, any situation that needs to be handled will throw an exception. i.e. Any status code that is not 2xx will cause an exception.
To handle situations where something went wrong, I don't have to manually check the status code of the result for every request. All I have to do is to write exception handling code. Like below:
try
{
var result = _table.Execute(insertOperation);
}
catch (Exception)
{
Log("Something went wrong in table operation.");
}
From this page:
REST API operations for Azure storage services return standard HTTP
status codes, as defined in the HTTP/1.1 Status Code Definitions.
So every successful operation against table service will return 2XX status code. To find out about the exact code returned, I would recommend checking out each operation on the REST API Documentation page. For example, Create Table operation returns 201 status code if the operation is successful.
Similarly, for errors in table service you will get error code in 400 range (that would mean you provided incorrect data e.g. 409 (Conflict) error if you're trying to create a table which already exists) or in 500 range (for example, table service is unavailable). You can find the list of all Table Service Error Codes here: https://msdn.microsoft.com/en-us/library/azure/dd179438.aspx.
Basically, any return in 2xx is "OK". In this example:
https://msdn.microsoft.com/en-us/library/system.net.httpstatuscode%28v=vs.110%29.aspx
203 Non-Authoritative Information:
Indicates that the returned metainformation is from a cached copy
instead of the
origin server and therefore may be incorrect.
This Azure white paper elaborates further:
http://go.microsoft.com/fwlink/?LinkId=153401
9.6.5 Error handling and reporting
The REST API is designed to look like a standard HTTP server interacting with existing HTTP clients
(e.g., browsers, HTTP client libraries, proxies, caches, and so on).
To ensure the HTTP clients handle errors properly, we map each Windows
Azure Table error to an HTTP status code.
HTTP status codes are less expressive than Windows Azure Table error
codes and contain less information about the error. Although the HTTP
status codes contain less information about the error, clients that
understand HTTP will usually handle the error correctly.
Therefore, when handling errors or reporting Windows Azure Table
errors to end users, use the Windows Azure Table error code along with
the HTTP status code as it contains more information about the error.
Additionally, when debugging your application, you should also consult
the human readable element of the XML error
response.
These links are also useful:
Microsoft Azure: Status and Error Codes
Clean way to catch errors from Azure Table (other than string match?)
If you are using Azure Storage SDK accessing Azure Table Storage, the SDK would throw a StorageException on the client side for unexpected Http Status Codes returned from the table storage service. To extract the actual HttpStatusCode you would need to wrap your code in a try {} catch(StorageException ex){} block. And then parse the actual exception object to extract the HttpStatusCode embedded in it.
Have a look at Azure Storage Exception parser I implemented in Nuget:
https://www.nuget.org/packages/AzureStorageExceptionParser/
This extracts HttpStatusCode and many other useful fields from Azure StorageExceptions. You can use the same library accross table, blob, queue clients etc. as they all follow the same StorageException pattern.
Note that there will be some exceptions thrown by the Azure Storage SDK that are not StorageExceptions, those are mostly client side request validation type of exceptions and naturally they do not contain any HttpStatusCode. (Hence you would need to have a catch for specifically StorageExceptions to extract HttpStatusCode s).
As a separate note, Azure Storage SDK has a fairly robust retry mechanism for failed requests. Below is the snippet from SDK source code where they decide if the failed response is retrieable or not.
https://github.com/Azure/azure-storage-net/blob/master/Lib/Common/RetryPolicies/ExponentialRetry.cs
if ((statusCode >= 300 && statusCode < 500 && statusCode != 408)
|| statusCode == 501 // Not Implemented
|| statusCode == 505 // Version Not Supported
|| lastException.Message == SR.BlobTypeMismatch)
{
return false; //aka. do not Retry if w are here otherwise Retry if within max retry count..
}