It doesn't happen regularly, but I was able to catch it in my debugger:
And then, because of that trailing comma, things further complicate. That error is not even well formatted JSON, which makes it throw DeserializationError, which is how it reaches my code. We can ignore this one, I just needed to bitch about it a little bit.
How do we find why is the "No server available to handle the request" happening? How to mitigate this?
If this is the expected behavior of an overloaded ES cluster, how do we properly handle this error specifically?
Here are the health metrics:
Here's context around the error:
Related
I want to treat 4xx HTTP responses from a function app (e.g. a 400 response after sending a HTTP request to my function app) as failures in application insights. The function app is being called by another service I control so a 4xx response probably means an implementation error and so I'd like to capture that to ultimately run an alert on it (so I can get an email instead of checking into Azure everytime).
If possible, I'd like it to appear here:
If not, are there any alternative approaches that might fit my use case?
Unless an unhandled exception occurs the function runtime will mark the invocation as succesful, whether the status code is actually denoting an error or not. Since this behavior is defined by the runtime there are 2 things you can do: throw an exception in the code of the function and/or remove exception handling so the invocation is marked as not succesful.
Since you ultimately want to create an alert, you better alert on this specific http status code using a "Custom log search" alert
requests
| where toint(resultCode) >= 400
We call REST based APIs hosted by Azure Functions and fail to implement a consistent error handling supporting App Insights and wonder what can be done about it:
If we don't handle exceptions of the function, then App Insights
reports a 'failure', but the service returns only the the error code to the caller, but no error content:
Hence, the client receives a 500 and thats it.
If we handle the exception and log it (to AppInsights) then App Insights stops reporting a 'failure' hence monitoring on function level is broken. We can query for the exception, but they are out-of-context (i.e. we can see the exception by a custom query only) and we don't know which function is impacted actually.
How to marry up the two needs:
Let the function fail so that AppInsights reports the failure (and monitor can alert)
Return a bit more meaningful error message to the caller than 500.
Example on how it looks in AppInsights:
Exception is visible on the Exceptions tab, but the underlying operation has not failed
UPDATE:
According to Microsoft, App Insight Failures are exclusive to unhandled exceptions. Still, open whether there is a way to at least pass-through an error message.
If don't handle exceptions of the function, then App Insights reports a 'failure', returns the error code, but no error content. Hence, the client receives a 500 and thats it.
App Insights doesn't return anything, so what do you mean with returns the error code?
If we handle the exception, LOG it (to AppInsights), return simple error message in code 500, then App Insights doesn't log this as 'failure' hence monitoring is not possible.
Can you show how you do the logging? Because as soon as you log an exception using App Insights you should see it under failures.
This should work:
try
{
...
}
catch(Exception e)
{
logger.LogError(e, e.Message);
return httpRequest.CreateResponse(HttpStatusCode.InternalServerError, e.Message);
}
I am using Dialogflow ES and once I got the webhook setup, I haven't been having issues. But after a few months, I just started getting a random error. It seems to be inconsistent as in sometimes I get it for a specific web call and other times it works fine. This is from the Raw API response:
"webhookStatus": {
"code": 3,
"message": "Webhook call failed. Error: [ResourceName error] Path '' does not match template 'projects/{project_id=*}/locations/{location_id=*}/agent/environments/{environment_id=*}/users/{user_id=*}/sessions/{session_id=*}/contexts/{context_id=*}'.."
}
The webhook is in GCP Functions in the same project. I have a simple "ping" function in the same agent that calls the webhook. That works properly and pings the function, records some notes in the function log (so I know the function is being called), and returns a response fine, so I know the webhook is connected and working for other intents in the same agent before and after I get the error above.
Other intents in the same agent work (and this one WAS working), but I get this error now. I also tried recreating the intent and I get the same behavior.
The project is linked to a billing account and I have been getting charged for it, so I don't think it is an issue with being on a trial or otherwise. Though the Dialogflow itself is in "trial", but the linked webhook function is billed.
Where can I find what this error means or where to look to resolve it?
After looking at this with fresh eyes, I found out what was happening.
The issue was a mal-formed output context. I was returning the bad output context sometimes (which explained why sometimes it worked and sometimes it didn't). Specifically, I was returning the parameters directly into the output context without the output context 'name' or 'parameters'. Everything looked like it was working and I didn't get any other errors, but apparently, when Dialogflow receives a bad web response, it generates the unhelpful error above.
I have set a node server with Express middleware. I get the ECONNABORTED error randomly on some files when loading an HTML file which triggers about 10 other loads (js, css, etc.). The exact error is:
{ [Error: Request aborted] code: 'ECONNABORTED' }
Generated by this simplified code (after I tried to debug the issue):
res.sendFile(res.locals.physicalUrl,function (err) {
if (err)
console.log(err);
...
}
Many posts talk about this error resulting from not specifying the full path name. That is not the situation here. I do specify the full path and indeed the error is randomly generated. There are times when the page and all its subsequent links load perfectly and there are times when they do not. I tried to flush the cache and did not find any pattern to connect it with this.
This specific error appears to be a a generic term for socket connection getting aborted and is discussed in the context of other applications like FTP.
Having realized that the node worker threads can be increased, I tried to do so using:
process.env.UV_THREADPOOL_SIZE = 20;
However, my understanding is that even absent this, at most the file transfer may have to wait for a worker thread to be free and not get aborted. I am not talking about big files here, all files are less than 1 MB.
I have a gut feeling that this has nothing to do with node directly.
Please point to any other possibilities (node or otherwise) to handle this error. Also, any other indirect solutions? Retrying a few times could be one but that would be clumsy. EDIT: No, I cannot retry. Headers are already sent with the error!
A SIDE NOTE:
Many examples on the use of sendFile skip using the callback thereby giving the impression that it is a synchronous call. It is not. Do use the callback at all times, check for success and only then move on to the "next" middleware or take appropriate steps if the send fails for whatever reason. Not doing so can make it difficult to debug the consequences in an asynchronous environment.
See https://stackoverflow.com/a/36949631/2798152
Could it be possible that in some cases you terminate the connection by calling res.end before the asynchronous call to res.sendFile ends?
If that's not the case - can you pastebin more of your application code?
Uninstalling and Re-installing MongoDB solved this for me.
I was facing the same problem. It started happening when I had to force restart my laptop because it became unresponsive. On restarting, trying to connect to mongo server using nodejs, always threw ECONNABORTED error
I am just curious about this. I am making a change in this project, that is using NetSuite web service, and sometimes it throws a SoapException at random, "Only one request may be made against a session at a time".
private NetSuiteService _service;
SessionResponse response = _service.login(passport); //if SoapException, retries??
Status status = response.status;
Reference.cs:
public partial class NetSuiteService :
System.Web.Services.Protocols.SoapHttpClientProtocol
My question is: If I am in debug mode, I trace this, and I hit F5, and it seems to automatically retry after exception is thrown (the code keeps running, with no try catch block implemented, no while loop) until successful (status.isSuccess == true). When I run it in release mode, as a windows service, the log shows it stops running after exception is thrown.
How is this possible? Is it better to catch this exception in a try catch block and retry?
NS Server refuses a request if its already processing one from the same user.
If you want to make sure that your request succeeds than you have to catch this exception and retry.
This was not the experience I had. We thought this related to netsuite sessions but turned out to be nothing to do with that at all and in fact was not even hitting netsuite (according to netsuite log). Turned out we were trying to execute too many commands in a single request and it totally refused to send it to netsuite. Never seen this error before, may be it is a new thing with the new version!