I have an Application API wrapped in API Manager on Azure cloud service. For whatever reason when I send in a JSON payload of 1000 records or more (which translates to around 200k chars) the request is dropped. No trace, no logging just dropped but if I truncate the payload everything works as expected. If I send the same 1000 record payload into the underlying service (not through API Manager) all works as expected. Is there a request or return size limit when using APIM?
My underlying service was matching inner response codes and applying to overall return code if they were the same. My inner codes were all 404 "item not found". Azure API manager treats a 404 as an error and drops large payloads. This is per their support. On a small return payload it will return a code of 404 and the message but on a large payload it is dropped. Reason I was trying to return 404 is because each record in the return payload contains a status. If the status is mixed return 207 for mixed status but if they are all the same - 200 for found and 404 for not found the overall service was returning the internal status as the overall status. This was bad design. My payload consisted of a search for items that did not exist in the DB and therefore was returning 404 as overall status with messages for each record indicating "Not found". APIM was dropping the return response in favor of a generic 404 url not found response. Switched the internal service to return only 200 or 207 depending on outcome and all is right.
{
"ProcessId": "2",
"Code": 404,
"Message": "Could not resolve token.",
"Token": "#!!!#xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
},
{
"ProcessId": "2",
"Code": 404,
"Message": "Could not resolve token.",
"Token": "#!!!#xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
},
Overall return code was set to 404 and APIM dropped the response. I fixed this by setting overall return code to 200 in this case.
Related
I have an express js project that has a try catch, when i respond with status code 500 i get no error detail returned.
Below you can see that when i return status code 200 i get a full response with the error and in this case 'Invalid time value'.
But when i change to status code 500 i get hardly any information, how do i send status code 500 but still get full error information?
this output is normal based on Hypertext Transfer Protocol, you can check the details about status codes here.
when status code is 200
The request has succeeded. The information returned with the response
is dependent on the method used in the request, for example:
GET an entity corresponding to the requested resource is sent in
the response;
HEAD the entity-header fields corresponding to the requested
resource are sent in the response without any message-body;
POST an entity describing or containing the result of the action;
TRACE an entity containing the request message as received by the
end server.
when status code is 500
500 Internal Server Error
The server encountered an unexpected condition which prevented it
from fulfilling the request
this outputs is Hypertext Transfer Protocol and its normal
As the topic implies, my logic app having a Get File Content from a sharepoint connector,which was working fine for all the files less than 20 MB. For the files greater than 20 MB getting timeout after 4 retries by giving the 500 Internal Server Error
I couldn't able to find this type of size limitations in the documentations.
I tried to use the chunks options but its only for upload not for retrieve
Some Other findings :
A file with 17 MB got succeeded at the second retry, however for files more than 20 MB it always getting failed even after 4 retries.
RAW Output:
{
"error": {
"code": 500,
"source": "logic-apis-northeurope.azure-apim.net",
"clientRequestId": "3a0bf64d-2b82-4aef-92ba-ff8b101e44bb",
"message": "BadGateway",
"innerError": {
"status": 500,
"message": "Request timed out. Try again later.\r\nclientRequestId: 3a0bf64d-2b82-4aef-92ba-ff8b101e44bb\r\nserviceRequestId: e0ce569f-96aa-d08b-1c7e-20a6ccf358c3",
"source": "https://xxxxx",
"errors": []
}
}
}
P.S I'm using on-prem sharepoint, i.e gateway is already using. However no timeout logs in the gateway,which makes me to eliminate the issue is not from gateway and from logic app
We can see logic app supports get data from sharepoint according to the on-premises data gateway in this document.
Then click limits on their payload size, we can see the limits of it.
Although the document above doesn't mention the limit of 20 MB for the data response, but I think when you request the data of 17 MB, it beyond exceed the limit. So it got succeeded at the second retry but not success at the first time.
Description:
The user inquires about the availability of the event on a given day.
Question:
Which response code should I send back if new event can not be created on this day? 404? 400? 422?
My choice:
I chose the "404 Not Found" status code:
{
"statusCode": 404,
"error": "Not Found",
"message": "The event can not be created on the selected date."
}
As a response to a POST request I would use the 422 Unprocessable Entity status code along with a description of why the request could not be processed.
If you only want to test if there is already an event at this day, perform a GET request (for example: /event/2019-06-01) to check if it exists.
If it does not exist yet, respond with a 404 not found.
Your question is entirely opinion based. You should try learning about various status codes and their reasons. In your case, you may find status code 410 useful, but again, this is entirely opinion based.
I am currently writing a script in nodejs to pull all DriveItems from the MS Graph API, the idea is to grab the permissions of each DriveItem so that a record can be kept and presented to the user. Our authentication with Graph is app based.
I am running asynchronous requests to MS graph which obviously results in a lot of simultaneous requests. After a few seconds of executing the script, MS graph starts returning the following errors from the endpoint and doesn't seem to recover. The status codes are a 500 internal server error, 504 gateway timeout, and 404 not found.
The number of errors returned isn't always the same; one request returns 856 failed requests, another returns 900.
Here are the different responses received from endpoints which represent true drives and driveitems:
Endpoint: https://graph.microsoft.com/v1.0/drives/(drive_id)/items/(driveitem_id)/permissions
500 Internal server error:
{
"error": {
"code": "-2146232060, Microsoft.SharePoint.SPException",
"message": "Exception from HRESULT: 0x80131904",
"innerError": {
"request-id": "794863b8-bbf1-4866-9604-f8fcec4afabe",
"date": "2016-11-18T05:41:35"
}
}
}
504 Gateway timeout:
{
"error": {
"code": "UnknownError",
"message": "",
"innerError": {
"request-id": "e91c3037-10b6-4523-88d3-7fc31107b719",
"date": "2016-11-18T05:46:21"
}
}
}
404 Not found error:
{
"error": {
"code": "itemNotFound",
"message": "The resource could not be found.",
"innerError": {
"request-id": "b9a8286d-ce8c-4353-b8f0-818fe0f609ea",
"date": "2016-11-18T05:41:35"
}
}
}
This errors occur for a period of time and then everything seems to work fine again.
I can confirm that these endpoints work fine when I use postman to get an individual endpoint and the same token as the script - it returns valid data. Also if I make the script synchronous, it works fine but this is not an acceptable solution as it is far too slow to process the thousands of files I am dealing with.
The only thing I can think of that would be a problem is the sheer amount of simultaneous requests happening? However I am not receiving a 429 too many requests error back from msgraph before this starts to happen.
Any help is greatly appreciated.
I want to do something when/if an insert operation on Azure Table Storage fails. Assume that I want to return false from the below code when I receive an error. _table is of type CloudTable and the code below works.
public bool InsertEntity(TableEntity entity)
{
var insertOperation = TableOperation.Insert(entity);
var result = _table.Execute(insertOperation);
return (result.HttpStatusCode == (int)System.Net.HttpStatusCode.OK);
}
I get the result 203 when the operation succeeds. But there are other possible results like "200 OK".
How can I write a piece of code that will allow me to understand from the status code that something went wrong?
Using the .NET SDK, any situation that needs to be handled will throw an exception. i.e. Any status code that is not 2xx will cause an exception.
To handle situations where something went wrong, I don't have to manually check the status code of the result for every request. All I have to do is to write exception handling code. Like below:
try
{
var result = _table.Execute(insertOperation);
}
catch (Exception)
{
Log("Something went wrong in table operation.");
}
From this page:
REST API operations for Azure storage services return standard HTTP
status codes, as defined in the HTTP/1.1 Status Code Definitions.
So every successful operation against table service will return 2XX status code. To find out about the exact code returned, I would recommend checking out each operation on the REST API Documentation page. For example, Create Table operation returns 201 status code if the operation is successful.
Similarly, for errors in table service you will get error code in 400 range (that would mean you provided incorrect data e.g. 409 (Conflict) error if you're trying to create a table which already exists) or in 500 range (for example, table service is unavailable). You can find the list of all Table Service Error Codes here: https://msdn.microsoft.com/en-us/library/azure/dd179438.aspx.
Basically, any return in 2xx is "OK". In this example:
https://msdn.microsoft.com/en-us/library/system.net.httpstatuscode%28v=vs.110%29.aspx
203 Non-Authoritative Information:
Indicates that the returned metainformation is from a cached copy
instead of the
origin server and therefore may be incorrect.
This Azure white paper elaborates further:
http://go.microsoft.com/fwlink/?LinkId=153401
9.6.5 Error handling and reporting
The REST API is designed to look like a standard HTTP server interacting with existing HTTP clients
(e.g., browsers, HTTP client libraries, proxies, caches, and so on).
To ensure the HTTP clients handle errors properly, we map each Windows
Azure Table error to an HTTP status code.
HTTP status codes are less expressive than Windows Azure Table error
codes and contain less information about the error. Although the HTTP
status codes contain less information about the error, clients that
understand HTTP will usually handle the error correctly.
Therefore, when handling errors or reporting Windows Azure Table
errors to end users, use the Windows Azure Table error code along with
the HTTP status code as it contains more information about the error.
Additionally, when debugging your application, you should also consult
the human readable element of the XML error
response.
These links are also useful:
Microsoft Azure: Status and Error Codes
Clean way to catch errors from Azure Table (other than string match?)
If you are using Azure Storage SDK accessing Azure Table Storage, the SDK would throw a StorageException on the client side for unexpected Http Status Codes returned from the table storage service. To extract the actual HttpStatusCode you would need to wrap your code in a try {} catch(StorageException ex){} block. And then parse the actual exception object to extract the HttpStatusCode embedded in it.
Have a look at Azure Storage Exception parser I implemented in Nuget:
https://www.nuget.org/packages/AzureStorageExceptionParser/
This extracts HttpStatusCode and many other useful fields from Azure StorageExceptions. You can use the same library accross table, blob, queue clients etc. as they all follow the same StorageException pattern.
Note that there will be some exceptions thrown by the Azure Storage SDK that are not StorageExceptions, those are mostly client side request validation type of exceptions and naturally they do not contain any HttpStatusCode. (Hence you would need to have a catch for specifically StorageExceptions to extract HttpStatusCode s).
As a separate note, Azure Storage SDK has a fairly robust retry mechanism for failed requests. Below is the snippet from SDK source code where they decide if the failed response is retrieable or not.
https://github.com/Azure/azure-storage-net/blob/master/Lib/Common/RetryPolicies/ExponentialRetry.cs
if ((statusCode >= 300 && statusCode < 500 && statusCode != 408)
|| statusCode == 501 // Not Implemented
|| statusCode == 505 // Version Not Supported
|| lastException.Message == SR.BlobTypeMismatch)
{
return false; //aka. do not Retry if w are here otherwise Retry if within max retry count..
}