Logic Apps "Get file Content" Sharepoint connector timed out for files over 20MB - azure

As the topic implies, my logic app having a Get File Content from a sharepoint connector,which was working fine for all the files less than 20 MB. For the files greater than 20 MB getting timeout after 4 retries by giving the 500 Internal Server Error
I couldn't able to find this type of size limitations in the documentations.
I tried to use the chunks options but its only for upload not for retrieve
Some Other findings :
A file with 17 MB got succeeded at the second retry, however for files more than 20 MB it always getting failed even after 4 retries.
RAW Output:
{
"error": {
"code": 500,
"source": "logic-apis-northeurope.azure-apim.net",
"clientRequestId": "3a0bf64d-2b82-4aef-92ba-ff8b101e44bb",
"message": "BadGateway",
"innerError": {
"status": 500,
"message": "Request timed out. Try again later.\r\nclientRequestId: 3a0bf64d-2b82-4aef-92ba-ff8b101e44bb\r\nserviceRequestId: e0ce569f-96aa-d08b-1c7e-20a6ccf358c3",
"source": "https://xxxxx",
"errors": []
}
}
}
P.S I'm using on-prem sharepoint, i.e gateway is already using. However no timeout logs in the gateway,which makes me to eliminate the issue is not from gateway and from logic app

We can see logic app supports get data from sharepoint according to the on-premises data gateway in this document.
Then click limits on their payload size, we can see the limits of it.
Although the document above doesn't mention the limit of 20 MB for the data response, but I think when you request the data of 17 MB, it beyond exceed the limit. So it got succeeded at the second retry but not success at the first time.

Related

Strange Internal server error on Synapse pipeline

I received the below error on Synapse pipeline. I am running pipeline with more cluster size with memory optimized clusters. I am just processing 7-8 JSON files of around 90MB of size each.
Error
{ "errorCode": "145", "message": "Internal Server Error in Synapse
batch operation: '[plugins.C4T-PRIV-SAW-CAS.IR-Test.19
WorkspaceType: CCID:]
[Monitoring] Livy Endpoint=[
https://hubservice1.westeurope.azuresynapse.net:8001/api/v1.0/publish/c1e53348-b457-4afd-a61d-76553bdd369c
]. Livy Id=[4] Job failed during run time with state=[dead].'.",
"failureType": "SystemError", "target": "DF_Load_CustomsShipment",
"details": [] }
The internal server errors are mostly noticed when there is an intermittent/transient issue with the dependent service.
If the issue is not auto resolved or you still experience the issue consistently, I would recommend raising a support ticket for the support engineer to investigate.

Bot Framework: httpStatusCode": 504 getting Failed to send activity: bot timed out

Occasionally the bot sends a timeout error. It seems that it happens by requests with larger amout of data. Is it possible to increase the timeout or cash buffer?
Request
https://directline.botframework.com/v3/directline/conversations/xxx/activities
Response
{
"error": {
"code": "BotError",
"message": "Failed to send activity: bot timed out"
},
"httpStatusCode": 504
}
Payload
17. x-ms-bot-agent:
DirectLine/3.0 (directlinejs; WebChat/4.9.0 (Full))
18. x-requested-with:
XMLHttpRequest
4. Request Payloadview source
1. {,…}
1. channelData: {clientActivityID: "", clientTimestamp: "2020-06-05T06:57:43.001Z"}
2. channelId: "webchat"
3. from: {id: "",…}
4. locale: "en-US"
5. text: "nohy"
6. textFormat: "plain"
7. timestamp: "2020-06-05T06:57:43.045Z"
8. type: "message"
Any idea guys?
You cannot increase the timeout limit. This is a limit imposed by the Direct Line service, and is set in place for performance and stability reasons. The timeout happens because your bot/code takes too long to respond back to the service, not because of how much data (unless that large data is contributing to the length of time to reply).
You should investigate your bot and those portions of it that take longer, to see if you can decrease the time taken. If you know that there are areas (say external calls to other services, etc) that will take a long time and is unavoidable; then you should implement proactive messaging to remedy that.
https://learn.microsoft.com/en-us/azure/bot-service/bot-builder-howto-proactive-message?view=azure-bot-service-4.0&tabs=csharp
https://github.com/microsoft/BotBuilder-Samples/tree/master/samples/csharp_dotnetcore/16.proactive-messages

Azure API Manager POST Request/Response Limit 65,535

I have an Application API wrapped in API Manager on Azure cloud service. For whatever reason when I send in a JSON payload of 1000 records or more (which translates to around 200k chars) the request is dropped. No trace, no logging just dropped but if I truncate the payload everything works as expected. If I send the same 1000 record payload into the underlying service (not through API Manager) all works as expected. Is there a request or return size limit when using APIM?
My underlying service was matching inner response codes and applying to overall return code if they were the same. My inner codes were all 404 "item not found". Azure API manager treats a 404 as an error and drops large payloads. This is per their support. On a small return payload it will return a code of 404 and the message but on a large payload it is dropped. Reason I was trying to return 404 is because each record in the return payload contains a status. If the status is mixed return 207 for mixed status but if they are all the same - 200 for found and 404 for not found the overall service was returning the internal status as the overall status. This was bad design. My payload consisted of a search for items that did not exist in the DB and therefore was returning 404 as overall status with messages for each record indicating "Not found". APIM was dropping the return response in favor of a generic 404 url not found response. Switched the internal service to return only 200 or 207 depending on outcome and all is right.
{
"ProcessId": "2",
"Code": 404,
"Message": "Could not resolve token.",
"Token": "#!!!#xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
},
{
"ProcessId": "2",
"Code": 404,
"Message": "Could not resolve token.",
"Token": "#!!!#xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
},
Overall return code was set to 404 and APIM dropped the response. I fixed this by setting overall return code to 200 in this case.

Problems with pulling OneDrive/SharePoint Permissions

I am currently writing a script in nodejs to pull all DriveItems from the MS Graph API, the idea is to grab the permissions of each DriveItem so that a record can be kept and presented to the user. Our authentication with Graph is app based.
I am running asynchronous requests to MS graph which obviously results in a lot of simultaneous requests. After a few seconds of executing the script, MS graph starts returning the following errors from the endpoint and doesn't seem to recover. The status codes are a 500 internal server error, 504 gateway timeout, and 404 not found.
The number of errors returned isn't always the same; one request returns 856 failed requests, another returns 900.
Here are the different responses received from endpoints which represent true drives and driveitems:
Endpoint: https://graph.microsoft.com/v1.0/drives/(drive_id)/items/(driveitem_id)/permissions
500 Internal server error:
{
"error": {
"code": "-2146232060, Microsoft.SharePoint.SPException",
"message": "Exception from HRESULT: 0x80131904",
"innerError": {
"request-id": "794863b8-bbf1-4866-9604-f8fcec4afabe",
"date": "2016-11-18T05:41:35"
}
}
}
504 Gateway timeout:
{
"error": {
"code": "UnknownError",
"message": "",
"innerError": {
"request-id": "e91c3037-10b6-4523-88d3-7fc31107b719",
"date": "2016-11-18T05:46:21"
}
}
}
404 Not found error:
{
"error": {
"code": "itemNotFound",
"message": "The resource could not be found.",
"innerError": {
"request-id": "b9a8286d-ce8c-4353-b8f0-818fe0f609ea",
"date": "2016-11-18T05:41:35"
}
}
}
This errors occur for a period of time and then everything seems to work fine again.
I can confirm that these endpoints work fine when I use postman to get an individual endpoint and the same token as the script - it returns valid data. Also if I make the script synchronous, it works fine but this is not an acceptable solution as it is far too slow to process the thousands of files I am dealing with.
The only thing I can think of that would be a problem is the sheer amount of simultaneous requests happening? However I am not receiving a 429 too many requests error back from msgraph before this starts to happen.
Any help is greatly appreciated.

Azure ML: Getting Error 503: NoMoreResources to any web service API even when I only make 1 request

Getting the following response even when I make one request (concurrency set to 200) to a web service.
{ status: 503, headers: '{"content-length":"174","content-type":"application/json; charset=utf-8","etag":"\"8ce068bf420a485c8096065ea3e4f436\"","server":"Microsoft-HTTPAPI/2.0","x-ms-request-id":"d5c56cdd-644f-48ba-ba2b-6eb444975e4c","date":"Mon, 15 Feb 2016 04:54:01 GMT","connection":"close"}', body: '{"error":{"code":"ServiceUnavailable","message":"Service is temporarily unavailable.","details":[{"code":"NoMoreResources","message":"No resources available for request."}]}}' }
The request-response web service is a recommender retraining web service with the training set containing close to 200k records. The training set is already present in my ML studio dataset, only 10-15 extra records are passed in the request. The same experiment was working flawlessly till 13th Feb 2016. I have already tried increasing the concurrency but still the same issue. I even reduced the size of the training set to 20 records, still didn't work.
I have two web service both doing something similar and both aren't working since 13th Feb 2016.
Finally, I created a really small experiment ( skill.csv --> split row ---> web output ) which doesn't take any input. It just has to return some part of the dataset. Did not work, response code 503.
The logs I got are as follows
{
"version": "2014-10-01",
"diagnostics": [{
.....
{
"type": "GetResourceEndEvent",
"timestamp": 13.1362,
"resourceId": "5e2d653c2b214e4dad2927210af4a436.865467b9e7c5410e9ebe829abd0050cd.v1-default-111",
"status": "Failure",
"error": "The Uri for the target storage location is not specified. Please consider changing the request's location mode."
},
{
"type": "InitializationSummary",
"time": "2016-02-15T04:46:18.3651714Z",
"status": "Failure",
"error": "The Uri for the target storage location is not specified. Please consider changing the request's location mode."
}
]
}
What am I missing? Or am I doing it completely wrong?
Thank you in advance.
PS: Data is stored in mongoDB and then imported as CSV
This was an Azure problem. I quote the Microsoft guy,
We believe we have isolated the issue impacting tour service and we are currently working on a fix. We will be able to deploy this in the next couple of days. The problem is impacting only the ASIA AzureML region at this time, so if this is an option for you, might I suggest using a workspace in either the US or EU region until the fix gets rolled out here.
To view the complete discussion, click here

Resources