I am using a HTTP connector to download a file using the get Request. Have set Allow chunking to On but still getting the error
"Http request failed as there is an error: 'Cannot write more bytes to the buffer than the configured maximum buffer size: 104857600.'."
Tested the endpoint with a HEAD request and it returns Range header as bytes which means it supports chunking but if I send the header "Range": "bytes=0-1023" getting below exception
BadRequest. The provided workflow action input is not valid.
How do I read this file from Logic apps and write to datalake ? Is the restriction coming from connector or logic apps? How can this be accomplished ?
Related
We have deployed an Azure Function App with the Python 3.8 stack. For some requests, we have received a BadHttpRequestException
Exception while executing function: Functions.***
Exception binding parameter 'req' Reading the request body timed out
due to data arriving too slowly. See MinRequestBodyDataRate.
But this exception is thrown from ASP .Net core . We are not sure what is causing this exception, network connectivity seems to fine.
Is it something to do with request payload size, if it is big and it resulting in slowness.
In Python Http Trigger Azure function where do set the MinRequestBodyDataRate
Good Day
I configured a Pipeline Copy Data job in Azure Data Factory to extract data from Jira with an API call using the rest API connector in Azure.
When i configure and test the connection it is successful.
Now when i try to preview the data in the Copy container i get the following error.
Does anyone know what this error means and how do i bypass it?
I believe i am not the first one trying to extract data from Jira via Rest API.
Thank you and Regards
Rayno
Error occurred when deserializing source JSON file ".Check if the data
is in valid JSON object format.Unexpected character encountered while
parsing value:<.Path".....
I think the error already indicates the root cause.You data format is invalid JSON format,you could try to simulate rest api invoke to make sure if the situation exists.ADF can't help you handle the illegal deserialization.
In addition,according to the connector doc,ADF supports JIRA connector.Maybe you could try to have a try on that.
Calling azure rest api (/consumption/usagedetails) in webactivity using azure data factory .It throws error The length of execution output is over limit (around 1M currently).Any suggestion
Please reference this blog: Web activity throws overlimit error when calling rest api, Jay Gong has given the answer.
Web activity has times out limitation for 1 minute. Also, based on the above error The length of execution ouput is over limit (around 1M currently)., web activity also has output size limitation for 1 MB.
You could find the limitation rules from Data Factory limits and some of the them could be adjusted if you ask for Contact Support.
According the Data Factory limits and your data size, choose the right Azure Data Factory component. Such as Yuvarajan said: You can create a HTTP link service and HTTP data set and pull the data from REST API.
Hope this helps.
Web activity has limitation of output size of 1 MB. You can create a HTTP link service and
HTTP data set and pull the data from REST API.
If you have SQL Server 2017 (14.x) and later (or SQL Managed Instance), you can use python to do this instead. Recommend to use Postman to generate the python script. As an additional extra, you can use the openjson function to convert turn this into tabular format too.
DECLARE #response NVARCHAR(MAX)
EXECUTE sp_execute_external_script #language = N'Python'
, #script = N'
import requests
import json
url = "https://..................................."
payload = json.dumps({
"api_key": "............................"
})
headers = {
''Subscription-Key'': ''.........................'',
''Content-Type'': ''application/json'',
''Ocp-Apim-Trace'': ''false''
}
response = requests.request("POST", url, headers=headers, data=payload)
response = response.text
'
,#params = N'#response NVARCHAR(MAX) OUTPUT'
,#response=#response OUTPUT
INSERT INTO dbo.mytable ([response])
SELECT #response
I want to do something when/if an insert operation on Azure Table Storage fails. Assume that I want to return false from the below code when I receive an error. _table is of type CloudTable and the code below works.
public bool InsertEntity(TableEntity entity)
{
var insertOperation = TableOperation.Insert(entity);
var result = _table.Execute(insertOperation);
return (result.HttpStatusCode == (int)System.Net.HttpStatusCode.OK);
}
I get the result 203 when the operation succeeds. But there are other possible results like "200 OK".
How can I write a piece of code that will allow me to understand from the status code that something went wrong?
Using the .NET SDK, any situation that needs to be handled will throw an exception. i.e. Any status code that is not 2xx will cause an exception.
To handle situations where something went wrong, I don't have to manually check the status code of the result for every request. All I have to do is to write exception handling code. Like below:
try
{
var result = _table.Execute(insertOperation);
}
catch (Exception)
{
Log("Something went wrong in table operation.");
}
From this page:
REST API operations for Azure storage services return standard HTTP
status codes, as defined in the HTTP/1.1 Status Code Definitions.
So every successful operation against table service will return 2XX status code. To find out about the exact code returned, I would recommend checking out each operation on the REST API Documentation page. For example, Create Table operation returns 201 status code if the operation is successful.
Similarly, for errors in table service you will get error code in 400 range (that would mean you provided incorrect data e.g. 409 (Conflict) error if you're trying to create a table which already exists) or in 500 range (for example, table service is unavailable). You can find the list of all Table Service Error Codes here: https://msdn.microsoft.com/en-us/library/azure/dd179438.aspx.
Basically, any return in 2xx is "OK". In this example:
https://msdn.microsoft.com/en-us/library/system.net.httpstatuscode%28v=vs.110%29.aspx
203 Non-Authoritative Information:
Indicates that the returned metainformation is from a cached copy
instead of the
origin server and therefore may be incorrect.
This Azure white paper elaborates further:
http://go.microsoft.com/fwlink/?LinkId=153401
9.6.5 Error handling and reporting
The REST API is designed to look like a standard HTTP server interacting with existing HTTP clients
(e.g., browsers, HTTP client libraries, proxies, caches, and so on).
To ensure the HTTP clients handle errors properly, we map each Windows
Azure Table error to an HTTP status code.
HTTP status codes are less expressive than Windows Azure Table error
codes and contain less information about the error. Although the HTTP
status codes contain less information about the error, clients that
understand HTTP will usually handle the error correctly.
Therefore, when handling errors or reporting Windows Azure Table
errors to end users, use the Windows Azure Table error code along with
the HTTP status code as it contains more information about the error.
Additionally, when debugging your application, you should also consult
the human readable element of the XML error
response.
These links are also useful:
Microsoft Azure: Status and Error Codes
Clean way to catch errors from Azure Table (other than string match?)
If you are using Azure Storage SDK accessing Azure Table Storage, the SDK would throw a StorageException on the client side for unexpected Http Status Codes returned from the table storage service. To extract the actual HttpStatusCode you would need to wrap your code in a try {} catch(StorageException ex){} block. And then parse the actual exception object to extract the HttpStatusCode embedded in it.
Have a look at Azure Storage Exception parser I implemented in Nuget:
https://www.nuget.org/packages/AzureStorageExceptionParser/
This extracts HttpStatusCode and many other useful fields from Azure StorageExceptions. You can use the same library accross table, blob, queue clients etc. as they all follow the same StorageException pattern.
Note that there will be some exceptions thrown by the Azure Storage SDK that are not StorageExceptions, those are mostly client side request validation type of exceptions and naturally they do not contain any HttpStatusCode. (Hence you would need to have a catch for specifically StorageExceptions to extract HttpStatusCode s).
As a separate note, Azure Storage SDK has a fairly robust retry mechanism for failed requests. Below is the snippet from SDK source code where they decide if the failed response is retrieable or not.
https://github.com/Azure/azure-storage-net/blob/master/Lib/Common/RetryPolicies/ExponentialRetry.cs
if ((statusCode >= 300 && statusCode < 500 && statusCode != 408)
|| statusCode == 501 // Not Implemented
|| statusCode == 505 // Version Not Supported
|| lastException.Message == SR.BlobTypeMismatch)
{
return false; //aka. do not Retry if w are here otherwise Retry if within max retry count..
}
I have a user receiving the following error in response to an ItemQueryRq with the QuickBooks Web Connector and IIS 7.
Version:
1.6
Message:
ReceiveResponseXML failed
Description:
QBWC1042: ReceiveResponseXML failed
Error message: There was an exception running the extensions specified in the config file. --> Maximum request length exceeded. See QWCLog for more details. Remember to turn logging on.
The log shows the prior request to be
QBWebConnector.SOAPWebService.ProcessRequestXML() : Response received from QuickBooks: size (bytes) = 3048763
In IIS 7, the max allowed content length is set to 30000000, so I'm not sure what I need to change to allow this response through. Can someone point me in the right direction?
Chances are, your web server is rejecting the Web Connector's HTTP request because you're trying to POST too much data to it. It's tough to tell for sure though, because it doesn't look like you have the Web Connector in VERBOSE mode, and you didn't really post enough of the log to be able to see the rest of what happened, and you didn't post the ItemQuery request you sent or an idea of how many items you're getting back in the response.
If I had to guess, you're sending a very generic ItemQueryRq to try to fetch ALL items, which has a high likelihood of returning A LOT of data, and thus having IIS reject the HTTP request.
Whenever you're fetching a large amount of data using the Web Connector, you should be using iterators. Iterators allow you to break up the result set into smaller chunks.
qbXML Iterator example
other qbXML examples
If you just need to determine if an item exists in QB you can simply add IncludeRetElement to your ItemQuery
So you should post something like
<ItemQueryRq requestID="55">
<FullName>Prepay Discount</FullName>
<IncludeRetElement>ListID</IncludeRetElement>
</ItemQueryRq>
And in Item query response just check the status code. If it is equal to 500 then it means that you should push your item into QB, if it is equal to 0 then it means that item exists
That workaround will save plenty of bytes in your response