I am trying to pull data from web API using HTTP connector (of Azure Data Factory) and load into Azure Data Lake. I am successfully able to create pipeline using Copy Wizard but getting below error while running the pipeline.
Copy activity encountered a user error at Source side: ErrorCode=UserErrorFailedToReadHttpFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to read data from http source file.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The underlying connection was closed: An unexpected error occurred on a send.,Source=System,''Type=System.IO.IOException,Message=Authentication failed because the remote party has closed the transport stream.,Source=System,'.
Thanks
Nutan Patel
According to this article, you could find:
The type property must be set to: Http.
So we couldn't use https as the http linked service, I suggest you could firstly changed your linked service's setting to enable http connected and test again.
Related
I am trying to create a data flow in Azure data factory but when I attempt to call the API connection, it throws me an error. This API connection was working fine when I used in copy activity but it does not work in data flow.
Here is the link to screenshot
Please tell me what wrong?
Previously we made logic app in azure where we used http request to retrieve xml file from our clients system.
It goes like that:
HTTP request --> response body is xml data --> we save that xml data in azure blob storage as xml file.
My question is how and if its possible to do the same thing in azure data factory?
Reason for us to move this process over to data factory is that we also need to execute sql server stored procedures there and in logic app there is that 2 minute timeout and some of our procedures run longer than 2 min.
If you're looking for a way to manually trigger an Azure Data Factory pipeline,
You can manually run your pipeline by using one of the following methods:
.NET SDK
Azure PowerShell module
REST API
Python SDK
The following sample command shows you how to run your pipeline by using the REST API manually:
POST
https://management.azure.com/subscriptions/mySubId/resourceGroups/myResourceGroup/providers/Microsoft.DataFactory/factories/myDataFactory/pipelines/copyPipeline/createRun?api-version=2017-03-01-preview
More information: Manual execution (on-demand) with JSON
There are more questions to be answered, however, like "can we increase the timeout for the Logic App" (yes, see HTTP request limits - Timeout duration), "does the Logic App need to wait for the Stored Procedures to complete" and "is Data Factory the best tool for the job". The best answer to your question depends on the answer to all of these questions.
Based on the information you provided, running the logic in a different way like a Logic App on an Integrated Service Environment or an Azure Function feels like the best option.
I am new to ADF. I am creating a small demo where I am trying to hit a sample API "https://dummy.restapiexample.com/api/v1/employees" and inserting data into the Employee table in the SQL database.
However, I would also like to add data flow to perform few select and other transformations on other related tables.
Below are the things that I have done so far in azure data factory:
Created a linked service called "EmployeeRestService" to communicate with the above api.
Created a linked service called "AzureSqlDatabase" to communicate with azure sql database
Created a dataset "EmployeeApiDataset" of type "Rest" and linked it with "EmployeeRestService".
Created a dataset of type Azure sql database and Selected "EmployeeTable".
Added one data flow (Now here I am getting the problem)
And here is the error snapshot:
In the data flow --> Source Settings, I selected "EmployeeApiDataset". But when I am trying to preview the data by enabling the debug, I am getting below error (refer above snapshot):
at Source 'EmployeeApi': Rest Sink - Error response received from the server (url:http://dummy.restapiexample.com/api/v1/employees,request body: None, request method: GET, status code: 429), response body: Some(
Is this a correct way of doing adding Rest data source in data flow?
Note: I do not want to create an Azure data lake to store API results.
Can anybody suggest to me why I am getting the above error and what is the correct way of doing this?
As #esqew mentioned in the comments, there is nothing wrong with your dataflow Source settings. As you are using online dummy Rest API as your source dataset, there are high chances that too many requests are made to call the Rest API by users.
I also got the same error when tried.
Also got the same 429 error when tested the URL.
I just hit the refresh and got the data in the source data preview.
I have a URl like this : wss://**** and i have a token for this URL. I need to copy files from this location using data factory.
I could not find any activity which could be used for this and how i create linked service for the connection. I tried using Web activity but it allows only HTTP or HTTPS.
Any leads would be helpful. Thanks.
As far as I know, Azure Data Factory doesn't support web socket secure link by now. This is all connectors that Azure Data Factory supports. And there is no connector for web socket secure link.
I have created an experiment and successfully published a web service which requires inputs.
When I schedule this web service as a HTTPS POST JOB it shows this error
Http Action - Response from host
'ussouthcentral.services.azureml.net': 'BadRequest' Response Headers:
x-ms-request-id: 51fb1d34-5bc7-4832-ad9f-b19826468ea0 Date: Mon, 11
May 2015 11:02:01 GMT Server: Microsoft-HTTPAPI/2.0 Body:
{"error":{"code":"BadArgument","message":"Invalid argument
provided.","details":[{"code":"MissingInputBlobInformation","target":"Inputs","message":"Missing
Azure storage blob information. Provide a valid connection string and
relative path or URI and try again."}]}}
My data is not located in Azure Blob Storage. I am am trying to pass this web input as part as a HTTPS POST BODY.
If you are using BES with web service input and output, you would need to provide the Storage information for the data.
With the Reader and Writer modules, you can remove the web service input and output ports.
Then when the web service is called, it executes without using the Storage blob. It will read from the Reader and write to the destination specified in the Writer.
I have uploaded a Visual Studio template to CodePlex that you can install. The NoInputOutput.aspx of that project does the above. And it should show you the workflow.
I was wondering if you were trying to call Batch execution service. You may want to go over this tutorial - http://azure.microsoft.com/en-us/documentation/articles/machine-learning-consume-web-services/#batch-execution-service-bes
I think for Batch execution, you would need to either upload your data into Azure Blob for batch scoring or publish experiment as a web service without input port.