I am calling Azure Synapse Pipeline Create Pipeline REST API with a Databricks Notebook JSON activity.
Error message in the Synapse Pipeline screen: Could not load resource 'PLDB2'. Please ensure no mistakes in the JSON and that referenced resources exist. Status: Cannot read properties of undefined (reading 'toLowerCase'), Possible reason: Cannot read properties of undefined (reading 'toLowerCase')
PostMan: https://mysynapseworkspace.dev.azuresynapse.net/pipelines/PLDB2?api-version=2020-12-01
Links:
https://learn.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline/create-or-update-pipeline?tabs=HTTP
https://learn.microsoft.com/en-us/azure/data-factory/transform-data-databricks-notebook?context=%2Fazure%2Fsynapse-analytics%2Fcontext%2Fcontext&tabs=synapse-analytics
Create Pipeline JSON in Postman Body
{
"name": "PLDB2",
"properties": {
"description": "Databricks Notebook",
"activities": [
{
"activity": {
"name": "MyActivity",
"description": "MyActivity description",
"type": "DatabricksNotebook",
"linkedServiceName": {
"referenceName": "LS_Databricks",
"type": "LinkedServiceReference"
},
"typeProperties": {
"notebookPath": "/Users/abc#myemail.com/Test_Job_1"
}
}
}
]
}
}
Synape Pipeline item is created with above error message:
{
"id": "/subscriptions/mysubscription/resourceGroups/myresourcegroup/providers/Microsoft.Synapse/workspaces/mysynapseworkspace/pipelines/PLDB2",
"name": "PLDB2",
"type": "Microsoft.Synapse/workspaces/pipelines",
"etag": "9400d55f-0000-0100-0000-63b180e40000",
"properties": {
"description": "Databricks Notebook",
"activities": [
{
"activity": {
"name": "MyActivity",
"description": "MyActivity description",
"type": "DatabricksNotebook",
"linkedServiceName": {
"referenceName": "LS_Databricks_Personify",
"type": "LinkedServiceReference"
},
"typeProperties": {
"notebookPath": "/Users/nlaw#quaerainsights.com/Test_Job_1"
}
}
}
],
"lastPublishTime": "2023-01-01T12:47:32Z"
}
}
In Azure Synapse, I can manually create and run a pipeline calling the Databricks "Test_Job_1" notebook described in the JSON.
What is the problem? Thx for help.
I found the problem. In the JSON, delete these 2 lines: "activity": { and its closing bracket "}"
{
"name": "Correct JSON",
"properties": {
"activities": [
{
"name": "MyActivity",
"description": "MyActivity description",
"type": "DatabricksNotebook",
"linkedServiceName": {
"referenceName": "LS_Databricks",
"type": "LinkedServiceReference"
},
"typeProperties": {
"notebookPath": "/Users/abc#myemail.com/Test_Job_1"
}
}
]
}
}
Related
To create an API Connection to Azure Service Bus using Managed Identity I'm using the following template:
"resources": [
{
"type": "MICROSOFT.WEB/CONNECTIONS",
"apiVersion": "2018-07-01-preview",
"name": "[parameters('connections_servicebus_name')]",
"location": "[parameters('connections_servicebus_location')]",
"kind": "V1",
"properties": {
"alternativeParameterValues": {},
"displayName": "[parameters('connections_servicebus_displayname')]",
"api": {
"name": "[parameters('connections_servicebus_name')]",
"displayName": "[parameters('connections_servicebus_displayname')]",
"id": "[concat(subscription().id, '/providers/Microsoft.Web/locations/', parameters('connections_servicebus_location'), '/managedApis/', 'servicebus')]",
"type": "Microsoft.Web/locations/managedApis"
},
"customParameterValues": {},
"parameterValueSet": {
"name": "managedIdentityAuth",
"values": {}
}
}
}
]
that is actually working except for the fact that the 'NamespaceEndpoint' information (like: sb://mySBNS.servicebus.windows.net) is not provided anywhere and so the field appear empty on Azure portal:
After adding it manually, the connection and the LogicApp that is using it, start to work.
What is the json field to provide that information??
You can refer to this question for full details:
"parameterValueSet": {
"name": "managedIdentityAuth",
"values": {
"namespaceEndpoint": {
"value": "sb://<servicebus-namespace-name>.servicebus.windows.net/"
}
}
}
I was successfully able to connect to my local machine using File System Linked Service. But now, when I am trying to access a folder from a network drive using a similar approach, I am unable to proceed. Please can anyone tell me where am I going wrong ?
The below is the working version when connected from my local machine.
{
"name": "Test_FileSystem",
"type": "Microsoft.DataFactory/factories/linkedservices",
"properties": {
"annotations": [],
"type": "FileServer",
"typeProperties": {
"host": "D:\\AZURE_FS",
"userId": "<xxx#org.com>",
"encryptedCredential": "eyJDcmVkZW50aWFsSWQiOiI0ZDYwMWI1Yi02YmI3LTRlN2YtOTBmYi0xNmIzZjI1MzQ3ZjciLC=="
},
"connectVia": {
"referenceName": "IR-FS-DEV",
"type": "IntegrationRuntimeReference"
}
}
}
However, when I use a similar approach and try connecting to a network drive,my connection is failing.
{
"name": "NetworkDrive_LS",
"type": "Microsoft.DataFactory/factories/linkedservices",
"properties": {
"annotations": [],
"type": "FileServer",
"typeProperties": {
"host": "\\\\<host>:<port>\\SD\\AZURE_FS",
"userId": "<userid>",
"encryptedCredential": "eyJDcmVkZW50aWFsSW=="
},
"connectVia": {
"referenceName": "ServerIR-VM",
"type": "IntegrationRuntimeReference"
}
} }
Also,I am using a Self Hosted IR. Is there something I am doing wrong? My error is as below :
File path \\<host>:<port>\SD\AZURE_FS is not supported. Check the configuration to make sure the path is valid. The given path's format is not supported.
Looks like I found my mistake -
As per the example from the documentation
{
"name": "FileLinkedService",
"properties": {
"type": "FileServer",
"typeProperties": {
"host": "<host>",
"userid": "<domain>\\<user>",
"password": {
"type": "SecureString",
"value": "<password>"
}
},
"connectVia": {
"referenceName": "<name of Integration Runtime>",
"type": "IntegrationRuntimeReference"
}
}
}
I changed to add the domain for the userid and also removed the port number. That solved.
{
"name": "NetworkDrive_LS",
"type": "Microsoft.DataFactory/factories/linkedservices",
"properties": {
"annotations": [],
"type": "FileServer",
"typeProperties": {
"host": "\\\\<host>\\SD\\AZURE_FS",
"userId": "<domain>\\<user>",
"encryptedCredential": "eyJDcmVkZW50aWFsSWQiOiIwY=="
},
"connectVia": {
"referenceName": "ServerIR-VM",
"type": "IntegrationRuntimeReference"
}
}
}
I'm trying to get data from Azure Table Storage using Azure Data Factory. I have a table called orders which has 30 columns. I want to take only 3 columns from this table (PartitionKey, RowKey and DeliveryDate). The DeliveryDate column has different data types like DateTime.Null (String value) and actual datetime values. When I want to preview the data i get the following error:
The DataSource looks like this:
{
"name": "Orders",
"properties": {
"linkedServiceName": {
"referenceName": "AzureTableStorage",
"type": "LinkedServiceReference"
},
"annotations": [],
"type": "AzureTable",
"structure": [
{
"name": "PartitionKey",
"type": "String"
},
{
"name": "RowKey",
"type": "String"
},
{
"name": "DeliveryDate",
"type": "String"
}
],
"typeProperties": {
"tableName": "Orders"
}
},
"type": "Microsoft.DataFactory/factories/datasets"}
I test your problem,it works.Can you show me more detail about this or there is something wrong about my test?
Below is my test data:
The dataset code:
{
"name": "Order",
"properties": {
"linkedServiceName": {
"referenceName": "AzureTableStorage",
"type": "LinkedServiceReference"
},
"annotations": [],
"type": "AzureTable",
"structure": [
{
"name": "PartitionKey",
"type": "String"
},
{
"name": "RowKey",
"type": "String"
},
{
"name": "DeliveryDate",
"type": "String"
}
],
"typeProperties": {
"tableName": "Table7"
}
},
"type": "Microsoft.DataFactory/factories/datasets"
}
I'm trying to transfer some data to Azure Search, but for some reason it fails with Invalid linked service reference. Name: AzureSearch1
I have set up an Azure Search Linked Service like this:
{
"name": "AzureSearch1",
"properties": {
"type": "AzureSearch",
"typeProperties": {
"url": "https://xxxxxx.search.windows.net",
"key": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "AzureKeyVault",
"type": "LinkedServiceReference"
},
"secretName": "Search-AdminKey"
}
},
"connectVia": {
"referenceName": "integrationRuntime1",
"type": "IntegrationRuntimeReference"
}
}
}
"Test connection" works fine.
Now, I'm trying to create an Azure Search Indexer like this:
{
"name": "AzureSearchIndex_PriceSheet",
"properties": {
"linkedServiceName": {
"referenceName": "AzureSearch1",
"type": "LinkedServiceReference"
},
"type": "AzureSearchIndex",
"typeProperties": {
"indexName": "pricesheet"
}
}
}
but it fails when I click "Preview Data" or "Import Schema" with this error:
Invalid linked service reference. Name: AzureSearch1. Activity ID:2fa29fe9-ca5d-4308-af62-xxxxxxxxx
The integration pipeline is set to "West Europe" and Azure Search in provisioned in that region too.
Any thoughts?
Thanks!
I tried to reproduce your issue but failed.Please refer to my working configuration:
My Azure Search Linked Service:
{
"name": "AzureSearch1",
"properties": {
"type": "AzureSearch",
"typeProperties": {
"url": "https://***.search.windows.net",
"key": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "AzureKeyVault1",
"type": "LinkedServiceReference"
},
"secretName": "testas"
}
}
},
"type": "Microsoft.DataFactory/factories/linkedservices"
}
My Azure Search Indexer:
{
"name": "AzureSearchIndex1",
"properties": {
"linkedServiceName": {
"referenceName": "AzureSearch1",
"type": "LinkedServiceReference"
},
"folder": {
"name": "azureSearch"
},
"type": "AzureSearchIndex",
"typeProperties": {
"indexName": "documentdb-index"
}
},
"type": "Microsoft.DataFactory/factories/datasets"
}
Perview Data:
I presumed it may because the secret which is stored in AKV has expired,then the link lost connection.I suggest you re-creating secret in AKV(just set default configuration) and try again.
Hope it helps you.Any concern,let me know.
I am trying to do a pipeline in Azure Data factory V1 which will do an Azure Batch Execution on a file. I implemented it using a blob storage as input and output and it worked. However, I am not trying to change the input and output to a folder in my data lake store. When I try to deploy it, it gives me the following error:
Entity provisioning failed: AzureML Activity 'MLActivity' specifies 'DatalakeInput' in a property that requires an Azure Blob Dataset reference.
How can I have the input and output as a datalakestore instead of a blob?
Pipeline:
{
"name": "MLPipeline",
"properties": {
"description": "use AzureML model",
"activities": [
{
"type": "AzureMLBatchExecution",
"typeProperties": {
"webServiceInput": "DatalakeInput",
"webServiceOutputs": {
"output1": "DatalakeOutput"
},
"webServiceInputs": {},
"globalParameters": {}
},
"inputs": [
{
"name": "DatalakeInput"
}
],
"outputs": [
{
"name": "DatalakeOutput"
}
],
"policy": {
"timeout": "02:00:00",
"concurrency": 3,
"executionPriorityOrder": "NewestFirst",
"retry": 1
},
"scheduler": {
"frequency": "Hour",
"interval": 1
},
"name": "MLActivity",
"description": "description",
"linkedServiceName": "MyAzureMLLinkedService"
}
],
"start": "2016-02-08T00:00:00Z",
"end": "2016-02-08T00:00:00Z",
"isPaused": false,
"hubName": "hubname",
"pipelineMode": "Scheduled"
}
}
Output dataset:
{
"name": "DatalakeOutput",
"properties": {
"published": false,
"type": "AzureDataLakeStore",
"linkedServiceName": "AzureDataLakeStoreLinkedService",
"typeProperties": {
"folderPath": "/DATA_MANAGEMENT/"
},
"availability": {
"frequency": "Hour",
"interval": 1
}
}
}
Input dataset:
{
"name": "DatalakeInput",
"properties": {
"published": false,
"type": "AzureDataLakeStore",
"linkedServiceName": "AzureDataLakeStoreLinkedService",
"typeProperties": {
"fileName": "data.csv",
"folderPath": "/RAW/",
"format": {
"type": "TextFormat",
"columnDelimiter": ","
}
},
"availability": {
"frequency": "Hour",
"interval": 1
}
}
}
AzureDatalakeStoreLinkedService:
{
"name": "AzureDataLakeStoreLinkedService",
"properties": {
"description": "",
"hubName": "xyzdatafactoryv1_hub",
"type": "AzureDataLakeStore",
"typeProperties": {
"dataLakeStoreUri": "https://xyzdatastore.azuredatalakestore.net/webhdfs/v1",
"authorization": "**********",
"sessionId": "**********",
"subscriptionId": "*****",
"resourceGroupName": "xyzresourcegroup"
}
}
}
The linked service was done following this tutorial based on data factory V1.
I assume there is some issue with AzureDataLakeStoreLinkedService. Please verify.
Depending on the authentication used for access data store, your AzureDataLakeStoreLinkedService json must look like below -
Using service principal authentication
{
"name": "AzureDataLakeStoreLinkedService",
"properties": {
"type": "AzureDataLakeStore",
"typeProperties": {
"dataLakeStoreUri": "https://<accountname>.azuredatalakestore.net/webhdfs/v1",
"servicePrincipalId": "<service principal id>",
"servicePrincipalKey": {
"type": "SecureString",
"value": "<service principal key>"
},
"tenant": "<tenant info, e.g. microsoft.onmicrosoft.com>",
"subscriptionId": "<subscription of ADLS>",
"resourceGroupName": "<resource group of ADLS>"
},
"connectVia": {
"referenceName": "<name of Integration Runtime>",
"type": "IntegrationRuntimeReference"
}
}
}
Using managed service identity authentication
{
"name": "AzureDataLakeStoreLinkedService",
"properties": {
"type": "AzureDataLakeStore",
"typeProperties": {
"dataLakeStoreUri": "https://<accountname>.azuredatalakestore.net/webhdfs/v1",
"tenant": "<tenant info, e.g. microsoft.onmicrosoft.com>",
"subscriptionId": "<subscription of ADLS>",
"resourceGroupName": "<resource group of ADLS>"
},
"connectVia": {
"referenceName": "<name of Integration Runtime>",
"type": "IntegrationRuntimeReference"
}
}
}
This is Microsoft Document for reference - Copy data to or from Azure Data Lake Store by using Azure Data Factory