I have an Azure Data Factory Copy Activity that is using a REST request to elastic search as the Source and attempting to map the response to a SQL table as the Sink. Everything works fine except when it attempts to map the data field that contains the dynamic JSON. I get the following error:
{
"errorCode": "2200",
"message": "ErrorCode=UserErrorUnsupportedHierarchicalComplexValue,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The retrieved type of data JObject with value {\"name\":\"department\"} is not supported yet, please either remove the targeted column or enable skip incompatible row to skip them.,Source=Microsoft.DataTransfer.Common,'",
"failureType": "UserError",
"target": "CopyContents_Paged",
"details": []
}
Here's an example of my mapping configuration:
"type": "TabularTranslator",
"mappings": [
{
"source": {
"path": "['_source']['id']"
},
"sink": {
"name": "ContentItemId",
"type": "String"
}
},
{
"source": {
"path": "['_source']['status']"
},
"sink": {
"name": "Status",
"type": "Int32"
}
},
{
"source": {
"path": "['_source']['data']"
},
"sink": {
"name": "Data",
"type": "String"
}
}
],
"collectionReference": "$['hits']['hits']"
}
The JSON in the data object is dynamic so I'm unable to do an explicit mapping for the nested fields within it. That's why I'm trying to just store the entire JSON object under data in a column of a SQL table.
How can I adjust my mapping configuration to allow this to work properly?
I posted this question on the MSDN forums and I was told that if you are using a tabular sink you can set this option "mapComplexValuesToString": true and it should allow complex JSON properties to get mapped correctly. This resolved my ADF copy activity issue.
I have the same problem a few days ago. You need to convert your JSON object to a Json String. It will solve your mapping problem (UserErrorUnsupportedHierarchicalComplexValue).
Try it and tell me if also resolves your error.
Related
I am passing the credentials and parameters required but I get the error
The value of the property 'index' is invalid: 'Index was out of range.
Must be non-negative and less than the size of the collection.
Parameter name: index'. Index was out of range. Must be non-negative
and less than the size of the collection. Parameter name: index
Activity ID: 36a4265d-3607-4472-8641-332f5656661d.
I had the same issue, the password contained a ' and that's causing the trouble. Changed the password with no symbols and it works like a charm
Seems the UI doesn't generate the linked service correctly. Using Microsoft Docs Example JSON I received the same index error when attempting to create the linked service. If I remove the password from the connection string and add it as a separate property I am able to successfully generate the linked service.
Microsoft Docs Example (Doesn't Work)
{
"name": "SnowflakeLinkedService",
"properties": {
"type": "Snowflake",
"typeProperties": {
"connectionString": "jdbc:snowflake://<accountname>.snowflakecomputing.com/?user=<username>&password=<password>&db=<database>&warehouse=<warehouse>&role=<myRole>"
},
"connectVia": {
"referenceName": "<name of Integration Runtime>",
"type": "IntegrationRuntimeReference"
}
}
}
Working Example
{
"name": "SnowflakeLinkedService",
"properties": {
"type": "Snowflake",
"typeProperties": {
"connectionString": "jdbc:snowflake://<accountname>.snowflakecomputing.com/?user=<username>&db=<database>&warehouse=<warehouse>",
"password": {
"type": "SecureString",
"value": "<password>"
}
},
"connectVia": {
"referenceName": "<name of Integration Runtime>",
"type": "IntegrationRuntimeReference"
}
}
}
We hit this same issue today, it was because our password had an ampersand (&) at the end. This seemed to mess up the connection string as it contained this:
&password=abc123&&role=MyRole
Changing the password to not include an ampersand fixed it
The documentation demonstrates how to create a parameter for a connected service but not how to actual pass in that parameter from a dataset or activity. Basically the connection string is coming from a lookup foreach loop and I want to connect to a storage table.
The connection looks like this. The test works when passing in a correct parameter:
{
"name": "StatsStorage",
"properties": {
"type": "AzureTableStorage",
"parameters": {
"connectionString": {
"type": "String"
}
},
"annotations": [],
"typeProperties": {
"connectionString": "#{linkedService().connectionString}"
}
}
}
The dataset is the following, I'm struggling to determine how to set the connectionString parameter for the connection. The dataset has two parameters, the connectionstring from the db and the tablename that it needs to connect to:
{
"name": "TestTable",
"properties": {
"linkedServiceName": {
"referenceName": "StatsStorage",
"type": "LinkedServiceReference"
},
"parameters": {
"ConnectionString": {
"type": "string"
},
"TableName": {
"type": "string"
}
},
"annotations": [],
"type": "AzureTable",
"schema": [],
"typeProperties": {
"tableName": {
"value": "#dataset().TableName",
"type": "Expression"
}
}
}
}
How do I set the connection string on the connection?
First, you can't make the whole connection string as an expression. You need provide accountName and accountKey sperately. Refer this post about how to do it. How to provide connection string dynamically for azure table storage/blob storage in Azure data factory Linked service
Then, if you are using ADF UI, it will guide you how to provide value for linked service. For example, if you have two dataset parameters, you could specify it as following.
If you want to see json code, you could click the code icon on the top left corner.
I am using azure blob as an example, but the azure table is almost the same.
Hope it could help.
I'm using npm BigQuery module for inserting data into bigquery. I have a custom field say params which is of type RECORD and accept any int,float or string value as a key value pair. How can I insert to such fields?
Looked into this, but could not find anything useful
[https://cloud.google.com/nodejs/docs/reference/bigquery/1.3.x/Table#insert]
If I understand correctly, you are asking for a map with ANY TYPE value, which is not support in BigQuery.
You may have a map with value type info with a record like below schema.
Your insert code needs to pick correct type_value to set.
{
"name": "map_field",
"type": "RECORD",
"mode": "REPEATED",
"fields": [
{
"name": "key",
"type": "STRING",
},
{
"name": "int_value",
"type": "INTEGER"
},
{
"name": "string_value",
"type": "STRING"
},
{
"name": "float_value",
"type": "FLOAT"
}
]
}
Background
I have an application which get list of runbooks jobs using REST API
I would like to apply $filter on properties parameters(see JSON in the end). Parameters is of type IDictionary<string,string>
Issue
$filter works fine for most of the properties but fails for IDictionary<string,string>
This is what I am trying https://management.azure.com/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Automation/automationAccounts/XXX/jobs?$filter=properties/parameters/any(keyValue: keyValue/owner eq 'Adam#test.com')&api-version=2015-10-31
Request fails with error {
"code": "BadRequest",
"message": "Could not find a property named 'owner' on type 'System.Collections.Generic.KeyValuePair_2OfString_String'."
}
Question
Is it possible to filter what I am trying?
If yes, then what I am doing wrong?
JSON response on which I want to apply filter
"value": [
{
"id": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Automation/automationAccounts/XXX/jobs/XXX",
"properties": {
"jobId": "XXX",
"runbook": {
"name": "HelloWorldRunbook"
},
"schedule": null,
"provisioningState": "Succeeded",
"status": "Completed",
"creationTime": "2018-06-17T05:44:12.197+00:00",
"startTime": "2018-06-17T05:44:21.227+00:00",
"lastModifiedTime": "2018-06-17T05:44:43.43+00:00",
"endTime": "2018-06-17T05:44:43.43+00:00",
"jobScheduleId": "7fc134ac-d8bd-464e-b041-6e6b50f83f0c",
"runOn": null,
"parameters": {
"Owner": "Adam#test.com",
"mailBox": "test_mailbox#test.com"
}
}
............removed for brevity
When I try to run List Job By Automation Account Rest API, there is no properties parameters returned in the response body.
If yes, then what I am doing wrong?
...
"parameters": {
"Owner": "Adam#test.com",
"mailBox": "test_mailbox#test.com"
}
...
But according to you mentioned responsed json format, the parameters properties is an object not a dictionary, so you could use $filter=properties/parameters/Owner eq 'Adam#test.com' to do that if you make sure that parameters is existed.
Get https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Automation/automationAccounts/{automationAccountName}/jobs?api-version=2015-10-31&$filter=properties/parameters/Owner eq 'Adam#test.com'
I am having some trouble understanding the URL which I should specify while importing data into Azure Blob storage from the OData feed on Dynamics Marketing(MDM), using Azure Data Factory.
I created a Odata Linked Service to import data and within this, for the path, I specified the location of the Odata URL specified within the OData settings for MDM. I get the below error.
Mashup operation failed. Error message from Mashup execution : ErrorCode=FailedMashupOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message='Type=System.NotSupportedException,Message=The given data source reference cannot test the connection. FailureReason: InvalidDataSourceLocationUrl.,Source=Microsoft.Data.Mashup,',Source=,'.
InvalidDataSourceLocationUrl error code shows it's wrong url.
The 'path' in odata data set should be a relative path, which works together with the root path in OData linked service.
E.g. if the odata endpoint your want is:
http: //services.odata.org/odata/odata.svc/Products?$select=Name,%20Description&$top=5
the linked service should be:
{
"name": "ODataLinkedService",
"properties": {
"type": "OData",
"typeProperties": {
"url": "http://services.odata.org/OData/OData.svc",
"authenticationType": "Anonymous"
}
}
}
and the dataset:
{
"name": "ODataDataset",
"properties": {
"type": "ODataResource",
"typeProperties": {
"path": "Products"
},
"linkedServiceName": "ODataLinkedService",
"structure": [],
"availability": {
"frequency": "Hour",
"interval": 1
},
"external": true,
"policy": {
"retryInterval": "00:01:00",
"retryTimeout": "00:10:00",
"maximumRetry": 3
}
}
}
and the query in Pipeline:
"?$select=Name, Description&$top=5"
Referenced by doc :https://learn.microsoft.com/en-us/azure/data-factory/data-factory-odata-connector