While loading data from Sybase DB in AWS Glue I encounter an error:
Py4JJavaError: An error occurred while calling o261.load.
: java.sql.SQLException: The identifier that starts with '__SPARK_GEN_JDBC_SUBQUERY_NAME' is too long. Maximum length is 30.
The code I use is:
spark.read.format("jdbc").
option("driver", "net.sourceforge.jtds.jdbc.Driver").
option("url", jdbc_url).
option("query", query).
option("user", db_username).
option("password", db_password).
load()
Is there any way to set this identifier as a custom one in order to have it shorter? What's interesting I am able to load all the data from a particular table by replacing query option with option("dbtable", table) but invoking a custom query is impossible.
Best Regards
I am trying to import data from a CSV file into a Dynamics 365 Account table. As I need to do some transformations I am using a dataflow rather than a basic copy activity.
I was having difficulties getting it to work using a dataflow for writing to a multi lookup field so I tried using a copy activity to see if that worked using the exact same source,sink and mappings. I was able to import the
data successfully with the copy activity. I'm confused as to why the data flow does not work using the same source,sink and mappings. Below are screenshots of the various elements I set up and configured. Would appreciate any suggestions to get the dataflow working.
I'm using a cut down version of what will ultimately be my source CSV file. This is just so I can concentrate on getting the writing to the lookup field working.
Source CSV file
Copy Activity Source
Copy Activity Sink
Dynamics 365 Sink
Dataflow Source
Dataflow Sink
Copy Activity Mapping
Dataflow Mapping
Copy Activity Success
Dataflow Failure
Dataflow Error
Details
{"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: DF-REST_001 - Rest - Error response received from the server (url:https://##############v9.0/accounts,request body: Some({"accountid":"8b0257ea-de19-4aaa-9945-############","name":"A User","ownerid":"7d64133b-daa8-eb11-9442-############","ownerid#EntityReference":"systemuser"}), request method: POST, status code: 400), response body: Some({"error":{"code":"0x0","message":"An error occurred while validating input parameters: Microsoft.OData.ODataException: A 'PrimitiveValue' node with non-null value was found when trying to read the value of the property 'ownerid'; however, a 'StartArray' node, a 'StartObject' node, or a 'PrimitiveValue' node with null value was expected.\r\n at Microsoft.OData.JsonLight.ODataJsonLightPropertyAndValueDeserializer.ValidateExpandedNestedResourceInfoPropertyValue(IJsonReader jsonReader, Nullable1 isCollection, String propertyName, IEdmTypeReference typeReference)\r\n at Microsoft.OData.JsonLight.ODataJsonLightResourceDeserializ","Details":"com.microsoft.dataflow.Issues: DF-REST_001 - Rest - Error response received from the server (url:https://dev-gc.crm11.dynamics.com/api/data/v9.0/accounts,request body: Some({"accountid":"8b0257ea-de19-4aaa-9945-############","name":"A User","ownerid":"7d64133b-daa8-eb11-9442-############","ownerid#EntityReference":"systemuser"}), request method: POST, status code: 400), response body: Some({"error":{"code":"0x0","message":"An error occurred while validating input parameters: Microsoft.OData.ODataException: A 'PrimitiveValue' node with non-null value was found when trying to read the value of the property 'ownerid'; however, a 'StartArray' node, a 'StartObject' node, or a 'PrimitiveValue' node with null value was expected.\r\n at Microsoft.OData.JsonLight.ODataJsonLightPropertyAndValueDeserializer.ValidateExpandedNestedResourceInfoPropertyValue(IJsonReader jsonReader, Nullable1 isCollection, String propertyName, IEdmTypeReference typeReference)\r\n at Microsoft.OData.JsonLight.ODataJsonLightResourceDeser"}
I am running into a same wall,
but a temporary solution here is to sink the dataflow output to a Csv/or similar file into ADLS and then use a Copy activity to extract those files and Upsert it into the Dynamics.
Other references: https://vishalgrade.com/2020/10/01/how-to-populate-multi-lookup-attribute-in-ce-using-azure-data-factory/
I am making JDBC connection to Denodo database using pyspark. The table that i am connecting to contains "TIMESTAMP_WITH_TIMEZONE" datatype for 2 columns. Since spark provides builtin jdbc connection to a handful of dbs only of which denodo is not a part, it is not able to recognize "TIMESTAMP_WITH_TIMEZONE" datatype and hence not able to map to any of its spark sql dataype.
To overcome this i am providing my custom schema(c_schema here) but this is not working as well and i am getting the same error. Below is the code snippet.
c_schema="game start date TIMESTAMP,game end date TIMESTAMP"
df = spark.read.jdbc("jdbc_url", "schema.table_name",properties={"user": "user_name", "password": "password","customSchema":c_schema,"driver": "com.denodo.vdp.jdbc.Driver"})
Please let me know how shall i fix this.
For anyone else facing this issue while connecting to denodo using spark,use CAST function to convert the datatype "TIMESTAMP_WITH_TIMEZONE" into any other datatype like String,Date or Timestamp etc. I had posted this question on denodo community page too and i have attached its official response.
CAST("PLANNED START DATE" as DATE) as "PLANNED_START_DATE"
I am working on importing data from Salesforce Report using ADF. I am using copy activity and have created salesforce linked service.
I did get the below syntax from Microsoft documentation https://learn.microsoft.com/en-us/azure/data-factory/connector-salesforce but I get the below error.
Error:
Failed to import source schema. ERROR [HY000] [Microsoft][Salesforce] (120) SOQL_FIRST mode prepare failure: SOQL error: [Microsoft][SQLEngine] (31480) syntax error near '"<<< ??? >>>query": "{call "Quality & Impact Survey Results by Tier"}"'. SQL error: [Microsoft][SQLEngine] (31480) syntax error near '"query"<<< ??? >>>: "{call "Reportname"}"'. Activity ID: a1373279-97e4-42e4-86e3-80d930ef9364
If anyone have worked on this then do let me know your thoughts.
I found out the syntax to extract data from Salesforce from ADF as per below.
{call "Reportname"}
I'm using the following MDX query:
SELECT [Measures].[Unita (libri)] ON COLUMNS,
TopCount([Anno].[Anno].Members, 10.0, [Measures].[Unita (libri)]) ON ROWS
FROM [Pubblicazioni]
But i always get back an error:
Error Occurred While getting Resultset
MDX Query Editor
An error occurred while rendering Pivot.jsp. Please see the log for details.
I can't access the log (for some reason the server is not writing logs) but i think it's not normal to get this error with this simple query. Any ideas about?