I am facing below mentioned issue while running the queries on Azure Databricks using Oracle Data integrator , please help me resolve this issue.
*Caused By: java.sql.SQLException: [Databricks]DatabricksJDBCDriver ERROR processing query/statement. Error Code: 0, SQL state: TStatus(statusCode:ERROR_STATUS, infoMessages:[org.apache.hive.service.cli.HiveSQLException:Configuration ENC_KEY_LEN is not available.:48:47, org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$:hiveOperatingError:HiveThriftServerErrors.scala:65, com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:runSafely:ErrorPropagationThriftHandler.scala:124, com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:ExecuteStatement:ErrorPropagationThriftHandler.scala:73
Related
When I am trying to load a query into a tabular dateset (from a devops docker image) I will get the following error:
raise DatasetValidationError(error_message + '\n' + str(e), e)
azureml.data.dataset_error_handling.DatasetValidationError: Cannot load any data from the datastore using the SQL query "<azureml.data.datapath.DataPath object at 0x>". Please make sure the datastore and query is correct.
Error Code: ScriptExecution.DatabaseConnection.Unexpected
Failed Step: 9ad57100-4870-49d2-a32f-1c9c15c244e0
Error Message: ScriptExecutionException was caused by DatabaseConnectionException.
DatabaseConnectionException was caused by UnexpectedException.
'MSSQL' encountered unexpected exception of type 'InvalidOperationException' with HResult 'x80131509' while opening connection.
Internal connection fatal error.
I believe that I have allowed the connection in firewall (I might not have done it quite right).
I don't get the error when I am running it from the notebook (on the compute instance).
Is the code that is running on the compute instance the exact same as the docker image? If so, it's likely that the firewall is causing connectivity issues. One way of checking this is to (be cautious) disable the firewall and see if the connectivity issue is gone.
If you're comfortable sharing the firewall/network configurations, that will also help troubleshoot this.
I configured Hive with mySQL as my metastore. I can enter hive shell and create tables successfully.
Spark version: 2.4.0
Hive version: 3.1.1
When I try to run a SparkSQL program using spark submit, I'm getting the below error.
2019-03-02 15:43:41 WARN HiveMetaStore:622 - Retrying creating default database after error: Error creating transactional connection factory
javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
......
......
Exception in thread "main" org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;
......
......
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;
Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "HikariCP" plugin to create a ConnectionPool gave an error : The connection pool plugin of type "HikariCP" was not found in the CLASSPATH!
Please let me know if anyone can help me in this regard.
I don't know if you have already solved this problem. There is my advice.
the default database connection is HikariCP in the hive-site.xml. You can search for this in the hive-site.xml: datanucleus.connectionPoolingType. The value is HikariCP. So you need to change it to dbcp since you use Mysql as your metastore.
And at last, don't forget about adding the mysql-connector-java-5.x.x.jar to the path like
/home/hadoop/spark-2.3.0-bin-hadoop2.7/jars
My script is listed below:
import pyodbc
cnxn = pyodbc.connect(driver=driver,
server=server,
user=username,
database=database,
Trusted_Connection='yes',
autocommit = True)
However, I got error message:
('HY000', '[HY000] [Microsoft][ODBC Driver 13 for SQL Server]SQL Server Network Interfaces: The Microsoft Online Services Sign-In Assistant encountered an error that might not occur if retried. Possible reasons include a transient network error. [x80048849]. (-2147186615) (SQLDriverConnect)')
Anyone has idea? My error came out when I tried to connect to azure. In the error message, it said retried may work, but I have tried several times, and I still saw this error. I see most of people have this issue when they run the query. Please feel free to leave your comment.
I have figured out the issue. Because it is connected to Azure, i dont need to include Trusted_Connection='yes', which is for windows authentication.
I am playing around with the Azure Spark-CosmosDB connector which lets you access CosmosDB nodes directly from a Spark cluster for analytics using Jupyter on HDINsight
I have been following the steps described here,including uploading the required jars to Azure storage and executing the %%configure magic to prepare the environment.
But it always seems to terminate due to an I/O exception when trying to open the jar (see yarn log below)
17/10/09 20:10:35 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.io.IOException: Error accessing /mnt/resource/hadoop/yarn/local/usercache/livy/appcache/application_1507534135641_0014/container_1507534135641_0014_01_000001/azure-cosmosdb-spark-0.0.3-SNAPSHOT.jar)
17/10/09 20:10:35 ERROR ApplicationMaster: RECEIVED SIGNAL TERM
17/10/09 20:10:35 INFO ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: java.io.IOException: Error accessing /mnt/resource/hadoop/yarn/local/usercache/livy/appcache/application_1507534135641_0014/container_1507534135641_0014_01_000001/azure-cosmosdb-spark-0.0.3-SNAPSHOT.jar)`
Not sure whether this is related to the jar not being copied to the worker nodes.
any idea? thanks, Nick
Database operation failed on server 'worker.database.windows.net,11146' with SQL Error Number '40197'. Error message from database execution : The service has encountered an error processing your request. Please try again. Error code 4815.
A severe error occurred on the current command. The results, if any, should be discarded..
I'm getting the following error. What does error code 4815 mean? Couldn't find any documentation regarding that error code.
The error 4815 means that the SQL Server engine received an invalid column length from the bcp client for colid. You might want to check the parameters again and try this task again.
Here is some documentation on the error codes. Hope this helps.
https://technet.microsoft.com/en-us/library/cc645613(v=sql.105).aspx