We have scala Jar file which is running on on-prem Hadoop cluster. It create hive tables on parquet file and do further spark processing on hive table.
May I know your suggestion on how to run such hive based spark processing in synapses spark pool? If possible, without changing our code?
Currently Spark only works with external Hive tables and non-transactional/non-ACID managed Hive tables. It doesn’t support Hive ACID/transactional tables now.
For more details, refer to Use external Hive Metastore for Synapse Spark Pool (Preview）
Related
Is there a difference between spark table and hive table? How they are different if both gets registered in hive metastore or not?
wanted to understand if a table can be registered in hive metastore which was created by spark but not hive?
We are trying to migrate pyspark script from on-premise which creates and drops tables in Hive with data transformations to GCP platform.
Hive is replaced by BigQuery.
In this case, the hive reads and writes is converted to bigquery reads and writes using spark-bigquery-connector.
However the problem lies with creation and dropping of bigquery tables via spark sql as spark sql will default run the create and drop queries on hive backed by hive metastore not on big query.
I wanted to check if there is plan to incorporate DDL statements support as well as part of spark-bigquery-connector.
Also, from architecture perspective is it possible to base the metastore for spark sql on bigquery so that any create or drop statement can be run on bigquery from spark.
I don't think Spark SQL will support BigQuery as metastore, nor BQ connector will support BQ DDL. On Dataproc, Dataproc Metastore (DPMS) is the recommended solution for Hive and Spark SQL metastore.
In particular, for no-prem to Dataproc migration, it is more straightforward to migrate to DPMS, see this doc.
I have a Hive Table on Azure HDInsight WASB, want to migrate / copy over from Production to QA environment, looks like I can only do it via export / import.
1) Export tables from parquet to files (metadata included)
2) AzCopy from Prod Storage to QA Storage
3) Import tables
Azure HDInsight supports only export/import Hive metastore.
If you want to retain your Hive tables after you delete an HDInsight cluster, use a custom metastore. You can then attach the metastore to another HDInsight cluster.
Note: An HDInsight metastore that is created for one HDInsight cluster version cannot be shared across different HDInsight cluster versions.
References:
How do I export a Hive metastore and import it on another cluster?
Hive metastore best practices
The Docs says "Every Databricks deployment has a central Hive metastore..." besides an external metastore for existing Hive installations.
I have an Azure Databricks workspace with an underlying spark cluster, and a datafiles stored on DBFS and Blob Storage. Do I need HDInsight cluster with external metastore to be able to create and use Hive tables? Or can I use the above mentioned central metastore to create Hive tables on data stored on DBFS or Blob storage?
#Gadam nope you do not. Azure Databricks provisions its own Hive Metastore, but if you are already using one with HDInsight, Databricks can be configured to also use it (an external metastore).
We are working on Azure ML and ADLS combination. Since HDInsight Cluster is working over ADLS, we are trying to use Hive Query and HDFS route and running into problems.
Request your help in solving the problem of reading data from hive query and writing to HDFS. Below is the error URL for reference:
https://studioapi.azureml.net/api/sharedaccess?workspaceId=025ba20578874d7086e6c495cc49a3f2&signature=ZMUCNMwRjlrksrrmsrx5SaGedSgwMmO%2FfSHvq190%2F1I%3D&sharedAccessUri=https%3A%2F%2Fesprodussouth001.blob.core.windows.net%2Fexperimentoutput%2Fccf9a206-730d-4773-b44e-a2dd8c6e87b9%2Fccf9a206-730d-4773-b44e-a2dd8c6e87b9.txt%3Fsv%3D2015-02-21%26sr%3Db%26sig%3DHkuFm8B2Ba1kEWWIwanqlv%2FcQPWVz0XYveSsZnEa0Wg%3D%26st%3D2017-10-16T18%3A31%3A06Z%26se%3D2017-10-17T18%3A36%3A06Z%26sp%3Dr
Azure Machine Learning supports Hive but not over ADLS.