Spark1.6 and Hive 0.14 integration issue - apache-spark

Question

I have been trying to integrate the latest spark 1.6 with hive 0.14.0. I am only trying to get the Thrift server to run. I have noticed that if I don't override the following configurations: (-conf spark.sql.hive.metastore.version=0.14.0 --conf spark.sql.hive.metastore.jars=maven) when invoking start-thrifstserver.sh spark script, then any create table queries fail in spark due to incompatibility issues between hive 1.2.1 which is used by spark 1.6 by default and my hive version running in prod. However, when I override those 2 configs, then when thrift server is started, it does not connect to my hive metastore uri as specified in hive-site.xml but rather it tires to connect to derby database and then Thrift server does not start properly. Am I missing some additional overrides?
Please see the thrift server log information below:
Loaded from file:/usr/lib/spark/lib/spark-assembly-1.6.0-hadoop2.6.0.jar
java.vendor=Oracle Corporation
java.runtime.version=1.7.0_79-b15
user.dir=/
os.name=Linux
os.arch=amd64
os.version=2.6.32-504.23.4.el6.x86_64
derby.system.home=null
Database Class Loader started - derby.database.classpath=''
16/01/26 16:35:20 INFO YarnClientSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (10.15.150.38:51475) with ID 20
16/01/26 16:35:20 INFO BlockManagerMasterEndpoint: Registering block manager 10.15.150.38:52107 with 9.9 GB RAM, BlockManagerId(20, 10.15.150.38, 52107)
16/01/26 16:35:20 INFO YarnClientSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (10.15.150.38:51479) with ID 48
16/01/26 16:35:20 INFO BlockManagerMasterEndpoint: Registering block manager 10.15.150.38:47973 with 9.9 GB RAM, BlockManagerId(48, 10.15.150.38, 47973)
16/01/26 16:35:20 WARN Configuration: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream#3cf4a477:an attempt to override final parameter: mapreduce.reduce.speculative; Ignoring.
16/01/26 16:35:20 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
16/01/26 16:35:21 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/01/26 16:35:21 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/01/26 16:35:22 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/01/26 16:35:22 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/01/26 16:35:22 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
16/01/26 16:35:22 INFO ObjectStore: Initialized ObjectStore
16/01/26 16:35:22 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/01/26 16:35:22 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
16/01/26 16:35:22 INFO HiveMetaStore: Added admin role in metastore
16/01/26 16:35:22 INFO HiveMetaStore: Added public role in metastore
16/01/26 16:35:22 INFO HiveMetaStore: No user is added in admin role, since config is empty
16/01/26 16:35:22 INFO HiveMetaStore: 0: get_all_databases
16/01/26 16:35:22 INFO audit: ugi=hive ip=unknown-ip-addr cmd=get_all_databases
16/01/26 16:35:22 INFO HiveMetaStore: 0: get_functions: db=default pat=*
16/01/26 16:35:22 INFO audit: ugi=hive ip=unknown-ip-addr cmd=get_functions: db=default pat=*
16/01/26 16:35:22 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
16/01/26 16:35:22 INFO SessionState: Created local directory: /tmp/06895c7e-e26c-42b7-b100-4222d0356b6b_resources
16/01/26 16:35:22 INFO SessionState: Created HDFS directory: /tmp/hive/hive/06895c7e-e26c-42b7-b100-4222d0356b6b
16/01/26 16:35:22 INFO SessionState: Created local directory: /tmp/hive/06895c7e-e26c-42b7-b100-4222d0356b6b
16/01/26 16:35:23 INFO SessionState: Created HDFS directory: /tmp/hive/hive/06895c7e-e26c-42b7-b100-4222d0356b6b/_tmp_space.db
16/01/26 16:35:23 WARN Configuration: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream#37f031a:an attempt to override final parameter: mapreduce.reduce.speculative; Ignoring.
16/01/26 16:35:23 INFO HiveContext: default warehouse location is /user/hive/warehouse
16/01/26 16:35:23 INFO HiveContext: Initializing HiveMetastoreConnection version 0.14.0 using maven.
Ivy Default Cache set to: /home/hive/.ivy2/cache
The jars for the packages stored in: /home/hive/.ivy2/jars
http://www.datanucleus.org/downloads/maven2 added as a remote repository with the name: repo-1
:: loading settings :: url = jar:file:/usr/lib/spark/lib/spark-assembly-1.6.0-hadoop2.6.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
org.apache.calcite#calcite-core added as a dependency
org.apache.calcite#calcite-avatica added as a dependency
org.apache.hive#hive-metastore added as a dependency
org.apache.hive#hive-exec added as a dependency
org.apache.hive#hive-common added as a dependency
org.apache.hive#hive-serde added as a dependency
com.google.guava#guava added as a dependency
org.apache.hadoop#hadoop-client added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]