Currently I can only see the file HANASPARKCTRL00P_5-70001262.RPM in SAP SWDC under HANA Platform Components. Does it mean that Ambari-managed installation is no longer supported, or I simply miss the required authorisation in the Services Portal?
There is no Ambari installation for Spark Controller 1.5. It is only possible as of Spark Controller 1.6. The installation instructions can be found in the HANA Admin Guide (current version is from 5/11/2016) -> chapter "9.2.3.1 Set up Spark Controller using Ambari" as well as in the attachments of SAP Note 2318054 "SAP HANA Spark Controller 1.6.0 SPS 11 and Higher (Compatible with Spark 1.5.2)".
Related
I was creating a recommendation engine in IBM Watson studio for that I needed to add spark service but now it is deprecated what I should use now.
You should utilize spark environments for your Watson studio project.
You can define that spark environment using Environments tab in project and then utilize that runtime when you create notebook or change service for existing notebook.
#Veer as #charles said Spark services are now accessible via Environments. When creating a notebook select a Spark compatible environment and import pyspark in your notebook.
I am planning to execute spark from KNIME analytics platform. For this I need to install KNIME spark executors in the KNIME analytics platform.
Can any one please let me know how to install KNIME spark executors in the KNIME analytics platform for hadoop distribution CDH 5.10.X.
I am referring the installation guide from the below link:
https://www.knime.org/knime-spark-executor
I could successfully configure/integrate spark in KNIME.
I did it in CDH 5.7.
I followed the following steps:
1.Downloaded knime-full_3.3.2.linux.gtk.x86_64.tar.gz.
2.Exract the above mentioned pacakge and run installation for KNIME.
3.After KNIME is installed goto File ->Install KNIME Extensions -> Install Bigdata extensions(Check all the Spark related extensions and proceed).
Follow this link:
https://tech.knime.org/installation-instructions#download
4.Till now only the Bigdata related extensions have been installed but they need license to be functional.
5.License needs to be purchased.However,free trail for 30 days can be availed after which it needs to be purchased.
Folow this link :
https://www.knime.org/knime-spark-executor
6.After plugins are installed we need to configure Spark-job-server.
For that we need to download the compatible version of spark-job-server for the hadoop version we have.
Folow this link for version of spark-job-server and its compatible version :
https://www.knime.org/knime-spark-executor
I'm pretty sure it's as easy as registering for the free trial (and buying the license for longer than 30 days) and then installing the software from the Help->Install New Software menu.
As of version KNIME 3.6 (latest), it should be possible to connect to Spark via Livy, no specific executor deployment on a KNIME Server. Still in preview, but it should do it.
https://www.knime.com/whats-new-in-knime-36
I want to upgrade my SPark component to 2.1.0 from its default 2.0.x.2.5 in Ambari.
I am using HDP 2.5.0 with Ambari 2.4.2.
Appreciate any idea to achieve this.
HDP 2.5 shipped with a technical preview of Spark 2.0, and also Spark 1.6.x. If you do not want to use either of those versions and you want Ambari to manage the service for you then you will need to write a custom service for the Spark version that you want. If you don't want ambari to manage the Spark instance you can follow similar instructions as provided on the Hortonworks Community Forum to manually install Spark 2.x without management.
Newer versions of Ambari (maybe 3.0) probably will support per-component upgrade/multiple component versions
See https://issues.apache.org/jira/browse/AMBARI-12556 for details.
I have Datastax Community 3.0.4 installed on Windws 8.1 and I am trying to use Datastax Studio 1.0.2. The question is that I use authentication in Cassandra and therefore I need to authenticate also in Datastax Studio.
How can I solve it? How can I authenticate in Datastax Studio?
Studio and Community Edition are not meant to work together.
DataStax Studio is meant for use with DataStax Enterprise (in particular, for use exploring graph data in DSE). The Community edition you have installed only contains a distribution of Apache Cassandra (+ OpsCenter) and not DataStax Enterprise. So if you want to use DataStax Studio, you're going to have to get a copy of DataStax Enterprise first.
Since you're on Windows (and not on Windows 10), your options are a little limited. DSE doesn't run on Windows natively, so you'll have to use a Virtual Machine of some kind. There is a Sandbox image available from the DataStax Academy downloads page for both VirtualBox or VMWare, or you can always create your own VM (running Ubuntu or the Linux flavor of your choice) and Install DSE yourself.
Good luck!
I am trying to build a simple project with Spark+Cassandra for a SQL-analytics demo.
I need to use Cassandra v2.0.14 (can't upgrade it for now). I am unable to find the correct version of Spark and Spark-cassandra-connector. I referred to Datastax's git project at - https://github.com/datastax/spark-cassandra-connector, and I know that the Spark and Spark-cassandra-connector versions need to match and be compatible with Cassandra. Hence, would like anyone to help pointing out the exact versions for Spark, Spark-Cassandra-connector. I tried using v1.1.0 and v1.2.1 for both Spark and Spark-Cassandra-connector - but unable to build the spark-cassandra-connector jat jar with neither the supplied sbt (fails because the downloaded sbt-launch jar just contains a 404 not found html), nor my local sbt v0.13.8 (fails for compilation error for "import sbtassembly.Plugin.", "import AssemblyKeys.")
The connector works with Cassandra 2.0 and 2.1 but some features may also work fine with 2.2 and 3.0 (not officially supported yet) using the older Java driver 2.1. This is because C* Java driver supports a wide range of Cassandra versions. The newer driver works with older C* versions, but also the older driver versions work with newer C* versions, excluding new C* features.
However, there is a one minor caveat with using C* 2.0:
Since version 1.3.0, we dropped the thrift client from the connector. This move was to simplify connectivity code and make it easier to debug - debugging one type of connection should be easier than two. It either connects or not, no more surprises of a kind "it writes fine, but can't connect for reading". Unfortunately, not all of the thrift functionality was exposed by the native protocol in C* 2.0 nor in the system tables. Therefore, if you use C* prior to version 2.1.5, automatic split sizing won't work properly and you have to tell the connector the preferred number of splits. This is to be set in ReadConf object passed at the creation of the RDD.
As for the interface between the Connector and Spark, there is much less freedom. Spark APIs change quite often and you typically need a connector dedicated to the Spark version you use. See the version table in the README.
(fails because the downloaded sbt-launch jar just contains a 404 not found html)
This looks like an SBT problem, not a connector problem.
I just tried to do sbt clean assembly on all v1.2.5, v1.3.0, b1.4 and it worked fine.
if you can upgrade version of spark then you can connect with spark with cassandra .
put following maven dependency in pom file :-
cassandra-all
cassandra-core
cassandra-mapping
cassandra-thrift
cassandra-client
spark-cassandra-connector
spark-cassandra-connector-java
this will be work.