I'm trying to use Cassandra with Spark but when I'm trying to execute some of the methods, I'm getting this kind of error:
Anyone know how to fix this?
I'm using Scala version 2.11.8, Spark version 2.0.0. and Cassandra 3.7.
Related
I want to use cdp7.1.8 with spark 3.3.0 but I am not able to find the compatibility version matrix for spark 3.3.0.
I am using maven as build tool.
I searched for documentation but did not find the solution.
Currently I have Spark version 1.6.2 installed.
I want to upgrade the Spark version to the newest 2.0.1. How do I do this without losing the existing configurations?
Any help would be appreciated.
If its maven or sbt application you simply change dependency version of spark and also migrate your code according to 2.0 so you will not lose you configurations. and for spark binary you can take backup of config folder.
There is no much change related to configuration, some method signatures are changed , major changes i observed was mapPartitions method signature and some changes to metrics/listener api, apart from new features.
I am currently working with Spark 1.6.1 and use it in both Jupyter ipython notebooks and from Java 8. For Java I can just modify my maven pom to import Spark 2.0, but I'm not sure how to do the equivalent in Ipython. I think I need to install 2.0, but is that doable since I already have Spark 1.6.1 installed? Can I have both versions on my macbook and select which one to use from pyspark? how?
Update: This is how I launch my Jupyter pyspark notebook => on terminal: % IPYTHON_OPTS="notebook" pyspark. How do I tell it to launch with Spark 2.0?
Can I have both versions on my macbook and select which one to use from pyspark
Yes.
Say you have extracted Spark to /opt/apache-spark folder. Then, in there, you could have both versions of 2.0.0 and 1.6.1.
Then, to run pyspark of version 2.0.0, you simply run
/opt/apache-spark/2.0.0/bin/pyspark
The real question you need to ask is why do you think you need both versions?
I am trying to build a simple project with Spark+Cassandra for a SQL-analytics demo.
I need to use Cassandra v2.0.14 (can't upgrade it for now). I am unable to find the correct version of Spark and Spark-cassandra-connector. I referred to Datastax's git project at - https://github.com/datastax/spark-cassandra-connector, and I know that the Spark and Spark-cassandra-connector versions need to match and be compatible with Cassandra. Hence, would like anyone to help pointing out the exact versions for Spark, Spark-Cassandra-connector. I tried using v1.1.0 and v1.2.1 for both Spark and Spark-Cassandra-connector - but unable to build the spark-cassandra-connector jat jar with neither the supplied sbt (fails because the downloaded sbt-launch jar just contains a 404 not found html), nor my local sbt v0.13.8 (fails for compilation error for "import sbtassembly.Plugin.", "import AssemblyKeys.")
The connector works with Cassandra 2.0 and 2.1 but some features may also work fine with 2.2 and 3.0 (not officially supported yet) using the older Java driver 2.1. This is because C* Java driver supports a wide range of Cassandra versions. The newer driver works with older C* versions, but also the older driver versions work with newer C* versions, excluding new C* features.
However, there is a one minor caveat with using C* 2.0:
Since version 1.3.0, we dropped the thrift client from the connector. This move was to simplify connectivity code and make it easier to debug - debugging one type of connection should be easier than two. It either connects or not, no more surprises of a kind "it writes fine, but can't connect for reading". Unfortunately, not all of the thrift functionality was exposed by the native protocol in C* 2.0 nor in the system tables. Therefore, if you use C* prior to version 2.1.5, automatic split sizing won't work properly and you have to tell the connector the preferred number of splits. This is to be set in ReadConf object passed at the creation of the RDD.
As for the interface between the Connector and Spark, there is much less freedom. Spark APIs change quite often and you typically need a connector dedicated to the Spark version you use. See the version table in the README.
(fails because the downloaded sbt-launch jar just contains a 404 not found html)
This looks like an SBT problem, not a connector problem.
I just tried to do sbt clean assembly on all v1.2.5, v1.3.0, b1.4 and it worked fine.
if you can upgrade version of spark then you can connect with spark with cassandra .
put following maven dependency in pom file :-
cassandra-all
cassandra-core
cassandra-mapping
cassandra-thrift
cassandra-client
spark-cassandra-connector
spark-cassandra-connector-java
this will be work.
I'm having a bit of trouble understanding exactly how to add jackson-module-scala to my datastax spark integration.
I am using Datastax Enterprise 4.5, planning to upgrade soon to 4.6.1 soon.
I am still new to scala, though I have some experience with maven. I understand that dse uses the spark-cassandra-connector under the hood, but I can't find where the connector code resides on a specific cassandra node, specifically the build.sbt file.
Would I even add the jackson-module-scala dependency to the connector build.sbt file? I grepped for an .sbt file but could not find one.
Any help here much appreciated.
sbt is the scala build tool.
You should have a scala project where you use spark.
this project should have a build.sbt file that lists dependencies and a lot more.
you should add the jackson dependency to that file