I cant run my very first simple spark program with scala ide.
I checked all my properties and i believe that are correct.
this is the link with the properties.
any help ?
The problem is that you are trying to include Scala 2.11.8 as a dependency in your application, while Spark artifacts rely on Scala 2.10.
You have two options to solve your problem:
Use Scala 2.10.x
Use Spark artifacts that rely on Scala 2.11 (e.g. spark-core_2.11 instead of spark-core_2.10)
Related
I'm new to spark and my understanding is this:
jars are like a bundle of java code files
Each library that I install that internally uses spark (or pyspark) has its own jar files that need to be available with both driver and executors in order for them to execute the package API calls that the user interacts with. These jar files are like the backend code for those API calls
Questions:
Why are these jar files needed. Why could it not have sufficed to have all the code in python? (I guess the answer is that originally Spark is written in scala and there it distributes its dependencies as jars. So to not have to create that codebase mountain again, the python libraries just call that javacode in python interpreter through some converter that converts java code to equivalent python code. Please if I have understood right)
You specify these jar files locations while creating the spark context via spark.driver.extraClassPath and spark.executor.extraClassPath. These are outdated parameters though I guess. What is the recent way to specify these jar files location?
Where do I find these jars for each library that I install? For example synapseml. What is the general idea about where the jar files for a package are located? Why do not the libraries make it clear where their specific jar files are going to be?
I understand I might not be making sense here and what I have mentioned above is partly just my hunch that that is how it must be happening.
So, can you please help me understand this whole business with jars and how to find and specify them?
Each library that I install that internally uses spark (or pyspark)
has its own jar files
Can you tell which library are you trying to install ?
Yes, external libraries can have jars even if you are writing code in python.
Why ?
These libraries must be using some UDF (User Defined Functions). Spark runs the code in java runtime. If these UDF are written in python, then there will be lot of serialization and deserialization time due to converting data into something readable by python.
Java and Scala UDFs are usually faster that's why some libraries ship with jars.
Why could it not have sufficed to have all the code in python?
Same reason, scala/java UDFs are faster than python UDF.
What is the recent way to specify these jar files location?
You can use spark.jars.packages property. It will copy to both driver and executor.
Where do I find these jars for each library that I install? For
example synapseml. What is the general idea about where the jar files
for a package are located?
https://github.com/microsoft/SynapseML#python
They have mentioned here what jars are required i.e. com.microsoft.azure:synapseml_2.12:0.9.4
import pyspark
spark = pyspark.sql.SparkSession.builder.appName("MyApp") \
.config("spark.jars.packages", "com.microsoft.azure:synapseml_2.12:0.9.4") \
.config("spark.jars.repositories", "https://mmlspark.azureedge.net/maven") \
.getOrCreate()
import synapse.ml
Can you try the above snippet?
I am trying to create a permanent function in spark using geomesa-spark-jts.
Geomesa-spark-jts has huge potential in the larger LocationTech community.
I started first by downloading geomesa-spark-jts which contain the following
The after that I have launched spark like this (I made sure that the jar is within the path)
Now whew I use ST_Translate which come with that package, it does give me a result
But the problem is when I try to define ST_Translate as a UDF , I get the following error
The functions you mentioned are already supported in GeoMesa 2.0.0 for Spark 2.2.0. http://www.geomesa.org/documentation/user/spark/sparksql_functions.html
The geomesa-accumulo-spark-runtime jar is a shaded jar that includes the code from geomesa-spark-jts. You might be hitting issues with having the classes defined in two different jars.
In order to use st_translate with hive, I believe that you would have to implement a new class that extends org.apache.hadoop.hive.ql.exec.UDF and invokes the GeoMesa function.
I am only trying to read a textfile into a pyspark RDD, and I am noticing huge differences between sqlContext.read.load and sqlContext.read.text.
s3_single_file_inpath='s3a://bucket-name/file_name'
indata = sqlContext.read.load(s3_single_file_inpath, format='com.databricks.spark.csv', header='true', inferSchema='false',sep=',')
indata = sqlContext.read.text(s3_single_file_inpath)
The sqlContext.read.load command above fails with
Py4JJavaError: An error occurred while calling o227.load.
: java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.csv. Please find packages at http://spark-packages.org
But the second one succeeds?
Now, I am confused by this because all of the resources I see online say to use sqlContext.read.load including this one: https://spark.apache.org/docs/1.6.1/sql-programming-guide.html.
It is not clear to me when to use which of these to use when. Is there a clear distinction between these?
Why is difference between sqlContext.read.load and sqlContext.read.text?
sqlContext.read.load assumes parquet as the data source format while sqlContext.read.text assumes text format.
With sqlContext.read.load you can define the data source format using format parameter.
Depending on the version of Spark 1.6 vs 2.x you may or may not load an external Spark package to have support for csv format.
As of Spark 2.0 you no longer have to load spark-csv Spark package since (quoting the official documentation):
NOTE: This functionality has been inlined in Apache Spark 2.x. This package is in maintenance mode and we only accept critical bug fixes.
That would explain why you got confused as you may have been using Spark 1.6.x and have not loaded the Spark package to have csv support.
Now, I am confused by this because all of the resources I see online say to use sqlContext.read.load including this one: https://spark.apache.org/docs/1.6.1/sql-programming-guide.html.
https://spark.apache.org/docs/1.6.1/sql-programming-guide.html is for Spark 1.6.1 when spark-csv Spark package was not part of Spark. It happened in Spark 2.0.
It is not clear to me when to use which of these to use when. Is there a clear distinction between these?
There's none actually iff you use Spark 2.x.
If however you use Spark 1.6.x, spark-csv has to be loaded separately using --packages option (as described in Using with Spark shell):
This package can be added to Spark using the --packages command line option. For example, to include it when starting the spark shell
As a matter of fact, you can still use com.databricks.spark.csv format explicitly in Spark 2.x as it's recognized internally.
The difference is:
text is a built-in input format in Spark 1.6
com.databricks.spark.csv is a third party package in Spark 1.6
To use third party Spark CSV (no longer needed in Spark 2.0) you have to follow the instructions on spark-csv site, for example provide
--packages com.databricks:spark-csv_2.10:1.5.0
argument with spark-submit / pyspark commands.
Beyond that sqlContext.read.formatName(...) is a syntactic sugar for sqlContext.read.format("formatName") and sqlContext.read.load(..., format=formatName).
Does Spark 2.0 support avro and parquet files? What versions?
I have downloaded spark-avro_2.10-0.1.jar and got this error during load:
Name: java.lang.IncompatibleClassChangeError
Message: org.apache.spark.sql.sources.TableScan
StackTrace: at java.lang.ClassLoader.defineClassImpl(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:349)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:154)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:727)
at java.net.URLClassLoader.access$400(URLClassLoader.java:95)
at java.net.URLClassLoader$ClassFinder.run(URLClassLoader.java:1182)
at java.security.AccessController.doPrivileged(AccessController.java:686)
at java.net.URLClassLoader.findClass(URLClassLoader.java:602)
You are just using the wrong dependency. You should use the spark-avro dependency that is compiles with Scala 2.11. You can find it here.
As for parquet, it's supported without any dependency to add to your application.
Does spark 2.0 supports avro and parquet files?
Avro format is not supportd in Spark 2.x out of the box. You have to use an external package, e.g. spark-avro.
Name: java.lang.IncompatibleClassChangeError
Message: org.apache.spark.sql.sources.TableScan
The reason for java.lang.IncompatibleClassChangeError is that you used spark-avro_2.10-0.1.jar that was compiled for Scala 2.10, but Spark 2.0 uses Scala 2.11 by default. This inevitably leads to this IncompatibleClassChangeError error.
You should rather load the spark-avro package using --packages command line option (as described in the official documentation of spark-avro in With spark-shell or spark-submit):
$ bin/spark-shell --packages com.databricks:spark-avro_2.11:3.2.0
using --packages ensures that this library and its dependencies will be added to the classpath. The --packages argument can also be used with bin/spark-submit.
Parquet format is the default format when loading or saving datasets.
// loading parquet datasets
spark.read.load
// saving in parquet format
mydataset.write.save
You may want to read up on Parquet Files support in the official documentation:
Spark SQL provides support for both reading and writing Parquet files that automatically preserves the schema of the original data.
Parquet 1.8.2 is used (as you can see in Spark's pom.xml)
In maven repository http://mvnrepository.com/artifact/org.apache.spark, apache-spark version 1.4.1 is available in 2 flavours.
spark-*_2.10 & spark-*_2.11
These seem to be Scala versions. Which of these is preferred if I am deploying apache-spark with java distribution?
The Scala SDK is not binary compatible between major releases (for example, 2.10 and 2.11). If you have Scala code that you will be using with Spark and that code is compiled against a particular major version of Scala (say 2.10) then you will need to use the compatible version of Spark. For example, if you are writing Spark 1.4.1 code in Scala and you are using the 2.11.4 compiler, then you should use Spark 1.4.1_2.11.
If you are not using Scala code then there should be no functional difference between Spark 1.4.1_2.10 and Spark 1.4.1_2.11 (if there is, it is most likely a bug). The only difference should be the version of the Scala compiler used to compile Spark and the corresponding libraries.
I don't think it matters if you are using java as the bytecode should be close enough. The current default for spark is 2.10, but you might get some minor gains if you choose 2.11. But, ultimately I don't think it matters
As zero323 mentions, there are some areas that might not be fully supported in 2.11, so as I stated above, 2.10 is the default for now and probably the safest route.