Currently working on PySpark. There is no map function on DataFrame, and one has to go to RDD for map function. In Scala there is a map on DataFrame, is there any reason for this?
Dataset.map is not part of the DataFrame (Dataset[Row]) API. It transforms strongly typed Dataset[T] into strongly typed Dataset[U]:
def map[U](func: (T) ⇒ U)(implicit arg0: Encoder[U]): Dataset[U]
and there is simply no place for Python in the strongly typed Dataset world. In general, Datasets are native JVM objects (unlike RDD it has not Python specific implementation) which depend heavily on rich Scala type system (even Java API is severely limited). Even if Python implemented some variant of the Encoder API, data would still have to be converted to RDD for computations.
In contrast Python implements its own map like mechanism with vectorized udfs, which should be released in Spark 2.3. It is focused on high performance serde implementation coupled with Pandas API.
That includes both typical udfs (in particular SCALAR and SCALAR_ITER variants) as well as map-like variants - GROUPED_MAP and MAP_ITER applied through GroupedData.apply and DataFrame.mapInPandas (Spark >= 3.0.0) respectively.
Related
Currently working on PySpark. There is no map function on DataFrame, and one has to go to RDD for map function. In Scala there is a map on DataFrame, is there any reason for this?
Dataset.map is not part of the DataFrame (Dataset[Row]) API. It transforms strongly typed Dataset[T] into strongly typed Dataset[U]:
def map[U](func: (T) ⇒ U)(implicit arg0: Encoder[U]): Dataset[U]
and there is simply no place for Python in the strongly typed Dataset world. In general, Datasets are native JVM objects (unlike RDD it has not Python specific implementation) which depend heavily on rich Scala type system (even Java API is severely limited). Even if Python implemented some variant of the Encoder API, data would still have to be converted to RDD for computations.
In contrast Python implements its own map like mechanism with vectorized udfs, which should be released in Spark 2.3. It is focused on high performance serde implementation coupled with Pandas API.
That includes both typical udfs (in particular SCALAR and SCALAR_ITER variants) as well as map-like variants - GROUPED_MAP and MAP_ITER applied through GroupedData.apply and DataFrame.mapInPandas (Spark >= 3.0.0) respectively.
From the DataSet and RDD documentation,
DataSet:
A Dataset is a strongly typed collection of domain-specific objects
that can be transformed in parallel using functional or relational
operations. Each dataset also has an untyped view called a DataFrame,
which is a Dataset of Row
RDD:
RDD represents an immutable,partitioned collection of elements that
can be operated on in parallel
Also, it is said the difference between them:
The major difference is, dataset is collection of domain specific
objects where as RDD is collection of any object. Domain object part
of definition signifies the schema part of dataset. So dataset API is
always strongly typed and optimized using schema where RDD is not.
I have two questions here;
what does it mean dataset is collection of domain specific objects while RDD is collection of any object,Given a case class Person, I thought DataSet[Person] and RDD[Person] are both collection of domain specific objects
dataset API is always strongly typed and optimized using schema where RDD is not Why is it said that dataset API always strongly typed while RDD not? I thought RDD[Person] is also strong typed
Strongly typed Dataset (not DataFrame) is a collection of record types (Scala Products) which are mapped to internal storage format using so called Encoders, while RDD can store arbitrary serializable (Serializable or Kryo serializable object). Therefore as a container RDD is much more generic than Dataset.
Following:
. So dataset API is always strongly typed (...) where RDD is not.
is an utter absurd, showing that you shouldn't trust everything you can find on the Internet. In general Dataset API has significantly weaker type protections, than RDD. This is particularly obvious when working Dataset[Row], but applies to any Dataset.
Consider for example following:
case class FooBar(id: Int, foos: Seq[Int])
Seq[(Integer, Integer)]((1, null))
.toDF.select($"_1" as "id", array($"_2") as "foos")
.as[FooBar]
which clearly breaks type safety.
After reading few great articles (this, this and this) about Spark's DataSets, I finishing with next DataSet's performance benefits over RDD:
Logical and physical plan optimization;
Strict typization;
Vectorized operations;
Low level memory management.
Questions:
Spark's RDD also builds physical plan and can combine/optimize multiple transformations at the same stage. Then what is the benefit of DataSet over RDD?
From the first link you can see an example of RDD[Person]. Does DataSet have advanced typization?
What do they mean by "vectorized operations"?
As I understand, DataSet's low memory management = advanced serialization. That means off-heap storage of serializable objects, where you can read only one field of an object without deserialization. But how about the situation when you have IN_MEMORY_ONLY persistence strategy? Will DataSet serialize everything any case? Will it have any performance benefit over RDD?
Spark's RDD also builds physical plan and can combine/optimize multiple transformations at the same stage. Than what is the benefit of DataSet over RDD?
When working with RDD what you write is what you get. While certain transformations are optimized by chaining, the execution plan is direct translation of the DAG. For example:
rdd.mapPartitions(f).mapPartitions(g).mapPartitions(h).shuffle()
where shuffle is an arbitrary shuffling transformation (*byKey, repartition, etc.) all three mapPartitions (map, flatMap, filter) will be chained without creating intermediate objects but cannot be rearranged.
Compared to that Datasets use significantly more restrictive programming model but can optimize execution using a number of techniques including:
Selection (filter) pushdown. For example if you have:
df.withColumn("foo", col("bar") + 1).where(col("bar").isNotNull())
can be executed as:
df.where(col("bar").isNotNull()).withColumn("foo", col("bar") + 1)
Early projections (select) and eliminations. For example:
df.withColumn("foo", col("bar") + 1).select("foo", "bar")
can be rewritten as:
df.select("foo", "bar").withColumn("foo", col("bar") + 1)
to avoid fetching and passing obsolete data. In the extreme case it can eliminate particular transformation completely:
df.withColumn("foo", col("bar") + 1).select("bar")
can be optimized to
df.select("bar")
These optimizations are possible for two reasons:
Restrictive data model which enables dependency analysis without complex and unreliable static code analysis.
Clear operator semantics. Operators are side effects free and we clearly distinguish between deterministic and nondeterministic ones.
To make it clear let's say we have a following data model:
case class Person(name: String, surname: String, age: Int)
val people: RDD[Person] = ???
And we want to retrieve surnames of all people older than 21. With RDD it can be expressed as:
people
.map(p => (p.surname, p.age)) // f
.filter { case (_, age) => age > 21 } // g
Now let's ask ourselves a few questions:
What is the relationship between the input age in f and age variable with g?
Is f and then g the same as g and then f?
Are f and g side effects free?
While the answer is obvious for a human reader it is not for a hypothetical optimizer. Compared to that with Dataframe version:
people.toDF
.select(col("surname"), col("age")) // f'
.where(col("age") > 21) // g'
the answers are clear for both optimizer and human reader.
This has some further consequences when using statically typed Datasets (Spark 2.0 Dataset vs DataFrame).
Have DataSet got more advanced typization?
No - if you care about optimizations. The most advanced optimizations are limited to Dataset[Row] and at this moment it is not possible to encode complex type hierarchy.
Maybe - if you accept overhead of the Kryo or Java encoders.
What does they mean by "vectorized operations"?
In context of optimization we usually mean loop vectorization / loop unrolling. Spark SQL uses code generation to create compiler friendly version of the high level transformations which can be further optimized to take advantage of the vectorized instruction sets.
As I understand, DataSet's low memory management = advanced serialization.
Not exactly. The biggest advantage of using native allocation is escaping garbage collector loop. Since garbage collections is quite often a limiting factor in Spark this is a huge improvement, especially in contexts which require large data structures (like preparing shuffles).
Another important aspect is columnar storage which enables effective compression (potentially lower memory footprint) and optimized operations on compressed data.
In general you can apply exactly the same types of optimizations using hand crafted code on plain RDDs. After all Datasets are backed by RDDs. The difference is only how much effort it takes.
Hand crafted execution plan optimizations are relatively simple to achieve.
Making code compiler friendly requires some deeper knowledge and is error prone and verbose.
Using sun.misc.Unsafe with native memory allocation is not for the faint-hearted.
Despite all its merits Dataset API is not universal. While certain types of common tasks can benefit from its optimizations in many contexts you may so no improvement whatsoever or even performance degradation compared to RDD equivalent.
For DataFrame, it is easy to generate a new column with some operation using a udf with df.withColumn("newCol", myUDF("someCol")). To do something like this in Dataset, I guess I would be using the map function:
def map[U](func: (T) ⇒ U)(implicit arg0: Encoder[U]): Dataset[U]
You have to pass the entire case class T as input to the function. If the Dataset[T] has a lot of fields/columns, it would seem very inefficient to be passing the entire row if you just wanted to make one extra column by operating on one of the many columns of T. My question is, is Catalyst smart enough to be able to optimize this?
Is Catalyst smart enough to be able to optimize this?
tl;dr No. See SPARK-14083 Analyze JVM bytecode and turn closures into Catalyst expressions.
There's currently no way Spark SQL's Catalyst Optimizer know what you do in your Scala code.
Quoting SPARK-14083:
One big advantage of the Dataset API is the type safety, at the cost of performance due to heavy reliance on user-defined closures/lambdas. These closures are typically slower than expressions because we have more flexibility to optimize expressions (known data types, no virtual function calls, etc). In many cases, it's actually not going to be very difficult to look into the byte code of these closures and figure out what they are trying to do. If we can understand them, then we can turn them directly into Catalyst expressions for more optimized executions.
And there's even your case mentioned:
df.map(_.name) // equivalent to expression col("name")
As you can see it's still open and I doubt anyone works on this currently.
What you could do to help Spark Optimizer is to select that one column and only then use map operator with a one-argument UDF.
That would certainly match your requirements of not passing the entire JVM object to your function, but would not get rid of this slow deserialization from an internal row representation to your Scala object (that would land on the JVM and occupy some space until a GC happens).
I tried to figure myself since I could not find a response anywhere.
Let's have a dataset which contains case classes with multiple fields:
scala> case class A(x: Int, y: Int)
scala> val dfA = spark.createDataset[A](Seq(A(1, 2)))
scala> val dfX = dfA.map(_.x)
Now if we check the optimized plan we get the following:
scala> val plan = dfX.queryExecution.optimizedPlan
SerializeFromObject [input[0, int, true] AS value#8]
+- MapElements <function1>, obj#7: int
+- DeserializeToObject newInstance(class A), obj#6: A
+- LocalRelation [x#2, y#3]
According to the more verbose plan.toJSON the DeserializeToObject step assumes both x and y to be present.
As you proof take for example the following snippet which uses reflection instead of directly touching the fields of A which still works.
val dfX = dfA.map(
_.getClass.getMethods.find(_.getName == "x").get.invoke(x).asInstanceOf[Int]
)
I created model using H2O's Sparkling Water. And now I'd like to apply it to huge Spark DF (populated with sparse vectors). I use python and pyspark, pysparkling. Basically I need to do map job with model.predict() function inside. But copying data into H2O context is huge overhead and not an option. What I think I gonna do is, extract POJO (Java class) model from h2o model and use it to do map in dataframe. My questions are:
Is there a better way?
How to write pyspark wrapper for java class, from which I intend to use only one method .score(double[] data, double[] result)
How to maximally reuse wrappers from Spark ML library?
Thank you!
In this case, you can:
1) use h2o.predict(H2OFrame) method to generate prediction, but you need to transform RDD to H2OFrame. It is not the perfect solution...however, for some cases, it can provide reasonable solution.
2) switch to JVM and call JVM directly via Spark's Py4J gateway
This is not fully working solution right now, since the method score0 needs to accept non-primitive types on H2O side and also to be visible (right now it is protected),
but at least idea:
model = sc._jvm.water.DKV.getGet("deeplearning.model")
double_class = sc._jvm.double
row = sc._gateway.new_array(double_class, nfeatures)
row[0] = ...
...
row[nfeatures-1] = ...
prediction = model.score0(row)
I created JIRA improvement for this case https://0xdata.atlassian.net/browse/PUBDEV-2726
However, workaround is to create a Java wrapper around model which would
expose right shape of score0 function:
class ModelWrapper extends Model {
public double[] score(double[] row) {
return score0(row)
}
}
Please see also hex.ModelUtils: https://github.com/h2oai/sparkling-water/blob/master/core/src/main/scala/hex/ModelUtils.scala
(again you can call them directly via Py4J gateway exposed by Spark)