SparkSQL joining parent/child datasets - apache-spark

Question

I am using SparkSQL 2.2.0 to load data from Cassandra and index it Elasticsearch. The data I have consists of customers (first table people) and orders (second table orders).
Table orders has a column person_id that points to the the corresponding customer.
My need is to query (and index later in Elasticsearch) the people table and the orders so I can have for each customer the number of orders she purchased.
The easiest approach I figured out is to read the two tables into org.apache.spark.sql.Dataset<Row>s and make a join on the person_id column. Then I groupBy(person_id).
That gives me a Dataset with two columns: person_id and count which I am obliged to join back with the people table so I can have the count with the other person data.
Dataset<Row> peopleWithOrders = people.join(orders, people.col("id").equalTo(orders.col("person_id")), "left_outer");
Dataset<Row> peopleOrdersCounts = peopleWithOrders.groupBy("id").count().withColumnRenamed("id", "personId");
Dataset<Row> peopleWithOrderCounts = people.join(personsOrdersCounts, people.col("id").equalTo(peopleOrdersCounts.col("personId")), "left_outer")
.withColumnRenamed("count", "nbrOfOrders")
.select("id", "name", "birthDate", "nbrOfOrders");
The people table has 1_000_000 rows and the orders one 2_500_000. Each customer has 2 or 3 orders.
I am using a MAC Book pro, with a 2,2 GHz Intel Core i7 Processor and 16 GB 1600 MHz DDR3 Memory. All of Cassandra, Spark 2.2 master and (single) worker are on the same machine.
These 3 joins take 15 to 20 seconds.
My question is: is there some room for performance gains. Do Window Aggregate Functions have benefits since I see ShuffleMapTask in the logs.
Thanks in advance

Answer 1

I think the first step is unnecessary. You could just do this:
Dataset<Row> peopleOrdersCounts = orders.groupBy("person_id").count();
Dataset<Row> peopleWithOrderCounts = people.join(personsOrdersCounts, people.col("id").equalTo(peopleOrdersCounts.col("personId")), "left_outer")
.withColumnRenamed("count", "nbrOfOrders")
.select("id", "name", "birthDate", "nbrOfOrders");
I hope this helps.