Spark / PySpark: Group by any item of nested list - apache-spark

Question

I´m still new to Spark / PySpark and have the following question. I got a nested list with ID´s in it:
result = [[411, 44, 61], [42, 33], [1, 100], [44, 42]]
The thing I´m trying to achieve is, that if any item of sublist matches an item in another sublist the both should be merged. The result should look like this:
merged_result = [[411, 44, 61, 42, 33, 44, 42], [1,100]]
The first list in "result" matches with the fourth list. The fourth list matches with the second, so all 3 should be merged into one list. The third list doesn´t match with any other list, so it stays the same.
I could achieve this by writing loops with Python.
result_after_matching = []
for i in result:
new_list = i
for s in result:
if any(x in i for x in s):
new_list = new_list + s
result_after_matching.append(set(new_list))
#merged_result = [[411, 44, 61, 42], [42,33,44], [1, 100], [44,42,33,411,61]]
As this is not the desired output I would need to repeat the loop and do another set() overt the "merged_result")
set([[411,44,61,42,33], [42,33,44,411,61],[1,100], [44,42,33,411,61]])
-> [[411, 44, 61, 42, 33], [1,100]]
As the list of lists, and the sublists gets bigger and bigger by time as new data will be incoming, this will not be the function to use.
Can anyone tell me if there is a function, in Spark / Pyspark, to match / merge / groupby / reduce these nested lists much easier and faster?!
Thanks a lot in advance!
MG

Answer 1

Most rdd or dataframe based solutions will probably be fairly inefficient. This is because the nature of your problem requires every element of your data set to be compared to every other element potentially multiple times. This makes it so that distributing the work across a cluster is at best inefficient.
Perhaps a different way to do this would be to reformulate this as a graph problem. If you treat each item in a list as a node on a graph, and each list as a subgraph, then the connected components of a parent graph constructed from the subgraphs will be the desired result. Here is an example using the networkx package in python:
import networkx as nx
result = [[411, 44, 61], [42, 33], [1, 100], [44, 42]]
g = nx.DiGraph()
for subgraph in result:
g.add_path(subgraph)
u = g.to_undirected()
output=[]
for component in nx.connected_component_subgraphs(u):
output.append(component.nodes())
print(output)
# [[33, 42, 411, 44, 61], [1, 100]]
This should be fairly efficient, but if your data is very large it will make sense to use a more scalable graph analysis tool. Spark does have a graph processing library called GraphX:
https://spark.apache.org/docs/latest/graphx-programming-guide.html
Unfortunately the pyspark implementation is lagging behind a bit. So if you intend to use something like this, you might be stuck using scala-spark or a different framework entirely for right now.

Answer 2

I think you can use aggregate action from RDD. Below I'm putting example implementation in Scala. Please note that I've used recursion, to make it more readable, but to improve performance it's good idea to reimplement those functions.
def overlap(s1: Seq[Int], s2: Seq[Int]): Boolean =
s1.exists(e => s2.contains(e))
def mergeSeq(s1: Seq[Int], s2: Seq[Int]): Seq[Int] =
s1.union(s2).distinct
def mergeSeqWithSeqSeq(s: Seq[Int], ss: Seq[Seq[Int]]): Seq[Seq[Int]] = ss match {
case Nil => Seq(s)
case h +: tail =>
if(overlap(h, s)) mergeSeqWithSeqSeq(mergeSeq(h, s), tail)
else h +: mergeSeqWithSeqSeq(s, tail)
}
def mergeSeqSeqWithSeqSeq(s1: Seq[Seq[Int]], s2: Seq[Seq[Int]]): Seq[Seq[Int]] = s1 match {
case Nil => s2
case h +: tail => mergeSeqWithSeqSeq(h, mergeSeqSeqWithSeqSeq(tail, s2))
}
val result = rdd
.aggregate(Seq.empty[Seq[Int]]) (
{case (ss, s) => mergeSeqWithSeqSeq(s, ss)},
{case (s1, s2) => mergeSeqSeqWithSeqSeq(s1, s2)}
)