Using Apache Spark to find frequent contiguous - apache-spark

Question

How can I use Apache Spark to find contiguous sequences

Answer 1

Try taking inital string, splitting it into unique subsequences of different length, then broadcasting intial sequence over them and filtering matched. Something like this would work in a spark-shell
val s = "AATTGTGTGTGTGATTTTTTAATG" //your string
val s_broadcast = sc.broadcast(s) //broadcast version
val A = 2 // min length of substring
val B = 3 // max length of substring
val C = 3 // min support
val L = s.size //length of the string
sc.parallelize(
for{
i <- A to B
j <- 0 to (L - i)
} yield (j,i+j)
) // generating paris of substrings
.map{case(j,i)=>s_broadcast.value.substring(j,i)}
.distinct // if optimization is needed, this step is a place to start
.filter(x=>s_broadcast.value.indexOf(x*C)>=0)
.collect
.map(_*C)
EDITED:
as an after though - here's code which will return LONGEST substrings. The previous code has C fixed, this one tries longest.
val s = "AATTGTGTGTGTGTGATTTTTTAATG" //your string
val s_broadcast = sc.broadcast(s) //broadcast version
val A = 2 // min length of substring
val B = 3 // max length of substring
val C = 3 // min support
val L = s.size //length of the string
sc.parallelize(
for{
i <- A to B
j <- 0 to (L - i)
} yield (j,i+j)
) // generating paris of substrings
.map{case(j,i)=>s_broadcast.value.substring(j,i)}
.distinct // if optimization is needed, this step is a place to start
.flatMap(x=>
for{
v <- C to L/A
} yield x->v
) //making "AB"->3 pairs, which will result in search for "ABABAB"
.filter{case(x,v)=>s_broadcast.value.indexOf(x*v)>=0}
.groupByKey //grouping same substrings of different length
.map{case(k,v)=>k->v.max} //getting longer substring
.collect //bringing substring to the driver
.map{case(k,v)=>k*v}