Spark MLlib Association Rules confidence is greater than 1.0 - apache-spark

Question

I was using Spark 2.0.2 to extract some association rules from some data, while when I get the result, I found I have some strange rules, such as the followings：
【[MUJI,ROEM,西单科技广场] => Bauhaus ] 2.0
“2.0” is the confidence of the rule printed, isn't it the meaning of "the probability of antecedent to consequent" and should be less than 1.0?

Answer 1

KEY WORD: transactions != freqItemset
SOLUTIONS: Use spark.mllib.FPGrowth instead, it accepts a rdd of transactions and can automatically calculate freqItemsets.
Hello, I found it. The reason of this phenomenon is because my input FreqItemset data freqItemsets is wrong. Let's to into detail. I simply use three original transactions ("a"),("a","b","c"),("a","b","d"), the frequency of them are all the same 1.
At the beginning, I thought spark would auto calculate sub-itemset frequency, the only thing I need to do is to create freqItemsets like this (the official example show us):
val freqItemsets = sc.parallelize(Seq(
new FreqItemset(Array("a"), 1),
new FreqItemset(Array("a","b","d"), 1),
new FreqItemset(Array("a", "b","c"), 1)
))
Here is the reason why it make mistakes, AssociationRules's params is FreqItemset, not the transactions, so I made a wrong understand of these two definition.
According to the three transactions, the freqItemsets should be
new FreqItemset(Array("a"), 3),//because "a" appears three times in three transactions
new FreqItemset(Array("b"), 2),//"b" appears two times
new FreqItemset(Array("c"), 1),
new FreqItemset(Array("d"), 1),
new FreqItemset(Array("a","b"), 2),// "a" and "b" totally appears two times
new FreqItemset(Array("a","c"), 1),
new FreqItemset(Array("a","d"), 1),
new FreqItemset(Array("b","d"), 1),
new FreqItemset(Array("b","c"), 1)
new FreqItemset(Array("a","b","d"), 1),
new FreqItemset(Array("a", "b","c"), 1)
You can do this statistical work your self use the following code
val transactons = sc.parallelize(
Seq(
Array("a"),
Array("a","b","c"),
Array("a","b","d")
))
val freqItemsets = transactions
.map(arr => {
(for (i <- 1 to arr.length) yield {
arr.combinations(i).toArray
})
.toArray
.flatten
})
.flatMap(l => l)
.map(a => (Json.toJson(a.sorted).toString(), 1))
.reduceByKey(_ + _)
.map(m => new FreqItemset(Json.parse(m._1).as[Array[String]], m._2.toLong))
//then use freqItemsets like the example code
val ar = new AssociationRules()
.setMinConfidence(0.8)
val results = ar.run(freqItemsets)
//....
Simply we can use FPGrowth instead of "AssociationRules", it accepts rdd of transactions.
val fpg = new FPGrowth()
.setMinSupport(0.2)
.setNumPartitions(10)
val model = fpg.run(transactions) //transactions is defined in the previous code
That's all.