How can I count occurrences of element in dataframe array? - apache-spark

Question

I have a dataframe that looks like this:
df = spark.sql("""
SELECT list
FROM categories
""")
df.show()
list
1,1,1,2,2,apple
apple,orange,1,2
And I want result something like this
list
frequency_count
1
4
2
3
apple
2
orange
1
This is what I tried.
count_df = df.withColumn('count', F.size(F.split('list', ',')))
count_df.show(truncate=False)
df.createOrReplaceTempView('tmp')
freq_sql = """
select list,count(*) count from
(select explode(flatten(collect_list(split(list, ',')))) list
from tmp)
group by list
"""
freq_df = spark.sql(freq_sql)
freq_df.show(truncate=False)
And I'm getting this error
AnalysisException: cannot resolve 'split(df.`list`, ',', -1)' due to
data type mismatch: argument 1 requires string type, however,
'df.`list`' is of array<string> type.;

Answer 1

You are currently trying to flatten a single list type value ,however flatten function for an array expects an array of arrays:
flatten(arrayOfArrays) - Transforms an array of arrays into a single array.
Hence the resultant error you are facing
You would need to explode the list before you try to split it and finally exploding it to transform the elements as rows and finally groupBy to get the required count
Data Preparation
sparkDF = sql.createDataFrame(
[
(["1,1,1,2,2,apple"],),
(["apple,orange,1,1"],),
],
("list",)
)
Explode
sparkDF.createOrReplaceTempView("dataset")
sql.sql("""
SELECT
explode(list) as exploded
,list
FROM dataset
""").printSchema()
root
|-- exploded: string (nullable = true)
|-- list: array (nullable = true)
| |-- element: string (containsNull = true)
+----------------+------------------+
| exploded| list|
+----------------+------------------+
| 1,1,1,2,2,apple| [1,1,1,2,2,apple]|
|apple,orange,1,1|[apple,orange,1,1]|
+----------------+------------------+
Group By
sql.sql("""
SELECT
exploded
,count(*) as count
FROM (
SELECT
EXPLODE(SPLIT(list,",")) as exploded
FROM (
SELECT
EXPLODE(list) as list
FROM dataset
)
)
GROUP BY 1
""").show()
+--------+-----+
|exploded|count|
+--------+-----+
| orange| 1|
| apple| 2|
| 1| 5|
| 2| 2|
+--------+-----+