How to Find all emoji from a PySpark Dataframe Column? - apache-spark

Question

I have a column in a spark data frame which has several messages. Here is a sample:
message = [
(1, "Sempre com #mariahcarey fazendo aquele aquecimento na voz antes dos shows. Quem lembra dessa? 🎤❤️"),
(2, "Happy Easter from the real bunny 💙🐰"),
(3, "Anakku aku udah diajak nonton malam mingguan kemarin😅🤣 tua😂 Haduhhh bener2 deh😂😂😂 #gadiiing #raffinagita1717")
]
rdd1 = sc.parallelize(message)
df=sqlContext.createDataFrame(rdd1,['id', 'message'])
I need to find all the emojies in the messages. Using the following code it is possible to find the first match:
import emoji
import re
emojis_list = map(lambda x: ''.join(x.split()), emoji.UNICODE_EMOJI.keys())
escape_list = '|'.join(re.escape(p) for p in emojis_list)
df.withColumn("emoji_in_post", fn.regexp_extract("message", escape_list, 0))
But I need all of them. So I tried to create a UDF using vanila python.
from pyspark.sql.types import ArrayType, StructType, StructField, StringType, IntegerType
import pyspark.sql.functions as fn
def find_all_emo(plain_text):
emo_list = re.findall(escape_list, plain_text)
return emo_list
search_all_emojis = fn.udf(lambda y: find_all_emo(y), ArrayType(StringType()))
But when applying that function to dataframe I am getting error.
TypeError: expected string or bytes-like object
If some one know the problem or has any better solution. Thanks in Advance.

Answer 1

So I found the problem. There is few rows when the value of message is null. So I had to extend the find all function.
def find_all_emo(plain_text):
if plain_text is None:
return None
emo_list = regex.findall(plain_text)
return emo_list
search_all_emojis = fn.udf(lambda y: find_all_emo(y), ArrayType(StringType()))
test = df.withColumn("emoji_in_post", search_all_emojis(fn.col("message")))
Now the output result is empty array or array with some emoji.