How to map String to Seq<String> in Spark in Java - apache-spark

Question

I want to use my own tokenizer to tokenize text stored as Dataset<String>, and get Dataset<Seq<String>> (so I can pass it to CountVectorizer).
Expected input (/tmp/fulltext.txt):
t1 t2 t3
t4 t5
Expected output:
[t1, t2, t3]
[t4, t5]
The tokenizer I write is (basically the thing it does now is the same as Tokenizer shipped with Spark, but I'll need to rewrite it to support tokenization of Chinese text, so I cannot use the official Tokenizer):
public class Utils {
public static Seq<String> segment(String text) {
String[] array = text.split(" ");
List<String> tokens = new ArrayList<>();
for (String term : array) {
tokens.add(term.toLowerCase());
}
return JavaConverters
.asScalaIteratorConverter(tokens.iterator())
.asScala()
.toSeq();
}
}
The Spark application I'm trying to make is
public class TokenizeTest {
public static void main(String[] args) {
SparkSession spark = SparkSession
.builder()
.appName("Tokenize Test")
.getOrCreate();
Dataset<String> rawText = spark
.read()
.textFile("/tmp/fulltext.txt")
.cache();
Encoder<Seq> listEncoder = Encoders.bean(Seq.class);
// Compilation error
Dataset<Seq<String>> newText = rawText
.map((MapFunction<String, Seq<String>>) s -> Utils.segment(s), listEncoder);
newText.show();
spark.stop();
}
}
I'm a beginner of Spark, the above code is just what I think will work (after reading the official guide). But it turns out the code of TokenizeTest doesn't compile at all. Do you think there is a way to fix it?

Answer 1

Using Scala collections like this won't work. For once Seq is not Bean compatible, for second it is generic.
If you want split just use arrays with segement defined as:
public class Utils {
public static String[] segment(String text) {
return text.split(" ");
}
}
and TokenizeTest defined as:
public class TokenizeTest {
public static void main(String[] args) {
SparkSession spark = SparkSession
.builder()
.appName("Tokenize Test")
.getOrCreate();
Dataset<String> rawText = spark
.read()
.textFile("/path/to/file")
.cache();
Encoder<String []> listEncoder = spark.implicits().newStringArrayEncoder();
Dataset<String []> newText = rawText
.map((MapFunction<String, String []>) s -> Utils.segment(s), listEncoder);
newText.show();
spark.stop();
}
}
In practice though, you might consider either org.apache.spark.sql.functions.split or org.apache.spark.ml.feature.Tokenizer instead of reinventing the wheel.