search replace using Apache spark java - apache-spark

Question

Problem Statement:
We need to replace the Synonyms of the words in a row with its equivalent words (from large collection of synonym list ~40000+ Key Value pairs) on a large dataset(50000 rows).
Example:
Input
Allen jeevi pramod Allen Armstrong
sandesh Armstrong jeevi
harsha Nischay DeWALT
Synonym list (key value pair)
//We have 40000 entries
Key | Value
------------------------------------
Allen | Apex Tool Group
Armstrong | Columbus McKinnon
DeWALT | StanleyBlack
Above Synonym list has to be used on the Input and the Output should be as shown in the below format.
Expected Output
Apex Tool Group jeevi pramod Apex Tool Group Columbus McKinnon
sandesh Columbus McKinnon jeevi
harsha Nischay StanleyBlack
We have tried with 3 approaches all of them has its own limitations
Approach 1
Using UDF
public void test () {
List<Row> data = Arrays.asList(
RowFactory.create(0, "Allen jeevi pramod Allen Armstrong"),
RowFactory.create(1, "sandesh Armstrong jeevi"),
RowFactory.create(2, "harsha Nischay DeWALT")
);
StructType schema = new StructType(new StructField[] {
new StructField("label", DataTypes.IntegerType, false, Metadata.empty()),
new StructField("sentence", DataTypes.StringType, false, Metadata.empty())
});
Dataset<Row> sentenceDataFrame = spark.createDataFrame(data, schema);
List<Row> data2 = Arrays.asList(
RowFactory.create("Allen", "Apex Tool Group"),
RowFactory.create("Armstrong","Columbus McKinnon"),
RowFactory.create("DeWALT","StanleyBlack")
);
StructType schema2 = new StructType(new StructField[] {
new StructField("label2", DataTypes.StringType, false, Metadata.empty()),
new StructField("sentence2", DataTypes.StringType, false, Metadata.empty())
});
Dataset<Row> sentenceDataFrame2 = spark.createDataFrame(data2, schema2);
UDF2<String, String, Boolean> contains = new UDF2<String, String, Boolean>()
{
private static final long serialVersionUID = -5239951370238629896L;
#Override
public Boolean call(String t1, String t2) throws Exception {
return t1.contains(t2);
}
};
spark.udf().register("contains", contains, DataTypes.BooleanType);
UDF3<String, String, String, String> replaceWithTerm = new UDF3<String,
String, String, String>() {
private static final long serialVersionUID = -2882956931420910207L;
#Override
public String call(String t1, String t2, String t3) throws Exception {
return t1.replaceAll(t2, t3);
}
};
spark.udf().register("replaceWithTerm", replaceWithTerm, DataTypes.StringType);
Dataset<Row> joined = sentenceDataFrame.join(sentenceDataFrame2, callUDF("contains", sentenceDataFrame.col("sentence"), sentenceDataFrame2.col("label2")))
.withColumn("sentence_replaced", callUDF("replaceWithTerm", sentenceDataFrame.col("sentence"), sentenceDataFrame2.col("label2"), sentenceDataFrame2.col("sentence2")))
.select(col("sentence_replaced"));
joined.show(false);
}
`
Input
Allen jeevi pramod Allen Armstrong
sandesh Armstrong jeevi
harsha Nischay DeWALT
Expected Output
Apex Tool Group jeevi pramod Apex Tool Group Columbus McKinnon
sandesh Columbus McKinnon jeevi
harsha Nischay StanleyBlack
Actual Output
Apex Tool Group jeevi pramod Apex Tool Group Armstrong
Allen jeevi pramod Allen Columbus McKinnon
sandesh Columbus McKinnon jeevi
harsha Nischay StanleyBlack
Issue with approach 1, if there are multiple synonym keys in the input dataset, that many rows are getting created as shown in the above example output.
Expected only one row with all the replacement
Approach 2.
Using ImmutableMap with replace function: Here we kept key and values pair in hashmap within ImmutableMap function, we called replace function to replace all the things
but if a row contains multiple keys then it ignores complete row without replacing single key…
try {
JavaSparkContext sc = new JavaSparkContext(new SparkConf().setAppName("SparkJdbcDs").setMaster("local[*]"));
SQLContext sqlContext = new SQLContext(sc);
SparkSession spark = SparkSession.builder()
.appName("JavaTokenizerExample").getOrCreate();
HashMap<String, String> options = new HashMap<String, String>();
options.put("header", "true");
Dataset<Row> dataFileContent = sqlContext.load("com.databricks.spark.csv", options);
dataFileContent=dataFileContent.withColumn("ManufacturerSource", regexp_replace(col("ManufacturerSource"),"[^a-zA-Z0-9\\s+]",""));
dataFileContent= dataFileContent.na().replace("ManufacturerSource",ImmutableMap.<String, String>builder()
.put("Allen", "Apex Tool Group"),
.put("Armstrong","Columbus McKinnon"),
.put("DeWALT","StanleyBlack")
//Here we have 40000 entries
.build()
);
dataFileContent.show(10,false);
} catch (Exception e) {
e.printStackTrace();
}
Here is the sample code and output:
Input
Allen jeevi pramod Allen Armstrong
sandesh Armstrong jeevi
harsha Nischay DeWALT
Expected Output
Apex Tool Group jeevi pramod Apex Tool Group Columbus McKinnon
sandesh Columbus McKinnon jeevi
harsha Nischay StanleyBlack
Actual Output
Allen jeevi pramod Allen Armstrong
sandesh Columbus McKinnon jeevi
harsha Nischay StanleyBlack
Approach 3
Using replace all within UDF
public static void main(String[] args) {
JavaSparkContext sc = new JavaSparkContext(new SparkConf().setAppName("JoinFunctions").setMaster("local[*]"));
SQLContext sqlContext = new SQLContext(sc);
SparkSession spark = SparkSession.builder().appName("StringSimiliarityExample").getOrCreate();
Dataset<Row> sourceFileContent = sqlContext.read()
.format("com.databricks.spark.csv")
.option("header", "true")
.load("source100.csv");
sourceFileContent.show(false);
StructType schema = new StructType(new StructField[] {
new StructField("label", DataTypes.IntegerType, false,
Metadata.empty()),
new StructField("sentence", DataTypes.StringType, false,
Metadata.empty()) });
Dataset<Row> sentenceDataFrame = spark.createDataFrame(data, schema);
UDF1 mode = new UDF1<String, String>() {
public String call(final String types) throws Exception {
return types.replaceAll("Allen", "Apex Tool Group")
.replaceAll("Armstrong","Columbus McKinnon")
.replaceAll("DeWALT","StanleyBlack")
//40000 more entries.....
}
};
sqlContext.udf().register("mode", mode, DataTypes.StringType);
sentenceDataFrame.createOrReplaceTempView("people");
Dataset<Row> newDF = sqlContext.sql("SELECT mode(sentence), label FROM people").withColumnRenamed("UDF(sentence)", "sentence");
newDF.show(false);
}
Output
Stackoverflow exception.
Here, we are getting stackoverflow exception. Because it resembles recursive function call.
Kindly, let us know if there are any other innovative approaches that can help to resolve this issue.

Answer 1

Neither will work, since you always have the issue of substring matches. For example:
ABC -> DE
ABCDE -> ABC
With text "ABCDEF HIJ KLM" what will the output be? It should be the same as the input, but your approach will at best output "DEDEF HIJ KLM" and at worst you will do a double replacement and get "DEF HIJ KLM". Either case is incorrect.
You could improve this by adding boundaries to replacements, perhaps using regex. A better way however would be to first tokenize your input correctly, apply token replacement (which can be exact match), and then un-tokenize back to original format. This may be as simple as splitting by space, but you should give proper though as to what token boundaries may exist. (Stops, hyphens, etc).