Search text for matching large number of strings - search

Question

I have a use case where i have to check if a text which i receive as input contains any of 3 million strings i have.
I tried regex matching but once the list of strings crossed 50k the performance is very bad
i am doing this for each word in the search list
inText = java.util.regex.Pattern.compile("\\b" + findStr + "\\b",
java.util.regex.Pattern.CASE_INSENSITIVE).matcher(intext).replaceAll(repl);
I understand we can use search indexes like lucene, but i feel those primarily work to search a particular text from predefined text, but my use case is the opposite, i need to send a large text and check if any of the pre defined strings are there in the text

Answer 1

I think, you could take at it the other way around. Your predefined strings are documents stored in the inverted index, and your incoming text is a query, that you will test against your documents. Since predefined strings will not change much it will be very performant.
I prepared some Elasticsearch code, that will do the trick.
public void add(String string, String id) {
IndexRequest indexRequest = new IndexRequest(INDEX, TYPE, id);
indexRequest.source(string);
index(INDEX, TYPE, id, string);
}
#Test
public void scoring() throws Exception {
// adding your predefined strings
add("{\"str\":\"string1\"}", "1");
add("{\"str\":\"alice\"}", "2");
add("{\"str\":\"bob\"}", "3");
add("{\"str\":\"string2\"}", "4");
add("{\"str\":\"melanie\"}", "5");
add("{\"str\":\"moana\"}", "6");
refresh(); // otherwise we would not anything
indexExists(INDEX); // verifies that index exists
ensureGreen(INDEX); // ensures cluster status is green
// querying your text separated by space, if the hits length is bigger than 0, you're good
SearchResponse searchResponse = client().prepareSearch(INDEX).setQuery(QueryBuilders.termsQuery("str", "string1", "string3", "melani")).execute().actionGet();
SearchHit[] hits = searchResponse.getHits().getHits();
assertThat(hits.length, equalTo(1));
for (SearchHit hit: hits) {
System.out.println(hit.getSource());
}
}