How to link/join multiple Lucene docs by AND operation - search

Question

I am beginner to lucene. Now I am blocked because of a search issue. We are developing an API to use lucene as search engine for our application and have to make lot of queries with different conditions as joined.
We store many entities into lucene as individual documents.
Each entity comes as number of records and stored into lucene as individual docs. Added below a sample structure of data,
Serial no. 1 --> 16 are docs into lucene.
1) "id": "1","sendr_name": "**sender1**", "recip_name": "**recipient1**", "subject": "**subject1**"
2) "id": "1","attachment": "**attachment1**"
3) "id": "1","domain": "**domain1**", "ip": "ip1"
5) "id": "1","mid": "**mid1**"
6) "id": "1","type": "type1"
7) "id": "2","sendr_name": "sender1", "recip_name": "recipient1", "subject": "subject1"
8) "id": "2","attachment": "attachment2"
9) "id": "2","domain": "domain1", "ip": "ip2"
10) "id": "2","mid": "mid2"
11) "id": "2","type": "type2"
12) "id": "3","sendr_name": "sender1", "recip_name": "recipient3", "subject": "subject3"
13) "id": "3","attachment": "attachment3"
14) "id": "3","domain": "domain1", "ip": "ip3"
15) "id": "3","mid": "mid3"
16) "id": "3","type": "type3"
Note : serial no. 1-16 are documents for different entities and field "id" get generated internally , so id value cannot use as query value by user.
My need is to extract specific entity or entities on specific condition.
+sendr_name:sender1 + recip_name:recipient1 +subject:subject1 +attachment:attachment1 +domain:domain1 +mid:mid1
This is to get an entity info(1-6 docs for an entity).
But above query fails to return result because attachment, mid and domain in different docs.
Is there any way that we can span AND condition to multiple docs? or is there anyway that we can join query on a field like doc1.id = doc2.id?
I request you all to provide your suggestions or help to solve this issue.

Answer 1

First of all, with plain Lucene, its not recommended to store heterogeneous documents in same index as that can have multitude of other problems in long run and other infrastructure problems.
Go through this SO Answer. You better use other top level techs like SOLR or ElasticSearch for that matter which are better capable to handle scenario that you describe.
You have not shown any code so its not clear if you are using Java or .NET or Lucene API version.
I am using Lucene 6.0 with Java and I think, its achievable with - BooleanQuery as top level container.
public static BooleanQuery.Builder buildQuery(final SearchBean searchBean) {
BooleanQuery.Builder finalQuery = new BooleanQuery.Builder();
finalQuery.add(buildDoc1Query(searchBean).build(), Occur.SHOULD);
finalQuery.add(buildDoc2Query(searchBean).build(), Occur.SHOULD);
....
....
return finalQuery;
}
i.e. first you build queries for each entity type depending on what all needed to be searched. SearchBean is a POJO that has all the searchable fields for all doc types combined.
private static BooleanQuery.Builder buildDoc1Query(SearchBean searchBean ) {
BooleanQuery.Builder doc1MatchQuery = new BooleanQuery.Builder();
if (StringUtils.isNotEmpty(searchBean.getSender_name())) {
doc2MatchQuery.add(new BoostQuery(new TermQuery(new Term(AppConstants.SENDER_NAME, searchBean.getSender_name())), MatchingBooster.SENDER_NAME), BooleanClause.Occur.MUST);
}
if (StringUtils.isNotEmpty(searchBean.getRecip_name())) {
doc2MatchQuery.add(new BoostQuery(new TermQuery(new Term(AppConstants.RECIP_NAME, searchBean.getRecip_name()())), MatchingBooster.RECIP_NAME), BooleanClause.Occur.MUST);
}
....
....
return doc2MatchQuery;
}
StringUtils is coming from Apache Commons library.
AppConstants contains indexed field names.
What is important here is - BooleanClause.Occur.MUST in child queries and Occur.SHOULD in master and that way you group child queries into one master query.
So you will get something like - (+sendr_name:sender1 + recip_name:recipient1 +subject:subject1) (+attachment:attachment1) ....so on.
Above will give you doc1 & doc2.
You can remove boosting part in above sample code ( BoostQuery) and can directly use TermQuery.
Hope it helps and let me know if I misunderstood your requirement.