Filtering computed values with Elasticsearch - node.js

Question

So we have a People schema and a Project schema, both of which are used on separate pages where you can filter the results. We are at a point where we are realising that we need to filter something like current projects, but on the PEOPLE list. However the data for that lives in Projects, and we are having trouble in finding a clean way to cross-reference the schemas and filter on a computed value.
Eg. Filter the list of people by projects that are currently active (ie. the current time).
There doesn't seem to be any way to do this in ES - my current option is stitching these together in the front-end and doing another "filter" there, but that feels hacky.
Displaying the data on the People list is no issue, it's the filtering.
Has anyone run into this situation before, and how did you resolve it? Appreciate any insight. THanks!

Answer 1

If I understand things correctly, you have two separate indices for people and projects.
Unfortunately, the only way to do things right is to denormalize your data in Elasticsearch and include projects on the people list.

Answer 2

You could also use Parent-child relationships.
Please read this excellent blog posts:
https://rockset.com/blog/can-i-do-sql-style-joins-in-elasticsearch/
and then Parent-Child Modeling in Elasticsearch https://blog.mimacom.com/parent-child-elasticsearch/
The previous link show examples in ElasticSearch 5.6 and 6. I'll provide examples that works in ES 7:
According to the post:
There are four common approaches to managing data in Elasticsearch:
Denormalization, Application-side joins, Nested objects, Parent-child relationships
One-to-one relationships: Object mapping
One-to-many relationships: Nested documents and the parent-child model
Many-to-many relationships: Denormalizing and application-side joins
Challenges with Parent-Child Relationships
Queries are more expensive and memory-intensive because of the join operation. There is an overhead to parent-child constructs, since they are separate documents that must be joined at query time. Need to ensure that the parent and all its children exist on the same shard. Storing documents with parent-child relationships involves implementation complexity.
I'll provide an example ready to execute on ES 7
curl -XDELETE "localhost:9200/project"
curl -XPUT "localhost:9200/project" \
-H 'Content-Type: application/json' \
-d'{
"settings": {
"number_of_shards": 1, "number_of_replicas": 0
},
"mappings": {
"properties": {
"project": { "type": "text" },
"people": { "type": "text" },
"active": { "type": "boolean" },
"project_relations": {
"type": "join",
"relations": {
"project": "people"
}
}
}
}
}'
curl -XPUT "localhost:9200/project/_bulk" \
-H 'Content-Type: application/json' \
-d'
{"index":{"_id":1}}
{"name":"ElasticSearch upgrade","project_relations":{"name":"project"}, "active": true}
{"index":{"_id":2}}
{"name":"Website","project_relations":{"name":"project"}, "active": true}
{"index":{"_id":3}}
{"name":"SQL Server migration","project_relations":{"name":"project"}, "active": false}
'
curl -XPUT "localhost:9200/project/_doc/4?routing=1" \
-H 'Content-Type: application/json' \
-d'
{"people":"Evaldas Buinauskas","project_relations":{"name":"people","parent":1}}
'
curl -XPOST "localhost:9200/project/_doc/5?routing=1" \
-H 'Content-Type: application/json' \
-d'
{"people":"xynon","project_relations":{"name":"people","parent":1}}
'
The routing must correspond to the project id
curl -XPOST "localhost:9200/project/_doc/6?routing=2" \
-H 'Content-Type: application/json' \
-d'
{"people":"Harry scherer","project_relations":{"name":"people","parent":2}}
'
curl -XPOST "localhost:9200/project/_doc/7?routing=2" \
-H 'Content-Type: application/json' \
-d'
{"people":"Darrel Simon","project_relations":{"name":"people","parent":2}}
'
curl -XPOST "localhost:9200/project/_doc/8?routing=3" \
-H 'Content-Type: application/json' \
-d'
{"people":"Juan Carlos Alafita","project_relations":{"name":"people","parent":3}}
'
Search people working on ElasticSearch upgrade and project is active
curl -XGET "localhost:9200/project/_search" \
-H 'Content-Type: application/json' \
-d'
{
"query": {
"has_parent": {
"parent_type": "project",
"query": { "bool": {"must": [ {"match": { "active": true } }, {"match": { "name": "ElasticSearch upgrade" } } ] }}
}
}
}' | jq
Results:
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "project",
"_type": "_doc",
"_id": "4",
"_score": 1,
"_routing": "1",
"_source": {
"people": "Evaldas Buinauskas",
"project_relations": {
"name": "people",
"parent": 1
}
}
},
{
"_index": "project",
"_type": "_doc",
"_id": "5",
"_score": 1,
"_routing": "1",
"_source": {
"people": "xynon",
"project_relations": {
"name": "people",
"parent": 1
}
}
}
]
}
}
Search people working on SQL Server migration and project inactive
curl -XGET "localhost:9200/project/_search" \
-H 'Content-Type: application/json' \
-d'
{
"query": {
"has_parent": {
"parent_type": "project",
"query": { "bool": {"must": [ {"match": { "active": false } }, {"match": { "name": "SQL Server migration" } } ] }}
}
}
}
' | jq
curl -XGET "localhost:9200/project/_search" \
-H 'Content-Type: application/json' \
-d'
{
"query": {
"has_parent": {
"parent_type": "project",
"query": { "match_all": {} }
}
}
}'