How to get count of documents that would be added if one selects another aggregation options of an array-field in elastic search - search

Question

Let's say we have four documents with a tags field. It can contain multiple strings, let's say foo, bar and baz.
docA.tags = ['foo']
docB.tags = ['bar']
docC.tags = ['foo', 'bar']
docD.tags = ['foo', 'baz']
I query the docs using aggregations so I get the four documents and a list of three buckets with the count that matches the specific tag.
buckets = [
{key: 'bar', doc_count: 2}, // docB, docC
{key: 'foo', doc_count: 3}, // docA, docC, docD
{key: 'baz', doc_count: 1} // docD
]
If I now run another query and add one of those tags – lets say foo – as a terms-filter to the query, I only get the docs (docA, docC, docD) that have this tag. That's what I want.
But I also get another list of possible aggregations with updated counts.
buckets = [
{key: 'bar', doc_count: 1}, // docC
{key: 'baz', doc_count: 1}, // docD
]
But these counts don't really match what's happening. They reflect the count of documents that match both of the tags, the one I selected in the first place (foo) AND the one of the bucket (bar or baz).
But if I then select a second tag – let's say baz – I get documents that have been tagged with foo OR baz. That's because I use the terms filter.
So what I really want is this
buckets = [
{key: 'bar', doc_count: 1}, //docB
{key: 'baz', doc_count: 0},
]
How can I achieve that the counts are appropriate. They should reflect the count of documents that would be added if I select the second tag. An example of this is here.
I already tried to use post_filter but that always gives me the first result. Than a min_doc_count-flag to the aggs, but this only shows me the combinations that would result in count=0.
I have a solution for this, but it seem pretty complicated to me. For this I would have to run another request for each aggregation where I invert the filter criteria. So in the upper example I have to make a query to all docs that don't have the tag foo and match the rest of the query. The aggregation results would be exactly what I needed.

Answer 1

It sounds like you're trying to do something a little atypical for facets/aggregations.
(However, it's not invalid... it makes a lot of sense to understand how the size of your selection will change through the application of a filter)
What I think you're asking for is:
Display results for: QUERY AND FILTER
Display term aggregation counts for: QUERY NOT FILTER
You mentioned you're doing subsequent request(s) for counts? You should be able to construct this aggregation request inside your main search request.
Structurally it's:
match: (QUERY) or match_all
aggregations:
filter: { not: (FILTER) }
aggregations: { terms: ... }
post_filter: (FILTER)
That post_filter is executed after the aggregations are calculated (but still applied to the search results) so your results will be what you expect.
The aggregations are working in the scope of the search query alone. (The postfilter has not been applied yet.)
The filter aggregation excludes all documents matching FILTER from the search query results before the Terms Aggregation calculates the counts.
(giving you the left outside edge of the Venn shown above, but just for the counts)