Compound index query - node.js

Question

I've got a query that i'm running that is pretty slow and am trying to do compound index on it, correctly, total newbie here. I've done the below index am just wondering if i've done it correctly, doesn't seem to have made a difference at all =/
Match.statics.getMatchesForDay = function (day, liveOnly, excludedAreas, excludedCompetitions, doneCallback) {
var include = "-_id match_id title date_utc date_iso _grouping match_info.period match_info.minute match_info.minute_extra match_info.full_time team_a team_b status time_utc";
var filter = {
date_utc: day,
score_coverage: true,
"_grouping._area.area_id": {
// Only include active areas
"$nin": excludedAreas
},
"_grouping._competition.competition_id": {
// Only include active competitions
"$nin": excludedCompetitions
}
};
if (liveOnly)
filter.status = "Playing";
this.find(filter).sort({
"_grouping._area.name": 1, //Sort by asc
"_grouping._competition.competition_id": 1, //Sort by asc
"date_iso": 1, //Sort by asc
}).select(include).exec(doneCallback);
};
Match.index({
date_utc: -1,
score_coverage: -1,
"_grouping._area.area_id": 1,
"_grouping._competition.competition_id": 1
});
Match.index({"_grouping._area.area_id": 1, "_grouping._competition.competition_id": 1, date_iso: 1});
My .explain() output:
{
"cursor" : "BtreeCursor date_utc_1_score_coverage_-1__grouping._area.area_id_1__grouping._competition.competition_id_1",
"isMultiKey" : false,
"n" : 358,
"nscannedObjects" : 358,
"nscanned" : 358,
"nscannedObjectsAllPlans" : 863,
"nscannedAllPlans" : 863,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 6,
"nChunkSkips" : 0,
"millis" : 7,
"indexBounds" : {
"date_utc" : [
[
"2015-03-15",
"2015-03-15"
]
],
"score_coverage" : [
[
true,
true
]
],
"_grouping._area.area_id" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
],
"_grouping._competition.competition_id" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
]
},
"server" : "xxxxx:27017",
"filterSet" : false
}
Unless it is working and i'm being picky, on a tiny collection it's around 200ms, on the larger ones 1-3s+. At the moment, the $nin is empty on both.
Thanks for looking.

Answer 1

That query won't be fast no matter what index you use. Let's look at the filter step-by-step:
var filter = {
date_utc: day,
Fine so far. An equality query can be indexed. Dates should have high enough selectivity. It's also likely to provide good data-locality.
score_coverage: true,
A boolean? Bad thing - indexes are essentially tree structures. If your datum is a single boolean, however, there's only two possible options: true and false. (low selectivity). This means that the tree has one true and one false branch that contain all the data 'below' it. That makes the tree become a linked list, essentially. It also destroys locality, because changing the value will have to rearrange entire sub trees. Move this to the end, and remove it from the index.
"_grouping._area.area_id": {
// Only include active areas
"$nin": excludedAreas
},
Indexes work like the letter marks in phone books. You're looking for "john doe"? Fine, look up the letter "D" visible from the outside (the index), then search for the "o" in "Do", and so forth. Suppose I gave you a phone book and asked you to find all the poeple that are NOT named "Doe". Does the index help? Not really, it would have been easy to skip "Doe". After all, you'll need to go through the whole thing anyway. Again, this is a problem of low selectivity.
"_grouping._competition.competition_id": {
// Only include active competitions
"$nin": excludedCompetitions
}
Same argument, $nin on large amounts of data is bad.
Now, the sorting:
this.find(filter).sort({
"_grouping._area.name": 1, //Sort by asc
"_grouping._competition.competition_id": 1, //Sort by asc
"date_iso": 1, //Sort by asc
}).select(include).exec(doneCallback);
Sorting is relatively expensive operation, so you'll want to ensure your indexes match the following rule because then the data is already sorted in the index:
equality criteria -- range criteria -- sort criteria
But now you have turned around the order of date-area-competitionId used in the equality and range criteria to area-competitionId-date for sorting.
Solving this requires understanding of the problem domain. I suggest you try to rearrange the data structure based on query selectivity / locality concerns. Queries should be simple.