The optimal data structure for filtering for objects that match criteria - object

Question

I'll try to present the problem as generally as I can, and a response in any language will do.
Suppose there are a few sets of varying sizes, each containing arbitrary values to do with a category:
var colors = ["red", "yellow", "blue"] // 3 items
var letters = ["A", "B", "C", ... ] // 26 items
var digits = [0, 1, 2, 3, 4, ... ] // 10 items
... // each set has fixed amount of items
Each object in this master list I already have (which I want to restructure somehow to optimize searching) has properties that are each a selection of one of these sets, as such:
var masterList = [
{ id: 1, color: "red", letter: "F", digit: 5, ... },
{ id: 2, color: "blue", letter: "Q", digit: 0, ... },
{ id: 3, color: "red", letter: "Z", digit: 3, ... },
...
]
The purpose of the search would be to create a new list of acceptable objects from the master list. The program would filter the master list by given search criteria that, for each property, contains a list of acceptable values.
var criteria = {
color: ["red", "yellow"],
letter: ["A", "F", "P"],
digit: [1, 3, 5],
...
};
I'd like to think that some sort of tree would be most applicable. My understanding is that it would need to be balanced, so the root node would be the "median" object. I suppose each level would be defined by one of the properties so that as the program searches from the root, it would only continue down the branches that fit the search criteria, each time eliminating the objects that don't fit given the particular property for that level.
However, I understand that many of the objects in this master list will have matching property values. This connects them in a graphical manner that could perhaps be conducive to a speedy search.
My current search algorithm is fairly intuitive and can be done with just the master list as it is. The program
iterates through the properties in the search criteria,
with each property iterates over the master list, eliminating a number of objects that don't have a matching property, and
eventually removes all the objects that don't fit the criteria. There is surely some quicker filtering system that involves a more organized data structure.
Where could I go from here? I'm open to a local database instead of another data structure I suppose - GraphQL looks interesting. This is my first Stack Overflow question, so my apologies for any bad manners 😶

Answer 1

As I don't have the context of number of sets and also on the number of elements in the each set. I would suggest you some very small changes which will make things at-least relatively fast for you.
To keep things mathematical, I will define few terms here:
number of sets - n
size of masterlist - k
size of each property in the search criteria - p
So, from the algorithm that I believe you are using, you are doing n iterations over the search criteria, because there can be n possible keys in the search criteria.
Then in each of these n iterations, you are doing p iterations over the allowed values of that particular set. Finally, in each of these np iterations you are iterating over the master list, with k iterations ,and checking if this value of record should be allowed or not.
Thus, in the average case, you are doing this in O(npk) time complexity.
So, I won't suggest to change much here.
The best you can do is, change the values in the search criteria to a set (hashset) instead of keeping it as a list, and then iterate over the masterlist. Follow this Python code:
def is_possible(criteria, master_list_entry):
for key, value in master_list_entry.items(): # O(n)
if not key in criteria or value not in criteria[key]: # O(1) average
return False
return True
def search(master_list, criteria):
ans = []
for each_entry in master_list: # O(k)
if is_possible(criteria, each_entry): # O(n), check above
ans.append(each_entry)
return ans
Just call search function, and it will return the filtered masterlist.
Regarding the change, change your search criteria to:
criteria = {
color: {"red", "yellow"}, # This is a set, instead of a list
letter: {"A", "F", "P"},
digit: {1, 3, 5},
...
}
As, you can see, I have mentioned the complexities along with each line, thus we have reduced the problem to O(nk) in average case.