Best suited data structure for prefix matching search - search

Question

I have to create a system of customer list (can be as large as 10million customers), each customer will have a unique ID and a unique ID consists of 10 letters, the first 3 are upper case letters and the last 7 are digits (ex: LQK0333208, HCK1646129,...). The system must perform two search operations in a fastest way (exact matching search and partial matching search):
For the exact matching search, users enter a complete Customer ID, and system displays details of the matching customer or an error message if there is no matching customer.
For the partial matching search, users enter several (at least 5 and at most 8) starting letters of Customer ID, and system displays details of the matching customers or an error message if there is no matching customer. If the number of matching customers is greater than 10, display only 10 of them.
So what the suitable data structure for this system? Currently, I am using AVL tree to handle the problem:
For exact matching search, I will perform a logarithmic search (left and right subtree): O(log(n)).
For partial matching search, I will perform a inorder search of the AVL Tree and check if each customer have the demanded prefix. This is a linear search: O(n).
But I want for partial matching search, the system will perform a better search in term of time complexity.
So any suggestion about the data structure is suitable for the system's requirement?
EDIT 1: I have tested the program with Trie Tree and Ternary Search Tree, but for larger dataset like (10 milions customer). There is no way I could store that in-memory data structure in the memory with a larger dataset like that. So any suggestions?
EDIT 2: I have tested the sorted array data structure and It works well with the data set of 10 million users. Actually, this was my first approach when I did not know anything about the Trie or Ternary tree. As far as I understand, first we will store all the customer in an array, then use some sort algorithms like quicksort to sort the array. Then perform binary search to search for the key, which is O(log(n)) to perform the search operation, quite good! But for a long term, when we need to add extra data to the array (not create the new one, but add to the array), for instance just one more customer, so adding the new element will take O(n) in worst case, as we need to find where to add and shift the element.
But for data structure like Trie or Ternary tree, when adding the new element, it might just require O(1) as we just need to traversal the tree to find the string. If we don't mind about the space complexity, I think trie or ternary tree are suit best for this project.

Answer 1

A suitable data structure for this is a trie. This is a tree of all prefixes, where each node (except the root) represents a character, and each possible path from root to a leaf will be a character sequence that corresponds to a valid ID.
A partial match means that there is a path from the root that ends in an internal node.
If implemented with an efficient child lookup, a match can in this particular use case be found in 10 steps. So if we consider 10 to be a constant, the match can be done in constant time, irrespective of how large (i.e. how wide) the tree is. This assumes that looking up a child by its character can be done in constant time (on average).
As in this particular use case the alphabet is limited (upper case only or digit only), a node can have at most 26 child entries, which could be stored in an array of that size, where the indexes map to the corresponding character. This will ensure constant time for stepping from a parent node to the relevant child node. Alternatively a hashing system can also be used (instead of an array with 26 slots).
Here is a demo implementation in JavaScript (using a plain object for the children, i.e. a "dictionary"):
class TrieNode {
constructor(data=null) {
this.children = {}; // Dictionary, <character, TrieNode>
this.data = data; // Non-null when this node represents the end of a valid word
}
addWord(word, data) {
let node = this; // the root of the tree
for (let ch of word) {
if (!(ch in node.children)) {
node.children[ch] = new TrieNode();
}
node = node.children[ch]; // Walk down the tree
}
node.data = data;
}
*getAllData() { // This method returns an iterator over all data in this subtree
if (this.data != null) yield this.data;
// Recursively yield all data in the children's subtrees
for (const child in this.children) yield* this.children[child].getAllData();
}
*find(prefix) { // This method returns an iterator over matches
let node = this;
// Find the node where this prefix ends:
for (let ch of prefix) {
if (!(ch in node.children)) return; // No matches
node = node.children[ch];
}
// Yield all data in this subtree
yield* node.getAllData();
}
}
class Customer {
constructor(id, name) {
this.id = id;
this.name = name;
}
toString() {
return this.name + " (" + this.id + ")";
}
}
// Demo
// Create some Customer data:
const database = [
new Customer('LQK0333208', 'Hanna'),
new Customer('LQK0333311', 'Bert'),
new Customer('LQK0339999', 'Joline'),
new Customer('HCK1646129', 'Sarah'),
new Customer('HCK1646130', 'Pete'),
new Customer('HCK1700012', 'Cristine')
];
// Build a trie for the database of customers
const trie = new TrieNode(); // The root node of the trie.
for (const customer of database) {
trie.addWord(customer.id, customer);
}
// Make a few queries
console.log("query: LQK0333");
for (const customer of trie.find("LQK0333")) console.log("found: " + customer);
console.log("query: HCK16461");
for (const customer of trie.find("HCK16461")) console.log("found: " + customer);
console.log("query: LQK0339999");
for (const customer of trie.find("LQK0339999")) console.log("found: " + customer);
console.log("query: LQK09 should not yield results");
for (const customer of trie.find("LQK09")) console.log("found: " + customer);
Sorted Array
Another approach is to store the Customer records in a sorted array. JavaScript has no such data structure, but splice is surprisingly fast in JavaScript, so you could just maintain a sorted order by inserting new entries in their sorted position. Binary search can be used to locate the index where to find or insert an entry:
class SortedArray {
constructor(keyField) {
this.arr = [];
this.keyField = keyField;
}
addObject(obj) {
const i = this.indexOf(obj[this.keyField]);
if (this.arr[i]?.[this.keyField] === obj[this.keyField]) throw "Duplicate not added";
this.arr.splice(i, 0, obj);
}
*find(prefix) { // This method returns an iterator over matches
for (let i = this.indexOf(prefix); i < this.arr.length; i++) {
const obj = this.arr[i];
if (!obj[this.keyField].startsWith(prefix)) return;
yield obj;
}
}
indexOf(key) {
let low = 0, high = this.arr.length;
while (low < high) {
const mid = (low + high) >> 1;
if (key === this.arr[mid][this.keyField]) return mid;
if (key > this.arr[mid][this.keyField]) {
low = mid + 1;
} else {
high = mid;
}
}
return low;
}
}
class Customer {
constructor(id, name) {
this.id = id;
this.name = name;
}
toString() {
return this.name + " (" + this.id + ")";
}
}
const database = [
new Customer('LQK0333208', 'Hanna'),
new Customer('LQK0333311', 'Bert'),
new Customer('LQK0339999', 'Joline'),
new Customer('HCK1646129', 'Sarah'),
new Customer('HCK1646130', 'Pete'),
new Customer('HCK1700012', 'Cristine')
];
const arr = new SortedArray("id");
for (const customer of database) {
arr.addObject(customer);
}
console.log("query: LQK0333");
for (const customer of arr.find("LQK0333")) console.log("found: " + customer);
console.log("query: HCK16461");
for (const customer of arr.find("HCK16461")) console.log("found: " + customer);
console.log("query: LQK0339999");
for (const customer of arr.find("LQK0339999")) console.log("found: " + customer);
console.log("query: LQK09 should not yield results");
for (const customer of arr.find("LQK09")) console.log("found: " + customer);