How to find most similar string using n-grams - rust

Question

I am trying to use n-grams to find the most similar string for each string within a list, currently I have this vector of strings
let arr = [
"Bilbo Baggins",
"Gandalf",
"Thorin",
"Balin",
"Kili",
"Fili",
"John",
"Frodo Baggins",
]
Using the following code I create the bigrams for each string and store them in a vector:
let arr = [
"Bilbo Baggins",
"Gandalf",
"Thorin",
"Balin",
"Kili",
"Fili",
"John",
"Frodo Baggins",
]
.iter()
.map(|elem|
elem
.len()
.rem(2)
.ne(&0)
.then_some(format!("{elem} "))
.unwrap_or(elem.to_string())
)
.map(|elem| elem.chars().array_chunks().collect::<Vec<[char; 2]>>())
.collect::<Vec<_>>();
Output:
[['B', 'i'], ['l', 'b'], ['o', ' '], ['B', 'a'], ['g', 'g'], ['i', 'n'], ['s', ' ']]
[['G', 'a'], ['n', 'd'], ['a', 'l'], ['f', ' ']]
[['T', 'h'], ['o', 'r'], ['i', 'n']]
[['B', 'a'], ['l', 'i'], ['n', ' ']]
[['K', 'i'], ['l', 'i']]
[['F', 'i'], ['l', 'i']]
[['J', 'o'], ['h', 'n']]
[['F', 'r'], ['o', 'd'], ['o', ' '], ['B', 'a'], ['g', 'g'], ['i', 'n'], ['s', ' ']]
Question is, how can I apply some sort of set logic to these vectors of bigrams to find the most similar string for each of the strings and get the somewhat the following output?:
'Bilbo Baggins' most similar string: 'Frodo Baggins'
'Gandalf' most similar string: None
'Thoring' most similar string: 'Balin'
'Balin' most similar string: 'Thorin'
'Kili' most similar string: 'Fili'
'Fili' most similar string: 'Kili'
'John' most similar string: None
'Frodo Baggins' most similar string: 'Bilbo Baggins'

Answer 1

There are many different algorithms to calculate distances between strings. The algorithm you are looking for with your bigrams is probably the cosine similarity function. It can be used to iterate over your bigrams and compute a value that represents the similarity between two vectors (or strings, in this case). It seems to favour matching to longer strings because there are more groups of characters repeated between them.
Here is an example of finding the closest names by cosine similarity:
use std::collections::HashSet;
const SIM_THRESHOLD: f32 = 0.22;
fn main() {
let arr = [
"Bilbo Baggins",
"Gandalf",
"Thorin",
"Balin",
"Kili",
"Fili",
"John",
"Frodo Baggins",
];
for name in arr {
let mut closest = (None, -1.0);
for other in arr.iter().filter(|&e| *e != name) {
let sim = str_diff(name, other);
if sim > closest.1 && sim > SIM_THRESHOLD {
closest = (Some(*other), sim);
}
}
println!(
"\"{}\" most similar string: {}",
name,
if let Some(name) = closest.0 {
format!("\"{}\"", name)
} else {
"None".to_string()
}
);
}
}
/// Returns the bigram cosine similarity between two strings. A `1` means the
/// strings are identical, while a `0` means they are completely different.
/// Returns NaN if both strings are fewer than 2 characters.
pub fn str_diff(a: &str, b: &str) -> f32 {
cos_sim(&ngram(&a, &b, 2))
}
// Returns the term frequency of `n` consecutive characters between two strings.
// The order of the terms is not guarenteed, but will always be consistent
// between the two returned vectors (order could be guarenteed with a BTreeSet,
// but that is slower).
fn ngram(s1: &str, s2: &str, n: usize) -> (Vec<u32>, Vec<u32>) {
let mut grams = HashSet::<&str>::new();
for i in 0..((s1.len() + 1).saturating_sub(n)) {
grams.insert(&s1[i..(i + n)]);
}
for i in 0..((s2.len() + 1).saturating_sub(n)) {
grams.insert(&s2[i..(i + n)]);
}
let mut q1 = Vec::new();
let mut q2 = Vec::new();
for i in grams {
q1.push(s1.matches(i).count() as u32);
q2.push(s2.matches(i).count() as u32);
}
(q1, q2)
}
// Returns the dot product of two slices of equal length. Returns an `Err` if
// the slices are not of equal length.
fn dot_prod(a: &[u32], b: &[u32]) -> Result<u32, &'static str> {
if a.len() != b.len() {
return Err("Slices must be of equal length");
}
let mut v = Vec::new();
for i in 0..a.len() {
v.push(a[i] * b[i]);
}
Ok(v.iter().sum())
}
// Returns the cosine similarity between two vectors of equal length.
// `S_c(A, B) = (A · B) / (||A|| ||B||)`
fn cos_sim((a, b): &(Vec<u32>, Vec<u32>)) -> f32 {
if a.len() != b.len() {
return f32::NAN;
}
let a_mag = (dot_prod(a, a).unwrap() as f32).sqrt();
let b_mag = (dot_prod(b, b).unwrap() as f32).sqrt();
// use `clamp` to constrain floating point errors within 0..=1
(dot_prod(a, b).unwrap() as f32 / (a_mag * b_mag)).clamp(0.0, 1.0)
}
"Bilbo Baggins" most similar string: "Frodo Baggins"
"Gandalf" most similar string: None
"Thorin" most similar string: "Balin"
"Balin" most similar string: "Bilbo Baggins"
"Kili" most similar string: "Fili"
"Fili" most similar string: "Kili"
"John" most similar string: None
"Frodo Baggins" most similar string: "Bilbo Baggins"
The Balin doesn't look quite right because cosine distance doesn't take into account string length. Another popular method is to find the Levenshtein distance (I used the Wagner Fischer algorithm to compute it), which is the number of insertions, deletions, or substitutions to transform one string into another
use std::cmp::min;
fn main() {
let arr = [
"Bilbo Baggins",
"Gandalf",
"Thorin",
"Balin",
"Kili",
"Fili",
"John",
"Frodo Baggins",
];
for name in arr {
let mut closest = (None, usize::MAX);
for other in arr.iter().filter(|&e| *e != name) {
let dist = distance(name, *other);
if dist < closest.1 && dist < min(name.len(), other.len()) {
closest = (Some(*other), dist);
}
}
println!(
"\"{}\" most similar string: {}",
name,
if let Some(name) = closest.0 {
format!("\"{}\"", name)
} else {
"None".to_string()
}
);
}
}
/// Calculates the Levenshtein Distance between 2 strings
fn distance(a: &str, b: &str) -> usize {
let a = a.chars().collect::<Vec<char>>();
let b = b.chars().collect::<Vec<char>>();
let mut d = vec![vec![0; b.len() + 1]; a.len() + 1];
let mut cost = 0;
for i in 1..=a.len() {
d[i][0] = i
}
for j in 1..=b.len() {
d[0][j] = j
}
for j in 1..=b.len() {
for i in 1..=a.len() {
if a[i-1] == b[j-1] {
cost = 0;
} else {
cost = 1;
}
d[i][j] = min(min(d[i - 1][j] + 1, d[i][j - 1] + 1), d[i - 1][j - 1] + cost);
}
}
d[a.len()][b.len()]
}
"Bilbo Baggins" most similar string: "Frodo Baggins"
"Gandalf" most similar string: None
"Thorin" most similar string: "Balin"
"Balin" most similar string: "Kili"
"Kili" most similar string: "Fili"
"Fili" most similar string: "Kili"
"John" most similar string: None
"Frodo Baggins" most similar string: "Bilbo Baggins"
Balin still isn't quite what you want since it takes the fewest modifications to get to Kili, but it definitely looks closer. Hopefully, this can help lead you closer to where you want to go, but you may need to use a combination of algorithms, or find one which weights beginnings/ends of words differently if you want the Balin to match Thorin.