Example:
> db.stuff.save({"foo":"bar"});
> db.stuff.find({"foo":"bar"}).count();
1
> db.stuff.find({"foo":"BAR"}).count();
0
You could use a regex.
In your example that would be:
db.stuff.find( { foo: /^bar$/i } );
I must say, though, maybe you could just downcase (or upcase) the value on the way in rather than incurring the extra cost every time you find it. Obviously this wont work for people's names and such, but maybe use-cases like tags.
UPDATE:
The original answer is now obsolete. Mongodb now supports advanced full text searching, with many features.
ORIGINAL ANSWER:
It should be noted that searching with regex's case insensitive /i means that mongodb cannot search by index, so queries against large datasets can take a long time.
Even with small datasets, it's not very efficient. You take a far bigger cpu hit than your query warrants, which could become an issue if you are trying to achieve scale.
As an alternative, you can store an uppercase copy and search against that. For instance, I have a User table that has a username which is mixed case, but the id is an uppercase copy of the username. This ensures case-sensitive duplication is impossible (having both "Foo" and "foo" will not be allowed), and I can search by id = username.toUpperCase() to get a case-insensitive search for username.
If your field is large, such as a message body, duplicating data is probably not a good option. I believe using an extraneous indexer like Apache Lucene is the best option in that case.
Starting with MongoDB 3.4, the recommended way to perform fast case-insensitive searches is to use a Case Insensitive Index.
I personally emailed one of the founders to please get this working, and he made it happen! It was an issue on JIRA since 2009, and many have requested the feature. Here's how it works:
A case-insensitive index is made by specifying a collation with a strength of either 1 or 2. You can create a case-insensitive index like this:
db.cities.createIndex(
{ city: 1 },
{
collation: {
locale: 'en',
strength: 2
}
}
);
You can also specify a default collation per collection when you create them:
db.createCollection('cities', { collation: { locale: 'en', strength: 2 } } );
In either case, in order to use the case-insensitive index, you need to specify the same collation in the find operation that was used when creating the index or the collection:
db.cities.find(
{ city: 'new york' }
).collation(
{ locale: 'en', strength: 2 }
);
This will return "New York", "new york", "New york" etc.
Other notes
The answers suggesting to use full-text search are wrong in this case (and potentially dangerous). The question was about making a case-insensitive query, e.g. username: 'bill' matching BILL or Bill, not a full-text search query, which would also match stemmed words of bill, such as Bills, billed etc.
The answers suggesting to use regular expressions are slow, because even with indexes, the documentation states:
"Case insensitive regular expression queries generally cannot use indexes effectively. The $regex implementation is not collation-aware and is unable to utilize case-insensitive indexes."
$regex answers also run the risk of user input injection.
If you need to create the regexp from a variable, this is a much better way to do it: https://stackoverflow.com/a/10728069/309514
You can then do something like:
var string = "SomeStringToFind";
var regex = new RegExp(["^", string, "$"].join(""), "i");
// Creates a regex of: /^SomeStringToFind$/i
db.stuff.find( { foo: regex } );
This has the benefit be being more programmatic or you can get a performance boost by compiling it ahead of time if you're reusing it a lot.
Keep in mind that the previous example:
db.stuff.find( { foo: /bar/i } );
will cause every entries containing bar to match the query ( bar1, barxyz, openbar ), it could be very dangerous for a username search on a auth function ...
You may need to make it match only the search term by using the appropriate regexp syntax as:
db.stuff.find( { foo: /^bar$/i } );
See http://www.regular-expressions.info/ for syntax help on regular expressions
db.company_profile.find({ "companyName" : { "$regex" : "Nilesh" , "$options" : "i"}});
db.zipcodes.find({city : "NEW YORK"}); // Case-sensitive
db.zipcodes.find({city : /NEW york/i}); // Note the 'i' flag for case-insensitivity
TL;DR
Correct way to do this in mongo
Do not Use RegExp
Go natural And use mongodb's inbuilt indexing , search
Step 1 :
db.articles.insert(
[
{ _id: 1, subject: "coffee", author: "xyz", views: 50 },
{ _id: 2, subject: "Coffee Shopping", author: "efg", views: 5 },
{ _id: 3, subject: "Baking a cake", author: "abc", views: 90 },
{ _id: 4, subject: "baking", author: "xyz", views: 100 },
{ _id: 5, subject: "Café Con Leche", author: "abc", views: 200 },
{ _id: 6, subject: "Сырники", author: "jkl", views: 80 },
{ _id: 7, subject: "coffee and cream", author: "efg", views: 10 },
{ _id: 8, subject: "Cafe con Leche", author: "xyz", views: 10 }
]
)
Step 2 :
Need to create index on whichever TEXT field you want to search , without indexing query will be extremely slow
db.articles.createIndex( { subject: "text" } )
step 3 :
db.articles.find( { $text: { $search: "coffee",$caseSensitive :true } } ) //FOR SENSITIVITY
db.articles.find( { $text: { $search: "coffee",$caseSensitive :false } } ) //FOR INSENSITIVITY
One very important thing to keep in mind when using a Regex based query - When you are doing this for a login system, escape every single character you are searching for, and don't forget the ^ and $ operators. Lodash has a nice function for this, should you be using it already:
db.stuff.find({$regex: new RegExp(_.escapeRegExp(bar), $options: 'i'})
Why? Imagine a user entering .* as his username. That would match all usernames, enabling a login by just guessing any user's password.
Suppose you want to search "column" in "Table" and you want case insensitive search. The best and efficient way is:
//create empty JSON Object
mycolumn = {};
//check if column has valid value
if(column) {
mycolumn.column = {$regex: new RegExp(column), $options: "i"};
}
Table.find(mycolumn);
It just adds your search value as RegEx and searches in with insensitive criteria set with "i" as option.
Mongo (current version 2.0.0) doesn't allow case-insensitive searches against indexed fields - see their documentation. For non-indexed fields, the regexes listed in the other answers should be fine.
For searching a variable and escaping it:
const escapeStringRegexp = require('escape-string-regexp')
const name = 'foo'
db.stuff.find({name: new RegExp('^' + escapeStringRegexp(name) + '$', 'i')})
Escaping the variable protects the query against attacks with '.*' or other regex.
escape-string-regexp
The best method is in your language of choice, when creating a model wrapper for your objects, have your save() method iterate through a set of fields that you will be searching on that are also indexed; those set of fields should have lowercase counterparts that are then used for searching.
Every time the object is saved again, the lowercase properties are then checked and updated with any changes to the main properties. This will make it so you can search efficiently, but hide the extra work needed to update the lc fields each time.
The lower case fields could be a key:value object store or just the field name with a prefixed lc_. I use the second one to simplify querying (deep object querying can be confusing at times).
Note: you want to index the lc_ fields, not the main fields they are based off of.
Using Mongoose this worked for me:
var find = function(username, next){
User.find({'username': {$regex: new RegExp('^' + username, 'i')}}, function(err, res){
if(err) throw err;
next(null, res);
});
}
If you're using MongoDB Compass:
Go to the collection, in the filter type -> {Fieldname: /string/i}
For Node.js using Mongoose:
Model.find({FieldName: {$regex: "stringToSearch", $options: "i"}})
The aggregation framework was introduced in mongodb 2.2 . You can use the string operator "$strcasecmp" to make a case-insensitive comparison between strings. It's more recommended and easier than using regex.
Here's the official document on the aggregation command operator: https://docs.mongodb.com/manual/reference/operator/aggregation/strcasecmp/#exp._S_strcasecmp .
You can use Case Insensitive Indexes:
The following example creates a collection with no default collation, then adds an index on the name field with a case insensitive collation. International Components for Unicode
/* strength: CollationStrength.Secondary
* Secondary level of comparison. Collation performs comparisons up to secondary * differences, such as diacritics. That is, collation performs comparisons of
* base characters (primary differences) and diacritics (secondary differences). * Differences between base characters takes precedence over secondary
* differences.
*/
db.users.createIndex( { name: 1 }, collation: { locale: 'tr', strength: 2 } } )
To use the index, queries must specify the same collation.
db.users.insert( [ { name: "Oğuz" },
{ name: "oğuz" },
{ name: "OĞUZ" } ] )
// does not use index, finds one result
db.users.find( { name: "oğuz" } )
// uses the index, finds three results
db.users.find( { name: "oğuz" } ).collation( { locale: 'tr', strength: 2 } )
// does not use the index, finds three results (different strength)
db.users.find( { name: "oğuz" } ).collation( { locale: 'tr', strength: 1 } )
or you can create a collection with default collation:
db.createCollection("users", { collation: { locale: 'tr', strength: 2 } } )
db.users.createIndex( { name : 1 } ) // inherits the default collation
I'm surprised nobody has warned about the risk of regex injection by using /^bar$/i if bar is a password or an account id search. (I.e. bar => .*#myhackeddomain.com e.g., so here comes my bet: use \Q \E regex special chars! provided in PERL
db.stuff.find( { foo: /^\Qbar\E$/i } );
You should escape bar variable \ chars with \\ to avoid \E exploit again when e.g. bar = '\E.*#myhackeddomain.com\Q'
Another option is to use a regex escape char strategy like the one described here Javascript equivalent of Perl's \Q ... \E or quotemeta()
Use RegExp,
In case if any other options do not work for you, RegExp is a good option. It makes the string case insensitive.
var username = new RegExp("^" + "John" + "$", "i");;
use username in queries, and then its done.
I hope it will work for you too. All the Best.
If there are some special characters in the query, regex simple will not work. You will need to escape those special characters.
The following helper function can help without installing any third-party library:
const escapeSpecialChars = (str) => {
return str.replace(/[-[\]{}()*+?.,\\^$|#\s]/g, "\\$&");
}
And your query will be like this:
db.collection.find({ field: { $regex: escapeSpecialChars(query), $options: "i" }})
Hope it will help!
Using a filter works for me in C#.
string s = "searchTerm";
var filter = Builders<Model>.Filter.Where(p => p.Title.ToLower().Contains(s.ToLower()));
var listSorted = collection.Find(filter).ToList();
var list = collection.Find(filter).ToList();
It may even use the index because I believe the methods are called after the return happens but I haven't tested this out yet.
This also avoids a problem of
var filter = Builders<Model>.Filter.Eq(p => p.Title.ToLower(), s.ToLower());
that mongodb will think p.Title.ToLower() is a property and won't map properly.
I had faced a similar issue and this is what worked for me:
const flavorExists = await Flavors.findOne({
'flavor.name': { $regex: flavorName, $options: 'i' },
});
Yes it is possible
You can use the $expr like that:
$expr: {
$eq: [
{ $toLower: '$STRUNG_KEY' },
{ $toLower: 'VALUE' }
]
}
Please do not use the regex because it may make a lot of problems especially if you use a string coming from the end user.
I've created a simple Func for the case insensitive regex, which I use in my filter.
private Func<string, BsonRegularExpression> CaseInsensitiveCompare = (field) =>
BsonRegularExpression.Create(new Regex(field, RegexOptions.IgnoreCase));
Then you simply filter on a field as follows.
db.stuff.find({"foo": CaseInsensitiveCompare("bar")}).count();
These have been tested for string searches
{'_id': /.*CM.*/} ||find _id where _id contains ->CM
{'_id': /^CM/} ||find _id where _id starts ->CM
{'_id': /CM$/} ||find _id where _id ends ->CM
{'_id': /.*UcM075237.*/i} ||find _id where _id contains ->UcM075237, ignore upper/lower case
{'_id': /^UcM075237/i} ||find _id where _id starts ->UcM075237, ignore upper/lower case
{'_id': /UcM075237$/i} ||find _id where _id ends ->UcM075237, ignore upper/lower case
For any one using Golang and wishes to have case sensitive full text search with mongodb and the mgo godoc globalsign library.
collation := &mgo.Collation{
Locale: "en",
Strength: 2,
}
err := collection.Find(query).Collation(collation)
As you can see in mongo docs - since version 3.2 $text index is case-insensitive by default: https://docs.mongodb.com/manual/core/index-text/#text-index-case-insensitivity
Create a text index and use $text operator in your query.
Related
I have a collection as
[
{
firstName: "John",
middleName: "F",
lastName: "Kennedy"
},
{
firstName: "Barack",
lastName: "Obama"
}
]
I am trying to create a function that searches the collection by name. I need to concat the names before trying to find a match. I tried the following
User.aggregate([
{
"$project": {
fullName: {
"$concat": [
"$firstName",
" ",
"$lastName"
]
}
}
},
{
$match: {
"fullName": {
$regex: /[a-z\\s]*oba[a-z\\s]*/i
}
}
}
])
It works for names without middle names. But I need this to work for names with middle names too. When I try to concat middleName, I get an error because the path middleName does not exist on all documents. I could not implement $cond and $exists operators properly make this work. Any kind of help is highly appreciated. Thanks!
One option is using $ifNull:
db.collection.aggregate([
{$project: {
fullName: {$concat: [
"$firstName",
" ",
{$ifNull: [
{$concat: ["$middleName", " "]},
""
]},
"$lastName"
]}
}}
])
See how it works on the playground example
As usual, #nimrod serok's good answer directly addresses the question that was asked and should be the accepted one. But there is an additional consideration to keep in mind associated with the following text in the question that won't fit in a comment:
I am trying to create a function that searches the collection by name. I need to concat the names before trying to find a match.
This approach is going to be very inefficient and probably won't deliver the performance or scalability that you need long term. It requires the database to first generate this fullName field for all candidate documents (which is all documents in the collection if there are no other filter conditions) and then search through them to look for matches. I would strongly recommend avoiding this.
In your particular situation, the answer is not simply storing a fullName field in the documents though. This is because your query is doing an unanchored and case-insensitive regex search which itself cannot use indexes efficiently. You may want to look into more efficient text searching options (such as those provided by Atlas Search) to support this functionality long term.
I am doing a mongo query in our express app with the search value passed by the user.
return await Orders.find({
orderId: {
$regex: `${search}`,
$options: "i"
}
})
the orderId is of the format POXXXXXXXXXX where X represents numbers. Often customer search with only the numeric part of the orderId. My question is if I modify my code as below will it make the query faster?
const filter = {
$regex: `${search}`
}
if(isNumeric(search)){
filter["$options"]= "i"
}
return await Orders.find(filter)
I tested these two queries but the results are inconsistent.
To answer the original question - no, conditional case sensitivity in the regexp options will not give you noticeable performance improvements.
Assuming users will search starting from leading numbers you can massively improve search speed by indexing the collection by orderId field.
The search like
let numberSearch = search.replace(/^PO/i, "")
return await Orders.find({
orderId: {
$regex: `^PO${numberSearch}`,
$options: "i"
}
})
normalises the input string and uses anchored regexp which can benefit from the index.
Please note the results will be slightly different to the original query. A document with orderId "PO123456" was a match in the OP query but is not in this one.
If you use Indexing with MongoDB then Your query search speed can improve. If you have not yet applied.
create index with following command,
db.Orders.createIndex({ orderId: 1 },{ background : true , unique : true })
about options ,
background : true
if we want the site live while index creating
unique: true
use it only when the unique field value
I'm trying to work with mongoose to implement a simple search feature that I plan to expand later.
Basically what I'm trying to do is take a search query from the user and put it inside the mongoose find() query, but I'm using the built-in " regular expression" way according to their documentation:
MyModel.find({ name: /john/i }, null, { skip: 10 })
Here, john is a static value, when I try to place a variable instead it doesn't work.
MyModel.find({ name: /req.query.name/i }, null, { skip: 10 })
How can I place a variable instead of the static content?
P.S: I tried using the JS regular expression and it did work but I want to do it the mongoose way.
MyModel.find({ name: new RegExp(req.query.name, "i") }, null, { skip: 10 }) //this works
You can use $regex of mongoose
MyModel.find({ name: {$regex: req.query.name, $options:'i'} }, null, { skip: 10 })
In order to create a dynamic regex from a string the only option is using new RegExp as you did. There is no way to to it with the literal /../-method (see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions).
However, in either case, the result will be a regex object — which will have the same methods and properties attached to them. Mongoose does not actually care which way you create the regex.
Instead of creating the regex in javascript you can let mongoose/mongodb do it, as Ibrahim's answer proposes.
Let's say I have this schema:
var mongoose = require("mongoose")
var userSchema = new mongoose.Schema({
name: {type: String},
// other fields
}, { collation: { locale: "en_US", strength: 1 } });
I use collation so that the search is case-insensitive
Then let's say I have a document with name "Dave"
{
name: "Dave",
// other fields
}
then, I search for it but without writing the whole word
var userList = {
.find({name: "da"})
.exec();
}
How can I make this work without using a regex expression? Which are quite slow. I have tried doing an index and then searching with the $text method but I don't know how to make it so that it searches only a specific field within the document.
I believe using REGEX is your best solution. What you are trying to do is literally what regex is designed for. Yeah it's slow, but any other option you try to implement will probably be slower.
Creating a text index, and using the $text is only designed to match full words so you cannot use this method.
If you are truly desperate, and really don't want to use regex you can try something else... Try creating and storing an object, with each possible substring in the document. Object lookup is O(1) time, which means it will be faster, but the tradeoff is you are storing an absurd amount of data in the database. If this is ok with you, then give 'er a try.
Let's use Dave for example. The object you store could look something like this:
{
"d": 1,
"da": 1,
"dav": 1,
"dave"" 1
}
We can store this object in a field called substrings. Then when we do the database lookup, it's as simple as:
User.find({ 'substrings.da': { $exists: true }})
But please consider using regex... It's so much simpler, so much cleaner and it's designed for exactly what you want.
I have a collection like this:
{
"_id" : ObjectId("5a7c49b02d2bbb28a4b2e6a2"),
"phone" : "Pinheiro",
"email" : "Pinheiro",
"variableParameters" : {
"loremIpsum" : "Do you see a little Asian child with a blank expression on his face sitting outside on a mechanical helicopter that shakes when you put quarters in it?",
"uf" : "Rio de Janeiro",
"city" : "Rio de Janeiro",
"end" : "RUA JARDIM BOTÂNICO 1060",
"tel" : "5521999999999",
"eml" : "teste#gmail.com",
"nome" : "Usuario de Teste"
}
}
And i want to query the "variableParameters" object, but like the name said, this properties are variable. So in some cases it will have "uf", but in other cases won't.
I'm actually doing a query that only matches the constant field from a mongoose schema:
{ 'phone': { $regex: filter, $options: 'i' } }
Is there any way that I can query "variableParameters" without knowing his child properties?
If you are unsure about the keys(since they are variable), then try using $text search
To use text search we need to index the variableParameters.
Case sensitive text search can also be performed but it comes with the impact on performance.
Please read https://docs.mongodb.com/manual/reference/operator/query/text/ for more information on text search
[SOLVED]
Thanks #Clement Amarnath for the help.
The solution is something like this:
_Events.find({ $text: { $search: 'searchText' } }, (err, events) => {
if (err) return Exceptions.HandleApiException(err, res);
res.send(events);
});
The $text parameter can have this properties:
{
$text: {
$search: <string>,
$language: <string>,
$caseSensitive: <boolean>,
$diacriticSensitive: <boolean>
}
}
$search A string of terms that MongoDB parses and uses to query the text index. MongoDB performs a logical OR search of the terms unless specified as a phrase. See Behavior for more information on the field.
$language Optional. The language that determines the list of stop words for the search and the rules for the stemmer and tokenizer. If not specified, the search uses the default language of the index. For supported languages, see Text Search Languages.
If you specify a language value of "none", then the text search uses simple tokenization with no list of stop words and no stemming.
$caseSensitive Optional. A boolean flag to enable or disable case sensitive search. Defaults to false; i.e. the search defers to the case insensitivity of the text index.
$diacriticSensitive Optional. A boolean flag to enable or disable diacritic sensitive search against version 3 text indexes. Defaults to false; i.e. the search defers to the diacritic insensitivity of the text index.
For more information, see MongoDB Documentation.
You can use $where to build your own matching function.
An example for a full-text match would be:
db.col.find({$where: function(){
return Object.values(this.variableParameters).includes("Rio de Janeiro")
}})