I am trying to execute a query which performs a full text search with mongoose. The filter object looks as follows and works using mongosh:
{
authorUid: 'X',
'$text': { '$search': 'searchStr' }
}
But, whenever trying to execute this with mongoose, I'm getting the following error:
MongoServerError: Missing expected field "$search"
The mongoose code is as follows:
let query = Collection.find(
filterObj,
{
_id: 1,
title: 1,
src: 1,
},
{ sort: { createdAt: -1 } },
);
I am quite certain the text search index in the collection is created without error. It is visible in MongoDB Compass and I don't think the query should have worked in mongosh if this wasn't the case.
To be complete, here is the command ran via mongosh:
db.collection.find(
{
authorUid: 'X',
'$text': { '$search': 'searchStr' }
},
{
_id: 1,
title: 1,
src: 1
},
{sort: {createdAt: -1}}
)
Is this one not equivalent with the one written for Mongoose? How can I perform this query using Mongoose?
-- EDIT --
Just to ensure the cause is not the filterObj, I tried doing the exact same thing with it hard coded. Using Mongoose, it looks as follows:
let query = Collection.find(
{
$text: {
$search: "math",
},
},
);
I'm still getting the same error. It is however still working via mongosh.
-- EDIT 2 --
I did get this to work using aggregates instead of .find. The following code returns the same as in the mongosh environment:
const res = await Collection.aggregate([
{
$match: {
$text: {
$search: "math",
},
},
},
]);
This is most definitely usable, but I'm still clueless as to why the initial query did not work.
Related
I know there are a LOT of questions regarding this subject. And while most work, they are really poor in performance when there are millions of records.
I have a collection with 10,000,000 records.
At first I was using mongoose paginator v2 and it took around 8s to get each page, with no filtering and 25s when filtering. Fairly decent compared to the other answers I found googling around. Then I read about aggregate (in some question about the same here) and it was a marvel, 7 ms to get each page without filtering, no matter what page it is:
const pageSize = +req.query.pagesize;
const currentPage = +req.query.currentpage;
let recordCount;
ServiceClass.find().count().then((count) =>{
recordCount = count;
ServiceClass.aggregate().skip(currentPage).limit(pageSize).exec().then((documents) => {
res.status(200).json({
message: msgGettingRecordsSuccess,
serviceClasses: documents,
count: recordCount,
});
})
.catch((error) => {
res.status(500).json({ message: msgGettingRecordsError });
});
}).catch((error) => {
res.status(500).json({ message: "Error getting record count" });
});
What I'm having issues with is when filtering. aggregate doesn't really work like find so my conditions are not working. I read the docs about aggregate and tried with [ {$match: {description: {$regex: regex}}} ] inside aggregate as a start but it did not return anything.
This is my current working function for filtering and pagination (which takes 25s):
const pageSize = +req.query.pagesize;
const currentPage = +req.query.currentpage;
const filter = req.params.filter;
const regex = new RegExp(filter, 'i');
ServiceClass.paginate({
$or:[
{code: { $regex: regex }},
{description: { $regex: regex }},
]
},{limit: pageSize, page: currentPage}).then((documents)=>{
res.status(200).json({
message: msgGettingRecordsSuccess,
serviceClasses: documents
});
}).catch((error) => {
res.status(500).json({ message: "Error getting the records." });
});
code and description are both indexes. code is a unique index and description is just a normal index. I need to search for documents which contains a string either in code or description field.
What is the most efficient way to filter and paginate when you have millions of records?
Below code will get the paginated result from the database along with the count of total documents for that particular query simultaneously.
const pageSize = +req.query.pagesize;
const currentPage = +req.query.currentpage;
const skip = currentPage * pageSize - pageSize;
const query = [
{
$match: { $or: [{ code: { $regex: regex } }, { description: { $regex: regex } }] },
},
{
$facet: {
result: [
{
$skip: skip,
},
{
$limit: pageSize,
},
{
$project: {
createdAt: 0,
updatedAt: 0,
__v: 0,
},
},
],
count: [
{
$count: "count",
},
],
},
},
{
$project: {
result: 1,
count: {
$arrayElemAt: ["$count", 0],
},
},
},
];
const result = await ServiceClass.aggregate(query);
console.log(result)
// result is an object with result and count key.
Hope it helps.
The most efficient way to filter and paginate when you have millions of records is to use the MongoDB's built-in pagination and filtering features, such as the skip(), limit(), and $match operators in the aggregate() pipeline.
You can use the skip() operator to skip a certain number of documents, and the limit() operator to limit the number of documents returned. You can also use the $match operator to filter the documents based on certain conditions.
To filter your documents based on the code or description field, you can use the $match operator with the $or operator, like this:
ServiceClass.aggregate([
{ $match: { $or: [{ code: { $regex: regex } }, { description: { $regex: regex } }] } },
{ $skip: currentPage },
{ $limit: pageSize }
])
You can also use the $text operator instead of $regex which will perform more efficiently when you have text search queries.
It's also important to make sure that the relevant fields (code and description) have indexes, as that will greatly speed up the search process.
You might have to adjust the query according to your specific use case and data.
$project: {
_id: 1,
edited: 1,
game: {
gta: {
totalUserNumber: {
$reduce: {
input: "$gta.users",
initialValue: 0,
in: { $add: [{ $size: "$$this" }, "$$value"] },
},
},
userList: "$gta.users", <----- paginating this
},
DOTA2: {
totalUserNumber: {
$reduce: {
input: "$dota2.users",
initialValue: 0,
in: { $add: [{ $size: "$$this" }, "$$value"] },
},
},
userList: "$dota2.users", <------ paginating this
},
},
.... More Games
},
I have this $project. I have paginated the list of games by using $facet,$sort, $skip and $limit after $project.
I am trying also trying to paginate each game's userList. I have done to get the total value in order to calculate the page number and more.
But, I am struggling to apply $sort and $limit inside the $project. So far, I have just returned the document and then paginated with the return value. However, I don't think this is very efficient and wondering if there is any way that I can paginate the field inside the $project.
Is there any way that I can apply $sort and $limit inside the $project, in order to apply pagination to the fields and return?
------ Edit ------
this is for paginating the field. Because, I am already paginating the document (game list), I could not find any way that I can paginate the field, because I could not find any way that I can apply $facet to the field.
e.g. document
[
gta: {
userID: ['aa', 'bb', 'cc' ......],
},
dota: {
userID: ['aa', 'bb', 'cc' ......],
}
....
]
I am using $facet to paginate the list of games (dota, gta, lol and more). However, I did not want to return all the userID. I had to return the entire document and then paginate the userID to replace the json doc.
Now, I can paginate the field inside the aggregate pipeline by using $function.
thanks to Mongodb sort inner array !
const _function = function (e) {
e // <---- will return userID array. You can do what you want to do.
return {
};
};
game
.collection("game")
.aggregate([
{},
{
$set: {
"game": {
$function: {
body: _function,
args: ["$userID"],
lang: "js",
},
},
},
},
])
.toArray();
By using $function multiple time, you will be able to paginate the field. I don' really know if this is faster or not tho. Plus, make sure you can use $function. I read that you can't use this if you are on the free tier at Atlas.
What you are looking for is the $slice Operator.
It requires three parameters.
"$slice": [<Array>, <start-N>, <No-Of.elements to fetch>]
userList: {"$slice": ["$dota2.users", 20, 10]} // <-- Will ignore first 20 elements in array and gets the next 10
I'm very new to mongodb and am having difficulty getting to a solution for my use case. For example I have the following document:
{
_id : ObjectId('5rtgwr6gsrtbsr6hsfbsr6bdrfyb'),
uuid : 'something',
mainArray : [
{
id : 1,
title: 'A',
array: ['lots','off','stuff']
},
{
id : 2,
title: 'B',
array: ['even','more','stuff']
}
]
}
I'd like to have the following returned:
{
uuid : 'something',
mainArray : [
{
id : 1,
title: 'A'
},
{
id : 2,
title: 'B'
}
]
}
I've tried various combinations of using findOne() and aggregate() with $slice and $project. With findOne(), if it worked at all, the who document would be returned. I am unable to test whether attempts at aggregating work because .then((ret)=>{}) promises don't seem to work in node.js for me with it (no issues with findOne). Calling a function like so
return db.myCollection.aggregate([
{
$match: {
_id:ObjectId(mongo_id)
}
},
{
$project : {
mainArray: {
id:1,
title:1
}
}
}
],function(err,res){
console.log(res)
return res
})
logs the entire function and not the droids I'm looking for.
You're missing toArray() method to obtain the actual result set. Instead you're returning the aggregation cursor object. Try this.
return db.myCollection.aggregate([matchCode,projectCode]).toArray().then(
data => {
console.log(data);
return data;
},
error => { console.log(error)});
The documnetation on aggregation cursor for MongoDB NodeJS driver can
be found here
http://mongodb.github.io/node-mongodb-native/3.5/api/AggregationCursor.html#toArray
This is an alternative solution (to the solution mentioned in the comment by #v1shva)
Instead of using aggregation you can use projection option of .findOne() operation.
db.myCollection.findOne(matchCode, {
projection: { _id: false, 'mainArray.array': false } // or { _id: -1, 'mainArray.array': -1 }
})
Mongodb returns non-matching elements in nested array
Here's my database sample:
const users = [{
'username':'jack',
'songs':[{
'song':'Another Love',
'likes':false
}, {
'song':"i'm into you",
'likes': true
}]
}, {
'username':'Stromae',
'songs':[{
'song':'Take me to church',
'likes':false
}, {
'song':"Habits",
'likes': true
}]
}];
I'm trying to find the following row:
const query = {'username':'Stromae' , 'songs.song':'Take me to church','songs.likes':true};
const result = await usersTable.find(query).project({'songs.$':1}).toArray();
as you see I'm trying to find a user who called "Stromae" and has a "Take me to church" song and he don't like it.
I'm expecting as result to be null, while the result is:
{ _id: 5d302809e734acbc5ffa2a8f,
songs: [ { song: 'Take me to church', likes: false } ] }
as you see from the result it ignores that I need "likes" field to be true.
As per my understanding you are trying to match data from 'songs' array which satisfying both the conditions for the 'song' and 'likes' both fields. But you haven't provided the logic like check both the fields for same array element. That's why it is checking this fields in whole 'songs' array.
To check condition for single array element you can use $elemMatch and for checking both the conditions are satisfying or not use the $and operator.
You can use your Mongo query as:
db.usersTable.find({
"username": "Stromae",
songs: {
$elemMatch: {
$and: [
{
"song": "Take me to church"
},
{
"likes": true
}
]
}
}
})
Just update your 'find' query, you might get your required result.
I'm trying to pop and retrieve an element out of an array stored in a document. I can't use $pop since it doesn't return the POPed element. I'm trying to use findAndModify() instead. It works in the shell console, but I'm having troubles getting it to work using the mongodb node.js driver (https://www.npmjs.org/package/mongodb).
my document structure looks like so:
{ _id: '1', queue: [1,2,3]}
In mongo shell, I do:
> db.collection.findAndModify({ query: { _id: 1 },
update: { $pop: { queue: -1 } },
fields: { queue: { $slice: 1 } }, new: false })
$slice ensures that the returning document shows the element that has just been poped. To clarify, I'm not interested in what is in the queue, I'm only interested in what I have just popped out of the queue.
returns:
< {_id: 1, "queue": [1]} // yes, it works!
Using the mongodb library, I don't know how to specify the $slice: 1, it doesn't seem to be supported in the options(?):
> db.collection('collection').findAndModify(
{ _id: 1 },
[],
{ $pop: { queue: -1 }, queue: { $slice: 1 } },
{ new: false },
function(error, results) {
if(error) {
console.log(error);
}
console.log(results);
}
);
returns:
< MongoError: exception: Field name duplication not allowed with modifiers
Basically - where should I put the "queue: {$slice: 1}" part in the nodejs query to make this work? Is it even supported in the node.js driver implementation?
Also, it doesn't seem like findAndModify() is meant to be used this way. If $pop was returning the POPed value, it would be ideal. Any suggestions on how to do this?
Thanks,
- Samir
It seems like that the node.js implementation does not support the 'fields' operand at all.
We've figured out this work around:
1) We store each element in it's own document, instead of an array within the same document.
2) Now findAndModify works like so:
db.collection('collection').findAndModify(
{}, // findAndModify will match it to the first document, if multiple docs are found
[],
{},
{remove: true, new: false }, // returns & removes document from collection
function(error, results) {
if(error) {
console.log(error);
}
console.log(results);
}
);
Some good links that helped us and might help you if you have a similar issue:
https://blog.serverdensity.com/queueing-mongodb-using-mongodb/
http://www.slideshare.net/mongodb/mongodb-as-message-queue