I am pretty confused to why making the searched fields to be "index" is making the query "theoretically" slower.
I have a not very big collection of items (6240) and all of them have the following structure.
const SomeSchema = new mongoose.Schema({
data: String,
from: {
type: Number,
},
to: {
type: Number,
},
timeStamp: {
type: Date,
default: new Date()
}
})
SomeSchema.set('toJSON', {
getters: true,
transform: (doc, ret) => {
delete ret.from
delete ret.to
return sanitizeSensitiveProperties(ret)
}
})
export const Some = mongoose.model('Some', SomeSchema, 'somethings')
The strange thing came when after trying to improve the query I changed the schema to be
...
from: {
type: Number,
index: true
},
to: {
type: Number,
index: true
},
...
With this schema I run the following query
db.rolls.find({from: {$lte: 1560858984}, to: {$gte: 1560858984}}).explain("executionStats")
This are the results NOTE THAT THE 1st ONE is the one without index
"executionTimeMillis" : 6,
"totalKeysExamined" : 0,
"totalDocsExamined" : 6240,
"executionTimeMillis" : 15,
"totalKeysExamined" : 2895,
"totalDocsExamined" : 2895,
Does this result make any sense, or is just the mongo .explain() function messing around?
As you can see I am using the Mongoose Driver in the version ^5.5.13 and I am using Mongo in the version 4.0.5
Related
I have a document in mongoDB structured like that
_id: ObjectId("generatedByMongo"),
name: {
required: true,
type: String,
trim: true
},
last: {
required: true,
type: String,
trim: true
},
grades: [{
grade: {
_id: ObjectId(""),
grade: Number,
date: date
}
}]
And to server I send array of objects containing 3 fields
[
{studentId}, {gradeId}, {newGrade}
]
What I'm trying to accomplish is I want to find in within that user collection grade with given gradeId and update it's value to newGrade. As far as I tried to do that I have done this
router.patch('/students/updateGrade',async(req,res) => {
const studentId = req.body.updateGradeArray[0].studentId;
const gradeId = req.body.updateGradeArray[0].gradeId;
const newGrade = req.body.updateGradeArray[0].newGrade;
try {
const student = await Student.find({_id: studentId})
.select({'grades': {$elemMatch: {_id: gradeId}}});
} catch(e) {
console.log(e);
}
}
);
If you intend to update just grade.grade(the number value), try this:
Student.updateOne(
// Find a document with _id matching the studentId
{ "_id": studentId },
// Update the student grade
{ $set: { "grades.$[selectedGrade].grade": newGrade } },
{ arrayFilters: [{ "selectedGrade._id": gradeId }] },
)
Why this should work:
Since you are trying to update a student document, you should be using one of MongoDB update methods not find. In the query above, I'm using the updateOne method. Inside the updateOne, I am using a combination of $set and $[identifier] update operators to update the student grade.
I hope this helps✌🏾
This question already has answers here:
Mongoose deleting (pull) a document within an array, does not work with ObjectID
(8 answers)
Closed 3 years ago.
I have a mongoose schema that looks like :-
{
"_id" : ObjectId("some_id"),
"repDet" : {
"devIDs" : [
"dev1_B37",
"dev2_B38",
"dev3_B26"
],
"sensors" : [
"D30"
],
"triggers" : [
"initial"
]
}
I want to pull "dev2_B38" from the array "devIDs" inside the object "repDet".
The following command works in mongo db :-
schedules.updateOne({ added_by: currentUser, "repDet.devIDs": "dev2_B38" },{$pull : {"repDet.devIDs": "dev2_B38"}})
perfectly.
But the same command doesnt work in mongoose. The mongoose version that I have tried on is
5.0.18 & 5.7.11.
what it does it removes all the other devIDs from the array and keeps just the "dev_B38" and also converting "devIDs" into a string from an array. The output looks like :-
{
"_id" : ObjectId("some_id"),
"repDet" : {
"devIDs" : "dev2_B38",
"sensors" : [
"D30"
],
"triggers" : [
"initial"
]
}
I have tried updateOne, update, findOneAndUpdate all of these.
I also tried modifying it like :-
schedules.updateOne({ added_by: currentUser, "repDet.devIDs": "dev2_B38" },{$pull : {"repDet" {"devIDs": "dev2_B38"}}})
but still it gave me the same result.
The exact schema is :-
var scheduleSchema = new mongoose.Schema({
title: { type: String },
message: { type: String },
devID: { type: String },
sensors: [{
sensorId: String,
sensorName: String,
unit: String
}],
selectedContacts: [{
_id: { type: mongoose.Schema.Types.ObjectId, ref: 'Contact' },
name: String,
mobiles: [Number],
emails: [String]
}],
repDet: { type: Object },
invocationTime: { type: String },
invocationDays: Array,
startTime: { type: String },
endTime: { type: String },
interval: { type: Number },
repeat: { type: Number },
url: String
});
Kindly help me with this.
Thank you
schedules.updateOne({ _id: currentUser},
{$pull : {"repDet.devIDs":"dev1_B37"}})
try this
I have a collection named Codes. This is how the Schema is defined:
import mongoose from 'mongoose'
import autoIncrement from 'mongoose-auto-increment';
const Schema = mongoose.Schema;
const CodesSchema = mongoose.Schema(
{
configId: { type: Number },
campaignId: { type: Number },
permissions: {
all: { type: Boolean, default: false },
users: { type: [Number], default: [] }
}
)
autoIncrement.initialize(mongoose.connection);
CodesSchema.plugin(autoIncrement.plugin, { model: 'Codes', field: 'campaignId' });
export default mongoose.model('Codes', CodesSchema)
There is one query that looks like this:
const query = {
$and:[
{$or: [
{'permissions.all': true},
{'permissions.users': 12}
]},
{configId: 3}
]
};
Codes.find(query, (err, res) => {
// do something with the result
})
This works fine, but if there is a huge number of documents in the database then this query is really slow.
Is there anything I can do to improve the performance of this specific query? I'm thinking that createIndex would help, but I'm not sure if that can be applied since there are $and and $or conditions.
UPDATE
I've added indexes this way:
CodesSchema.index({configId: 1, 'permissions.all': 1, 'permissions.users': 1});
But running the query with .explain('executionStats') option returns:
{
"executionSuccess" : true,
"nReturned" : 6,
"executionTimeMillis" : 0,
"totalKeysExamined" : 10,
"totalDocsExamined" : 10,
}
Which doesn't seems right because the number of docs examined is greater than the number of docs returned.
The index itself is correct.
It must be CodesSchema.index, not Code.index.
Ensure you call Code.syncIndexes to update indexes dbside.
The "explain" part - you should check winningPlan.
If no indexes are used by the query, it should be something like
"winningPlan" : {
"stage" : "COLLSCAN",
"filter" : {
When the index is being used it changes to
"winningPlan" : {
"stage" : "FETCH",
"inputStage" : {
"stage" : "OR",
"inputStages" : [
{
"stage" : "IXSCAN",
"keyPattern" : {
When I execute the following code (a larger example, boiled down to the essentials)
var mongoose = require("mongoose");
var LocationSchema = new mongoose.Schema({
userName: String,
loc: {
'type': { type: String, enum: "Point", default: "Point" },
coordinates: { type: [Number] }
}
})
LocationSchema.index({ category: 1, loc: "2dsphere" });
var Location = mongoose.model("location", LocationSchema);
var mongoDB = 'mongodb://user1:test#ds042417.mlab.com:42417/locationdemo';
mongoose.Promise = global.Promise;
mongoose.connect(mongoDB, { useMongoClient: true });
var testUser = Location({
userName: "Tester",
loc: { coordinates: [12.44, 55.69] }
});
testUser.save(function (err) {
if (err) {
return console.log("UPPPPs: " + err);
}
console.log("User Saved, Try to find him:");
let query = Location.find({
loc:
{
$near:
{
$geometry:
{
type: "Point",
coordinates: [12.50, 55.71]
},
$maxDistance: 600000
}
}
})
query.exec(function (err, docs) {
if (err) {
return console.log("Err: " + err);
}
console.log("Found: " + JSON.stringify(docs))
})
});
I get this error:
Err: MongoError: error processing query: ns=locationdemo.locationsTree: GEONEAR field=loc maxdist=600000 isNearSphere=0
Sort: {}
Proj: {}
planner returned error: unable to find index for $geoNear query
But the index is there (see line 10) and the screenshot from mlab below. What am I doing wrong?:
You are breaking a rule of how you can use a an index in general. Whilst it is true that there is no restriction that a "2dsphere" index be the "first" property in a compound index, it is however very important that your "queries" actually address the first property in order for the index to be selected.
This is covered in Prefixes from the manual on compound indexes. In excerpt:
{ "item": 1, "location": 1, "stock": 1 }
The index has the following index prefixes:
{ item: 1 }
{ item: 1, location: 1 }
For a compound index, MongoDB can use the index to support queries on the index prefixes. As such, MongoDB can use the index for queries on the following fields:
the item field,
the item field and the location field,
the item field and the location field and the stock field.
However, MongoDB cannot use the index to support queries that include the following fields since without the item field, none of the listed fields correspond to a prefix index:
the location field,
the stock field, or
the location and stock fields.
Because your query references "loc" first and does not include "category", the index does not get selected and MongoDB returns the error.
So in order to use the index you have defined, you need to actually query "category" as well. Amending your listing:
var mongoose = require("mongoose");
mongoose.set('debug',true);
var LocationSchema = new mongoose.Schema({
userName: String,
category: Number,
loc: {
'type': { type: String, enum: "Point", default: "Point" },
coordinates: { type: [Number] }
}
})
//LocationSchema.index({ loc: "2dsphere", category: 1 },{ "background": false });
LocationSchema.index({ category: 1, loc: "2dsphere" });
var Location = mongoose.model("location", LocationSchema);
var mongoDB = 'mongodb://localhost/test';
mongoose.Promise = global.Promise;
mongoose.connect(mongoDB, { useMongoClient: true });
var testUser = Location({
userName: "Tester",
category: 1,
loc: { coordinates: [12.44, 55.69] }
});
testUser.save(function (err) {
if (err) {
return console.log("UPPPPs: " + err);
}
console.log("User Saved, Try to find him:");
let query = Location.find({
category: 1,
loc:
{
$near:
{
$geometry:
{
type: "Point",
coordinates: [12.50, 55.71]
},
$maxDistance: 600000
}
}
})
query.exec(function (err, docs) {
if (err) {
return console.log("Err: " + err);
}
console.log("Found: " + JSON.stringify(docs))
})
});
As long as we include "category" everything is fine:
User Saved, Try to find him:
Mongoose: locations.find({ loc: { '$near': { '$geometry': { type: 'Point', coordinates: [ 12.5, 55.71 ] }, '$maxDistance': 600000 } }, category: 1 }, { fields: {} })
Found: [{"_id":"59f8f87554900a4e555d4e22","userName":"Tester","category":1,"__v":0,"loc":{"coordinates":[12.44,55.69],"type":"Point"}},{"_id":"59f8fabf50fcf54fc3dd01f6","userName":"Tester","category":1,"__v":0,"loc":{"coordinates":[12.44,55.69],"type":"Point"}}]
The alternate case is to simply "prefix" the index with the location. Making sure to drop previous indexes or the collection first:
LocationSchema.index({ loc: "2dsphere", category: 1 },{ "background": false });
As well as you probably should be in the habit of setting "background": true, else you start running into race conditions on unit tests where the index has not finished being created before unit test code attempts to use it.
My first solution to this problem was to create the index via the mLab web-interface which worked like a charm.
I have tried the solution suggested by Neil, but that still fails. The detailed instructions related to indexes, given by Neil however, did point me toward the solution to the problem.
It was a timing problem (which you not always see if you run the database locally) related to that my test code did the following:
Created the index, created a Location document (which first time will also create the collection), and then in the callback provided by save, I tried to find the user. It seems that the index was not yet created here, which is what gave the error.
If I delay the find method a second, using setTimeout it works fine.
But still, thanks to Neil for valuable information about the right way of using indexes (background) :-)
My lat & lng numbers are being converted to strings. My section integers are still the correct data type of Number. How do I set up model so that I can get my lat & lng back out as Float rather than String?
I'm storing latLng data in my db. Right now I have my data type set to Number for lat & lng. When I check out my db I see this:
{
"_id" : ObjectId("563bd98a105249f325bb8a7e"),
"lat" : 41.8126189999999980,
"lng" : -87.8187850000000054,
"created" : ISODate("2015-11-05T22:34:50.511Z"),
"__v" : 0,
"section" : 0,
}
But when I get my data back out using express I get this:
{
"_id": "563bd98a105249f325bb8a7e",
"lat" : "41.8126189999999980",
"lng" : "-87.8187850000000054",
"__v": 0,
"section" : 0,
"created" : "2015-11-05T22:34:50.511Z",
}
My model:
var WaypointSchema = new Schema({
lat: {
type: Number
},
lng: {
type: Number
},
section: {
type: Number
}
created: {
type: Date,
default: Date.now
}
});
mongoose.model('Waypoint', WaypointSchema);
Express controller:
exports.list = function(req, res) {
Waypoint.find().sort('-created').populate('user', 'displayName').exec(function(err, waypoints) {
if (err) {
return res.status(400).send({
message: errorHandler.getErrorMessage(err)
});
} else {
res.jsonp(waypoints);
}
});
};
While the mongoDB fully supports float type, the mongoose supports only type of Number which is integer. If you try to save to mongoDB float number using mongooses type of Number it will be converted to string.
To sort this out, you will need to load some plugin for mongoose which will extend its value types. There are some plugins which work best with currencies or dates, but in your case I would use https://www.npmjs.com/package/mongoose-double.
Your model after changes would look something like this:
var mongoose = require('mongoose')
require('mongoose-double')(mongoose);
var SchemaTypes = mongoose.Schema.Types;
var WaypointSchema = new Schema({
lat: {
type: SchemaTypes.Double
},
lng: {
type: SchemaTypes.Double
},
section: {
type: Number
}
created: {
type: Date,
default: Date.now
}
});
mongoose.model('Waypoint', WaypointSchema);
Hope it helps.
As of the current version of mongoose (v5.12.6), It supports Decimal128 which can be used for this.
var mongoose = require('mongoose');<br>
var Schema = mongoose.Schema;<br>
var Waypoint = new Schema({<br>
lat: {<br>
type: SchemaTypes.Double<br>
},<br>
lng: {<br>
type: SchemaTypes.Double<br>
},<br>
section: {<br>
type: Number<br>
}<br>
point: {<br>
type: [Number],<br>
index: '2d'<br>
},<br>
}, {<br>
timestamps: true<br>
})<br>
event.index({<br>
Point: '2dsphere'<br>
});<br>
module.exports = mongoose.model('Waypoint', Waypoint);<br>
waypoint.save(point: [parseFloat(values.latitude), parseFloat(values.longitude)],)