Is there a way to access mongodb node.js driver functionality while still using mongoose for schema definition? - node.js

What I am really trying to do is to make indexes for filtering and string matching of documents based on their property values.
I know that mongodb has built in operators such as $text that are very helpful with this sort of functionality.
I'm not sure how to access these operators while using mongoose, or if there are any methods i need to use to access them.
I want to use mongoose to still define schema and models but need the functionality of native mongodb.
Is this possible?

Below are my views, Please add if I miss anything or if something needs to be modified or well-explained :
1. You will still be able to use mongoDB's native functionalities on using Mongoose models.
2. Mongoose is a kind of wrapper on top of native mongoDB-driver.
3. It would be very useful if you want to have schema based collections/data.
4. Additionally it would provide few more features than native mongoDB's driver. You might see few syntax differences between those two.
5. Few examples like `.findByIdAndUpdate()` & `.populate()` are mongoose specific, which has equivalent functionalities available in mongoDB driver/mongoDB as well.
6. In general it's quiet common to define mongoose models and use those over mongoDB's functionality in coding(As in node.js - You would write all of your same DB queries on Mongoose models, queries that you execute in DB).
Point 2 :
Mongoose is an object document modeling (ODM) layer that sits on top of Node's MongoDB driver. If your coming from SQL, it's similar to an ORM for a relational database.
Point 3 :
In code if you're using mongoose models to implement your write queries, unless you define a field in model - it wouldn't be added to DB though you pass it in request. Additionally you can do multiple things like making a field unique/required etc.. it's kind of making your mongoDB data look like schema based. If your collections data is more like random data(newsfeed kind of thing where fields are not same for each document & you can't predict data) then you might not care of using mongoose.
Point 6 :
Let's say you use mongo shell or a client like mongo compass/robo3T and execute a query that's like this :
db.getCollection('yourCollection').find(
{
$text: {
$search: 'employeeName',
$diacriticSensitive: false
},
country: 'usa'
},
{
employee_id: 1,
name: 1
}
).sort({ score: { $meta: 'textScore' } });
you would do same on mongoose model(As yourCollectionModel is already defined) :
yourCollectionModel.find(
{
$text: {
$search: 'employeeName',
$diacriticSensitive: false
},
country: 'usa'
},
{
employee_id: 1,
name: 1
}
).sort({ score: { $meta: 'textScore' } });
You would see key functionality difference more on writes rather than reads while using mongoose, though all the above is not about performance - If you ask me, I can say you might see much performance gains using mongoose.
Ref : Mongoose Vs MongoDB Driver

Related

how to set mongoose indexes correctly and test them

I want to set 2 indexes for now, perhaps a 3rd but wanted to know how I can test if they are actually working? Do I need to use with mongo shell or is there a way to check using Node.js during development? I also saw an example of the indexes being created in mongoDb Compass. I am using mongoDb Atlas so wondered if I must just set the index in Compass or do I still need to do it in my mongoose schema?
Also, the mongoose docs say you should set autoIndex to false. Is the below then correct?
const mongoose = require("mongoose");
const Schema = mongoose.Schema;
const userSchema = new Schema({
firstName: {
type: String,
},
lastName: {
type: String,
},
});
userSchema.set("autoIndex", false);
userSchema.index({ firstName: 1, lastName: 1 });
module.exports = mongoose.model("User", userSchema);
There are a bunch of different questions here, let's see if we can tackle them in order.
I want to set 2 indexes for now, perhaps a 3rd
This isn't a question from your side, but rather from mine. What are the indexes that you are considering and what queries will you be running?
The reason I ask is because I only see a single index definition provided in the question ({ firstName: 1, lastName: 1 }) and no query. Normally indexes are designed specifically to support the queries, so the first step towards ensuring a successful indexing strategy is to make sure they align appropriately with the anticipated workload.
how I can test if they are actually working? Do I need to use with mongo shell or is there a way to check using Node.js during development?
There are a few ways to approach this, which include:
Using the explain() method to confirm that the winningPlan is using the index as expected. This is often done via the MongoDB Shell or via Compass.
Using the $indexStats aggregation stage to confirm that usage counters of the index are incrementing as expected when the application runs.
Taking a look at some of the tabs in the Atlas UI such as Performance Advisor or the Profiler which may help alert you to unoptimized operations and missing indexes.
I am using mongoDb Atlas so wondered if I must just set the index in Compass or do I still need to do it in my mongoose schema?
You can use Compass (or the Atlas UI, or the MongoDB Shell) to create your indexes. I would recommend against doing this in the application directly.
Also, the mongoose docs say you should set autoIndex to false. Is the below then correct?
As noted above, I would go further and remove index creation from the application code altogether. There can be some unintended side effects of making the application directly responsible for index management, which is one of the reasons that Mongoose no longer recommends using the autoIndex functionality.

mongoose query using sort and skip on populate is too slow

I'm using an ajax request from the front end to load more comments to a post from the back-end which uses NodeJS and mongoose. I won't bore you with the front-end code and the route code, but here's the query code:
Post.findById(req.params.postId).populate({
path: type, //type will either contain "comments" or "answers"
populate: {
path: 'author',
model: 'User'
},
options: {
sort: sortBy, //sortyBy contains either "-date" or "-votes"
skip: parseInt(req.params.numberLoaded), //how many are already shown
limit: 25 //i only load this many new comments at a time.
}
}).exec(function(err, foundPost){
console.log("query executed"); //code takes too long to get to this line
if (err){
res.send("database error, please try again later");
} else {
res.send(foundPost[type]);
}
});
As was mentioned in the title, everything works fine, my problem is just that this is too slow, the request is taking about 1.5-2.5 seconds. surely mongoose has a method of doing this that takes less to load. I poked around the mongoose docs and stackoverflow, but didn't really find anything useful.
Using skip-and-limit approach with mongodb is slow in its nature because it normally needs to retrieve all documents, then sort them, and after that return the desired segment of the results.
What you need to do to make it faster is to define indexes on your collections.
According to MongoDB's official documents:
Indexes support the efficient execution of queries in MongoDB. Without indexes, MongoDB must perform a collection scan, i.e. scan every document in a collection, to select those documents that match the query statement. If an appropriate index exists for a query, MongoDB can use the index to limit the number of documents it must inspect.
-- https://docs.mongodb.com/manual/indexes/
Using indexes may cause increased collection size but they improve the efficiency a lot.
Indexes are commonly defined on fields which are frequently used in queries. In this case, you may want to define indexes on date and/or vote fields.
Read mongoose documentation to find out how to define indexes in your schemas:
http://mongoosejs.com/docs/guide.html#indexes

Mongoose Schema vs Mongo Validator

Mongo 3.2 have document validation, can we use the same to define a schema instead of using mongoose to do so.? For example :
Mongoose
userschema = mongoose.Schema({
org: String,
username: String,
fullname: String,
password: String,
email: String
});
MongoDB
db.createCollection(
"example",{
validator:{
$and:[
{ "org":{$type:"string"}},
{ "username":{$type:"string"}},
{ "fullname":{$type:"double"}},
{"password":$type:"string"}},
{"email":{$type:"string"}}
]
},
validationLevel:"strict",
validationAction:"error"
})
What ar ethe difference between these tow and can we provide an optional field using validator as in schema ?
I use both because they each have different limitations:
Mongoose validators do not run on all types of update queries, and validators only run on paths with values in the update doc because the validators can't know if, for example, a required field is already defined in the database but not in your client's memory (see issue). This is a major reason to use MongoDB validators [in addition to Mongoose validators].
update validators only run on $set and $unset operations (and $push and $addToSet in >= 4.8.0).
So you can have a field with required: true in your Mongoose schema, but an update operation will not actually require that field! A MongoDB validator can solve this:
db.runCommand({collMod: "collection", validator: {myfield: {$exists: true}}})
MongoDB for the most part cannot reference other fields during validation. For example, you can't say {field1: {$lte: field2}}. Mongoose validators can reference other fields.
You can do some very basic types of cross-field referencing though:
{validator: {myfield1: "Value 1", $and: [/* other validators */]}
This comes in handy if you're using Mongoose discriminators (inheritance) and have different requirements for each child type.
MongoDB does not provide "nice" errors in case of validation failure; it simply says something like writeError: {code: 121, errmsg: "Document failed validation}. Mongoose will typically say something like Path 'foo.bar' failed validation.
MongoDB is fixing this in v4.6.
Abilities that they share:
Type validation. Mongoose by default attempts to cast values to the type specified in the schema. MongoDB with the $type attribute will cause a validation failure in case of a type mismatch.
Min and max number values. Mongoose uses min and max attributes on the schema. MongoDB uses $lt, $lte, $gt and $gte.
String enums. Mongoose uses enum: [values]. MongoDB uses $in: [values].
String length validation. Mongoose: minlength: 2, maxlength: 10. MongoDB, use a regex: {fieldname: {$regex: /.{2,10}/}}.
Array length validation. Mongoose you have to use a custom validator. MongoDB: {fieldName: {$size: 2}}.
String RegExp matching. Mongoose you have to use a custom validator.
The first bullet point is a major one. MongoDB does not have transactionsnow has transactions, but it does have powerful (and cheap) atomic updates. You often times can't reliably or safely read -> change -> validate -> write with MongoDB, so using MongoDB native validators is critical in these cases.
Since the last answer, MongoDB 4.0 have been released.
the $jsonSchema feature now have more options than base mongoose Schema validator. (you can add custom validator in mongoose, though).
the use of allOf, oneOf, anyOf and not operator permit to do complex matching, similar to Mongoose discriminator.
with the $exec command, it is possible to compare the value of two field of the same document like so :
db.createCollection("test", {
validator : {
$expr : {$gte: ["$budget", "$spend"]}
}
})
will validate that the value of the field budget must be greater or equal than the value of spend.
(example adapted from mongodb documentation)
MongoDB still have the problem of non informative error message.
Personnally, I validate my data client side (making request to database if necessary to check for uniqueness). This way, the validation of mongodb have error only if there is concurent modification (someone modified the data between the moment you check and the moment you save). when there is mongodb error, I can simply rerun the client side validation to see what wrong.
I think Mongoose is used to it's fullest when used with a find-modify-save strategy, witch permit to use all the feature. this strategy need the use of versioning or locking to prevent concurrent modification.
when going for atomic update (using mongodb operator, update or findAndModify), with the current state of mongodb validation, I would be tempted to not use mongoose (or only use it for the connection management)

Difference between MongoDB and Mongoose

I wanted to use the mongodb database, but I noticed that there are two different databases with either their own website and installation methods: mongodb and mongoose. So I came up asking myself this question: "Which one do I use?".
So in order to answer this question I ask the community if you could explain what are the differences between these two? And if possible pros and cons? Because they really look very similar to me.
I assume you already know that MongoDB is a NoSQL database system which stores data in the form of BSON documents. Your question, however is about the packages for Node.js.
In terms of Node.js, mongodb is the native driver for interacting with a mongodb instance and mongoose is an Object modeling tool for MongoDB.
mongoose is built on top of the mongodb driver to provide programmers with a way to model their data.
EDIT:
I do not want to comment on which is better, as this would make this answer opinionated. However I will list some advantages and disadvantages of using both approaches.
Using mongoose, a user can define the schema for the documents in a particular collection. It provides a lot of convenience in the creation and management of data in MongoDB. On the downside, learning mongoose can take some time, and has some limitations in handling schemas that are quite complex.
However, if your collection schema is unpredictable, or you want a Mongo-shell like experience inside Node.js, then go ahead and use the mongodb driver. It is the simplest to pick up. The downside here is that you will have to write larger amounts of code for validating the data, and the risk of errors is higher.
Mongo is NoSQL Database.
If you don't want to use any ORM for your data models then you can also use native driver mongo.js: https://github.com/mongodb/node-mongodb-native.
Mongoose is one of the orm's who give us functionality to access the mongo data with easily understandable queries.
Mongoose plays as a role of abstraction over your database model.
One more difference I found with respect to both is that it is fairly easy to connect to multiple databases with mongodb native driver while you have to use work arounds in mongoose which still have some drawbacks.
So if you wanna go for a multitenant application, go for mongodb native driver.
From the first answer,
"Using Mongoose, a user can define the schema for the documents in a particular collection. It provides a lot of convenience in the creation and management of data in MongoDB."
You can now also define schema with mongoDB native driver using
##For new collection
db.createCollection("recipes",
validator: { $jsonSchema: {
<<Validation Rules>>
}
}
)
##For existing collection
db.runCommand({
collMod: "recipes",
validator: { $jsonSchema: {
<<Validation Rules>>
}
}
})
##full example
db.createCollection("recipes", {
validator: {
$jsonSchema: {
bsonType: "object",
required: ["name", "servings", "ingredients"],
additionalProperties: false,
properties: {
_id: {},
name: {
bsonType: "string",
description: "'name' is required and is a string"
},
servings: {
bsonType: ["int", "double"],
minimum: 0,
description:
"'servings' is required and must be an integer with a minimum of zero."
},
cooking_method: {
enum: [
"broil",
"grill",
"roast",
"bake",
"saute",
"pan-fry",
"deep-fry",
"poach",
"simmer",
"boil",
"steam",
"braise",
"stew"
],
description:
"'cooking_method' is optional but, if used, must be one of the listed options."
},
ingredients: {
bsonType: ["array"],
minItems: 1,
maxItems: 50,
items: {
bsonType: ["object"],
required: ["quantity", "measure", "ingredient"],
additionalProperties: false,
description: "'ingredients' must contain the stated fields.",
properties: {
quantity: {
bsonType: ["int", "double", "decimal"],
description:
"'quantity' is required and is of double or decimal type"
},
measure: {
enum: ["tsp", "Tbsp", "cup", "ounce", "pound", "each"],
description:
"'measure' is required and can only be one of the given enum values"
},
ingredient: {
bsonType: "string",
description: "'ingredient' is required and is a string"
},
format: {
bsonType: "string",
description:
"'format' is an optional field of type string, e.g. chopped or diced"
}
}
}
}
}
}
}
});
Insert collection Example
db.recipes.insertOne({
name: "Chocolate Sponge Cake Filling",
servings: 4,
ingredients: [
{
quantity: 7,
measure: "ounce",
ingredient: "bittersweet chocolate",
format: "chopped"
},
{ quantity: 2, measure: "cup", ingredient: "heavy cream" }
]
});
Mongodb and Mongoose are two different drivers to interact with MongoDB database.
Mongoose : object data modeling (ODM) library that provides a rigorous modeling environment for your data. Used to interact with MongoDB, it makes life easier by providing convenience in managing data.
Mongodb: native driver in Node.js to interact with MongoDB.
mongo-db is likely not a great choice for new developers.
On the other hand mongoose as an ORM (Object Relational Mapping) can be a better choice for the new-bies.
If you are planning to use these components along with your proprietary code then please refer below information.
Mongodb:
It's a database.
This component is governed by the Affero General Public License (AGPL) license.
If you link this component along with your proprietary code then you have to release your entire source code in the public domain, because of it's viral effect like (GPL, LGPL etc)
If you are hosting your application over the cloud, the (2) will apply and also you have to release your installation information to the end users.
Mongoose:
It's an object modeling tool.
This component is governed by the MIT license.
Allowed to use this component along with the proprietary code, without any restrictions.
Shipping your application using any media or host is allowed.
Mongoose is built untop of mongodb driver, the mongodb driver is more low level. Mongoose provides that easy abstraction to easily define a schema and query. But on the perfomance side Mongdb Driver is best.
Mongodb and Mongoose are two completely different things!
Mongodb is the database itself, while Mongoose is an object modeling tool for Mongodb
EDIT: As pointed out MongoDB is the npm package, thanks!
MongoDB is The official MongoDB Node.js driver allows Node.js applications to connect to MongoDB and work with data.
On the other side Mongoose it other library build on top of mongoDB. It is more easier to understand and use. If you are a beginner than mongoose is better for you to work with.

MongoDB 2.6 Production Ready Text Search - How To Use Skip For Pagination

In MongoDB 2.6, the text-search is supposedly production ready and we can now use skip. I'd like to use text-search and skip for pagination in my, but I'm not yet sure how to implement it.
Right now, I'm using Mongoose and the Mongoose-text-search plugin, but I don't believe either of them support skip in MongoDB's text search, so I guess I'll need to use the native MongoClient...
My app connects to the database via Mongoose using:
//Bootstrap db connection
var db = mongoose.connect(config.db, function(e) {
Now, how can I use the native MongoClient to execute a full text search for my Products model, with a skip parameter. Here is what I had using Mongoose and Mongoose-text-search, but there is no way to add in skip:
Product = mongoose.model('Product')
var query = req.query.query;
var skip = req.query.skip;
var options = {
project: '-created', // do not include the `created` property
filter: filter, // casts queries based on schema
limit: 20,
language: 'english',
lean: true
};
Product.textSearch(query, options, function (err, response) {
});
The main difference introduced in 2.6 versions of MongoDB is that you can issue a "text search" query using the standard .find() interface so the old methods for textSearch would no longer need to be applied. This is basically how modifiers such as limit and skip can be applied.
But keep in mind that as of writing the current Mongoose dependency is for an earlier version of the MongoDB node driver that existed prior to the release of MongoDB 2.6. As Mongoose actually wraps the main methods and does some syntax checking of it's own, it is probably likely ( as in untried by me ) that using the Mongoose methods will currently fail.
So what you will need to do is get the underlying driver method for .find(), and also now use the $text operator instead:
Product.collection.find(
{ "$text": { "$search": "term" } },
{ "sort": { "score": { "$meta": "textScore" } }, "skip": 25, "limit": 25 },
function(err,docs) {
// processing here
});
Also noting that $text operator does not sort the results by "score" for relevance by default, but this is passed to the "sort" option using the new $meta operator, which is also introduced in MongoDB 2.6.
So alter your skip and limit values and you have paging on text search results and with a cursor. Just be wary of large data returns as skip and limit are not really efficient ways to move through a large cursor. Better to have another key where you can range match, even though counter-intuitive to "relevance matching".
So, text search facilities are a bit "better" but not "perfect". As always, if you really need more and/or more performance, look to an external solution.
Feel free to try a similar operation with the Mongoose implementation of .find() as well. But have my reservations from past experience that there is generally some masking and checking going on there, so hence the description of usage with the "native" node driver.

Resources