I am trying to sanitize user input in mongoose. I though that using mongoose middleware would help, but it seems that I am either wrong or I am doing something wrong.
The reason I am trying to use Mongoose middleware (and not Express middleware) is that I have a document that can have a nested document - however, that nested document can be a standalone document as well. I am trying to create a "single point of truth" for my documents so that I can sanitize only in one place.
The following code does not seem to work:
Organization.pre("validate", function (next) {
this.subdomain = this.trim().toLowerCase();
next();
});
PS. I am also using mongoose-validator, which in turn uses node-validator to validate the user input - node validator also has some sanitize methods, maybe I should use them somehow?
In this case I think it would be better to add trim: true to the Organization schema definition for subdomain:
subdomain: { type: String, trim: true }
Related
I am fairly new to MongoDB and Mongoose, I am really confused about why some of the middleware works at the document and some works on query. I am also confused about why some of the query methods return documents and some return queries. If a query is returning document it is acceptable, but why a query return query and what really it is.
Adding more to my question what is a Document function and Model or Query function, because both of them have some common methods like updateOne.
Moreover, I have gathered all these doubts from the mongoose documentation.
Tl;dr: the type of middleware most commonly defines what the this variable in a pre/post hook refers to:
Middleware Hook
'this' refers to the
methods
Document
Document
validate, save, remove, updateOne, deleteOne, init
Query
Query
count, countDocuments, deleteMany, deleteOne, estimatedDocumentCount, find, findOne, findOneAndDelete, findOneAndRemove, findOneAndReplace, findOneAndUpdate, remove, replaceOne, update, updateOne, updateMany
Aggregation
Aggregation object
aggregate
Model
Model
insertMany
Long explanation:
Middlewares are nothing, but built-in methods to interact with the database in different ways. However, as there are different ways to interact with the database, each with different advantages or preferred use-cases, they also behave differently to each other and therefor their middlewares can behave differently, even if they have the same name.
By themselves, middlewares are just shorthands/wrappers for the mongodbs native driver that's being used under the hood of mongoose. Therefor, you can usually use all middlewares, as if you were using regular methods of objects without having to care if it's a Model-, Query-, Aggregation- or Document-Middleware, as long as it does what you want it to.
However, there are a couple of use-cases where it is important to differentiate the context in which these methods are being called.
The most prominent use-case being hooks. Namely the *.pre() and the *.post() hooks. These hooks are methods that you can "inject" into your mongoose setup, so that they are being executed before or after specific events.
For example:
Let's assume I have the following Schema:
const productSchema = new Schema({
name: 'String',
version: {
type: 'Number',
default: 0
}
});
Now, let's say you always want to increase the version field with every save, so that it automatically increases the version field by 1.
The easiest way to do this would be to define a hook that takes care of this for us, so we don't have to care about this when saving an object. If we for example use .save() on the document we just created or fetched from the database, we'd just have to add the following pre-hook to the schema like this:
productSchema.pre('save', function(next) {
this.version = this.version + 1; // or this.version += 1;
next();
});
Now, whenever we call .save() on a document of this schema/model, it will always increment the version before it is actually being saved, even if we only changed the name.
However, what if we don't use the .save() or any other document-only middleware but e.g. a query middleware like findOneAndUpdate() to update an object?
Then, we won't be able to use the pre('save') hook, as .save() won't be called. In this case, we'd have to implement a similar hook for findOneAndUpdate().
Here, however, we finally come to the differences in the middlewares, as the findOneAndUpdate() hook won't allow us to do that, as it is query hook, meaning it does not have access to the actual document, but only to the query itself. So if we e.g. only change the name of the product the following middleware would not work as expected:
productSchema.pre('findOneAndUpdate', function(next) {
// this.version is undefined in the query and would therefor be NaN
this.version = this.version + 1;
next();
});
The reason for this is, that the object is directly updated in the database and not first "downloaded" to nodejs, edited and "uploaded" again. This means, that in this hook this refers to the query and not the document, meaning, we don't know what the current state of version is.
If we were to increment the version in a query like this, we'd need to update the hook as follows, so that it automatically adds the $inc operator:
productSchema.pre('findOneAndUpdate', function(next) {
this.$inc = { version: 1 };
next();
});
Alternatively, we could emulate the previous logic by manually fetching the target document and editing it using an async function. This would be less efficient in this case, as it would always call the db twice for every update, but would keep the logic consistent:
productSchema.pre('findOneAndUpdate', async function() {
const productToUpdate = await this.model.findOne(this.getQuery());
this.version = productToUpdate.version + 1;
next();
});
For a more detailed explanation, please the check the official documentation that also has a designated paragraph for the problem of having colliding naming of methods (e.g. remove() being both a Document and Query middleware method)
I have a very specific question. I have a web project that is using Express (Node.JS) and MLab (MongoDB/Mongoose). I've manually edited several records in a collection (yeah, I know, bad idea) and am using one of those fields in a Mongoose search. The schema is defined as follows: (relevant part only)
user: {
id: {
type: mongoose.Schema.Types.ObjectId,
ref: "Registration"
},
username: String,
type: String
}
My search is as follows:
Master.find({$or: [{'user.type': 'committee'}, {'user.type': 'admin'}]}, function(err, foundUsers) {
do stuff
});
The search works just fine (using 'user.type'), but the user object in each record is undefined in foundUsers.
What am I missing?
Thanks!
Found it. I tried to pull a fast one and add something to the user record that wasn't in the Registration Schema. Mongo was smarter than I was in this case.
Mongo 3.2 have document validation, can we use the same to define a schema instead of using mongoose to do so.? For example :
Mongoose
userschema = mongoose.Schema({
org: String,
username: String,
fullname: String,
password: String,
email: String
});
MongoDB
db.createCollection(
"example",{
validator:{
$and:[
{ "org":{$type:"string"}},
{ "username":{$type:"string"}},
{ "fullname":{$type:"double"}},
{"password":$type:"string"}},
{"email":{$type:"string"}}
]
},
validationLevel:"strict",
validationAction:"error"
})
What ar ethe difference between these tow and can we provide an optional field using validator as in schema ?
I use both because they each have different limitations:
Mongoose validators do not run on all types of update queries, and validators only run on paths with values in the update doc because the validators can't know if, for example, a required field is already defined in the database but not in your client's memory (see issue). This is a major reason to use MongoDB validators [in addition to Mongoose validators].
update validators only run on $set and $unset operations (and $push and $addToSet in >= 4.8.0).
So you can have a field with required: true in your Mongoose schema, but an update operation will not actually require that field! A MongoDB validator can solve this:
db.runCommand({collMod: "collection", validator: {myfield: {$exists: true}}})
MongoDB for the most part cannot reference other fields during validation. For example, you can't say {field1: {$lte: field2}}. Mongoose validators can reference other fields.
You can do some very basic types of cross-field referencing though:
{validator: {myfield1: "Value 1", $and: [/* other validators */]}
This comes in handy if you're using Mongoose discriminators (inheritance) and have different requirements for each child type.
MongoDB does not provide "nice" errors in case of validation failure; it simply says something like writeError: {code: 121, errmsg: "Document failed validation}. Mongoose will typically say something like Path 'foo.bar' failed validation.
MongoDB is fixing this in v4.6.
Abilities that they share:
Type validation. Mongoose by default attempts to cast values to the type specified in the schema. MongoDB with the $type attribute will cause a validation failure in case of a type mismatch.
Min and max number values. Mongoose uses min and max attributes on the schema. MongoDB uses $lt, $lte, $gt and $gte.
String enums. Mongoose uses enum: [values]. MongoDB uses $in: [values].
String length validation. Mongoose: minlength: 2, maxlength: 10. MongoDB, use a regex: {fieldname: {$regex: /.{2,10}/}}.
Array length validation. Mongoose you have to use a custom validator. MongoDB: {fieldName: {$size: 2}}.
String RegExp matching. Mongoose you have to use a custom validator.
The first bullet point is a major one. MongoDB does not have transactionsnow has transactions, but it does have powerful (and cheap) atomic updates. You often times can't reliably or safely read -> change -> validate -> write with MongoDB, so using MongoDB native validators is critical in these cases.
Since the last answer, MongoDB 4.0 have been released.
the $jsonSchema feature now have more options than base mongoose Schema validator. (you can add custom validator in mongoose, though).
the use of allOf, oneOf, anyOf and not operator permit to do complex matching, similar to Mongoose discriminator.
with the $exec command, it is possible to compare the value of two field of the same document like so :
db.createCollection("test", {
validator : {
$expr : {$gte: ["$budget", "$spend"]}
}
})
will validate that the value of the field budget must be greater or equal than the value of spend.
(example adapted from mongodb documentation)
MongoDB still have the problem of non informative error message.
Personnally, I validate my data client side (making request to database if necessary to check for uniqueness). This way, the validation of mongodb have error only if there is concurent modification (someone modified the data between the moment you check and the moment you save). when there is mongodb error, I can simply rerun the client side validation to see what wrong.
I think Mongoose is used to it's fullest when used with a find-modify-save strategy, witch permit to use all the feature. this strategy need the use of versioning or locking to prevent concurrent modification.
when going for atomic update (using mongodb operator, update or findAndModify), with the current state of mongodb validation, I would be tempted to not use mongoose (or only use it for the connection management)
I am trying to use the new unstable version of mongoose >4.0.0 to validate update queries.
say that i want to update a schema using the following query
schema.update({_id:'blah'},{a:'blah'},function(err){
//do your thing
})
so lets say i have the following schema,
var schema = new Schema({
a:{type:String}
});
schema.pre('update',function(next){
var findQuery=this._conditions; // gives {_id:'blah'}
// how do i get {a:'blah'}????
next();
});
how do i get the update query of {set:{a:'blah'}} in the pre middleware so i can do some checks before executing the update?
alternatively i know that the update query can be accessed in the post middleware, in
schema.post('update',function(){
var findQuery=this._conditions; // gives {_id:'blah'}
var updateQuery=this._update; //gives {$set:{a:'blah'}}
next();
});
but thats too late, i need this in the pre middleware to check before actually updating the db.
i tried looking through the 'this' object of the pre middleware but cannot find the updateQuery object anywhere and this._update is undefined in the pre middleware.
Is there a way to do this?
thanks
In case you're still looking for a solution that works on array operations, it looks like in newer versions of mongoose (at least 4.0.7+), this._update is defined in the pre-middleware.
I found a work around through this particular example, however it doesnt quite solve my actual problem. what you can do in mongoose version ~4.0.0 is to let the pre middleware specify to go through the model validation on update.
schema.pre('update',function(next){
this.options.runValidators = true; // make sure any changes adhere to schema
})
basically, you can then specify the validators inside the schema
var schema = new Schema({
a:{
type:String,
validate:[...] //the validation you want to run
}
});
you can choose to skip the validation on a normal save operation by using the this.isNew check inside validation functions.
this code will run validate:[...] on any $set and $unset to a in your update query.
however, it doesn't work on array operations like $push or $addToSet for some reason. so if your updating an array, it won't run the validation code at all! hence it doesn't solve the actual problem im faced with. but it can work with the example provided for anyone that comes across this particular problem
I have a mongoose User schema built like this:
var UserSchema = new Schema({
username: { type: String, required: true, index: { unique: true } },
password: { type: String, required: true },
salt: { type: String, required: true}
});
I want to be able to send this user object to the client side of my application but I don't want to sned the password or salt fields.
So I added he following code to my user model module
U
serSchema.methods.forClientSide = function() {
console.log('in UserSchema.methods.forClientSide');
console.log(this);
//var userForClientSide=_.omit(this,'passsword','salt');
var userForClientSide={_id:this._id, username:this.username };
console.log(userForClientSide);
return userForClientSide;
}
I have required the underscore module (its installed locally via a dependency in my package.js).
not the commented out line - I was expecting it to omit the password and salt fields of the user object but it did not do anything :( the logged object had the full set of properties.
when replaced with the currently used like var userForClientSide={_id:this._id, username:this.username }; it gets the results I want but:
1) I want to know why does the _.omit not work.
2) I don't like my current workaround very much because it actually selects some properties instead of omitting the ones I don't like so if I will add any new propertes to the scema I will have to add them here as well.
This is my first attempt at writing something using node.js/express/mongodb/mongoose etc. so It is very possible hat I am missing some other better solution to this issue (possibly some feature of mongoose ) feel free to educate me of the right way to do things like this.
so basically I want to know both what is the right way to do this and why did my way not work.
thanks
1) I want to know why does the _.omit not work.
Mongoose uses defineProperty and some heavy metaprogramming. If you want to use underscore, first call user.toJSON() to get a plain old javascript object that will work better with underscore without all the metaprogramming fanciness, functions, etc.
A better solution is to use mongo/mongoose's fields object and pass the string "-password -salt" and therefore just omit getting these back from mongo at all.
Another approach is to use the mongoose Transform (search for "tranform" on that page). Your use case is the EXACT use case the documentation uses as an example.
You can also make your mongoose queries "lean" by calling .lean() on your query, in which case you will get back plain javascript objects instead of mongoose model instances.
However, after trying each of these things, I'm personally coming to the opinion that there should be a separate collection for Account that has the login details and a User collection, which will make leaking the hashes extremely unlikely even by accident, but any of the above will work.