Prevent _id field from being supplied in a MongoDB query - node.js

I'm using Mongoose in NodeJS to control a MongoDB database.
I'm creating an API and for obvious security reasons, I want to prevent the auto generated document _id field from getting replaced by a manually generated one in the API request.
Schema:
{ name: String }
Creating a document:
const record = {
_id: '5e35517cc894c90327a34baf'
name: 'bob'
}
const insertRecords = async () => {
await Quiz.create(record);
};
insertRecords();
Results in the following document:
{
_id: '5e35517cc894c90327a34baf'
name: 'bob'
}
As can be seen, the _id supplied in the query, as long as it's a valid ObjectID, would replace the _id that was supposed to be auto generated by mongo.
Is there a way to check if this _id field is in the query so that I can reject the API request? The .create method triggers the pre save middleware hook which would always have the _id of the final document so I cannot depend on it to know whether the _id was in the query or it's the auto generated one.
The only option I found is to disable the _id field altogether but this does not make sense.

Solution #1 - Use .create() method with an explicit object.
It's actually easier than you think. This is self-explanatory - we only define what we want to allow. Mongoose will ignore anything that's not in the object.
const record = {
_id: '5e35517cc894c90327a34baf'
name: 'bob'
}
const insertRecords = async () => {
await Quiz.create({
name: record.name // only allow names.
});
};
insertRecords();
Solution #2 - Define a function to clear unwanted objects.
You can define a helper function to clear out unwanted fields.
const filterObj = (obj, ...allowedFields) => {
const newObject = {};
// If the current field is one of the allowed fields, keep them in the new object.
Object.keys(obj).forEach((el) => {
if (allowedFields.includes(el)) {
newObject[el] = obj[el];
}
});
return newObject;
};
How to use:
const filteredRecord = filterObj(record, 'name'); // arbitrary list of allowed fields. In this case, we'll only allow 'name'.
await Quiz.create(filteredRecord);

Related

Mongo changeStream for a find query

I want to make a query on mongo collection and when the collection changes I want to run the query again. But to optimize it, I don't want to run the query on every collection change, only when documents matching the query changes. I have following code:
const query = { author: someUserID };
const fetch = async () => await collection.find(query).toArray();
const watcher = collection
.watch([{ $match: { fullDocument: query } }])
.on("change", () => fetch().then(sendData)); // This does not work
fetch().then(sendData); // This works
On first run it fetches documents and executes sendData but when new document is inserted, the event is not triggered. When I wun collection.watch() without arguments, it works.
Where is the problem?
Thanks.
EDIT: I want to be able to reuse the query for .find() and for .watch().
The $match stage in that example is essentially
{$match: { fullDocument: { author: someUserId }}}
That will match only if the fullDocument is exactly { author: someUserId } with no other fields or values.
In order to match the author while permitting other fields in the document, use dotted notation, like
const query = { "fullDocument.author": someUserId };
and match like:
{$match: query }

Mongoose - remove _id from lean ( using toJSON() ?)

I'm having a NODE.JS project using mongoose 5.x
My model have toJSON method which removes the _id & __v fields perfectly
mySchema.method("toJSON", function toJSON() {
const {__v, _id, ...object} = this.toObject();
return {
id: _id,
...object
};
});
so when fetching data from the db:
const data = myModel.findOne({_id: id});
I get an object that when serialized to the user:
res.json(data);
It doesn't contain the _id and __v fields as required.
The problem is when I use lean():
const data = myModel.findOne({_id: id}).lean();
the data object contains those fields.
I can remove them manually when using lean
but I would prefer to find a way to sanitize the data object in both cases with the same mechanism.
any suggestions?
Thanks in advance.
Not sure if this is what you want but maybe:
const data = myModel.findOne({_id: id}).lean().then(res => {
delete res._id
return res
})
When JSON.stringify() is called on an object, a check if done if it has a property called toJSON. It's not specific to Mongoose, as it works on plain objects:
const myObj = {
_id: 123,
foo: 'bar',
// `toJSON() {...}` is short for `toJSON: function toJSON() {...}`
toJSON() {
// Because "this" is used, you can't use an arrow function.
delete this._id;
return this;
}
};
console.log(JSON.stringify(myObj));
// {"foo":"bar"}
Mongoose doesn't have an option to automatically inject a toJSON function into objects returned by lean(). But that's something you can add.
First, create a function that:
Takes an object with properties
Listens to when Mongoose runs a find query
Tells Mongoose that after the query, it should change the result
The change: merge the result with the object from step 1.
function mergeWithLeanObjects(props) {
// Return a function that takes your schema.
return function(schema) {
// Before these methods are run on the schema, execute a function.
schema.pre(['find', 'findOne'], function() {
// Check for {lean:true}
if (this._mongooseOptions.lean) {
// Changes the document(s) that will be returned by the query.
this.transform(function(res) {
// [].concat(res) makes sure its an array.
[].concat(res).forEach(obj => Object.assign(obj, props));
return res;
});
}
});
};
}
Now add it to your schema:
mySchema = new mongoose.Schema({
foo: String
});
mySchema.plugin(mergeWithLeanObjects({
toJSON() {
delete this._id;
delete this.__v;
return this;
}
}));
Then test it:
const full = await myModel.findOne();
const lean = await myModel.findOne().lean();
console.log(full);
// Logs a Mongoose document, all properties.
{ _id: new ObjectId("62a8b39466768658e7333154"), foo: 'bar', __v: 1 }
console.log(JSON.stringify(full));
// Logs a JSON string, all properties.
{"_id":"62a8b39466768658e7333154","foo":"bar","__v":1}
console.log(lean);
// Logs an Object, all properties.
{ _id: new ObjectId("62a8b39466768658e7333154"),
foo: 'bar', __v: 1, toJSON: [Function: toJSON] }
console.log(JSON.stringify(lean));
// Logs a JSON string, filtered properties.
{"foo":"bar"}
If you want to re-use the plugin with the same settings on multiple schemas, just save the function that mergeWithLeanObjects returns somewhere.
// file: clean.js
module.exports = mergeWithLeanObjects({
toJSON() {
delete this._id;
delete this.__v;
return this;
}
});
// file: some-schema.js
schema1.plugin(require('./clean.js'));
// file: some-other-schema.js
schema2.plugin(require('./clean.js'));
There's also mongoose.plugin() to add the function to all schemas.
Try this to retrieve the _id from a document
myModel.findOne({_id: id}, function(err, doc) {
if(err)
return 'do something with this err'
console.log(doc._id)
})

Mongoose find and findById inconsistency

I'm testing mongoose queries, and I found this bizarre behaviour. I have a document with _id: "5de64b376c79643fa847e86b", and if I call the findById method the document is returned just fine. But if I call the find method on the collection, giving as args an array with the same ID value than nothing is returned, while I expect an array of one element being the document. To summarize:
mongooseModel.findById(mongoose.Types.ObjectId("5de64b376c79643fa847e86b")) // Works
mongooseModel.find({ _id: { $in: [mongoose.Types.ObjectId("5de64b376c79643fa847e86b")] } }) //Doesn't work
What's the difference between the two, and why the second one doesn't work?
EDIT: This is kinda the code to access that method.
I define a DataSource in the ApolloServer configuration
const app = new ApolloServer({
...
dataSources: () => ({
source: new SourceAPI(DocumentModel)
})
...
});
where SourceAPI is the DataSource class and DocumentModel is the mongoose model.
SourceAPI is defined like this
class SourceAPI {
async get(ids) {
return await DocumentModel.find({
_id: {
$in: ids
}
});
}
}
Now, inside the GraphQL resolver I finally call the API method to get the documents, like this
const findResolver = () =>
DocumentSchema.get("$findById").wrapResolve(next => async rp => {
let ids = [];
ids.push(mongoose.Types.ObjectId(rp.args._id));
return await rp.context.dataSources.source.get(ids);
});
where DocumentSchema is the GraphQL Schema for the Document Model generated using graphql-compose-mongoose package. The get("$findById") and wrapResolve methods are also from that package. What I do is using these methods to get the GraphQL query parameters and pass them to the API method (in this case I'm just grabbing an ID for test).
If I change the API method to something like this
async get(id) {
return await DocumentModel.findById(id);
}
and the resolver method to this
const findResolver = () =>
DocumentSchema.get("$findById").wrapResolve(next => async rp => {
return await rp.context.dataSources.source.get(mongoose.Types.ObjectId(rp.args._id));
});
everything works
$in is used to find items which holds an array, but _id is just a JSON key/value pair. So you cannot use $in. If your object looks like this then you can use $in
{
id: [ObjectId("5de64b376c79643fa847e86b"),
ObjectId("5de64b376c79643fa847e86b")]
}

How to remove mongo specific fields from result (NodeJS, Mongoose)

I want to remove all Mongo specific fields (like '_id') from query result. Is there a simple method to do this or should I remove fields manually? If yes, then which are that fields and how to do that?
I'm using NodeJS and Mongoose
You can use select() method for remove the field from your query:
Model.find({}).select("-removed_field").then (resp => {
// your code
});
You should specified the "-" before field name, to be remove this field.
If you want remove several fields - you can specified their as array:
Model.find({}).select(["-removed_field1", "-removed_field2" ... ]).then (resp => {
// your code
});
Also you can to select only specified fields, using this method without "-"
Model.find({}).select(["field1", "field2" ... ]).then (resp => {
// your code
});
If you want hide _id property you can use text argument with prefix - which will exclude this or that field from the result, for get sepecifict fields you should pass like this:
Entity.find({ ... }, 'field1 field2', function(err, entity) {
console.log(entity); // { field1: '...', field2: '...' }
});
You can specify a field to be excluded from results by using the optional 2nd parameter projection string of the find method:
Model.find({}, "-a -b").then (res => {
// objects in the res array will all have the
// 'a' and 'b' fields excluded.
});
https://mongoosejs.com/docs/api.html#model_Model.find (see projection)
you can use mongoose instance method two show specific fields from all documents
const userSchema = new mongoose.Schema({
email: {
type: String,
},
name: {
type: String,
maxlength: 128,
index: true,
trim: true,
},
});
userSchema.method({
transform() {
const transformed = {};
const fields = ['name', 'email'];
fields.forEach((field) => {
transformed[field] = this[field];
});
return transformed;
},
});
module.exports = mongoose.model('User', userSchema);
if You want to remove any specific fields like _id, You can try in two ways:
Suppose Here you try to find a user using User Model
User.find({ email: email }, { _id: 0 });
OR
const user = User.find({ email: email });
delete user._doc._id;
OP mentioned "from result", as far as I understood, it means, removing from the query result i.e. query result will contain the field, but will be removed from the query result.
A SO answer here mentions, that to modify a query result (which are immutable), we've to convert the result to Object using toObject() method (making it mutable).
To remove a field from a query result,
let immutableQueryResult = await Col.findById(idToBeSearched)
let mutableQueryResult = immutableQueryResult.toObject()
delete mutableQueryResult.fieldToBeRemoved
console.log(mutableQueryResult)
Another way of getting the mutable result is using the _doc property of the result:
let immutableQueryResult = await Col.findById(idToBeSearched)
let mutableQueryResult = immutableQueryResult._doc // _doc property holds the mutable object
delete mutableQueryResult.fieldToBeRemoved
console.log(mutableQueryResult)

MongoDB: handling auto-incrementing model id's instead of Mongo's native ObjectID

Due to a management decision, we are using userId for the users collection, postId for the posts collection, and topicId for the topics collection, instead of '_id' for each collection as the unique identifier.
This causes a few problems getting started - one of the problems I have encountered is with upserts -
Using Mongoose, we have a schema that restricts userId to be a unique value - but when doing an update on a user model, with upsert set to true, MongoDB appears to only look at the ObjectIds of a collection to see if the same one exists - it doesn't check to see if a model already exists with the same userId - therefore Mongo does an insert instead of an update.
let me illustrate this with some data:
let's say the user's collection has one document:
{
_id:'561b0fad638e99481ab6d84a'
userId:3,
name:'foo'
}
we then run:
User.update({userId:3},{"$set":{name:'bar'},{upsert:true},function(err,resp){
if(err){
// "errMessage": "insertDocument :: caused by :: 11000 E11000 duplicate key error index: app42153482.users.$userId_1 dup key: { : 3 }",
}
});
one would think that MongoDB would find the existing document with userId:3 and udpate it, so there must be something I am doing wrong since it's giving me the duplicate key error?
Typically the default value ObjectId is more ideal for the _id. Here, in this situation you can either override the default _id or you can have your own field for id(like userId in your case).
Use a separate counters collection to track the last number sequence used. The _id field contains the sequence name and the seq field contains the last value of the sequence.
Insert into the counters collection, the initial value for the userid:
db.counters.insert( {
_id: "userid",
seq: 0 } )
Create a getNextSequence function that accepts a name of the sequence. The function uses the findAndModify() method to atomically increment the seq value and return this new value:
function getNextSequence(name) {
var ret = db.counters.findAndModify(
{
query: { _id: name },
update: { $inc: { seq: 1 } },
new: true
}
);
return ret.seq;
}
Use this getNextSequence() function during insert().
db.users.insert(
{
_id: getNextSequence("userid"),
name: "Sarah C."
}
)
db.users.insert(
{
_id: getNextSequence("userid"),
name: "Bob D."
}
)
This way you can maintain as many sequences as you want in the same counter collection. For the upsert issue, check out the Optimistic Loop block in this link Create an auto-increment sequence field.
The second approach is to use a mongoose middleware like mongodb-autoincrement.
Hope it helps.
I don't know which versions of MongoDB and Mongoose you are using, but I couldn't reproduce your problem with MongoDB 3.0 and Mongoose 4.1.10.
I made a sample for you which will create and save a new user, update (using upsert) it, and create another one through an upsert. Try running this code:
"use strict";
var mongoose=require("mongoose");
var Schema = require('mongoose').Schema;
var ObjectId = mongoose.Schema.Types.ObjectId;
// Connect to test
mongoose.connect("mongodb://localhost:27017/test");
// Lets create your schema
var userSchema = new Schema({
_id: ObjectId,
userId: {type: Number, unique: true },
name: String
});
var User = mongoose.model("User", userSchema, "Users");
User.remove() // Let's prune our collection to start clean
.then( function() {
// Create our sample record
var myUser = new User({
_id:'561b0fad638e99481ab6d84a',
userId:3,
name:'foo'
});
return myUser.save();
})
.then( function() {
// Now its time to update (upsert userId 3)
return User.update({userId:3},{"$set":{name:'bar'}},{upsert:true});
})
.then( function() {
// Now its time to insert (upsert userId 4)
return User.update({userId:4},{"$set":{name:'bee'}},{upsert:true});
})
.then( function() {
// Lets show what we have inserted
return User.find().then(function(data) {console.log(data)});
})
.catch( function(err) {
// Show errors if anything goes wrong
console.error("ERROR", err);
})
.then( function() {
mongoose.disconnect();
});
Following the documentation (of MongoDB 3.0) upsert:true will only not insert a non-existing document if your query conditions match on the _id field.
See: https://docs.mongodb.org/manual/reference/method/db.collection.update/#mongodb30-upsert-id
Why are you not using the user_name for a user as unique id?
Because auto-incrementing fields as ids are a bad practice to use in a mongodb environment, especially if you want to use sharding
=> all your inserts will occur on the latest shard
=> the mongodb cluster will have to rebalance often / redistribute the data around.
(Currently this will not occur on your system as you still use the generated _id field)
You can off course also create a unique index on the user_id field:
https://docs.mongodb.org/manual/core/index-unique/#index-type-unique

Resources