mongoose - have optional additional implicit fields - node.js

I have a schema with a field in which I can store anything : new Schema({settings : {}}).
I have a database with
I want to keep this ability to add data without adding new fields, but for some of the fields have default values if they are not present.
I can do the following :
new Schema({
settings : {
key : { type : String, default: "abc" }
// I want to be able to add data that contains more than just "key"
}
});
I just want to make sure that when requesting the data from this schema, I will still get all the data, and not just the keys explicitly defined ?
It seems to work, but I want to make sure that I can still :
read all the data
still write arbitrary data (ie. not necessarily defined in the schema)
Are there rules on mongo/mongoose that would prevent me from doing one of these two things (I'm very unsure for the writing part) ? If there is such a "feature", how can it be done ?
Note : I saw this question. Correct me if I am wrong, but the fields are not implicit (like in the first case with {}), and have to be defined (it's actually the opposite question).
Edit : I now saw also this question that addresses my concerns (even if the accepted solution sounds more like a workaround to me). But in my case I already have data stored so (1 - disable strict) would mean writing a lot of validation code to be safe (because a lot of keys, this is the biggest collection of the app), and (2 - mixed schemas) would require to migrate the data of this specific sub-element... In short : I would still welcome a solution to my particular problem.

I think you will want to build your own custom validation here rather than rely on the defauly schema type validation methods. Luckily, mongoose has a facility for this:
var async = require('async'),
mongoose = require('mongoose'),
Schema = mongoose.Schema;
mongoose.connect('mongodb://localhost/test');
var testSchema = new Schema({
settings: {}
});
var Test = mongoose.model( 'Test', testSchema, "test" );
testSchema.path('settings').validate(function(value) {
return Object.keys(value).indexOf("key") != -1;
},'Error "settings" must contain "key" as "settings.key"');
//var test = new Test({ settings: { "key": "something" } });
var test = new Test({ settings: { } });
test.save(function(err) {
try {
if (err) throw err;
console.log(test);
} catch (e) {
console.log(e);
}
});
So basically, I have set up the validate function there for the "settings" path in the schema to look for the presence of "key" with it's own object. Where that "key" does not exist, an exception is reported in the errors.
Like any such errors, it will be returned within the err object when you .save() the object, thus blocking the write. It can be then be acted on to handle the error however you want, with the message that was defined reported.
So that is a self contained "test" where you can alternately uncomment the valid data for the object and successfully save that object without errors being reported.
Alternately you can do a "pre save" to fill in some default data:
testSchema.pre("save",function(next) {
this.settings = (this.settings) ? this.settings : {};
if (Object.keys(this.settings).indexOf("key") == -1)
this.setting.key = "abc";
next();
});
Which fills in a default if it is not already there.

Try to use like this way
new Schema({
settings : {}
});
var modelObj = new myModel();
modelObj.settings.key = "keyval";
modelObj.settings.key1 = "keyval";
modelObj.settings.key2 = "keyval";
modelObj.settings.key3 = "keyval";
modelObj.save(function(err){
//handle
});

Related

Validate relationship existance in MongoDB with Mongoose

I'm using Mongoose and its not in an advanced stage, so I need some help with some specific points. I will try to keep my examples clear and without much context for it.
First of all, I'm doing some relationships in my schemas. Before I create or edit any of them, I'm verifying if the provided ObjectId exists in database when necessary.
VehicleSchema = new Schema({
name: String,
})
PersonSchema = new Schema({
name: String,
vehicle: ObjectId //relation with vehicle
})
PersonSchema.pre('save', (next) => {
// ...
if (!Vehicles.countDocuments({ _id: this.vehicle }) throw new Error('blabla')
// ...
}
Is there any better way to do this or is this the best way possible to make sure that my doc exists?
I was thinking about three possibilities to help this be faster, but I'm not sure if is secure and consistent:
Create a custom ObjectId that indicates the modelName of my Schema in it. Something like:
function createObjectIdByModelName(modelName) {
return new ObjectId(`${modelName}-${uuid.v4()}`)
}
and then:
function validateObjectIdByModelName(_id, expectedModel) {
const modelName = mongoose.model.get(_id).modelName
return modelName === expectedModel
}
Use some cache package like recachegoose or speedgoose
Make my requests have an "origin" where I could create some rules like:
// This is a simple example of course, but the idea is that
// if the origin of my request is my frontend, I would trust in it, so my validation
// would be ignored. Otherwise I validate it normally.
if (origin !== 'frontend') {
if (!Vehicles.countDocuments({ _id: this.vehicle }) throw new Error('blabla')
}
What do you think about? This is blowing my mind for weeks now.

How can I dynamically generate Mongoose discriminators (at runtime?)

TL;DR: Is there a safe way to dynamically define a mongoose discriminator at runtime?
I have an app with a MongoDB collection where users have some control over the underlying schema.
I could add one or two fixed, required fields and just use mongoose.Mixed for the remainder that users can change, but I'd like to make use of Mongoose's validation and discriminators if I can.
So, what I've got is a second collection Grid where the users can define the shape they'd like their data to take, and in my main model Record, I've added a function to dynamically generate a discriminator from the definition in the second collection.
The code for my Record model looks like this:
const mongoose = require("mongoose")
const recordSchema = new mongoose.Schema({
fields: {
type: Array,
required: true
}
}, {
discriminatorKey: "grid"
})
const Record = mongoose.model("Record", recordSchema)
module.exports = grid => {
// Generate a mongoose-compatible schema from the grid's field definitions
const schema = grid.fields.map(field => {
if(field.type === "string") return { [field.name]: String }
if(field.type === "number") return { [field.name]: Number }
if(field.type === "checkbox") return { [field.name]: Boolean }
return { [field.name]: mongoose.Mixed }
})
return Record.discriminator(grid._id, new mongoose.Schema(schema))
}
This is inside an Express app, and I use the model in my middleware handlers something like this:
async (req, res) => {
const grid = await Grid.findById(req.params.id)
const Record = await GenerateRecordModel(grid)
const records = await Record.find({})
res.json({
...grid,
records
})
}
This works great on the first request, but after that I get an error Discriminator with name “ ” already exists.
I guess this is because only one discriminator with its name per model can exist.
I could give every discriminator a unique name whenever the function is called:
return Record.discriminator(uuidv4(), new mongoose.Schema(schema), grid._id)
But I imagine that this isn't a good idea because discriminators seem to persist beyond the lifetime of the request, so am I laying the groundwork for a memory leak?
I can see two ways forward:
COMPLICATED? Define all discriminators when the app boots up, rather than just when a HTTP request comes in, and write piles of extra logic to handle the user creating, updating or deleting the definitions over in the Grid collection.
SIMPLER? Abandon using discriminators, just use mongoose.Mixed so anything goes as far as mongoose is concerned, and write any validation myself.
Any ideas?

Preventing NoSQL injection: Isn't mongoose supposed to convert inputs based on given schema?

Looking to prevent NoSQL injection attacks for a node.js app using mongodb.
var mongoose = require('mongoose'); // "^5.5.9"
var Schema = mongoose.Schema;
var historySchema = new Schema({
userId: {
type: String,
index: true,
},
message: {},
date: {
type: Date,
default: Date.now,
}
});
var history = mongoose.model('history', historySchema);
// the following is to illustrate the logic, not actual code
function getHistory(user){
history.find({userId: user}, function(err, docs) {
console.log(docs)
}
}
Based on this answer to a similar question, my understanding is that using mongoose and defining the field as string should prevent query injection. However, by changing the user input to a query object, it is possible to return all users. For example:
getHistory({$ne: 1}) // returns the history for all users
I am aware of other ways to prevent this type of attack before it gets to the mongoose query, like using mongo-sanitize. But I'd like to know if there's something wrong with the way I defined the schema or if one can't expect mongoose to convert inputs according to the schema.
Thanks in advance!
this part is good enough, you do not need anything else there. There is method that receives string and uses the string.
The best approach is to validate the input that can be modified (usually HTTP request) on top level before processing anything (I can recommend https://github.com/hapijs/joi its easy to use and you can check if there all required fields and if all fields are in correct format).
So put the validation into middleware just before it hits your controller. Or at the beginning of your controller.
From that point you are in full control of all the code and you believe what you got through your validation, so it cannot happen that someone pass object instead of string and get through.
Following the "skinny controllers, fat model" paradigm, it would be best to expose a custom validation schema from your model to be used in your controller for POST and PUT requests. This means that any data that attempts to enter your database will first be sanitized against a validation schema. Every Mongoose model should own its own validation schema.
My personal favorite for this is Joi. It's relatively simple and effective. Here is a link to the documentation: https://www.npmjs.com/package/#hapi/joi
A Joi schema permits type checking (i.e., Boolean vs. String vs. Number, etc), mandatory inputs if your document has the field required, and other type-specific enforcement such as "max" for numbers, enumerable values, etc.
Here is an example you'd include in your model:
const Joi = require('joi');
...
function validateHistory(history) {
const historySchema = {
userId: Joi.string(),
message: Joi.object(),
date: Joi.date()
}
return Joi.validate(history, historySchema);
}
...
module.exports.validate = validateHistory;
And then in your controller you can do:
const {
validate
} = require('../models/history');
...
router.post('/history', async (req, res) => {
const {
error
} = validate(req.body.data);
if (error) return res.status(400).send(error.details[0].message);
let history = new History({
userID: req.body.user,
message: req.body.message,
date: req.body.date
})
history = await history.save();
res.send(history);
});
*Note that in a real app this route would also have an authentication callback before handling the request.

Saving Schema-less Records with Mongoose?

So I've been trying to save CSP reports into Mongoose with a Mixed schema and have ran into a snag of sorts.
If I try to save anything using the "schema-less" way, it only saves the default _v and _id fields
ViolationSchema = new Schema({});
Violation = mongoose.model('CSPViolation', ViolationSchema);
... wait for POST ...
new Violation( req.body ).save( callback );
// { _id : <some_id>, _v : <some_hash> }
If I set a field in the schema to be Mixed and add a .markModified() to the field, it will save.
ViolationSchema = new Schema({ report : { type : Mixed } });
Violation = mongoose.model('CSPViolation', ViolationSchema);
... wait for POST ...
var v = new Violation( { report : req.body } );
v.markModified('report');
v.save( callback );
// report saved under v.report.<actual_report>
I thought about using native MongoDB-style collection.insert, however it doesn't look like the model has an insert method (nor the schema for that matter).
I suppose I could also go over each key in the report I'm saving and manually mark it as modified, but I'd like to avoid that just to store a report such as this.
Any ideas how I can blindly save a mixed schema type using Mongoose?
It looks like this can be done by setting { strict : false } on the schema. This ensures that Mongoose will save any fields that weren't declared in the original schema.
Normally this isn't something you would enable on 95% of your data, it just fits perfectly with what I'm trying to do currently.
Example
ViolationSchema = new Schema({ type: Mixed }, { strict : false });
Violation = mongoose.model('CSPViolation', ViolationSchema);
... wait for POST ...
new Violation( req.body ).save( callback );
// Saves with full data

How do I get this nested schema working in Mongoose?

So I'm trying to figure out how to how to save multiple commands in a command list but everything I've tried hasn't worked. This is how I have it set up so far but when it saves, it saves in the format of
"command_list" : [ { "action" : "goto,goto", "target" : "http://www.google.com,http://www.cnn.com" } ]
when I really want something like
"command_list" : [ "command" : { "action" : "goto", "target" : "http://www.google.com" },
"command" : { "action" : "goto", "target" : "http://www.cnn.com" } ]
where there are multiple commands. So far my app.js is storing the data like this
var configSample = new Configurations({
command_list_size: request.body.command_list_size,
command_list: [ {action: request.body.action, target: request.body.target}]
});
and the model looks like this
var mongoose = require("mongoose");
var command = mongoose.Schema({
action: String,
target: String
});
var configSchema = mongoose.Schema({
command_list_size: Number,
command_list: [command]
});
module.exports = mongoose.model('Configurations', configSchema);
So how do I get that nesting action going? Thanks!
It looks like you're not packing the data right when you send it to the server. If you use the following:
command_list: [ {action: request.body.action, target: request.body.target}]
it's going to grab all of the actions and lump them together and do the same with the targets. You'd be better off sending an array to your server with the documents already nested in them.
The other option would be to parse the data to pull out the elements once you receive it on your server, but I think it'd be easier to just package it right in the first place.
ADDITION:
If you wanted to split what you have, you could use the String.split() method and rebuild the object:
// not certain the chaining will work like this, but you get the idea. It works
// on the string values you'll receive
var actions = response.body.action.split(',');
var targets = response.body.target.split(',');
// the Underscore library provides some good tools to manipulate what we have
// combined the actions and targets arrays
var combinedData = _.zip(actions, targets);
// go through the combinedData array and create an object with the correct keys
var commandList = _.map(combinedData, function(value) {
return _.object(["action", "target"], value)
});
There may be a better way to create the new object, but this does the trick.
EDIT:
I created a question about trying to refactor the above code here.

Resources