From node, I access a database with objects like
animals: [
{
name: monkey,
diet: banana,
tame: false,
},
{
name: donkey,
diet: carrot,
tame: true,
}
// [...]
]
I'd like to give access to most of the data to the clients, but make sure that the tame property is not exposed.
Using node and lodash's pick(), I could somehow whitelist the data, e.g.,
// retrieve data
// [...]
// whitelist
return {
name: _.pick(animal, 'name'),
diet: _.pick(animal, 'diet'),
};
but this is somewhat tedious, particularly if the selection of keys depends on other factors (e.g., the user who tries to access the data).
What's are good whitelisting/redaction patterns/modules for node?
A lot of this depends on the Database you use. Most databases allow you to select only specific columns/fields in the query itself. MongoDB also does this.
If you use mongoose you can actually enforce this per model:
function filter(document, animal) {
delete animal.tame;
return animal;
};
var options = {
toJSON: {transform: filter},
toObject: {transform: filter}
};
var animalSchema = new Schema({
name: { type: String, trim: true, required: true },
tame: { type: boolean, required: true },
secret: { type: String, required: true, select: false }
},options);
var Animal = mongoose.model('Animal', animalSchema);
var dog = new Animal({name:"rex", tame:true, secret:"rexrex"});
dog.save();
dog.toJSON(); // will not have "tame" property
dog.toJSON({transform: filter}) // dynamic filter
dog.toObject(); // will not have "tame" property
Animal.findOne(); // result objects will not have "secret" property
As you see, you can:
Set a transform function to execute on document when exporting to json or object.
Mark a field with select:false , and it will not show up in any mongoose Model based queries. (You still can do a custom query though.)
In case you are processing a lot of objects, consider writing a Transform Stream. Then you can:
Animal.find().stream().pipe(myTransformStream).pipe(clientResponse)
Related
I've a simple question related to my scenario. I've a react frontend where admin have a form to define it's own sign up form for the users.
Admin will send a set of fields with it's data types
Backend NodeJs will take those fields and run some script/function to create table in a database with the data types accordingly
It also needs to create a model for the respective table dynamically
I'm just finding a way to get it done. Any alternate solution is welcomed or any suggestions to refine my scenario?
Thanks in advance
You can just map all indicated fields into field definitions for Sequielize like:
fieldName: {
field: 'field_name',
type: DataTypes.TEXT,
allowNull: false
},
and call sequelize.define which will return you a Sequilize model that you can use to execute queries:
// NOTE "sequelize" should be an instance of Sequelize, i.e. an existing connection.
function registerModel(tableName, modelName, fields) {
const model = sequelize.define(modelName, {
...fields
}, {
tableName: tableName,
timestamps: false,
});
return model;
}
Register a new model:
const fields = [{
fieldName: {
field: 'field_name',
type: DataTypes.TEXT,
allowNull: false
},
}]
const newModel = registerModel('some_table', 'someModel', fields);
Use the registered model:
const items = await newModel.findAll({
where: {
// some conditions
}
})
Let's say we have :
const mealSchema = Schema({
_id: Schema.Types.ObjectId,
title: { type: string, required: true },
sauce: { type: string }
});
How can we make sauce mandatory if title === "Pasta" ?
The validation needs to work on update too.
I know that a workaround would be
Find
update manually
Then save
But the risk is that if I add a new attribute (let's say "price"), I forget to update it manually too in the workaround.
Document validators
Mongoose has several built-in validators.
All SchemaTypes have the built-in required validator. The required validator uses the SchemaType's checkRequired() function to determine if the value satisfies the required validator.
Numbers have min and max validators.
Strings have enum, match, minlength, and maxlength validators.
For your case you could do something like this
const mealSchema = Schema({
_id: Schema.Types.ObjectId,
title: { type: string, required: true },
sauce: {
type: string,
required: function() {
return this.title === "pasta"? true:false ;
}
}
});
If the built-in validators aren't enough, you can define custom validators to suit your needs.
Custom validation is declared by passing a validation function. You can find detailed instructions on how to do this in the SchemaType#validate().
Update Validators
this refers to the document being validated when using document validation. However, when running update validators, the document being updated may not be in the server's memory, so by default the value of this is not defined. So, What's the solution?
The context option lets you set the value of this in update validators to the underlying query.
In your case, we can do something like this:
const mealSchema = Schema({
_id: Schema.Types.ObjectId,
title: { type: string, required: true },
sauce: { type: string, required: true }
});
mealSchema.path('sauce').validate(function(value) {
// When running update validators with
// the `context` option set to 'query',
// `this` refers to the query object.
if (this.getUpdate().$set.title==="pasta") {
return true
}else{
return false;
}
});
const meal = db.model('Meal', mealSchema);
const update = { title:'pasta', sauce:false};
// Note the context option
const opts = { runValidators: true, context: 'query' };
meal.updateOne({}, update, opts, function(error) { assert.ok(error.errors['title']); });
Not sure if this answers your question. Hope this adds some value to your final solution.
Haven't tested it, pls suggest an edit if this solution needs an upgrade.
Hope this helps.
I have the following mongoose schema used in my MEAN app:
// schema
var categorySchema = new Schema ({
sId: String,
name: String,
parentId: String,
merchants: {},
attributes: {}, /* used to generate pivots and then discarded. */
pivots: [],
_id: {type: String, select: false},
/*other elements in db also not returned by using select: false*/
});
here's the problem. I have a mongodb that is not created by my app, rather its actual schema is defined elsewhere. I have access to this data but want it in a completely different format then what is actually in the database. This is working great by using:
categorySchema.options.toJSON = {
transform: function(doc, ret, options) {
however the schema doesn't represent the full API contract because the "attributes" field in the Schema is deleted in the transform. Pivots aren't in the database but are needed in the schema for mongoose to return it. Thankfully I like this, I want the schema to reflect exactly what I am returning, not what is in the database because frankly, it's a mess and I'm heavily transforming it, so I can give it to other engineers and use it for automated testing.
How do I get attributes out of the schema but still able to use in the transform?
turns out mongoose has function transforms. So I can do:
merchants: { type: {}, get: objToArr},
and that function is called.
just be sure to set:
Schema.set('toObject', { getters: true });
Schema.set('toJSON', { getters: true });
to true.
I'm looking to create a new Document that is saved to the MongoDB regardless of if it is valid. I just want to temporarily skip mongoose validation upon the model save call.
In my case of CSV import, some required fields are not included in the CSV file, especially the reference fields to the other document. Then, the mongoose validation required check is not passed for the following example:
var product = mongoose.model("Product", Schema({
name: {
type: String,
required: true
},
price: {
type: Number,
required: true,
default: 0
},
supplier: {
type: Schema.Types.ObjectId,
ref: "Supplier",
required: true,
default: {}
}
}));
var data = {
name: 'Test',
price: 99
}; // this may be array of documents either
product(data).save(function(err) {
if (err) throw err;
});
Is it possible to let Mongoose know to not execute validation in the save() call?
[Edit]
I alternatively tried Model.create(), but it invokes the validation process too.
This is supported since v4.4.2:
doc.save({ validateBeforeSave: false });
Though there may be a way to disable validation that I am not aware of one of your options is to use methods that do not use middleware (and hence no validation). One of these is insert which accesses the Mongo driver directly.
Product.collection.insert({
item: "ABC1",
details: {
model: "14Q3",
manufacturer: "XYZ Company"
},
}, function(err, doc) {
console.log(err);
console.log(doc);
});
You can have multiple models that use the same collection, so create a second model without the required field constraints for use with CSV import:
var rawProduct = mongoose.model("RawProduct", Schema({
name: String,
price: Number
}), 'products');
The third parameter to model provides an explicit collection name, allowing you to have this model also use the products collection.
I was able to ignore validation and preserve the middleware behavior by replacing the validate method:
schema.method('saveWithoutValidation', function(next) {
var defaultValidate = this.validate;
this.validate = function(next) {next();};
var self = this;
this.save(function(err, doc, numberAffected) {
self.validate = defaultValidate;
next(err, doc, numberAffected);
});
});
I've tested it only with mongoose 3.8.23
schema config validateBeforeSave=false
use validate methed
// define
var GiftSchema = new mongoose.Schema({
name: {type: String, required: true},
image: {type: String}
},{validateBeforeSave:false});
// use
var it new Gift({...});
it.validate(function(err){
if (err) next(err)
else it.save(function (err, model) {
...
});
})
I'm trying to hide certain fields on my GET output for my REST server. I have 2 schema's, both have a field to embed related data from eachother into the GET, so getting /people would return a list of locations they work at and getting a list of locations returns who works there. Doing that, however, will add a person.locations.employees field and will then list out the employees again, which obviously I don't want. So how do I remove that field from the output before displaying it? Thanks all, let me know if you need any more information.
/********************
/ GET :endpoint
********************/
app.get('/:endpoint', function (req, res) {
var endpoint = req.params.endpoint;
// Select model based on endpoint, otherwise throw err
if( endpoint == 'people' ){
model = PeopleModel.find().populate('locations');
} else if( endpoint == 'locations' ){
model = LocationsModel.find().populate('employees');
} else {
return res.send(404, { erorr: "That resource doesn't exist" });
}
// Display the results
return model.exec(function (err, obj) {
if (!err) {
return res.send(obj);
} else {
return res.send(err);
}
});
});
Here is my GET logic. So I've been trying to use the query functions in mongoose after the populate function to try and filter out those references. Here are my two schema's.
peopleSchema.js
return new Schema({
first_name: String,
last_name: String,
address: {},
image: String,
job_title: String,
created_at: { type: Date, default: Date.now },
active_until: { type: Date, default: null },
hourly_wage: Number,
locations: [{ type: Schema.ObjectId, ref: 'Locations' }],
employee_number: Number
}, { collection: 'people' });
locationsSchema.js
return new Schema({
title: String,
address: {},
current_manager: String, // Inherit person details
alternate_contact: String, // Inherit person details
hours: {},
employees: [{ type: Schema.ObjectId, ref: 'People' }], // mixin employees that work at this location
created_at: { type: Date, default: Date.now },
active_until: { type: Date, default: null }
}, { collection: 'locations' });
You should specify the fields you want to fetch by using the select() method. You can do so by doing something like:
if( endpoint == 'people' ){
model = PeopleModel.find().select('locations').populate('locations');
} else if( endpoint == 'locations' ){
model = LocationsModel.find().select('employees').populate('employees');
} // ...
You can select more fields by separating them with spaces, for example:
PeopleModel.find().select('first_name last_name locations') ...
Select is the right answer but it also may help to specify it in your schema so that you maintain consistency in your API and I've found it helps me to not remember to do it everywhere I perform a query on the object.
You can set certain fields in your schema to never return by using the select: true|false attribute on the schema field.
More details can be found here: http://mongoosejs.com/docs/api.html#schematype_SchemaType-select
SOLUTION!
Because this was so hard for me to find i'm going to leave this here for anybody else. In order to "deselect" a populated item, just prefix the field with "-" in your select. Example:
PeopleModel.find().populate({path: 'locations', select: '-employees'});
And now locations.employee's will be hidden.
If you remember from you SQL days, SELECT does a restriction on the table(s) being queried. Restrict is one of the primitive operations from the relational model and continues to be a useful feature as the relational model has evolved. blah blah blah.
In mongoose, the Query.select() method allows you to perform this operation with some extra features. Particularly, not only can you specify what attributes (columns) to return, but you can also specify what attributes you want to exclude.
So here's the example:
function getPeople(req,res, next) {
var query = PeopleModel.find().populate({path: 'locations', select: '-employees'});
query.exec(function(err, people) {
// error handling stuff
// process and return response stuff
});
}
function getLocations(req,res, next) {
var query = LocationModel.find().populate({path: 'employees', select: '-locations'});
query.exec(function(err, people) {
// error handling stuff
// processing and returning response stuff
});
}
app.get('people', getPeople);
app.get('locations', getLocations);
Directly from the Mongoose Docs:
Go to http://mongoosejs.com/docs/populate.html and search for "Query conditions and other options"
Query conditions and other options
What if we wanted to populate our fans array based on their age,
select just their names, and return at most, any 5 of them?
Story
.find(...)
.populate({
path: 'fans',
match: { age: { $gte: 21 }},
select: 'name -_id',
options: { limit: 5 }
})
.exec()
I just wanted to remark, for the simplicity of the endpoint you may be able to get away with this way to define the endpoints. However, in general this kind of dispacher pattern is not necessary and may pose problems later in development when developing with Express.