mongoose-encryption and updating objects - node.js

I know the mongoose-encryption doc states:
update will work fine on unencrypted and unauthenticated fields, but will not work correctly if encrypted or authenticated fields are involved.
And I've observed that when I use the mongoose create method that my fields are encrypted into the _ct field. However if I then use findByIdAndUpdate to update my object I see the fields are created in plain text (as output from mongodb console via find command).
From save
> db.tenants.find().pretty()
{
"_id" : ObjectId("554b7f8e7806c204e0c7589e"),
"_ac" : BinData(0,"YdJjOUJhzDWuDE5oBU4SH33O4qM2hbotQTsF6NzDnx4hWyJfaWQiLCJfY3QiXQ=="),
"_ct" : BinData(0,"YaU4z/UY3djGCKBcgMaNIFHeNp8NJ9Woyh9ahff0hRas4WD80V80JE2B8tRLUs0Qd9B7IIzHsq6O4pYub5VKJ1PIQA+/dbStZpOH/KfvPoDC6DzR5JdoAu+feU7HyFnFCMY81RZeJF5BKJylhY1+mG4="),
"__v" : 0
}
After findByIdAndUpdate
> db.tenants.find().pretty()
{
"_id" : ObjectId("554b7f8e7806c204e0c7589e"),
"_ac" : BinData(0,"YdJjOUJhzDWuDE5oBU4SH33O4qM2hbotQTsF6NzDnx4hWyJfaWQiLCJfY3QiXQ=="),
"_ct" : BinData(0,"YaU4z/UY3djGCKBcgMaNIFHeNp8NJ9Woyh9ahff0hRas4WD80V80JE2B8tRLUs0Qd9B7IIzHsq6O4pYub5VKJ1PIQA+/dbStZpOH/KfvPoDC6DzR5JdoAu+feU7HyFnFCMY81RZeJF5BKJylhY1+mG4="),
"__v" : 0,
"userId" : ObjectId("55268f43cbfc87be221cd611"),
"social" : "123-45-6789",
"last" : "bar",
"first" : "foo"
}
Is there a recommended strategy for updating objects and maintaining the encryption with mongoose-encryption?

As you quoted, the documentation for mongoose-encryption clearly tells that it does not work for update.
https://github.com/joegoldbeck/mongoose-encryption
Mongoose update hook is little tricky as well.
What you can do potentially is model your collection in such a way that fields which needs to be encrypted are a separate collection altogether and in the paren collection just link them via ids.
Person = {
_id: <ObjectId>
name: Blah
..
..
documents: [
{ 'doc_id': <ObjectId1> },
{ 'doc_id': <ObjectId2> },
]
}
Documents = [
{
"_id" : <ObjectId1>,
"_ac" : BinData(0,"YdJjOUJhzDWuDE5oBU4SH33O4qM2hbotQTsF6NzDnx4hWyJfaWQiLCJfY3QiXQ=="),
"_ct" : BinData(0,"YaU4z/UY3djGCKBcgMaNIFHeNp8NJ9Woyh9ahff0hRas4WD80V80JE2B8tRLUs0Qd9B7IIzHsq6O4pYub5VKJ1PIQA+/dbStZpOH/KfvPoDC6DzR5JdoAu+feU7HyFnFCMY81RZeJF5BKJylhY1+mG4="),
"__v" : 0
}
...
...
]
This will increase code reuse as well.

I have implemented an strategy that i don´t think it is most efficient but it works.
I need to have all my data in database encrypted so i can´t use the above approach.
What i did is to create an update function that finds the document i want to modify, then i construct a new schema object and assing the _id of the found document to the new object.
Then i delete the original document and after that save the new object wich has the original _id. The only problem i found is that mongoose throw an error because duplicated _id that is printed in the console but it still works and _id aren´t duplicated.
I have tried replacing the_id and traking the document with another property but it still throw that error, anyway data is stored as expected.
exports.update= (req, res, next) => {
Solucion.findOne({_id: req.params.id})
.then(document => {
if (!document) {
res.status(404).json({
message: notFoundMessage,
data: null,
error: null
})
} else {
const solucion = new Solucion({
_id: document._id,
identificacion: document.identificacion,
informacion: document.informacion,
estado: req.body
})
Solucion.deleteOne({_id: document._id})
.then(() => {return solucion.save()})
.then(result=> {
return res.status(201).json({
message: editedSavedMessage,
data: result,
error: null
});
})
.catch(err => {
errorHandler.errorHandler(err, res);
})
}
})
};
UPDATE 29/07/2020
I have found that if you use the save method using the same _id, data is stored encrypted but Mongo creates your schema structure but with all values set to null.
Beyond that it seems to work as expected as data is not visible in DB.

Related

How to Update only passed value in mongodb using mongoose?

I have built an API that updates records in MongoDB using mongoose, but currently what happening is if I am passing only 4 field values in the JSON file of postman and try to update then all the values are updating with value null except that 4 fields which I had passed in JSON so can anyone help me how I can pass dynamic field and value that update only passed values of collection not all the fields of collection.
Passes JSON :
{
"preferance_id" : "60fe9ba1766d10d65c64083c",
"is_active": true,
"price_blur":true,
"affiliate_commission":27,
"language_code": "en"
}
Update call which I have passed in node js
transaction.update(PreferencesMasterName,
{ _id: new mongoose.Types.ObjectId(preferance_id) }, {
subscriptin_vat : subscriptin_vat,
popular_normal : popular_normal,
popular_crawled : popular_crawled,
price_blur : price_blur,
blur_rule : blur_rule,
affiliate_commission : affiliate_commission,
red_lock : red_lock,
automatic_dummy_price : automatic_dummy_price,
...
is_active: is_active
})
I want to pass dynamic field and values here instead of this because due to this other values are set will null value. So, can anyone have an idea how to do this?
You can do something like this:
const data = res.body; // should be an object that needs to updated
transaction.update({_id: PreferanceMasterName._id}, data, {new: true }, ( error, obj ) => {
if( error ) {
console.error( JSON.stringify( error ) );
}
console.log( obj );
});
In certain cases new doesn't work, you can use : { returnOriginal: false };
for more details, you can check this thread there are multiple ways you can do this.
Please check update how to use it.

How to update a field using its previous value in MongoDB/Mongoose

For example, I have some documents that look like this:
{
id: 1
name: "foo"
}
And I want to append another string to the current name field value.
I tried the following using Mongoose, but it didn't work:
Model.findOneAndUpdate({ id: 1 }, { $set: { name: +"bar" } }, ...);
Edit:
From Compatibility Changes in MongoDB 3.6:
MongoDB 3.6.1 deprecates the snapshot query option.
For MMAPv1, use hint() on the { _id: 1} index instead to prevent a cursor from returning a document more than once if an intervening write operation results in a move of the document.
For other storage engines, use hint() with { $natural : 1 } instead.
Original 2017 answer:
You can't refer to the values of the document you want to update, so you will need one query to retrieve the document and another one to update it. It looks like there's a feature request for that in OPEN state since 2016.
If you have a collection with documents that look like:
{ "_id" : ObjectId("590a4aa8ff1809c94801ecd0"), "name" : "bar" }
Using the MongoDB shell, you can do something like this:
db.test.find({ name: "bar" }).snapshot().forEach((doc) => {
doc.name = "foo-" + doc.name;
db.test.save(doc);
});
The document will be updated as expected:
{ "_id" : ObjectId("590a4aa8ff1809c94801ecd0"), "name": "foo-bar" }
Note the .snapshot() call.
This ensures that the query will not return a document multiple times because an intervening write operation moves it due to the growth in document size.
Applying this to your Mongoose example, as explained in this official example:
Cat.findById(1, (err, cat) => {
if (err) return handleError(err);
cat.name = cat.name + "bar";
cat.save((err, updatedCat) => {
if (err) return handleError(err);
...
});
});
It's worth mentioning that there's a $concat operator in the aggregation framework, but unfortunately you can't use that in an update query.
Anyway, depending on what you need to do, you can use that together with the $out operator to save the results of the aggregation to a new collection.
With that same example, you will do:
db.test.aggregate([{
$match: { name: "bar" }
}, {
$project: { name: { $concat: ["foo", "-", "$name"] }}
}, {
$out: "prefixedTest"
}]);
And a new collection prefixedTest will be created with documents that look like:
{ "_id" : ObjectId("XXX"), "name": "foo-bar" }
Just as a reference, there's another interesting question about this same topic with a few answers worth reading: Update MongoDB field using value of another field
If this is still relevant, I have a solution for MongoDB 4.2.
I had the same problem where "projectDeadline" fields of my "project" documents were Array type (["2020","12","1"])
Using Robo3T, I connected to my MongoDB Atlas DB using SRV link. Then executed the following code and it worked for me.
Initial document:
{
_id : 'kjnolqnw.KANSasdasd',
someKey : 'someValue',
projectDeadline : ['2020','12','1']
}
CLI Command:
db
.getCollection('mainData')
.find({projectDeadline: {$not: {$eq: "noDeadline"}}})
.forEach((doc) => {
var deadline = doc.projectDeadline;
var deadlineDate = new Date(deadline);
db
.mainData
.updateOne({
_id: doc._id},
{"$set":
{"projectDeadline": deadlineDate}
}
)}
);
Resulting document:
{
_id : 'kjnolqnw.KANSasdasd',
someKey : 'someValue',
projectDeadline : '2020-12-01 21:00:00.000Z'
}

Trying to upsert mongodb subdocument array with node.js

I have a straightforward mongo collection with an array of subdocuments. I'm trying to do the oft asked "upsert a subdocument in an array". I have read all questions on this topic, but can't seem to get it to work.
Data structure for game_managers:
{
"_id" : ObjectId("555cf465715ff974fb09221f"),
"game_id" : "123456789",
"players" : [
{
"request_email" : "thebigcheese#foobar.com",
"request_notes" : "I love mongo!",
"user_id" : ObjectId("551eb55f555b404d68b88063")
},
{
"request_email" : "morecowbell#example.com",
"request_notes" : "I love oysters!",
"user_id" : ObjectId("551eb55f555b404d68b88063")
}
]
}
When I try to Create / Update with the following code, it always overwrites the first element. I can't get it to even
var col = db.mongo.collection('game_managers');
// Upsert a game manager record for the game
col.update( {game_id:game.place_id}, {$setOnInsert:{game_id:game.game_id}}, { upsert: true }, function(err, result, upserted) {
// Append or update game manager record.
col.update(
{game_id:game.place_id},
{$addToSet: {"players":fields}},
function(err, result) {
next();
}
);
});
I modelled the code from this similar question however it doesn't apply to arrays of subdocuments. I do not want to $pull, and then $push a new element, as the subdocument will ultimately have timestamps and some comments[{},{},{}] subdocs on them.

Filter subdocument and trigger

i have collection of objects inside an invitation, having hard time to filter particular object and trigger it's boolean field.
Document:
"Invitation" : [
{
"__v" : 0,
"userID" : ObjectId("54afaabd88694dc019d3b628"),//ObjectId of personA
"__t" : "USER",
"_id" : ObjectId("54b5022b583973580c706784"),
"Accepted" : false
},
{
"__v" : 0,
"userID" : ObjectId("54af6ce091324fd00f97a15f"),//ObjectId of personB
"__t" : "USER",
"_id" : ObjectId("54bde39cdd55dd9016271f14"),
"Accepted" : false
}
]
here i have only two objects inside Invitation array,it can be more than two.
Let's say personA and personB send me Invitation, so two different invitation objects are inserted into database having different fields, with objectId of both persons(userID in above document), now if i accept only invitation of personA, it should trigger accepted field of personA object only, here is what i tried so far, but not working as per expectation.
Controller:
User.find({_id: req.user._id},'Invitation',function(err,docs) {
if (err) {
console.log(err);
}
var results = [];
async.each(docs,function(doc,callback) {
async.each(doc.Invitation,function(invite,callback) {
User.findOneAndUpdate(
{'_id': doc._id, 'Invitation._id': invite._id},
{'$set': {'Invitation.$.Accepted': !invite.Accepted}},
function(err,doc) {
results.push(doc);
callback(err);
}
);
},callback);
},function(err) {
if (err)
console.log(err);
console.log('end'+results);
});
});
finally i am looking for a query which can be used to filter a single element or object, like if i accept invitation of personA then Accepted field of personA object should be set to true.
i would be really helpful if some logic is provided.
Thank you
Not a very clear question. But it seems all you really need to do here is just match the only sub-document you want to update in the first place:
User.find(
{
"_id": "req.user._id",
"Invitation._id": personA.id
},
{ "Invitation.$": 1 },
function(err,docs) {
// and continue
}
);
This is the form of the positional $ operator in a "projection" context. Where only the "singular" matched element is returned.
Once you have a "singular" result, then all the other code works as designed.
I should know after all because I wrote it for you. Not that you are paying any decent respect to that.
Update on Aggregate in Mongodb
Toggle boolean value of subdocuments
Or personA.userID or whatever makes it work.
Just use the unique identifier for the "user" where you expect that to match the query conditions.
You can do this:
db.user.update({"invitation.userID": 1}, {"$set" : {"invitation.$.Accepted" : true}});
Replacing the value 1 with the user ID you want to update.
The code is in the syntax of MongoShell, simply convert to driver syntax you are using
The operator used was the $. According to the documentation: The positional $ operator identifies an element in an array to update without explicitly specifying the position of the element in the array. To project, or return, an array element from a read operation, see the $ projection operator.
For more details see: http://docs.mongodb.org/manual/reference/operator/update/positional/

Migrating using mongify

I'm using mongify to migrate a mysql database into mongodb.
Doing that, 2 questions appeared:
1- How can i declare my translation file in order to have a embedded array of ids that references to the objects (that are stored in a different collection and can be retrieved through populate), instead of just embedding as json objects.
2- Embedded objects can have an unique id as objects in colections do?. On other projects i've used that approach to query for embedded objects, but if that id is not present i should use a different field.
Unfortunately the first request isn't possible with Mongify at the moment, it requires a custom script to do that.
I could give you more details if you want to send me your translation file (Make sure to remove any sensitive data).
As for number two, the embedded object will get a unique ID. You don't need to do anything special.
Hope that answers your questions.
from mongify isn't possible but in mongodb you can transform data as follows:
//find posts has array of objects
db.getCollection('posts').find({'_tags.0': {$exists: true}}).forEach( function (post) {
var items = [];
var property = '_tags';
post[property].forEach(function(element){
if(element._id !== undefined){
items.push(element._id);
}
});
if(items.length>0){
post[property] = items;
db.posts.update({_id:post._id},post);
}
});
Source Document:
{
"_id" : ObjectId("576aa0389863482f64051c81"),
"id_post" : 130155,
"_tags" : [
{
"_id" : ObjectId("576a9efd9863482f64000044")
},
{
"_id" : ObjectId("576a9efd9863482f6400004b")
},
{
"_id" : ObjectId("576a9efd9863482f64000052")
},
{
"_id" : ObjectId("576a9efd9863482f6400005a")
}
]
}
Final Document:
{
"_id" : ObjectId("576aa0389863482f64051c81"),
"id_post" : 130155,
"_tags" : [
ObjectId("576a9efd9863482f64000044"),
ObjectId("576a9efd9863482f6400004b"),
ObjectId("576a9efd9863482f64000052"),
ObjectId("576a9efd9863482f6400005a")
]
}

Resources