Mongoose (MongoDB): Exclude Properties in a Dictionary Like Schema Type - node.js

I have a Schema of the following structure:
var schema = mongoose.Schema({
answers: {type: mongoose.Schema.Types.Mixed}
});
I use the answers field as an object (associative array to implement something like a dictionary). Here is an example:
{
"__v": 0,
"_id": {
"$oid": "53a0251c50d0536c1bfc6006"
},
"answers": {
"fea": {
"viewed": false
},
"3d2": {
"viewed": true,
"value": true
},
"4fr": {
"viewed": true,
"value": true
},
"84h": {
"viewed": false
},
...
}
}
In a query I want to select only the "value" field of each entry. How is that possible through the select syntax? This of course doesn't work:
XY.find(...)
.select({'answers': true, 'answers.*.value': false})
.exec(...);
Maybe I have to design the data in another fashion?
Best regards,
Kersten

You should never model with "explicit values" as the "key" names. This is very bad practice. Consider what you would do in a SQL database. Would you create "fields/columns" for the different "names" of the things you want?
No you would not. You have a generic field that specifies a "type" and then you have others that hold the data. Nothing changes here:
{
"_id": {
"$oid": "53a0251c50d0536c1bfc6006"
},
"answers": [
{ "type": "fea", "viewed": false },
{ "type": "3d2", "viewed": true, "value": true },
{ "type": "4fr", "viewed": true, "value": true },
{ "type": "84h", "viewed": false },
...
]
}
Now this is easy to use something like the aggregation framework to make the projection of the content like you want:
With Modern MongoDB 2.6 and onwards you can use $map and $setDifference to filter the array without using $unwind:
Model.aggregate(
[
{ "$project": {
"answers": {
"$setDifference": [
{
"$map": {
"input": "$answers",
"as": "el",
"in": {
"$cond": [
1,
{
"type": "$$el.type",
"value": { "$ifNull": [ "$$el.value", false ] }
},
false
]
}
}
},
[false]
]
}
}}
],
function(err,result) {
}
);
Or with older versions pre 2.6:
Model.aggregate(
[
{ "$unwind": "$answers" },
{ "$group": {
"_id": "$_id",
"answers": {
"$push": {
"type": "$answers.type",
"value": { "$ifNull": [ "$answers.value", false ] }
}
}
}}
],
function(err,result) {
}
);
Of course you can "filter" the array results to certain conditions by either adding a logical evaluation as the first argument to the $cond operator in the $map implementation. Or by using a $match pipeline stage in between the $unwind and $group stages.
Either form allows you to re-shape the result without problem and is the fastest way to process this, which is a strong advantage of using arrays as opposed to embedded objects whose keys are actually a "data" item.
If you are stuck with this then you need to process with JavaScript evaluation like mapReduce. This runs much slower than the aggregation framework due to the need to invoke and run in a JavaScript interpreter process:
Model.mapReduce(
{
"map": function() {
for ( var k in this.answers ) {
this.answers[k] = this.answers[k].hasOwnProperty("value")
? this.answers[k].value : false;
}
var id = this._id;
delete this._id;
emit( id, this );
},
"reduce": function(){}
},
function(err,docs) {
}
)
But really, consider changing your structure as it makes things much more flexible for queries and other operations.

Related

How to filter sub values from the object using ExpressJS and MongoDB?

I have a data from MongoDB where I'm getting all the data of the agent, and I want to get the testimonials with isDisplayed is true only. Is there a way in ExpressJS & MongoDB where I can filter the testimonial key?
Here's what I tried
const getSingleAgent = expressAsync(async (req, res) => {
const agent = await Agents.findOne({
_id: req.params.id,
});
res.json(agent);
} else {
res.status(404);
throw new Error("Agent not found.");
}
});
Actual Result
{
"_id": "63ea901a85d4fbd62fb887b3",
"name": "test namee",
"isDeclined": false,
"testimonials": [
{
"isDisplayed": true,
"name": "test123123123123123",
},
{
"isDisplayed": false,
"name": "test123123123123123",
},
{
"isDisplayed": false,
"name": "test#gmail.com",
},
],
}
Expected Result
{
"_id": "63ea901a85d4fbd62fb887b3",
"name": "test namee",
"isDeclined": false,
"testimonials": [
{
"isDisplayed": true,
"name": "test123123123123123",
},
],
}
You can use $elemMatch into a projection stage like this:
db.collection.find({
"_id": "63ea901a85d4fbd62fb887b3",
"testimonials.isDisplayed": true
},
{
"name": 1,
"isDeclined": 1,
"testimonials": {
"$elemMatch": {
"isDisplayed": true
}
}
})
Example here
But be careful. using $elemMatch will only return the first match. Look at this example where there are two true but only one is returned.
As explained into docs:
The $elemMatch operator limits the contents of an field from the query results to contain only the first element matching the $elemMatch condition.
So if you have or want more than one value you can use and aggregation pipelina and $filter into a $project stage:
db.collection.aggregate([
{
"$match": {
"_id": "63ea901a85d4fbd62fb887b3",
"testimonials.isDisplayed": true
}
},
{
"$project": {
"name": 1,
"isDeclined": 1,
"testimonials": {
"$filter": {
"input": "$testimonials",
"cond": {
"$eq": [
"$$this.isDisplayed",
true
]
}
}
}
}
}
])
Example here
Note how in this example if exists two true values, both are displayed.

Mongoose - How to query field in the last object of an array of objects

I have MongoDB documents structured like this:
{
"_id": "5d8b987f9f8b9f9c8c8b9f9",
"targetsList": [
{
"target": "user",
"statusList": [
{
"date": "2018-01-01",
"type": "OK"
},
{
"date": "2018-01-02",
"type": "FAILD"
}
]
}
]
}
And I want to count all documents that in their "targetList" array, there is an object with "target"=="user" - and also that object conatin on the last element of its "statusList" array, an object with "type" != "FAILD".
Any ideas on how to implement this kind of query?
Mongo playground:
https://mongoplayground.net/p/3bCoHRnh-KQ
In this example, I expected the count to be 1, because only the second object meets the conditions.
An aggregation pipeline
1st step - Filtering out where "targetsList.target": "user"
2nd step - $unwind on targetsList to get it out of array
3rd step - getting the last element of the targetsList.statusList array using $arrayElemAt
4th step - getting the results where that last element is not FAILD
5th step - getting the count
demo - you can try removing parts of the pipeline to see what the intermediate results are
db.collection.aggregate([
{
$match: {
"targetsList.target": "user"
}
},
{
$unwind: "$targetsList"
},
{
$project: {
"targetsList.statusList": {
$arrayElemAt: [
"$targetsList.statusList",
-1
]
},
}
},
{
$match: {
"targetsList.statusList.type": {
$ne: "FAILD"
}
}
},
{
$count: "withoutFailedInLastElemCount"
}
])
Unless it's crucial that the element be the last index, this should work for your case.
db.collection.find({
"targetsList.statusList.type": {
$in: [
"FAILD"
]
}
})
This will retrieve documents where the type value is FAILD. To invert this you can swap $in for $nin.
Updated playground here
Here's another way to do it with a leading monster "$match".
db.collection.aggregate([
{
"$match": {
"targetsList.target": "user",
"$expr": {
"$reduce": {
"input": "$targetsList",
"initialValue": false,
"in": {
"$or": [
"$$value",
{
"$ne": [
{
"$last": "$$this.statusList.type"
},
"FAILD"
]
}
]
}
}
}
}
},
{
"$count": "noFailedLastCount"
}
])
Try it on mongoplayground.net.

Mongodb Aggregate with natural order

I am looking for a way to query database to fetch last saved entry in the collection.
I have a function which saves the document to the collection this is hwo my saved collection looks like.
{
"_id": {
"$oid": "5ebbf2b4586b4946226e2c88"
},
"name": "Stept_1",
"description": "",
"coordinates": {
"pose": {
"$numberInt": "0"
},
"x": {
"$numberDouble": "-9.760518723445719"
},
"y": {
"$numberDouble": "-3.4586615766853854"
},
"z": {
"$numberInt": "0"
}
},
"depth": {
"$numberInt": "1"
},
"_neighbours": [],
"optional": {},
"__v": {
"$Reference": "1111111"
}
}
Each document is saved with name in ascending order e.g. name:Step_1, Step_2 etc.
I have tried fetching the last saved documents using aggregate method like
db.collection.aggregate([
{ $sort: { name: -1 } },
{ $group: { _id: "$Reference", name: { $first: "$name" } } }
])
This returns the data in ascending order till name:Step_10 i.e. "Step_1,2,3,..." but once after name:Step_1o if I restart the app and again start fetching the last saved document the index of ascending order returns to name:Step_9 which then lead some duplicate entries of few documents. What I am looking for is it should always return follow natural order(i.e. Step_1,2,3,..11,12,13 etc) while fetching those documents.
Thanks in advance.
You can use collation to specify numericOrdering like so:
db.collection.aggregate([
{ $sort: { name: -1 } },
{ $group: { _id: "$Reference", name: { $first: "$name" } } }
],
{ collation : {
locale: "en_US",
numericOrdering: true
}}
)
From mongo docs:
numericOrdering boolean
Optional. Flag that determines whether to compare numeric strings as numbers or as strings.
If true, compare as numbers; i.e. "10" is greater than "2".
If false, compare as strings; i.e. "10" is less than "2".
Default is false.

MongoDB: Concatenate Multiple Arrays

I have 3 arrays of ObjectIds I want to concatenate into a single array, and then sort by creation date. $setUnion does precisely what I want, but I'd like to try without using it.
Schema of object I want to sort:
var chirpSchema = new mongoose.Schema({
interactions: {
_liked : ["55035390d3e910505be02ce2"] // [{ type: $oid, ref: "interaction" }]
, _shared : ["507f191e810c19729de860ea", "507f191e810c19729de860ea"] // [{ type: $oid, ref: "interaction" }]
, _viewed : ["507f1f77bcf86cd799439011"] // [{ type: $oid, ref: "interaction" }]
}
});
Desired result: Concatenate _liked, _shared, and _viewed into a single array, and then sort them by creation date using aggregate pipeline. See below
["507f1f77bcf86cd799439011", "507f191e810c19729de860ea", "507f191e810c19729de860ea", "55035390d3e910505be02ce2"]
I know I'm suppose to use $push, $each, $group, and $unwind in some combination or other, but I'm having trouble piecing together the documenation to make this happen.
Update: Query
model_user.aggregate([
{ $match : { '_id' : { $in : following } } }
, { $project : { 'interactions' : 1 } }
, { $project : {
"combined": { $setUnion : [
"$interactions._liked"
, "$interactions._shared"
, "$interactions._viewed"
]}
}}
])
.exec(function (err, data) {
if (err) return next(err);
next(data); // Combined is returning null
})
If all the Object _id values are "unique" then $setUnion is your best option. It is of course not "ordered" in any way as it works with a "set", and that does not guarantee order. But you can always unwind and $sort.
[
{ "$project": {
"combined": { "$setUnion": [
{ "$ifNull": [ "$interactions._liked", [] ] },
{ "$ifNull": [ "$interactions._shared", [] ] },
{ "$ifNull", [ "$interactions._viewed", [] ] }
]}
}},
{ "$unwind": "$combined" },
{ "$sort": { "combined": 1 } },
{ "$group": {
"_id": "$_id",
"combined": { "$push": "$combined" }
}}
]
Of course again since this is a "set" of distinct values you can do the old way instead with $addToSet, after processing $unwind on each array:
[
{ "$unwind": "$interactions._liked" },
{ "$unwind": "$interactions._shared" },
{ "$unwind": "$interactions._viewed" },
{ "$project": {
"interactions": 1,
"type": { "$const": [ "liked", "shared", "viewed" ] }
}}
{ "$unwind": "$type" },
{ "$group": {
"_id": "$_id",
"combined": {
"$addToSet": {
"$cond": [
{ "$eq": [ "$type", "liked" ] },
"$interactions._liked",
{ "$cond": [
{ "$eq": [ "$type", "shared" ] },
"$interactions._shared",
"$interactions._viewed"
]}
]
}
}
}},
{ "$unwind": "$combined" },
{ "$sort": { "combined": 1 } },
{ "$group": {
"_id": "$_id",
"combined": { "$push": "$combined" }
}}
]
But still the same thing applies to ordering.
Future releases even have the ability to concatenate arrays without reducing to a "set":
[
{ "$project": {
"combined": { "$concatArrays": [
"$interactions._liked",
"$interactions._shared",
"$interactions._viewed"
]}
}},
{ "$unwind": "$combined" },
{ "$sort": { "combined": 1 } },
{ "$group": {
"_id": "$_id",
"combined": { "$push": "$combined" }
}}
]
But still there is no way to re-order the results without procesing $unwind and $sort.
You might therefore consider that unless you need this grouped across multiple documents, that the basic "contenate and sort" operation is best handled in client code. MongoDB has no way to do this "in place" on the array at present, so per document in client code is your best bet.
But if you do need to do this grouping over multiple documents, then the sort of approaches as shown here are for you.
Also note that "creation" here means creation of the ObjectId value itself and not other properties from your referenced objects. If you need those, then you perform a populate on the id values after the aggregation or query instead, and of course sort in client code.

How to pull one instance of an item in an array in MongoDB?

According to the documents:
The $pull operator removes from an existing array all instances of a value or values that match a specified condition.
Is there an option to remove only the first instance of a value? For example:
var array = ["bird","tiger","bird","horse"]
How can the first "bird" be removed directly in an update call?
So you are correct in that the $pull operator does exactly what the documentation says in that it's arguments are in fact a "query" used to match the elements that are to be removed.
If your array content happened to always have the element in the "first" position as you show then the $pop operator does in fact remove that first element.
With the basic node driver:
collection.findOneAndUpdate(
{ "array.0": "bird" }, // "array.0" is matching the value of the "first" element
{ "$pop": { "array": -1 } },
{ "returnOriginal": false },
function(err,doc) {
}
);
With mongoose the argument to return the modified document is different:
MyModel.findOneAndUpdate(
{ "array.0": "bird" },
{ "$pop": { "array": -1 } },
{ "new": true },
function(err,doc) {
}
);
But neither are of much use if the array position of the "first" item to remove is not known.
For the general approach here you need "two" updates, being one to match the first item and replace it with something unique to be removed, and the second to actually remove that modified item.
This is a lot more simple if applying simple updates and not asking for the returned document, and can also be done in bulk across documents. It also helps to use something like async.series in order to avoid nesting your calls:
async.series(
[
function(callback) {
collection.update(
{ "array": "bird" },
{ "$unset": { "array.$": "" } },
{ "multi": true }
callback
);
},
function(callback) {
collection.update(
{ "array": null },
{ "$pull": { "array": null } },
{ "multi": true }
callback
);
}
],
function(err) {
// comes here when finished or on error
}
);
So using the $unset here with the positional $ operator allows the "first" item to be changed to null. Then the subsequent query with $pull just removes any null entry from the array.
That is how you remove the "first" occurance of a value safely from an array. To determine whether that array contains more than one value that is the same though is another question.
It's worth noting that whilst the other answer here is indeed correct that the general approach here would be to $unset the matched array element in order to create a null value and then $pull just the null values from the array, there are better ways to implement this in modern MongoDB versions.
Using bulkWrite()
As an alternate case to submitting two operations to update in sequence as separate requests, modern MongoDB release support bulk operations via the recommended bulkWrite() method which allows those multiple updates to be submitted as a single request with a single response:
collection.bulkWrite(
[
{ "updateOne": {
"filter": { "array": "bird" },
"update": {
"$unset": { "array.$": "" }
}
}},
{ "updateOne": {
"filter": { "array": null },
"update": {
"$pull": { "array": null }
}
}}
]
);
Does the same thing as the answer showing that as two requests, but this time it's just one. This can save a lot of overhead in server communication, so it's generally the better approach.
Using Aggregation Expressions
With the release of MongoDB 4.2, aggregation expressions are now allowed in the various "update" operations of MongoDB. This is a single pipeline stage of either $addFields, $set ( which is an alias of $addFields meant to make these "update" statements read more logically ), $project or $replaceRoot and it's own alias $replaceWith. The $redact pipeline stage also applies here to some degree. Basically any pipeline stage which returns a "reshaped" document is allowed.
collection.updateOne(
{ "array": "horse" },
[
{ "$set": {
"array": {
"$concatArrays": [
{ "$slice": [ "$array", 0, { "$indexOfArray": [ "$array", "horse" ] }] },
{ "$slice": [
"$array",
{ "$add": [{ "$indexOfArray": [ "$array", "horse" ] }, 1] },
{ "$size": "$array" }
]}
]
}
}}
]
);
In this case the manipulation used is to implement the $slice and $indexOfArray operators to essentially piece together a new array which "skips" over the first matched array element. Theses pieces are joined via the $concatArrays operator, returning a new array absent of the first matched element.
This is now probably more effective since the operation which is still a single request is now also a single operation and would incur a little less server overhead.
Of course the only catch is that this is not supported in any release of MongoDB prior to 4.2. The bulkWrite() on the other hand may be a newer API implementation, but the actual underlying calls to the server would apply back to MongoDB 2.6 implementing actual "Bulk API" calls, and even regresses back to earlier versions by the way all core drivers actually implement this method.
Demonstration
As a demonstration, here is a listing of both approaches:
const { Schema } = mongoose = require('mongoose');
const uri = 'mongodb://localhost:27017/test';
const opts = { useNewUrlParser: true, useUnifiedTopology: true };
mongoose.Promise = global.Promise;
mongoose.set('debug', true);
mongoose.set('useCreateIndex', true);
mongoose.set('useFindAndModify', false);
const arrayTestSchema = new Schema({
array: [String]
});
const ArrayTest = mongoose.model('ArrayTest', arrayTestSchema);
const array = ["bird", "tiger", "horse", "bird", "horse"];
const log = data => console.log(JSON.stringify(data, undefined, 2));
(async function() {
try {
const conn = await mongoose.connect(uri, opts);
await Promise.all(
Object.values(conn.models).map(m => m.deleteMany())
);
await ArrayTest.create({ array });
// Use bulkWrite update
await ArrayTest.bulkWrite(
[
{ "updateOne": {
"filter": { "array": "bird" },
"update": {
"$unset": { "array.$": "" }
}
}},
{ "updateOne": {
"filter": { "array": null },
"update": {
"$pull": { "array": null }
}
}}
]
);
log({ bulkWriteResult: (await ArrayTest.findOne()) });
// Use agggregation expression
await ArrayTest.collection.updateOne(
{ "array": "horse" },
[
{ "$set": {
"array": {
"$concatArrays": [
{ "$slice": [ "$array", 0, { "$indexOfArray": [ "$array", "horse" ] }] },
{ "$slice": [
"$array",
{ "$add": [{ "$indexOfArray": [ "$array", "horse" ] }, 1] },
{ "$size": "$array" }
]}
]
}
}}
]
);
log({ aggregateWriteResult: (await ArrayTest.findOne()) });
} catch (e) {
console.error(e);
} finally {
mongoose.disconnect();
}
})();
And the output:
Mongoose: arraytests.deleteMany({}, {})
Mongoose: arraytests.insertOne({ array: [ 'bird', 'tiger', 'horse', 'bird', 'horse' ], _id: ObjectId("5d8f509114b61a30519e81ab"), __v: 0 }, { session: null })
Mongoose: arraytests.bulkWrite([ { updateOne: { filter: { array: 'bird' }, update: { '$unset': { 'array.$': '' } } } }, { updateOne: { filter: { array: null }, update: { '$pull': { array: null } } } } ], {})
Mongoose: arraytests.findOne({}, { projection: {} })
{
"bulkWriteResult": {
"array": [
"tiger",
"horse",
"bird",
"horse"
],
"_id": "5d8f509114b61a30519e81ab",
"__v": 0
}
}
Mongoose: arraytests.updateOne({ array: 'horse' }, [ { '$set': { array: { '$concatArrays': [ { '$slice': [ '$array', 0, { '$indexOfArray': [ '$array', 'horse' ] } ] }, { '$slice': [ '$array', { '$add': [ { '$indexOfArray': [ '$array', 'horse' ] }, 1 ] }, { '$size': '$array' } ] } ] } } } ])
Mongoose: arraytests.findOne({}, { projection: {} })
{
"aggregateWriteResult": {
"array": [
"tiger",
"bird",
"horse"
],
"_id": "5d8f509114b61a30519e81ab",
"__v": 0
}
}
NOTE : The example listing is using mongoose, partly because it was referenced in the other answer given and partly to also demonstrate an important point with the aggregate syntax example. Note the code uses ArrayTest.collection.updateOne() since at the present release of Mongoose ( 5.7.1 at time of writing ) the aggregation pipeline syntax to such updates is being removed by the standard mongoose Model methods.
As such the .collection accessor can be used in order to get the underlying Collection object from the core MongoDB Node driver. This would be required until a fix is made to mongoose which allows this expression to be included.
As mentioned in this Jira this feature will never exist properly.
The approach I recommend using would be via the aggregation pipeline update syntax as proposed in a different answer, however that answer has some edge cases where it fails - for example if the element does not exist in the array, here is a working version for all edge cases.
ArrayTest.updateOne({},
[
{
"$set": {
"array": {
"$concatArrays": [
{
$cond: [
{
$gt: [
{
"$indexOfArray": [
"$array",
"horse"
]
},
0
]
},
{
"$slice": [
"$array",
0,
{
"$indexOfArray": [
"$array",
"horse"
]
}
]
},
[]
]
},
{
"$slice": [
"$array",
{
"$add": [
{
"$indexOfArray": [
"$array",
"horse"
]
},
1
]
},
{
"$size": "$array"
}
]
}
]
}
}
}
])
Mongo Playground

Resources