MongoDB: Concatenate Multiple Arrays - node.js

I have 3 arrays of ObjectIds I want to concatenate into a single array, and then sort by creation date. $setUnion does precisely what I want, but I'd like to try without using it.
Schema of object I want to sort:
var chirpSchema = new mongoose.Schema({
interactions: {
_liked : ["55035390d3e910505be02ce2"] // [{ type: $oid, ref: "interaction" }]
, _shared : ["507f191e810c19729de860ea", "507f191e810c19729de860ea"] // [{ type: $oid, ref: "interaction" }]
, _viewed : ["507f1f77bcf86cd799439011"] // [{ type: $oid, ref: "interaction" }]
}
});
Desired result: Concatenate _liked, _shared, and _viewed into a single array, and then sort them by creation date using aggregate pipeline. See below
["507f1f77bcf86cd799439011", "507f191e810c19729de860ea", "507f191e810c19729de860ea", "55035390d3e910505be02ce2"]
I know I'm suppose to use $push, $each, $group, and $unwind in some combination or other, but I'm having trouble piecing together the documenation to make this happen.
Update: Query
model_user.aggregate([
{ $match : { '_id' : { $in : following } } }
, { $project : { 'interactions' : 1 } }
, { $project : {
"combined": { $setUnion : [
"$interactions._liked"
, "$interactions._shared"
, "$interactions._viewed"
]}
}}
])
.exec(function (err, data) {
if (err) return next(err);
next(data); // Combined is returning null
})

If all the Object _id values are "unique" then $setUnion is your best option. It is of course not "ordered" in any way as it works with a "set", and that does not guarantee order. But you can always unwind and $sort.
[
{ "$project": {
"combined": { "$setUnion": [
{ "$ifNull": [ "$interactions._liked", [] ] },
{ "$ifNull": [ "$interactions._shared", [] ] },
{ "$ifNull", [ "$interactions._viewed", [] ] }
]}
}},
{ "$unwind": "$combined" },
{ "$sort": { "combined": 1 } },
{ "$group": {
"_id": "$_id",
"combined": { "$push": "$combined" }
}}
]
Of course again since this is a "set" of distinct values you can do the old way instead with $addToSet, after processing $unwind on each array:
[
{ "$unwind": "$interactions._liked" },
{ "$unwind": "$interactions._shared" },
{ "$unwind": "$interactions._viewed" },
{ "$project": {
"interactions": 1,
"type": { "$const": [ "liked", "shared", "viewed" ] }
}}
{ "$unwind": "$type" },
{ "$group": {
"_id": "$_id",
"combined": {
"$addToSet": {
"$cond": [
{ "$eq": [ "$type", "liked" ] },
"$interactions._liked",
{ "$cond": [
{ "$eq": [ "$type", "shared" ] },
"$interactions._shared",
"$interactions._viewed"
]}
]
}
}
}},
{ "$unwind": "$combined" },
{ "$sort": { "combined": 1 } },
{ "$group": {
"_id": "$_id",
"combined": { "$push": "$combined" }
}}
]
But still the same thing applies to ordering.
Future releases even have the ability to concatenate arrays without reducing to a "set":
[
{ "$project": {
"combined": { "$concatArrays": [
"$interactions._liked",
"$interactions._shared",
"$interactions._viewed"
]}
}},
{ "$unwind": "$combined" },
{ "$sort": { "combined": 1 } },
{ "$group": {
"_id": "$_id",
"combined": { "$push": "$combined" }
}}
]
But still there is no way to re-order the results without procesing $unwind and $sort.
You might therefore consider that unless you need this grouped across multiple documents, that the basic "contenate and sort" operation is best handled in client code. MongoDB has no way to do this "in place" on the array at present, so per document in client code is your best bet.
But if you do need to do this grouping over multiple documents, then the sort of approaches as shown here are for you.
Also note that "creation" here means creation of the ObjectId value itself and not other properties from your referenced objects. If you need those, then you perform a populate on the id values after the aggregation or query instead, and of course sort in client code.

Related

Mongoose - How to query field in the last object of an array of objects

I have MongoDB documents structured like this:
{
"_id": "5d8b987f9f8b9f9c8c8b9f9",
"targetsList": [
{
"target": "user",
"statusList": [
{
"date": "2018-01-01",
"type": "OK"
},
{
"date": "2018-01-02",
"type": "FAILD"
}
]
}
]
}
And I want to count all documents that in their "targetList" array, there is an object with "target"=="user" - and also that object conatin on the last element of its "statusList" array, an object with "type" != "FAILD".
Any ideas on how to implement this kind of query?
Mongo playground:
https://mongoplayground.net/p/3bCoHRnh-KQ
In this example, I expected the count to be 1, because only the second object meets the conditions.
An aggregation pipeline
1st step - Filtering out where "targetsList.target": "user"
2nd step - $unwind on targetsList to get it out of array
3rd step - getting the last element of the targetsList.statusList array using $arrayElemAt
4th step - getting the results where that last element is not FAILD
5th step - getting the count
demo - you can try removing parts of the pipeline to see what the intermediate results are
db.collection.aggregate([
{
$match: {
"targetsList.target": "user"
}
},
{
$unwind: "$targetsList"
},
{
$project: {
"targetsList.statusList": {
$arrayElemAt: [
"$targetsList.statusList",
-1
]
},
}
},
{
$match: {
"targetsList.statusList.type": {
$ne: "FAILD"
}
}
},
{
$count: "withoutFailedInLastElemCount"
}
])
Unless it's crucial that the element be the last index, this should work for your case.
db.collection.find({
"targetsList.statusList.type": {
$in: [
"FAILD"
]
}
})
This will retrieve documents where the type value is FAILD. To invert this you can swap $in for $nin.
Updated playground here
Here's another way to do it with a leading monster "$match".
db.collection.aggregate([
{
"$match": {
"targetsList.target": "user",
"$expr": {
"$reduce": {
"input": "$targetsList",
"initialValue": false,
"in": {
"$or": [
"$$value",
{
"$ne": [
{
"$last": "$$this.statusList.type"
},
"FAILD"
]
}
]
}
}
}
}
},
{
"$count": "noFailedLastCount"
}
])
Try it on mongoplayground.net.

How to sum values of third level nested array of objects across all documents in MongoDB?

I have a mongoose document having the following Schema:
Products
{
"section":"",
"category":"Food & Drink",
"sub_category":"Main Dish",
"product_code":"ST",
"title":"Steak",
"description":"Served with sauted vegetables",
"tags":[
],
"warranty":"None",
"product_variants":[
{
"variant_code":"ST1",
"variant_title":"Rib Eye",
"images":[
],
"status":"Active",
"variant_details":[
{
"size":"6oz",
"local_price":800,
"local_discount":"0",
"foreign_price":0,
"foreign_discount":"0",
"inventory":[
{
"branch_id":{
},
"quantity":94
}
]
},
{
"size":"10oz",
"local_price":1000,
"local_discount":"0",
"foreign_price":0,
"foreign_discount":"0",
"inventory":[
{
"branch_id":{
},
"quantity":147
}
]
},
{
"size":"12oz",
"local_price":1200,
"local_discount":"0",
"foreign_price":0,
"foreign_discount":"0",
"inventory":[
{
"branch_id":{
},
"quantity":199
}
]
}
]
}
]
}
The above document shows only one object in the product_variants field but please note that there could be several objects as well. I need to sum the quantity for each size and product variant.
How would I do that using aggregate function? I am using mongoose in node js environment.
Query
(its based on the last comment in the previous answer, similar query but multiplies that quantity with the local price)
Test code here
db.collection.aggregate([
{
"$unwind": "$product_variants"
},
{
"$unwind": "$product_variants.variant_details"
},
{
"$unwind": "$product_variants.variant_details.inventory"
},
{
"$set": {
"total_local_price": {
"$multiply": [
"$product_variants.variant_details.inventory.quantity",
"$product_variants.variant_details.local_price"
]
}
}
},
{
$group: {
_id: null, // or "$_id" if you want only for 1 document
total_qty: {
$sum: "$total_local_price"
}
}
}
])
You can use this aggregation query:
Fisrt $project to get only the quantity values. It generates the following output:
"array": [
[
[
94
],
[
147
],
[
199
]
]
So next step is to use $unwind three times to flat the array.
And $group by _id using $sum
yourModel.aggregate([{
"$project": {
"array": "$product_variants.variant_details.inventory.quantity"
}
},
{
"$unwind": "$array"
},
{
"$unwind": "$array"
},
{
"$unwind": "$array"
},
{
"$group": {
"_id": "$_id",
"size": {
"$sum": "$array"
}
}
}])
Example here
Edit
As Takis _ suggested into the comments if you want to get all values from your entire collection (not only for each document) you can $group using null as this example

Query by data already in the object

I'm writing a query that gets data from "coll2" based on data that is inside "coll1".
Coll1 has the following data structure:
{
"_id": "asdf",
"name": "John",
"bags": [
{
"type": "typ1",
"size": "siz1"
},
{
"type": "typ2",
"size": "siz2"
}
]
}
Coll2 has the following data structure:
{
_id: "qwer",
coll1Name: "John",
types: ["typ1", "typ3"],
sizes: ["siz1", "siz4"]
}
{
_id: "zxcv",
coll1Name: "John",
types: ["typ2", "typ3"],
sizes: ["siz1", "siz2"]
}
{
_id: "fghj",
coll1Name: "John",
types: ["typ2", "typ3"],
sizes: ["siz1", "siz4"]
}
I want to get all the documents in coll2 that have the same Type+Size combo as in coll1 using the $lookup stage of the aggregation pipeline. I understand that this can be achieved by using the $lookup pipeline and $expr but I cant seem to figure out how to dynamically make a query to pass into the $match stage.
The output I would like to get for the above data would be:
{
_id: "qwer",
coll1Name: "John",
types: ["typ1", "typ3"],
sizes: ["siz1", "siz4"]
}
{
_id: "zxcv",
coll1Name: "John",
types: ["typ2", "typ3"],
sizes: ["siz1", "siz2"]
}
You can use $lookup to get the data from Col2. Then you need to check if there's any element in Col2 ($anyElemenTrue) that matches with Col1. $map and $in can be used here. Then you just need to $unwind and promote Col2 to root level using $replaceRoot
db.Col1.aggregate([
{
$lookup: {
from: "Col2",
localField: "name",
foreignField: "coll1Name",
as: "Col2"
}
},
{
$project: {
Col2: {
$filter: {
input: "$Col2",
as: "c2",
cond: {
$anyElementTrue: {
$map: {
input: "$bags",
as: "b",
in: {
$and: [
{ $in: [ "$$b.type", "$$c2.types" ] },
{ $in: [ "$$b.size", "$$c2.sizes" ] },
]
}
}
}
}
}
}
}
},
{
$unwind: "$Col2"
},
{
$replaceRoot: {
newRoot: "$Col2"
}
}
])
You are correct in your approach to use $lookup with the pipeline field to filter the input documents in the $match pipeline
The $expr expression should typically follow
"$expr": {
"$and": [
{ "$eq": [ "$name", "$$coll1_name" ] },
{ "$setEquals": [ "$bags.type", "$$types" ] },
{ "$setEquals": [ "$bags.size", "$$sizes" ] }
]
}
where the first match expression in the $and conditional { "$eq": [ "$name", "$$coll1_name" ] } checks to see if the name field in coll1 collection matches the coll1Name field in the input documents from coll2.
Of course the fields from coll2 should be defined in a variable in the pipeline with the let field for the $lookup pipeline to access them.
The other match filters are basically checking if the arrays are equal where "$bags.type" from coll1 resolves to an array of types i.e. [ "typ1", "typ3" ] for example.
On getting the output field from $lookup which happens to be an array, you can filter the documents in coll2 on that array field where there can be some empty lists as a resul of the above $lookup pipeline $match filter:
{ "$match": { "coll1Data.0": { "$exists": true } } }
Overall your aggregate pipeline operation would be as follows:
db.getCollection('coll2').aggregate([
{ "$lookup" : {
"from": "coll1",
"let": { "coll1_name": "$coll1Name", "types": "$types", "sizes": "$sizes" },
"pipeline": [
{ "$match": {
"$expr": {
"$and": [
{ "$eq": [ "$name", "$$coll1_name" ] },
{ "$setEquals": [ "$bags.type", "$$types" ] },
{ "$setEquals": [ "$bags.size", "$$sizes" ] }
]
}
} }
],
"as": "coll1Data"
} },
{ "$match": { "coll1Data.0": { "$exists": true } } },
{ "$project": { "coll1Data": 0 } }
])

Finding only an item in an array of arrays by value with Mongoose

Here is an example of my Schema with some data:
client {
menus: [{
sections: [{
items: [{
slug: 'some-thing'
}]
}]
}]
}
And I am trying to select it like this:
Schema.findOne({ client._id: id, 'menus.sections.items.slug': 'some-thing' }).select('menus.sections.items.$').exec(function(error, docs){
console.log(docs.menus[0].sections[0].items[0].slug);
});
Of course "docs.menus[0].sections[0].items[0].slug" only works if there is only one thing in each array. How can I make this work if there is multiple items in each array without having to loop through everything to find it?
If you need more details let me know.
The aggregation framework is good for finding things in deeply nested arrays where the positional operator will fail you:
Model.aggregate(
[
// Match the "documents" that meet your criteria
{ "$match": {
"menus.sections.items.slug": "some-thing"
}},
// Unwind the arrays to de-normalize as documents
{ "$unwind": "$menus" },
{ "$unwind": "$menus.sections" },
{ "$unwind": "$menus.sections.items" }
// Match only the element(s) that meet the criteria
{ "$match": {
"menus.sections.items.slug": "some-thing"
}}
// Optionally group everything back to the nested array
// One step at a time
{ "$group": {
"_id": "$_id",
"items": { "$push": "$menus.sections.items.slug" }
}},
{ "$group": {
"_id": "$_id",
"sections": {
"$push": { "items": "$items" }
}
}},
{ "$group": {
"_id": "$_id",
"menus": {
"$push": { "sections": "$sections" }
}
}},
],
function(err,results) {
}
)
Also see the other aggregation operators such as $first for keeping other fields in your document when using $group.

Non strict behavior of $nin in mongodb

Is there a non strict $nin version in mongodb? for example
Let's say that we have a model called User and a Model called task
var TaskSchema = new Schema({
user_array: [{user: Schema.ObjectId}],
});
A quick sample would be this
task1 : [user1, user2, user4, user7]
task2 : [user2, user 5, user7]
if I have a list of user
[user1, user7]
I want to select the task that has the least overlapping in the user_array, in this case task2, I know $nin strictly returns the task that contains neither user1 or user7, but I would like to know if there are operation where $nin is non strict.
Alternatively, I could have write a DP function to this for me
Any advice would be appreciated
Thanks
Well in MongoDB version 2.6 and upwards you have the $setIntersection and $size operators available so you can perform an .aggregate() statement like this:
db.collection.aggregate([
{ "$project": {
"user_array": 1,
"size": { "$size": {
"$setIntersection": [ "$user_array", [ "user1", "user7" ] ]
}}
}},
{ "$match": { "size": { "$gt": 1 } },
{ "$sort": { "size": 1 }},
{ "$group": {
"_id": null
"user_array": { "$first": "$user_array" }
}}
])
So those operators help to reduce the steps required to find the least matching document.
Basically the $setIntersection returns the matching elements in the array to the one it is being compared with. The $size operator returns the "size" of that resulting array. So later you filter out with $match any documents where neither of the items in the matching list were found in the array.
Finally you just sort and return the item with the "least" matches.
But it can still be done in earlier versions with some more steps. So basically your "non-strict" implementation becomes an $or condition. But of course you still need to count the matches:
db.collection.aggregate([
{ "$project": {
"_id": {
"_id": "$_id",
"user_array": "$user_array"
},
"user_array": 1
}}
{ "$unwind": "$user_array" },
{ "$match": {
"$or": [
{ "user_array": "user1" },
{ "user_array": "user7" }
]
}},
{ "$group": {
"_id": "$_id",
"size": { "$sum": 1 }
}},
{ "$sort": { "size": 1 } },
{ "$group": {
"_id": null,
"user_array": { "$first": "$_id.user_array" }
}}
])
And that would do the same thing.

Resources