In order to expand the JSON-API capabilities of my node.js application, I'm trying to sort a query based on relationships (AKA other documents), although I don't want to return them.
According to the JSON-API documentation:
a sort field of author.name could be used to request that the primary data be sorted based upon the name attribute of the author relationship.
E.g. db.collection('books').find({}) returns:
[
{
type: "book",
id: "2349",
attributes: {
title: "My Sweet Book"
},
relationships: {
author: {
data: {
type: "authors",
id: "9"
}
}
}
},
{} // etc ...
]
db.collection('authors').find({id: "9"}) returns:
[
{
type: "author",
id: "9",
attributes: {
name: "Hank Moody"
}
}
]
Now I need some way to do something similar to e.g.:
db.collection('books').find({}).sort({"author.name": -1})
I think I need to convert the query to an aggregation so I can use the $lookup operator, but I'm not sure how to use localField and foreignField.
db.collection('books').aggregate([
{$match: {}},
{$lookup: {from: "authors", localField: "attributes.author.data.id", foreignField: "id", as: "temp.author"}},
{$sort: {"$books.temp.author.name": -1}},
{$project: {temp: false}},
])
Notes
This will be a global function for fetching JSON-API data.
This means we don't know wether a sort key is an attribute or a relationship.
Most servers run LTS versions and have MongoDB 3.2
You can try below aggregation.
$lookup to join to authors collection followed by $unwind to flatten the book_author array for applying $sort on name field and $project with exclusion to remove book_author field ( only works starting Mongo 3.4 version ). For lower versions you have to include all the other fields you want to keep and excluding book_author field in the $project stage.
db.collection('books').aggregate([{
$lookup: {
from: "authors",
localField: "relationships.author.data.id",
foreignField: "id",
as: "book_author"
}
}, {
$unwind: "$book_author"
}, {
$sort: {
"book_author.attributes.name": -1
}
}, {
$project: {
"book_author": 0
}
}])
Related
I'm using a aggregate query to retrieve data from multiple collections, however there is a strange behavior that I dont seem to understand.
I need to lookup throw two collections, thus the lookup inside the pipeline. And also use the _id from the collection I'm making the aggregation(campaignadgroups) to match on the second nested collection (broadcastplans)
This is my query:
db.getCollection('campaignadgroups').aggregate([
{
$match: { "campaign_id": ObjectId("5fc8f7125148d7d0a19dcbcb")} // hardcoded just for tests
},
{
$lookup: {
from: "broadcastreports",
let: {campaignadgroupid: "$_id"},
pipeline: [
{
$match: {"reported_at": { $gte:ISODate("2020-12-01T15:56:58.743Z"), $lte: ISODate("2020-12-03T15:56:58.743Z")} }
},
{
$lookup: {
from: "broadcastplans",
localField: "broadcast_plan_id",
foreignField: "_id",
as: "broadcastplan"
}
},
{$unwind: "$broadcastplan"},
{
$match: { "broadcastplan.campaign_ad_group_id": {$eq: "$$campaignadgroupid"} // The problem happens here
}
}
],
as: "report"
}
},
])
The issue is that by matching with $$campaignadgroupid the report documents is empty.
However, if I replace the variable with the hardcoded id like ObjectId("5fc8f7275148d7d0a19dcbcc") I get the documents that I pretend.
For reference I'm debugging this issue on Robot3T so I can then translate to mongoose later.
I tried already to use $toObjectId however the _ids are not strings but ObjectIds already.
Thank you very much
Ok this is why I love and hate to code. After 3h debugging after asking here I immediately discovered the issue... I just needed to change from
$match: { "broadcastplan.campaign_ad_group_id": {$eq: "$$campaignadgroupid"}
to
$match: { $expr: { $eq: ["$broadcastplan.campaign_ad_group_id", "$$campaignadgroupid"]}
I am having trouble with aggregation in Mongodb. I have a model User and UserExtra with extra details about the user. I need to do a $lookup to connect UserExtra to User output. But want to be able to filter results, based on age, gender and city, which are part of UserExtra. My current query is as follows, and it should return results but returs empty array.
const match = {
'userExtra.age': {
$gt: dateTo.toISOString(),
$lt: dateFrom.toISOString()
},
'userExtra.gender': gender
}
if (city) match['userExtra.city'] = city;
const users = await User.aggregate([
{
$lookup: {
from: 'user_extra',
localField: 'userExtra',
foreignField: '_id',
as: 'userExtra'
}
},
{
$match: match
},
{
$unwind: "$userExtra"
}
]);
res.send(users);
You have to unwind the lookup result first, then match pipeline.
let pipelineQuery = [
{
$lookup: {
from: 'user_extra',
localField: 'userExtra',
foreignField: '_id',
as: 'userExtra'
}
},
{
$unwind: "$userExtra"
},
{
$match: match
},
]
Hey guys I figured it out. I was using toIsoString() method, which turns the date into a string. All I did was remove the method and just passed the dateTo and dateFrom dates and now it works.
I would suggest testing each part of query one by one..First just run the lookup part in aggregate to see what the result looks like.After that run match.and subsequently add unwind.
As the question title says, I'm trying to use an array field returned from a $match stage to query another collection in the next stage using $lookup and the $in operator to retrieve all documents that have at least one category inside this array. (I'm using Mongoose in Node, by the way)
I want to match a "configurations" collection by '_id' that have this simplified schema:
{
title: {type: String, required: true},
categories: {
allow: {type: Boolean, required: true},
list: [
{
name: {type: String, required: true},// DENORMALIZED CATEGORY NAME
_id: {type: mongoose.Schema.Types.ObjectId}
}
]
}
}
And in the next stage I want to aggregate all "partners" that belongs to at least one of those categories array. "partners" have the following schema:
{
company: {type: String, required: true},
categories: [
{type: mongoose.Schema.Types.ObjectId}
]
}
This is what I'm doing right now:
configuration.aggregate([
{$match: {_id: ObjectID(configurationId)}},
{
$lookup: {
from: "partners",
pipeline: [
{
$match: {
active: true,// MATCH ALL ACTIVE PARTNERS
categories: {
$in: {// HERE IS THE PROBLEM: I CAN'T RETRIEVE AN ARRAY FROM $map OPERATOR
$map: {// MAP CONFIGURATION CATEGORY LIST TO OUTPUT AN ARRAY ONLY WITH ID OBJECTS
input: '$categories.list',
as: 'category',
in: '$$category._id'
}
}
}
}
},
{ $project: { _id: 1, company: 1 } }
],
as: "partners"
}
},
])
The $map operator works as expected in a $project stage, but in this case I just can't use it's result as an array to be used with $in operator.
Is there any way to do this?
Thanks!
UPDATE
Doing like #Veeram suggested eliminates the need of $map operator in the $lookup stage:
{
"$lookup":{
"from":"partners",
"let":{"categories_id":"$categories.list._id"},
"pipeline":[
{"$match":{"active":true,"$expr":{"$in":["$categories","$$categories_id"]}}},
{"$project":{"_id":1,"company":1}}
],
"as":"partners"
}
}
But the problem persists with the $in operator. Like I've commented, this $in use case is the same as the 4th example in the official documentation (docs.mongodb.com/manual/reference/operator/aggregation/in), and it results in a false statement, because we are trying to check if an array ("$categories") is an element of another array ("$$categories_id"), which will fail, because the elements of "$$categories_id" are id objects and not arrays.
Does anyone know if there is any workaround for this?
Thanks!
You don't need to use $map. You can use dot notation to access the ids.
$let is required to access the values from local collection and $expr to compare the document fields.
Something like
{
"$lookup":{
"from":"partners",
"let":{"categories_id":"$categories.list._id"},
"pipeline":[
{"$match":{
"active":true,
"$expr":{
"$gt":[
{"$size":{"$setIntersection":["$categories","$$categories_id"]}},
0
]
}
}},
{"$project":{"_id":1,"company":1}}
],
"as":"partners"
}
}
I'm trying to use CASL for authorization check of nested items.
It uses mongoose for query data and check access.
My domain is that:
A "User" could has more "Vehicles"
A "Document" must have a Vehicle
Schema:
vehicle { users: [ {type: objectId, ref: 'user'} ] }
document { vehicle: {type: objectId, ref: 'vehicle' }}
To find the vehicle "by user" I do:
db.getCollection('vehicle').find(
{ users: {$in: [ ObjectId("5ae1a957d67500018efa2c9d") ]} }
)
That works.
In the documents collection, the data has records such as this:
{
"_id": ObjectId("5aeaad1277e8a6009842564d"),
"vehicle": ObjectId("5aea338b82d8170096b52ce9"),
"company": "Allianz",
"price": 500,
"date_start": ISODate("2018-05-02T22:00:00.000Z"),
"date_end": ISODate("2019-05-02T22:00:00.000Z"),
"createdAt": ISODate("2018-05-03T06:32:50.590Z"),
"updatedAt": ISODate("2018-05-03T06:32:50.590Z"),
"__v": 0
}
To find the document "by user" I do:
db.getCollection('document').find(
{ "vehicle.users": {$in: [ ObjectId("5ae1a957d67500018efa2c9d") ]} }
)
It doesn't work. Is possibile to do that in one single "find" query?
You can't do it in a simple MongoDB find() query, because the data about vehicle users exists in the vehicle collection, not the documents collection.
However, it is possible with an aggregation pipeline using the $lookup operator to link the data in two different collections. The aggregation would be something like this:
db.document.aggregate([
{$lookup: {
"from": "vehicle",
"localField": "vehicle",
"foreignField": "_id",
"as": "vehicleDetails",
}},
{$match: {"vehicleDetails.users" : ObjectId("5ae1a957d67500018efa2c9d")}}
])
You will probably need to add more stages to reshape the data the way you need it, but the key is to use $lookup to link the data from the two collections, then use $match to filter the set of results.
In order for this query to work you need to store users ids array in vehicle document. Neither Mongo nor CASL doesn't manage external references automatically.
Alternative solutions:
So, I see few ways:
Retrieve ids of all vehicles when you define rules. This works good in case if amount of vehicles not big (<= 1000)
const vehicleIds = await getVehicleIds(user)
can(['read', 'update'], 'document', { vehicle: { $in: vehicleIds } })
Denormalize your scheme. For example, add additional user_id field to vehicle document
Think whether you can embed document as subdocument to vechicle, something like this:
vehicle {
documents: [Document],
users: [ {type: objectId, ref: 'user'} ]
}
Just don't define rule per documents and enforce them in routes (REST or GraphQL doesn't matter).
app.get('/vehicle/:id/documents', async (req, res) => {
const vehicle = await Vehicle.findById(req.params.id)
req.ability.throwUnlessCan('read', vehicle)
const documents = Document.find({ vehicle: vehicle.id })
res.send({ documents })
})
I have a group object, which is included an members property. The members property is an array of objects. That object is having following properties.
{"id" : "1", "status" : 1}
My requirement is to get the list of users, who are having status 1, of a given group object (by a given id).
This is possible to do, with a simple get by ID query and a foreach. But want to know, is there any advance query to do this.
My overall Group object as follows.
var UserSchema = new Schema({
user_id:{
type : Schema.ObjectId,
ref : 'User',
default : null
},
status:{
type : Number,
default : null /* 1 - pending | 2 - rejected | 3 - accepted*/
},
});
var GroupSchema = new Schema({
type:{
type : Number,
default : 1 /* 1 - group | 2 - community*/
},
name:{
type:String,
default:null
},
members:[UserSchema]
},{collection:"groups"});
Thank You in advance.
You can use the aggregation framework to filter the documents in the groups collection by the given group id and the members array status field. This will be your initial pipeline stage which is $match operator-driven.
The next pipeline step should be the $filter operator which selects a subset of the members array based on a given condition. This is necessary since the previous pipeline only filters at document level, not at array/field level.
Once you get the filtered array you can then apply the $lookup function as means to "populate" your members list. However, since localField is an array and you want to match the elements inside it against a foreignField which is a single element, you'll need to $unwind the array as one stage of the aggregation pipeline before applying the $lookup operator.
The following example demonstrates how you can apply all the above steps in your case:
Group.aggregate([
{
"$match": {
"_id": groupId,
"members.status": 1
}
},
{
"$filter": {
"input": "$members",
"as": "member",
"cond": { "$eq": ["$$member.status", 1] }
}
}
{ "$unwind": "$members" },
{
"$lookup": {
"from": "users"
"localField": "members.user_id",
"foreignField": "_id",
"as": "member"
}
}
]).exec(function(err, results) {
if (err) throw err;
console.log(results);
});
The results will contain a list of documents which have both the group and user attributes.
If your MongoDB version does not support the $filter and $lookup operators introduced in version 3.2. X and newer, then consider using the $setDifference and $map operator combo to filter the array elements in a $project pipeline.
The $map operator in essence creates a new array field that holds values as a result of the evaluated logic in a subexpression to each element of an array. The $setDifference operator then returns a set with elements that appear in the first set but not in the second set; i.e. performs a relative complement of the second set relative to the first. In this case it will return the final members array that has elements not related to the parent documents via the status property.
Execute the aggregate operation after the $project pipeline step and since the documents returned are plain javascript objects, not Mongoose Documents (any shape of document can be returned), you need to cast the results to Mongoose Documents so that you can use populate function on the field with the results.
The following example demonstrates the above workaround:
Group.aggregate([
{
"$match": {
"_id": groupId,
"members.status": 1
}
},
{
"$project": {
"type": 1, "name": 1,
"members": {
"$setDifference": [
{
"$map": {
"input": "$members",
"as": "member",
"in": {
"$cond": [
{ "$eq": [ "$$member.status", 1 ] },
"$$member",
false
]
}
}
},
[false]
]
}
}
}
]).exec(function(err, result) {
if (err) throw err;
var docs = result.map(function(doc) { return new Group(doc) });
Group.populate(docs, { "path": "members" }, function(err, results) {
if (err) throw err;
console.log(JSON.stringify(results, undefined, 4 ));
res.json(results);
});
});
Assuming your collection name is groups, you can query like this :
db.groups.find({"members.status":1}, {"members":1});
This will fetch all the users that has status 1. If you want to query based on a specific user_id (assuming user id as "1A" here) , you can add in the object like this :
db.groups.find({"members.status":1, "members.user_id":"1A"}, {"members":1});