Many-to-Many in MongoDB - node.js

I have two MongoDB models "Users" and "Roles". Each user can have multiple roles and each role can be assigned to many users. To keep the relation simple I would like to store the reference between both models only in the "Users" model which is already working as expected. But when I'm loading all roles at once with .find({}), I would also like to know how many users are assigned to these roles (to check if a role can be modified).
I'm using Node.js + ExpressJS and mongoose. This is what I already have:
var userSchema = new Schema({
username: String,
...
roles : [ {
type : Number,
ref : 'Role'
} ]
});
var roleSchema = new Schema({
_id : Number,
name : String
});
--------------------------------------------------
function getRoles(request, response) {
Role.find({}, function(err, roles) {
....
response.send(roles);
});
}
But now I wonder how I would achive the count query per role and still be able to send the result in one response.
I would like to avoid to send a single request per role and try to do the count within the getRoles() function. In a relational database I would do something like this:
select r.*,
(select count(*) from user2role u2r where u2r.role_id = r.role_id)
from role r;
But what's the MongoDB equivalent?
Any help and hints will be appreciated.
Thanks, Gerry

To achieve the user count query per role with the given schema, you could either use the count() method as follows:
function getUsersCount(roleName, req, res) {
Role.findOne({"name": roleName}, function(err, role) {
var roleId = role._id;
// var countQuery = User.where({ "role": roleId }).count();
// User.count({ "roles": roleId }).count(callback)
// User.count({ "roles": roleId }, callback)
User.where({ "roles": roleId }).count(function (err, count) {
if (err) return handleError(err);
console.log("There are %d users with the role %d", count, roleName);
res.send(count);
})
});
}
or you can use the aggregate() method like:
// Find the user count for a given role
function getUsersCount(roleName, req, res) {
Role.findOne({"name": roleName}, function(err, role) {
var roleId = role._id;
var pipeline = [
{
"$match": { "roles": roleId } /* filter the documents that only have the roleId in the role array */
},
{ "$unwind": "$roles" },
{
"$match": { "roles": roleId }
},
{
"$group": {
"_id": null, "count": { "$sum": 1 }
}
},
{
"$project": {
"_id": 0, "count": 1
}
}
];
Users.aggregate(pipeline, function (err, result) {
if (err) return handleError(err);
console.log(result); // [ { count: 55 } ] for example
res.send(result);
});
// Or use the aggregation pipeline builder.
Users.aggregate()
.match({ "roles": roleId })
.unwind("roles")
.match({ "roles": roleId })
.group({ "_id": null, "count": { "$sum": 1 } })
.select("-id count")
.exec(function (err, result) {
if (err) return handleError(err);
console.log(result); // [ { count: 55 } ] for example
res.send(result);
});
});
}
-- EDIT --
One approach that you could take is to use the lean() method to get pure JavaScript objects that you can manipulate to add the extra userCount field for each role. With lean() it's possible because documents returned from queries with the lean option enabled are plain javascript objects', not MongooseDocuments. Thus with the roles returned from the find() cursor, you can convert them into plain JS objects
that you can iterate over using the native JavaScript map() method, get the number of users per role using another query that takes into consideration the count() method explained above and return the modified object.
The following demonstrates how the function is implemented:
function getRoles(request, response) {
Role.find({}).lean().exec(function(err, roles) {
var result = roles.map(function(r){
var obj = {},
countQuery = User.where({ "roles": r._id }).count();
obj["userCount"] = countQuery;
obj["_id"] = r._id;
obj["name"] = r.name;
return obj;
});
response.send(result);
});
}

As per your above explanation about the schema design, i can assume your schema is as below :
Roles collection :
{
role_id :
name :
}
User collection :
{
user :
role_id : [ ]
....
}
you can use aggregation to achieve this :
db.users.aggregate([{$unwind: "$role_id"},
{$group: {_id: "$role_id", count : {"$sum":1}} },
]);

Related

MongoDB - find one and add a new property

Background: Im developing an app that shows analytics for inventory management.
It gets an office EXCEL file uploaded, and as the file uploads the app convert it to an array of JSONs. Then, it comapers each json object with the objects in the DB, change its quantity according to the XLS file, and add a timestamp to the stamps array which contain the changes in qunatity.
For example:
{"_id":"5c3f531baf4fe3182cf4f1f2",
"sku":123456,
"product_name":"Example",
"product_cost":10,
"product_price":60,
"product_quantity":100,
"Warehouse":4,
"stamps":[]
}
after the XLS upload, lets say we sold 10 units, it should look like that:
{"_id":"5c3f531baf4fe3182cf4f1f2",
"sku":123456,
"product_name":"Example",
"product_cost":10,
"product_price":60,
"product_quantity":90,
"Warehouse":4,
"stamps":[{"1548147562": -10}]
}
Right now i cant find the right commands for mongoDB to do it, Im developing in Node.js and Angular, Would love to read some ideas.
for (let i = 0; i < products.length; i++) {
ProductsDatabase.findOneAndUpdate(
{"_id": products[i]['id']},
//CHANGE QUANTITY AND ADD A STAMP
...
}
You would need two operations here. The first will be to get an array of documents from the db that match the ones in the JSON array. From the list you compare the 'product_quantity' keys and if there is a change, create a new array of objects with the product id and change in quantity.
The second operation will be an update which uses this new array with the change in quantity for each matching product.
Armed with this new array of updated product properties, it would be ideal to use a bulk update for this as looping through the list and sending
each update request to the server can be computationally costly.
Consider using the bulkWrite method which is on the model. This accepts an array of write operations and executes each of them of which a typical update operation
for your use case would have the following structure
{ updateOne :
{
"filter" : <document>,
"update" : <document>,
"upsert" : <boolean>,
"collation": <document>,
"arrayFilters": [ <filterdocument1>, ... ]
}
}
So your operations would follow this pattern:
(async () => {
let bulkOperations = []
const ids = products.map(({ id }) => id)
const matchedProducts = await ProductDatabase.find({
'_id': { '$in': ids }
}).lean().exec()
for(let product in products) {
const [matchedProduct, ...rest] = matchedProducts.filter(p => p._id === product.id)
const { _id, product_quantity } = matchedProduct
const changeInQuantity = product.product_quantity - product_quantity
if (changeInQuantity !== 0) {
const stamps = { [(new Date()).getTime()] : changeInQuantity }
bulkOperations.push({
'updateOne': {
'filter': { _id },
'update': {
'$inc': { 'product_quantity': changeInQuantity },
'$push': { stamps }
}
}
})
}
}
const bulkResult = await ProductDatabase.bulkWrite(bulkOperations)
console.log(bulkResult)
})()
You can use mongoose's findOneAndUpdate to update the existing value of a document.
"use strict";
const ids = products.map(x => x._id);
let operations = products.map(xlProductData => {
return ProductsDatabase.find({
_id: {
$in: ids
}
}).then(products => {
return products.map(productData => {
return ProductsDatabase.findOneAndUpdate({
_id: xlProductData.id // or product._id
}, {
sku: xlProductData.sku,
product_name: xlProductData.product_name,
product_cost: xlProductData.product_cost,
product_price: xlProductData.product_price,
Warehouse: xlProductData.Warehouse,
product_quantity: productData.product_quantity - xlProductData.product_quantity,
$push: {
stamps: {
[new Date().getTime()]: -1 * xlProductData.product_quantity
}
},
updated_at: new Date()
}, {
upsert: false,
returnNewDocument: true
});
});
});
});
Promise.all(operations).then(() => {
console.log('All good');
}).catch(err => {
console.log('err ', err);
});

Can't find a easy way out of multiple async for each node js (sails)

So here's the deal :
I have an array of objects with a child array of objects
askedAdvices
askedAdvice.replayAdvices
I'm looping trough the parent and foreach looping trough the childs and need to populate() two obejcts (I'm using sails)
The child looks like :
askedAdvices = {
replayAdvices : [{
bookEnd : "<ID>",
user : "<ID>"
}]
}
So my goal is to cycle and populate bookEnd and user with two findOne query, but I'm going mad with the callback hell.
Here's the Models code :
AskedAdvices Model
module.exports = {
schema : false,
attributes: {
bookStart : {
model : 'book'
},
replayAdvices : {
collection: 'replybookend'
},
user : {
model : 'user',
required : true
},
text : {
type : "text"
}
}
};
ReplyBookEnd Model
module.exports = {
schema : false,
attributes: {
bookEnd : {
model : 'book'
},
user : {
model : 'user',
required : true
},
text : {
type : "text"
}
}
};
Here's the Method code :
getAskedAdvices : function(req, res) {
var queryAskedAdvices = AskedAdvices.find()
.populate("replayAdvices")
.populate("user")
.populate("bookStart")
queryAskedAdvices.exec(function callBack(err,askedAdvices){
if (!err) {
askedAdvices.forEach(function(askedAdvice, i){
askedAdvice.replayAdvices.forEach(function(reply, i){
async.parallel([
function(callback) {
var queryBook = Book.findOne(reply.bookEnd);
queryBook.exec(function callBack(err,bookEndFound) {
if (!err) {
reply.bookEnd = bookEndFound;
callback();
}
})
},
function(callback) {
var queryUser = User.findOne(reply.user)
queryUser.exec(function callBack(err,userFound){
if (!err) {
reply.user = userFound;
callback();
}
})
}
], function(err){
if (err) return next(err);
return res.json(200, reply);
})
})
})
} else {
return res.json(401, {err:err})
}
})
}
I can use the async library but need suggestions
Thanks folks!
As pointed out in the comments, Waterline doesn't have deep population yet, but you can use async.auto to get out of callback hell. The trick is to gather up the IDs of all the children you need to find, find them with single queries, and then map them back onto the parents. The code would look something like below.
async.auto({
// Get the askedAdvices
getAskedAdvices: function(cb) {
queryAskedAdvices.exec(cb);
},
// Get the IDs of all child records we need to query.
// Note the dependence on the `getAskedAdvices` task
getChildIds: ['getAskedAdvices', function(cb, results) {
// Set up an object to hold all the child IDs
var childIds = {bookEndIds: [], userIds: []};
// Loop through the retrieved askedAdvice objects
_.each(results.getAskedAdvices, function(askedAdvice) {
// Loop through the associated replayAdvice objects
_.each(askedAdvice.replayAdvices, function(replayAdvice) {
childIds.bookEndIds.push(replayAdvice.bookEnd);
childIds.userIds.push(replayAdvice.user);
});
});
// Get rid of duplicate IDs
childIds.bookEndIds = _.uniq(childIds.bookEndIds);
childIds.userIds = _.uniq(childIds.userIds);
// Return the list of IDs
return cb(null, childIds);
}],
// Get the associated book records. Note that this task
// relies on `getChildIds`, but will run in parallel with
// the `getUsers` task
getBookEnds: ['getChildIds', function(cb, results) {
Book.find({id: results.getChildIds.bookEndIds}).exec(cb);
}],
getUsers: ['getChildIds', function(cb, results) {
User.find({id: results.getChildIds.userIds}).exec(cb);
}]
}, function allTasksDone(err, results) {
if (err) {return res.serverError(err);
// Index the books and users by ID for easier lookups
var books = _.indexBy(results.getBookEnds, 'id');
var users = _.indexBy(results.getUsers, 'id');
// Add the book and user objects back into the `replayAdvices` objects
_.each(results.getAskedAdvices, function(askedAdvice) {
_.each(askedAdvice.replayAdvices, function(replayAdvice) {
replayAdvice.bookEnd = books[replayAdvice.bookEnd];
replayAdvice.user = users[replayAdvice.bookEnd];
});
});
});
Note that this is assuming Sails' built-in Lodash and Async instances; if you're using newer versions of those packages the usage of async.auto has changed slightly (the task function arguments are switched so that results comes before cb), and _.indexBy has been renamed to _.keyBy.

Modify mongoose response data

I have a an API call that returns all messages a user has recieved as JSON.
The model data looks something like this:
{
sender: ObjectId,
reciever: ObjectId,
message: String
}
What is the proper way to modify the JSON the API responds with?
I want to end up with data grouped like so:
{
<senderid>:[ all of the messages from this sender],
<other_sender>:[ all of the messages from this sender]
}
Do I have to manually do this in javascript, or is there a faster way to do this taking advantage of mongoose?
Using the aggregation framework will be ideal for this task. You could run the following aggregation pipeline that makes use of the $group operator step to group the data to process them. The group pipeline operator is similar to the SQL's GROUP BY clause. In SQL, you can't use GROUP BY unless you use any of the aggregation functions. The same way, you have to use an aggregation function in MongoDB as well. In this instance, use the $push accumulator operator to create the array of messages.
Since Model.aggregate() returns plain objects you would then transform the resulting array to the desired hash key using lodash library's _.indexBy() method:
var pipeline = [
{
"$group": {
"_id": "$sender",
"messages": { "$push": "$message" }
}
}
];
Model.aggregate(pipeline,
function(err, res) {
if (err) return handleError(err);
var hashmap = _.chain(res)
.indexBy('_id')
.mapValues('messages')
.value();
console.log(JSON.stringify(hashmap, undefined, 4));
}
);
// Or use the aggregation pipeline builder.
Model.aggregate()
.group({ "_id": "$sender", "messages": { "$push": "$message" } })
.exec(function (err, res) {
if (err) return handleError(err);
var hashmap = _.chain(res)
.indexBy('_id')
.mapValues('messages')
.value();
console.log(JSON.stringify(hashmap, undefined, 4));
});
Check the demo below.
var data = [
{ _id: 'user1', messages: ['msg1', 'msg2'] },
{ _id: 'user2', messages: ['msg3', 'msg1'] },
{ _id: 'user3', messages: ['msg6', 'msg3'] },
{ _id: 'user4', messages: ['msg4', 'msg8'] }
];
var hashmap = _.chain(data)
.indexBy('_id')
.mapValues('messages')
.tap(log)
.value();
function log(value) {
pre.innerHTML += JSON.stringify(value, null, 4) + "\n"
}
<script src="//cdnjs.cloudflare.com/ajax/libs/lodash.js/3.10.1/lodash.min.js"></script>
<pre id="pre"></pre>

Mongoose update or create many values in one query

I have a document like this:
{
"_id" : ObjectId("565e906bc2209d91c4357b59"),
"userEmail" : "abc#example.com",
"subscription" : {
"project1" : {
"subscribed" : false
},
"project2" : {
"subscribed" : true
},
"project3" : {
"subscribed" : false
},
"project4" : {
"subscribed" : false
}
}
}
I'm using express to for my post web service call like this:
router.post('/subscribe', function(req, res, next) {
MyModel.findOneAndUpdate(
{
userEmail: req.body.userEmail
},
{
// stuck here on update query
},
{
upsert: true
}, function(err, raw) {
if (err) return handleError(err);
res.json({result: raw});
}
)
});
My req contains data like this:
{
userEmail: "abc#example.com",
subscription: ["project1", "project4"]
}
So these are the steps I would like to perform on this call:
Check user exists, otherwise create the user. For instance, if abc#example.com doesn't exist, create a new document with userEmail as abc#example.com.
If user exists, check project1 and project4 exists in subscription object. If not create those.
If project1 and project4 exists in subscription, then update the subscribed to true.
I'm not sure whether I can achieve all the above 3 steps with a single query. Kindly advise.
Vimalraj,
You can accomplish this using $set. The $set operator replaces the value of a field with the specified value. According to the docs:
If the field does not exist, $set will add a new field with the specified value, provided that the new field does not violate a type constraint. If you specify a dotted path for a non-existent field, $set will create the embedded documents as needed to fulfill the dotted path to the field.
Since your req.subscription will be an array you'll have to build your query. to look like this:
{
$set: {
"subscription.project1":{"subscribed" : true},
"subscription.project4":{"subscribed" : true}
}
You can use reduce to create an object from req.subscription = ["project1","project4"] array
var subscription= req.subscription.reduce(function(o,v){
o["subscription." + v] = {subscription:true};
return o;
},{});
Then your code becomes:
router.post('/subscribe', function(req, res, next) {
var subscription= req.subscription.reduce(function(o,v){
o["subscription." + v] = {subscription:true};
return o;
},{});
MyModel.findOneAndUpdate(
{
userEmail: req.body.userEmail
},
{
$set: subscription
},
{
upsert: true
}, function(err, raw) {
if (err) return handleError(err);
res.json({result: raw});
}
)
});

MongoDB and NodeJS get related data from 3 collections

i have a mongoDB query to get data with $group and $count.
This data contains the _id from other documents collection.
How can i get the other documents by its _id in NodeJS and MongoDB asyncrohnous?
db.orders.aggregate([
{$match: { 'works.TechnicianId': {$in:['53465f9d519c94680327965d','5383577a994be8b9a9e3f01e']},
'works.Date': {$gte: ISODate("2013-05-21T06:40:20.299Z"), $lt: ISODate("2016-05-21T06:40:20.299Z")}}},
{$unwind: "$works" },
{$group: {_id: "$works.TechnicianId",total:{$sum:'$works.price'},ordersId: { $push: "$_id" }}},
])
This is the result:
{
"result" : [
{
"_id" : "53465f9d519c94680327965d",
"total" : 198,
"ordersId" : [
ObjectId("537b5ea4c61b1d1743f4341f"),
ObjectId("537b4633021d75bd36863f29")
]
},
{
"_id" : "5383577a994be8b9a9e3f01e",
"total" : 22,
"ordersId" : [
ObjectId("537b5ea4c61b1d1743f4341f"),
ObjectId("537b4633021d75bd36863f29")
]
}
],
"ok" : 1
}
Now i need to get from orders collection the documents with id from ordersId, and from other collection the documents with _id from the result _id field.
I try with this:
var collection = db.collection('orders');
var result = [];
collection.aggregate([
{
$match: {
'works.TechnicianId': {
$in: ids
},
'works.Date': {
$gte: new Date(startDate),
$lt: new Date(endDate)
}
}
},
{
$unwind: "$works"
},
{
$group: {
_id: "$works.TechnicianId",
total: {
$sum: '$works.price'
},
orderId: {
$push: "$_id"
}
}
}
],
function (e, docs) {
if (e) {
error(e);
}
var usersCollection = db.collection('users');
_.each(docs, function (doc) {
usersCollection.findOne({_id: new ObjectID(doc._id)}, function (e, doc) {
doc.tech = doc;
});
doc.orders = [];
_.each(doc.orderId, function (queryOrder) {
collection.findOne({_id: new ObjectID(queryOrder._id)}, function (e, order) {
doc.orders.push(order);
});
});
success(docs);
});
});
But the success its called before all the _.eachs are finished..Any help, or idea?
Edit:
I try with Q promises, this is my code:
var usersCollection = db.collection('users');
var promises = [];
_.each(reports, function (report) {
var promise = usersCollection.findOne({_id: new ObjectID(report._id)}).then(
function (e, orderUserReported) {
if (e) {
error(e);
}
report.tech = orderUserReported;
_.each(orderUserReported.orderId, function (queryOrder) {
collection.findOne({_id: new ObjectID(queryOrder._id)}, function (e, order) {
report.orders.push(order);
});
});
});
promises.push(promise);
});
Q.allSettled(promises).then(success(reports));
and the error:
/Users/colymore/virteu/aa/server/node_modules/mongodb/lib/mongodb/connection/base.js:245
throw message;
^
TypeError: Cannot call method 'then' of undefined
Because of asynchronous execution you have to wait until results are returned. There are several options available:
async library https://github.com/caolan/async
promises https://github.com/kriskowal/q
Async is closer to your current code, you could use async.parallel https://github.com/caolan/async#parallel to wait untill you get data back
Update
Mongoose functions don't return Q promises, so you need to convert mongoose calls to promises by using something like Q.denodeify(User.findOne.bind(models.User))({ _id: userId}).then(...
For your case Q.denodeify(userCollection.findOne.bind(userCollection))({_id: new ObjectID(report._id)}).then(...
Short answer: Use promises. Look at Q.allSettled ( https://github.com/kriskowal/q )
Just run success asynchronously when all subtask are done.
Also using https://github.com/iolo/mongoose-q package may be helpful to not combine mongoose promises with Q ones if you want use mongoose in your mongo.

Resources