Nodejs mongodb API performance - node.js

I want to create an API which will query users in my mongodb and returns the data of all users. For each user i need to perform additional query to get count in 2 other schema as below:
const getUsersSummary = async (req, res, next) => {
try {
const users = await User.fetch();
const usersWithCount = await Promise.all(
users.map(async (user) => {
let tests;
let enrollments;
try {
tests = await Test.countDocuments(
{ user: user._id }
);
enrollments = await Enrollment.countDocuments(
{ user: user._id }
);
} catch (e) {
tests = 0;
enrollments = 0;
}
return {
_id: user._id,
name: user.name,
address: user.address,
tests: tests,
enrollments: enrollments,
};
})
);
return res.json({
users: usersWithCount,
});
} catch (err) {
next(err);
}
};
I want to know if this is a good way to do it.
The code works. But i am concerned about the performance and load it will put on my server.

I think you might consider use the MongoDB aggregation framework. You can combine user data with trial and registration data using the $lookup operator to join the user data with the tests and enrollments data, and then use the $group operator to get the count of tests and enrollments for each user. This will reduce the number of database queries and improve the performance of your API.
const aggregate = User.aggregate([
{
$lookup: {
from: "tests",
localField: "_id",
foreignField: "user",
as: "tests"
}
},
{
$lookup: {
from: "enrollments",
localField: "_id",
foreignField: "user",
as: "enrollments"
}
},
{
$project: {
_id: 1,
name: 1,
address: 1,
tests: { $size: "$tests" },
enrollments: { $size: "$enrollments" }
}
}
]);
const usersWithCount = await aggregate.exec();
return res.json({
users: usersWithCount,
});

Related

MongoDB aggregate: return collection if object id same between two collection

This is my code
async isCaseBelongToRiderCompany(userId, caseId): Promise<boolean> {
const employeeIdFromCase = await this.connection.db
.collection('cases')
.findOne({
_id: new mongo.ObjectID(caseId)
})
const employeeIdFromUser = await this.connection.db
.collection('users-permissions_user')
.findOne({
_id: new mongo.ObjectID(userId)
})
if(employeeIdFromCase.provider){
if(employeeIdFromUser.employer){
if(employeeIdFromCase.provider.toString() === employeeIdFromUser.employer.toString()){
return true;
}
}else{
return false
}
}else{
return false
}
}
I want to change this to mongo aggregate so that it will reduce to a single call to MongoDB instead of needing to await two times.
Is there any way I get convert this logic to aggregate?
You can use the following pipeline using $lookup:
async isCaseBelongToRiderCompany(userId, caseId): Promise<boolean> {
const employeeIdFromUser = await this.connection.db
.collection('users-permissions_user')
.aggregate([
{
$match: {
_id: new mongo.ObjectID(userId),
},
},
{
$lookup: {
from: 'cases',
let: { employer: '$employer', caseId: caseId },
pipeline: [
{
$match: {
$expr: {
$and: [
{
$eq: [
'$_id',
'$$caseId',
],
},
{
$eq: [
'$provider',
'$$employer',
],
},
],
},
},
},
],
as: 'cases',
},
},
{
$match: {
'cases.0': { $exists: true },
},
},
]).toArray();
return employeeIdFromUser.length > 0;
}
I personally do not recommend it, the aggregation pipeline has some overhead, combined with the fact that this $lookup syntax does not utilize indexes if you're on Mongo version lesser than 5 makes this less efficient by a mile.
The two separate calls is in fact the best approach as they are both indexed and very lean, you can just make the code a little cleaner:
async isCaseBelongToRiderCompany(userId, caseId): Promise<boolean> {
const employeeIdFromCase = await this.connection.db
.collection('cases')
.findOne({
_id: new mongo.ObjectID(caseId)
})
const employeeIdFromUser = await this.connection.db
.collection('users-permissions_user')
.findOne({
_id: new mongo.ObjectID(userId)
})
return employeeIdFromCase?.provider?.toString() === employeeIdFromUser?.employer?.toString()
}
Additionally you can execute both promises at the same time using Promise.all, like so:
async isCaseBelongToRiderCompany(userId, caseId): Promise<boolean> {
const [employeeIdFromCase, employeeIdFromUser] = await Promise.all([
this.connection.db
.collection('cases')
.findOne({
_id: new mongo.ObjectID(caseId)
}),
this.connection.db
.collection('users-permissions_user')
.findOne({
_id: new mongo.ObjectID(userId)
})
])
return employeeIdFromCase?.provider?.toString() === employeeIdFromUser?.employer?.toString()
}

How to look up into db in mongo db using node js

I am learning NodeJS and Mongo Db and I got stuck. Could you please help me If you know solution?
I have 2 questions
1.)I have 2 collections in mongo db
I tried to find out which projects user has. For example michaela has projects :nodejs, test2
jasmin has projects: test2 (I don't want to know who are coders for current project but I want to know what are projects for current user)
data what I want to get should look something like this
[{
id: "some value"
username: "michaela"
email:"some value"
password:"some value"
projects:"[{
project_name:"node js"
project_description:"learn node js"
deadline:"2021-11-12"
status:"todo"
},
{
project_name:"test2"
project_description:"learn node js"
deadline:"2021-11-12"
status:"todo"
}]
},
id: "some value"
username: jasmin"
email:"some value"
password:"some value"
projects:"[
{
project_name:"test2"
project_description:"learn node js"
deadline:"2021-11-12"
status:"todo"
}]
}]
I don't understand exactly how look up works. I tried following example(test) but it doesn't work
I also don't understand how to connect 2 db for example allMembers and all projects I merged into one array but I don't think this is correct way how to do it.
this is my model
const usersCollection = require('../db').db().collection("users")
const projectsCollection = require('../db').db().collection("projects")
Team.prototype.getAllMembers = function(){
return new Promise (async (resolve, reject) => {
try{
let allMembers = await usersCollection.find({}).toArray()
let allprojects = await projectsCollection.find({}).toArray()
let test = await usersCollection.aggregate([
{$lookup: {from: "projects", localField: "username", foreignField: "coder", as: "projects"}},
{$project: {
username:1,
projects:{
name:{$arrayElemAt: ["$projects.project_name", 0]},
description:{$arrayElemAt: ["$projects.project_description", 0]},
},
email:1,
password:1,
}}
]).toArray()
console.log("test", test)
let arr3 = [...allMembers, ...allprojects]
resolve(arr3)
}catch{
reject()
}
})
}
this is my controller
exports.home = function(req, res) {
//when user is log in
if (req.session.user) {
let teamMember = new Team(req.body)
teamMember.getAllMembers().then((data)=>{
console.log("members data", data)
if(data){
res.render('members', {data: data })
}else{
res.send("on this page are no data")
}
}).catch((err)=>{
res.send("problem")
})
//when user is not log in, flash tie errory z db automaticky vymaze po pouziti
} else {
res.render('login', {errors: req.flash('errors'), regErrors: req.flash('regErrors')})
}
}
and this is how it looks in db
Thank you in advance
I think you should aggregate on user collection like this match user name and then project first element of projects
lookup in here put project to projects field
db.users.aggregate([
{
$match: {
username: "jasmin"
}
},
{
"$lookup": {
"from": "projects",
"localField": "username",
"foreignField": "coder",
"as": "projects"
}
},
{
"$project": {
username: 1,
email:1,
password:1,
projects: 1
}
}
])
https://mongoplayground.net/p/U6hNw2WdQLE
if you want to get all data with just first project in that use this
db.users.aggregate([
{
"$lookup": {
"from": "projects",
"localField": "username",
"foreignField": "coder",
"as": "projects"
}
},
{
"$project": {
username: 1,
email:1,
password:1,
projects:{$arrayElemAt: ["$projects", 0]}
}
}
])

How to get an document in a table and join it with a second table mogodb, Getting Error : .find(...).aggregate is not a function

I am working with mongodb and I have two tables, a users and a books one. I want to get an specific user and join it with the second table. Example:
Document Example of table users:
var user = {
_id: ObjectId('asduiasbdiuasbduiasbduib'),
username: 'raos',
last_login: '12-01-2020'
}
Document Example of table book:
var book = {
_id: ObjectId('jkngsjilsdhioh'),
bookTitle: 'Lost in space',
available: false,
rented_by: 'asduiasbdiuasbduiasbduib'
}
I tried the next thing:
F.db.collection('users').find({_id: 'asduiasbdiuasbduiasbduib'}).aggregate([
{
$lookup:{
from: 'books',
localField: '_id',
foreignField: 'rented_by',
as: 'books'
}
}
]).toArray((err, res) => {
console.log(res);
});
// Error showed:
//F.db.collection(...).find(...).aggregate is not a function
function something(userId) {
return new Promise((resolve, reject) => {
var aggregate = user.aggregate();
aggregate
.match({
_id: ObjectId(userId)
})
.lookup({
localField: '_id',
from: 'books',
foreignField: 'rented_by',
as: 'book'
})
aggregate.exec(function (err, books) {
if (err) return reject(err);
return resolve(books);
});
});
}
Does it solved your problem?

Mongodb join query from nodejs

I am new to MongoDB and trying to join two query and store result in a single model. I want to fetch client name from another collection while fetching client task.
Model:-
const mongoose = require('mongoose');
var ObjectId = mongoose.Schema.Types.ObjectId;
const ClientTaskSchema = new mongoose.Schema({
clientId: {
type: Number
},
taskId: {
type: Number
},
clientTaskId: {
type: Number
},
active: {
type: Boolean
}
});
module.exports = mongoose.model('ClientTask', ClientTaskSchema);
Controller:-
module.exports.getClientByTask = function(req, res) {
var query = url.parse(req.url,true).query;
ClientTask.find({taskId: query.taskId}, function(err, clientTask) {
if (err) throw err;
if (!clientTask) {
res.status(200).json({ success: false, message: 'Somthing went wrong. Please contact admin.'});
}
else {
res.status(200).json({ success: true, message: 'Successfull', data: clientTask});
}
});
};
One option is to pass clientId as a reference:
clientId: {
type: mongoose.Schema.Types.ObjectId, ref: 'Client / or whatever your model'
}
Then you'll be able to use Mongoose's populate method http://mongoosejs.com/docs/populate.html
ClientTask
.find({ taskId: query.taskId })
.populate('clientId', { name: 1 }).exec(function (err, clientTask) {
if (!clientTask) {
res.status(404).json({ message: 'Client task not found' })
}
// your logic
});
You can fetch aggregated data with mongodb aggregate. To Calculates aggregate values for the data in a collection:
$lookup Performs a left outer join to an unsharded collection in the same database to filter in documents from the “joined” collection for processing. To each input document, the $lookup stage adds a new array field whose elements are the matching documents from the “joined” collection. The $lookup stage passes these reshaped documents to the next stage.
In your case:
Model.ClientTask.aggregate([
{
$lookup:
{
from: "client",
localField: "_id",
foreignField: "clientId",
as: "clientData"
},
},
{
$project: {
"name": clientData.name, // This name will be client name from client collections
"taskId": 1,
"clientTaskId": 1,
"active": 1
}
}
],
function (err, response) {
console.log(err, response)
});

Is there any way to make join faster, when dealing large amount of data?

I am using mongo and nodejs.
Current I have the following collections:
entryMore {
entryId
...
}
filesize {
entryId
...
}
entryIam {
entryId
...
}
entryStatus {
entryId
...
}
entryPubStatus {
entryId
....
}
They are all one to one mapping with entryId
I am joining all of the collections in nodejs with the following code
// db
const { MongoClient } = require('mongodb');
const fs = require('fs');
const Json2csvParser = require('json2csv').Parser;
// run catch err
run()
.catch(error => console.error(error.stack));
async function run() {
// connect
const client = await MongoClient.connect('mongodb://user:pass#127.0.0.1:27017');
// db
const db = client.db('kal');
// The only way to use `$lookup` in MongoDB 3.2 and 3.4
const docs = await db.collection('entryMore').aggregate([
// filesize
{
$lookup: {
from: 'filesize',
localField: 'entryId',
foreignField: 'entryId',
as: 'filesize'
}
},
{
// deconstruct and map to one by one
$unwind: '$filesize'
},
// Iam
{
$lookup: {
from: 'entryIam',
localField: 'entryId',
foreignField: 'entryId',
as: 'entryIam'
}
},
{
$unwind: '$entryIam'
},
// entry status
{
$lookup: {
from: 'entryStatus',
localField: 'entryId',
foreignField: 'entryId',
as: 'entryStatus'
}
},
{
$unwind: '$entryStatus'
},
// pub status
{
$lookup: {
from: 'entryPubStatus',
localField: 'entryId',
foreignField: 'entryId',
as: 'entryPubStatus'
}
},
{
$unwind: '$entryPubStatus'
},
// Final
{
$project: {
_id: 0,
entryId: 1,
name: 1,
//description: 1,
userId: 1,
creatorId: 1,
msDuration: 1,
categories: 1,
categoriesIds: 1,
createdAt: 1,
updatedAt: 1,
tags: 1,
downloadUrl: 1
}
}
]);
const json2csvParser = new Json2csvParser({field});
const csv = json2csvParser.parse(docs);
await writeFile('./csv/entrySheet.csv', csv);
console.log('done!');
process.exit();
}
Each collection has 97k records. It takes whole date to generate the csv. I wonder is there any way to improve it?
I agree with what is written by #NeilLunn. But to still clarify your questions, $lookup performs an equality match on the foreignField to the localField from the input documents. If a document in the from collection does not contain the foreignField, the $lookup treats the value as null for matching purposes.
So go ahead and create an index on your foreign field so you'll be doing 97k index hits and not 97k table scans. It is bound to get the time way lower

Resources