Goal: Get the students based on the given library name. The student model has the library name linked in the database.
What's happening:
1: Retrieving the name that is linked to the given ID (library ID aquired with useParams().id).
2: Looking for all students based on that found library name.
Result: Empty response. I feel like the problem has to do with the line library_name = library.name;. Simply that value hasn't set yet when the second query starts to execute? Because when I log the result right after this line, with res.send(library_name); the name is showing correctly.
app.get("/students/:id", (req, res) => {
const id = req.params.id;
let library_name = "";
LibraryModel.findById(id, (err, library) => {
library_name = library.name;
});
StudentModel.find({library: library_name}, (err, students) => {
if (err) {
res.send(err);
} else {
res.send(students);
}
});
});
You are right. library_name gehts only set in the callback function you passed, which happens after the StudentModel.find(...) gets called. Basically you are currently performing these 2 calls in parallel.
There are three ways to resolve this issue.
Moving second call to callback function
app.get("/students/:id", (req, res) => {
const id = req.params.id;
let library_name = "";
LibraryModel.findById(id, (err, library) => {
library_name = library.name;
StudentModel.find({library: library_name}, (err, students) => {
if (err) {
res.send(err);
} else {
res.send(students);
}
});
});
});
Use promises
To avoid what's known as "callback hell" can you also use promises instead of callback functions and await them:
app.get("/students/:id", async (req, res) => {
const id = req.params.id;
try {
const library = await LibraryModel.findById(id);
const students = await StudentModel.find({library: library.name});
res.send(students);
} catch (err) {
res.send(err);
}
});
Use a single aggregation pipeline
You can also merge these two separate databases queries into a single aggregation pipeline. You would need to first use $lookup and afterwards use $match to filter for the specific entries. Nevertheless there is additional information on the schemas needed in order to build this query.
Another Hint
I assume you are trying to create a RESTful API. You might want to review your path structure, since a RESTful approach would expect the ':id' to be the ID of a student, not the ID of a library.
It looks like GET '/libraries/:id/students' makes more sense in your case.
Related
I'm learning node.js currently and this is my first ever project in it. It's (supposedly) a simple to-do list app where there are multiple lists I can load/edit/save/remove.
In the todo_list.ejs file I have a div where I list all the collection names:
<div id="list_container" class="lists">
<ul id="col_list" class="collection-list">
<% lists.forEach(list => { %>
<li class="collection-list-item">
<div class="list-name-container">
<a href="/<%=list.name %>" class="list-link">
<span class="list-name" name="list_name"><%=list.name %></span>
</a>
</div>
</li>
<% }) %>
</ul>
</div>
looks like this:
When I click on the link of a list. I try to use the following code to load in a new list (which is a mongodb collection):
app.route("/:list_name").get((req, res) => {
MongoClient.connect(process.env.DB_CONNECT, (err, db) => {
if(err) throw err;
var database = db.db("myFirstDatabase");
const cursor = database.collection(req.params.list_name).find({}); /* stuck here */
database.listCollections().toArray((err, collections) => {
if(err) throw err;
db.close();
collections.forEach(element => {
if(element.name == req.params.list_name){
current_list = element;
current_list_name = element.name;
}
});
task.find({}, (err, todo_tasks) => { /*currently using the model.find() method to list all the documents which always looks at the "tasks" collection*/
res.render("todo_list.ejs", { tasks: todo_tasks, lists: collections, curr_list: current_list_name });
});
});
});
});
I commented where I'm stuck in the code above. I'm trying to get a mongodb collection by name, and then load all its contents onto a list after, but I don't know how to find a collection by name. Reading through the node.js documentation lead me to the cursor object, which has a ton of info and properties I have no clue what to do with...
Is there a simple way to find a collection by name and get a list of it's documents?
EDIT 1:
this is where i add tasks:
//ADD TASK TO LIST
app.post('/', async (req, res) => {
const tsk = new task({ /*the mongodb model for tasks*/
content: req.body.content,
deadline: req.body.deadline
});
try {
await tsk.save();
res.redirect("/");
} catch(e) {
console.log(e);
res.redirect("/");
}
});
I will not address the EJS part in this answer as I'm not qualified and the code you provided seems all good. However, I'll review the back-end part.
Also, since I don't know what kind of coding background you have (if any), this answer will contain a lot of explanation on perhaps simple concepts.
Summary
From your second code snippet, there are a couple things that are to be discussed:
Asynchronous code
The database connection and generalities
Actual implementation
Code conception
[EDIT]: Save/Edit implementations
There is also a lot more to cover depending on the knowledge of OP, such as try/catch clauses, MongoDB models validation, the usage of express's Router and more but I will only edit my answer if needed.
Asynchronous code
For the rest of the answer, most of the code will be surrounded by async/await keywords. These are necessary for the code to work properly.
Basically, JS being a language that is made for the web, you sometimes need to wait for network or database requests to be done before you do any other action. That's where the callbacks, the promises or the async/await syntax (which is syntactic sugar for promises) come in handy.
Let's say you need, like your example, to retrieve a list of tasks:
app.route("/:list_name").get((req, res) => {
MongoClient.connect(process.env.DB_CONNECT, (err, db) => {
if(err) throw err;
var database = db.db("myFirstDatabase");
const cursor = database.collection(req.params.list_name).find({}); /* stuck here */
console.log(cursor);
// ..........
});
});
JS being asynchronous by default, if you run this code, chances are high that cursor will be undefined. The reason for that is that the code doesn't wait for the database.collection(............. to finish in order to continue the execution. But with the help of the aforementioned callback/promises/async-await, our code can now wait until this instruction is done.
You can read on async/await here and here, and see here that MongoDB examples are using async/await as well, but you will see in the following sections more "practical" usages of it.
Keep in mind that what you are using (whether it is callbacks, promises or async/await syntax) is completely up to you and your preferences.
Database connection
As the code is currently written, everytime a user clicks on any item on your list, a connection to MongoDB will be established, and that connection doesn't belong to the route handler. Your back-end app should connect to the database once (at least for this case, it could prove useful to initiate multiple connections for some advanced cases), and close the connection when your back-end app stops (generally not the case with an API).
Atlas cloud databases for example, have a limit of 500 connections. Meaning that if, let's say, 501 users click on an item simultaneously on your front-end list, the best case scenario is someone doesn't get what he asked, but it could be worse.
For this matter, you have several options. One would be to go with a framework that helps you leverage some of the code and boilerplate, such as Mongoose or work with the native MongoDB driver which we will do, since you seem to already work with that and I strongly believe working with the lowest layer first will make you learn higher-level frameworks way faster.
Now, let's tackle the problem. We want to put the database connection somewhere else where it'll be called once. Again, there's several options you can go with, but I like to create a class for it, and exporting a new instance to do what I want anywhere in my code. Here is a (really) simple example of what my minimal go-to looks like:
mongo-client.js:
const { MongoClient } = require('mongodb');
class MongoCli {
constructor() {
let url = `mongodb://testuser:my_sup3r_passwOrd#127.0.0.1:27017/?authSource=my_database_name`;
this.client = new MongoClient(url, { useUnifiedTopology: true });
}
async init() {
if (this.client) {
await this.client.connect();
this.db = this.client.db('test');
} else
console.warn("Client is not initialized properly");
}
}
module.exports = new MongoCli();
Actual implementation
Of course, this code on his own won't work and we need to call and wait for it, before defining routes. So, right before app.route("/:list_name")............, call this: await MongoCli.init();.
Here is what my (again, really) simple server.js look like (I have separated the mongo-client code from the server):
const express = require('express');
const MongoCli = require('./mongo-cli.js');
const server = async () => {
const app = express();
await MongoCli.init();
app.route("/:list_name").get(async (req, res) => {
});
return app;
};
module.exports = server;
Now, let's start implementing what you really want from the beginning, a.k.a once a user click on a topic of tasks, it will display all the tasks on the topic he clicked:
const express = require('express');
const MongoCli = require('./mongo-cli.js');
const server = async () => {
const app = express();
await MongoCli.init();
app.route("/:list_name").get(async (req, res) => {
// we will query the collection specified by req.params.list_name
// then, .find({}) indicates we want all the results (empty filter)
// finally, we call .toArray() to transform a Cursor to a human-readable array
const tasks = await MongoCli.db.collection(req.params.list_name).find({}).toArray();
// making sure we got what we needed, you can remove the line below
console.log(tasks);
// return a HTTP 200 status code, along with the results we just queried
res.status(200).json(tasks);
});
return app;
};
module.exports = server;
Quite simple, right?
Keep in mind my server.js might not look quite as yours since there are many ways to handle this and it is to the developer to find his own preferred method, but you get the idea.
Code conception
We got our GET route going, we get the results when we call the route, everything's great! ... not quite.
What happens now if we have, say, 1500 topics of tasks? Should we really create 1500 different collections, knowing that a task consist of a description, a status, a deadline, eventually a name? Sure, we can do it, but it doesn't mean we have to.
Instead, what about creating one and only collection tasks, and adding a key topic to it?
Considering the above sentences, here's what the route would now look like:
const express = require('express');
const MongoCli = require('./mongo-cli.js');
const server = async () => {
const app = express();
await MongoCli.init();
app.route("/:topic_wanted").get(async (req, res) => {
// we now know the collection is named 'tasks'
// then, .find({topic: req.params.topic_wanted}) indicates we want all the results where the key 'topic' corresponds to req.params.topic_wanted
// finally, we call .toArray() to transform a Cursor to a human-readable array
const tasks = await MongoCli.db.collection('tasks').find({topic: req.params.topic_wanted}).toArray();
// making sure we got what we needed
console.log(tasks);
// return a HTTP 200 OK, along with the results we just queried
res.status(200).json(tasks);
});
return app;
};
module.exports = server;
Last words
I hope I'm not too off-topic and my answer could help you.
Also, I saw while writing the answer that you need to figure out how to post tasks now. Please let me know in the comments if you need further information/explanation or even help for posting tasks.
EDIT (added):
Save/Edit implementations
Seeing your implementation of creating a new task, I assume you already use mongoose. Unfortunately, when declaring a model in Mongoose, it will automatically search for (or create if it doesn't exist) the collection having the same name of your declared model, except in lowercase and pluralized (see here for more info). Meaning you can't declare a new task and assign it to a collection named "users" for example.
That's where the part 4 of this answer, "Code conception", comes into play. Otherwise, the code you edited-in has no "major" flaw.
Try this, this should work.
Changes that I made :-
MongoDb connect callback function changed to async.
Add toArray() function in the end of database.collection(req.params.list_name).find({});
And made the above function to await.
You can choose .then or async/await, it is up to you!
app.route("/:list_name").get((req, res) => {
MongoClient.connect(process.env.DB_CONNECT,async (err, db) => {
if(err) throw err;
var database = db.db("myFirstDatabase");
const todo_tasks = await database.collection(req.params.list_name).find({}).toArray(); /* add '.toArray()' */
database.listCollections().toArray((err, collections) => {
if(err) throw err;
db.close();
collections.forEach(element => {
if(element.name == req.params.list_name){
current_list = element;
current_list_name = element.name;
}
});
res.render("todo_list.ejs", { tasks: todo_tasks, lists: collections, curr_list: current_list_name });
});
});
});
After some improvements :-
app.route("/:list_name").get((req, res) => {
// Connecting to MongoDb database
MongoClient.connect(process.env.DB_CONNECT, async (err, db) => {
if (err) throw err;
// Choosing 'myFirstDatabase' database
const database = db.db("myFirstDatabase");
let todo_tasks = [];
let collections = [];
let current_list_name = "";
// Getting selected list items(todo tasks) to array
try {
todo_tasks = await database.collection(req.params.list_name).find({}).toArray(); // Change :- Add '.toArray()'
} catch (err) {
if (err) throw err;
}
// Getting collections names
try {
collections = await database.listCollections().toArray();
db.close();
} catch (err) {
if (err) throw err;
}
// Getting selected list details
collections.forEach(element => {
if (element.name === req.params.list_name) {
current_list = element; // I don't understand what this code here
current_list_name = element.name;
}
});
// Rendering front end
res.render("todo_list.ejs", {
tasks: todo_tasks,
lists: collections,
curr_list: current_list_name,
});
});
});
To sort the documents inside Ads Collection I am using the below query which takes parameters from the URL and its working perfectly.
router.get("/", auth, async (req, res) => {
let query;
let queryStr = JSON.stringify(req.query);
queryStr = queryStr.replace(
/\b(gt|gte|lt|lte|in)\b/g,
(match) => `$${match}`
);
console.log(queryStr);
query = Ads.find(JSON.parse(queryStr));
const ads = await query;
res.status(200).json({ data: ads });
});
I am using the text operator in the Ads Collection for searching with the below route .
router.get("/find/:query", (req, res) => {
let query = req.params.query;
Ads.find(
{
$text: { $search: query },
},
function (err, result) {
if (err) throw err;
if (result) {
res.json(result);
} else {
res.send(
JSON.stringify({
error: "Error",
})
);
}
}
);
});
Both the routes are working perfectly but How can I merge the above two in one?
For e.g, I want to do a text search on the first route after getting a response, and similarly for the second route, after getting a response I want to apply the query parameters and get a response .
How can I merge the above two to get the desired output?
From what I can infer, are you looking for a pipeline where the parallel running of the two is possible:
Please have a look at MongoDB Aggregate which works as a pipeline. Here you can have two pipelines for the same input and have different output, or output of one can be transferred to next level to process.
Mongodb Aggregate - Facet Command Link
[1]:https://docs.mongodb.com/manual/reference/operator/aggregation/facet/#pipe._S_facet
Mongodb Aggregate links
[2]: https://docs.mongodb.com/manual/reference/operator/query/
I am trying to get results from my "Books.find" and push it into my books array. I want to then res.send it.
I suspect this has something to do with some kind of asynchronous and scope rubbish.
What's the solution?
This is currently my code.
exports.timeline = function(req, res) {
var Followers = mongoose.model('Follow');
Followers.find({'follower': req.user.username}, function(err, followerResult) {
var name = [req.user.username];
var books = [];
function addName(username) {
name.push(username);
}
for(var user in followerResult) {
addName(followerResult[user]['followed']);
}
function getDataByUsername(username) {
function addBookArray(result) {
books.push(result);
return result;
}
var Books = mongoose.model('Book');
Books.find({'username': username}).exec(function (err, result) {
addBookArray(result);
});
}
for(var usernames in name) {
getDataByUsername(name[usernames]);
}
console.log(books);
res.send(books);
});
}
You're right, the problem is that Find is asynchronous and you send your response before you receive the result.
To deal with this kind of issues, you have several choices:
The powerful package Async to organize your async loops
The aggregation of MongoDB to let your DB join your data
Node Js is asynchronous . Your code is never wait for your query result.
Following are some options you can try :
use Async npm package
use Promise
Promise example :-
Promise.all([qry1, qry2]).then(res=>{ console.log(res) }).catch(err=>{console.log(err);})
here qry1 and qry2 are your mongo query
If you don't want to work with the async package, you can try to work with the $in functionality of mongoose and mongodb. As in, you get the list of users, and then find the list of books whoose userid is inside the user list.
Something along the line of :
exports.timeline = function(req, res) {
var Followers = mongoose.model('Follow');
Followers.find({'follower': req.user.username}, function(err, followerResult) {
var name = [req.user.username];
var books = [];
function addName(username) {
name.push(username);
}
for(var user in followerResult) {
addName(followerResult[user]['followed']);
}
Books.find({"username" : { $in: name }}).exec(function (err, books) {
console.log(books);
res.send(books);
});
});
}
Hope this helps.
I am trying to execute an asynchronous database lookup where each item in an array is looked up and its value added to a sum total.
Right now I have the following code which works as long as the main function is located in the same file as the database lookups:
// async is the async.js library
// Products is a mongoose Schema
var user = {
cart: [11, 22, 33],
orderCost = 0;
}
function getTotal (user, callback) {
async.each(user.cart, findProduct, function (err) {
if (err) {
throw err;
}
callback(user);
});
}
function findProduct (skunumber, callback) {
Products.findOne({sku: skunumber}, function (err, product) {
user.orderCost += product.toObject().currentPrice;
callback();
});
}
function main () {
getTotal(user);
}
main();
However, I would like the database functions, in this case Products.findOne to be located in a different file. When this is done, te findProduct function will no longer have access to the user object which means the user.orderCost += will fail.
Is there an agreed way to avoid this problem? Does the async library have a way to account for this or is there a way in Node directly?
1) you are querying db multiple times to get results you can get in one query like this
Products.find({sku: {$in: cart}}, function (err, products) {
// use for loop to get total cost here
});
2) now change your code to use the above, you can place this function anywhere you want as long as it has access to Products variable
function getTotal(cart,callback) {
Products.find({sku: {$in: cart}, function (err, products) {
// get total
callback (total);
});
}
3) use it
getTotal(user.cart, function(total){
console.log(total);
});
Update according to comments
To only get same number of products from db (since there are multiple products with the same sku) as there's in the cart use code below
Products
.find({sku: {$in: cart}})
.limit(cart.length)
.exec( function (err, products) {
// use for loop to get total cost here
});
given the async nature of mongoose (or sequelize, or redis) queries, what do you do when you have multiple queries you need to make before rendering the view?
For instance, you have a user_id in a session, and want to retrieve some info about that particular user via findOne. But you also want to display a list of recently logged in users.
exports.index = function (req, res) {
var current_user = null
Player.find({last_logged_in : today()}).exec(function(err, players) {
if (err) return res.render('500');
if (req.session.user_id) {
Player.findOne({_id : req.session.user_id}).exec(function(err, player) {
if (err) return;
if (player) {
current_user = player
}
})
}
// here, current_user isn't populated until the callback fires
res.render('game/index', { title: 'Battle!',
players: players,
game_is_full: (players.length >= 6),
current_user: current_user
});
});
};
So res.render is in the first query callback, fine. But what about waiting on the response from findOne to see if we know this user? It is only called conditionally, so I can't put render inside the inner callback, unless I duplicate it for either condition. Not pretty.
I can think of some workarounds -
make it really async and use AJAX on the client side to get the current user's profile. But this seems like more work than it's worth.
use Q and promises to wait on the resolution of the findOne query before rendering. But in a way, this would be like forcing blocking to make the response wait on my operation. Doesn't seem right.
use a middleware function to get the current user info. This seems cleaner, makes the query reusable. However I'm not sure how to go about it or if it would still manifest the same problem.
Of course, in a more extreme case, if you have a dozen queries to make, things might get ugly. So, what is the usual pattern given this type of requirement?
Yep, this is a particularly annoying case in async code. What you can do is to put the code you'd have to duplicate into a local function to keep it DRY:
exports.index = function (req, res) {
var current_user = null
Player.find({last_logged_in : today()}).exec(function(err, players) {
if (err) return res.render('500');
function render() {
res.render('game/index', { title: 'Battle!',
players: players,
game_is_full: (players.length >= 6),
current_user: current_user
});
}
if (req.session.user_id) {
Player.findOne({_id : req.session.user_id}).exec(function(err, player) {
if (err) return;
if (player) {
current_user = player
}
render();
})
} else {
render();
}
});
};
However, looking at what you're doing here, you'll probably need to look up the current player information in multiple request handlers, so in that case you're better off using middleware.
Something like:
exports.loadUser = function (req, res, next) {
if (req.session.user_id) {
Player.findOne({_id : req.session.user_id}).exec(function(err, player) {
if (err) return;
if (player) {
req.player = player
}
next();
})
} else {
next();
}
}
Then you'd configure your routes to call loadUser wherever you need req.player populated and the route handler can just pull the player details right from there.
router.get("/",function(req,res){
var locals = {};
var userId = req.params.userId;
async.parallel([
//Load user Data
function(callback) {
mongoOp.User.find({},function(err,user){
if (err) return callback(err);
locals.user = user;
callback();
});
},
//Load posts Data
function(callback) {
mongoOp.Post.find({},function(err,posts){
if (err) return callback(err);
locals.posts = posts;
callback();
});
}
], function(err) { //This function gets called after the two tasks have called their "task callbacks"
if (err) return next(err); //If an error occurred, we let express handle it by calling the `next` function
//Here `locals` will be an object with `user` and `posts` keys
//Example: `locals = {user: ..., posts: [...]}`
res.render('index.ejs', {userdata: locals.user,postdata: locals.posts})
});
Nowadays you can use app.param in ExpressJS to easily establish middleware that loads needed data based on the name of parameters in the request URL.
http://expressjs.com/4x/api.html#app.param