I'm learning node.js currently and this is my first ever project in it. It's (supposedly) a simple to-do list app where there are multiple lists I can load/edit/save/remove.
In the todo_list.ejs file I have a div where I list all the collection names:
<div id="list_container" class="lists">
<ul id="col_list" class="collection-list">
<% lists.forEach(list => { %>
<li class="collection-list-item">
<div class="list-name-container">
<a href="/<%=list.name %>" class="list-link">
<span class="list-name" name="list_name"><%=list.name %></span>
</a>
</div>
</li>
<% }) %>
</ul>
</div>
looks like this:
When I click on the link of a list. I try to use the following code to load in a new list (which is a mongodb collection):
app.route("/:list_name").get((req, res) => {
MongoClient.connect(process.env.DB_CONNECT, (err, db) => {
if(err) throw err;
var database = db.db("myFirstDatabase");
const cursor = database.collection(req.params.list_name).find({}); /* stuck here */
database.listCollections().toArray((err, collections) => {
if(err) throw err;
db.close();
collections.forEach(element => {
if(element.name == req.params.list_name){
current_list = element;
current_list_name = element.name;
}
});
task.find({}, (err, todo_tasks) => { /*currently using the model.find() method to list all the documents which always looks at the "tasks" collection*/
res.render("todo_list.ejs", { tasks: todo_tasks, lists: collections, curr_list: current_list_name });
});
});
});
});
I commented where I'm stuck in the code above. I'm trying to get a mongodb collection by name, and then load all its contents onto a list after, but I don't know how to find a collection by name. Reading through the node.js documentation lead me to the cursor object, which has a ton of info and properties I have no clue what to do with...
Is there a simple way to find a collection by name and get a list of it's documents?
EDIT 1:
this is where i add tasks:
//ADD TASK TO LIST
app.post('/', async (req, res) => {
const tsk = new task({ /*the mongodb model for tasks*/
content: req.body.content,
deadline: req.body.deadline
});
try {
await tsk.save();
res.redirect("/");
} catch(e) {
console.log(e);
res.redirect("/");
}
});
I will not address the EJS part in this answer as I'm not qualified and the code you provided seems all good. However, I'll review the back-end part.
Also, since I don't know what kind of coding background you have (if any), this answer will contain a lot of explanation on perhaps simple concepts.
Summary
From your second code snippet, there are a couple things that are to be discussed:
Asynchronous code
The database connection and generalities
Actual implementation
Code conception
[EDIT]: Save/Edit implementations
There is also a lot more to cover depending on the knowledge of OP, such as try/catch clauses, MongoDB models validation, the usage of express's Router and more but I will only edit my answer if needed.
Asynchronous code
For the rest of the answer, most of the code will be surrounded by async/await keywords. These are necessary for the code to work properly.
Basically, JS being a language that is made for the web, you sometimes need to wait for network or database requests to be done before you do any other action. That's where the callbacks, the promises or the async/await syntax (which is syntactic sugar for promises) come in handy.
Let's say you need, like your example, to retrieve a list of tasks:
app.route("/:list_name").get((req, res) => {
MongoClient.connect(process.env.DB_CONNECT, (err, db) => {
if(err) throw err;
var database = db.db("myFirstDatabase");
const cursor = database.collection(req.params.list_name).find({}); /* stuck here */
console.log(cursor);
// ..........
});
});
JS being asynchronous by default, if you run this code, chances are high that cursor will be undefined. The reason for that is that the code doesn't wait for the database.collection(............. to finish in order to continue the execution. But with the help of the aforementioned callback/promises/async-await, our code can now wait until this instruction is done.
You can read on async/await here and here, and see here that MongoDB examples are using async/await as well, but you will see in the following sections more "practical" usages of it.
Keep in mind that what you are using (whether it is callbacks, promises or async/await syntax) is completely up to you and your preferences.
Database connection
As the code is currently written, everytime a user clicks on any item on your list, a connection to MongoDB will be established, and that connection doesn't belong to the route handler. Your back-end app should connect to the database once (at least for this case, it could prove useful to initiate multiple connections for some advanced cases), and close the connection when your back-end app stops (generally not the case with an API).
Atlas cloud databases for example, have a limit of 500 connections. Meaning that if, let's say, 501 users click on an item simultaneously on your front-end list, the best case scenario is someone doesn't get what he asked, but it could be worse.
For this matter, you have several options. One would be to go with a framework that helps you leverage some of the code and boilerplate, such as Mongoose or work with the native MongoDB driver which we will do, since you seem to already work with that and I strongly believe working with the lowest layer first will make you learn higher-level frameworks way faster.
Now, let's tackle the problem. We want to put the database connection somewhere else where it'll be called once. Again, there's several options you can go with, but I like to create a class for it, and exporting a new instance to do what I want anywhere in my code. Here is a (really) simple example of what my minimal go-to looks like:
mongo-client.js:
const { MongoClient } = require('mongodb');
class MongoCli {
constructor() {
let url = `mongodb://testuser:my_sup3r_passwOrd#127.0.0.1:27017/?authSource=my_database_name`;
this.client = new MongoClient(url, { useUnifiedTopology: true });
}
async init() {
if (this.client) {
await this.client.connect();
this.db = this.client.db('test');
} else
console.warn("Client is not initialized properly");
}
}
module.exports = new MongoCli();
Actual implementation
Of course, this code on his own won't work and we need to call and wait for it, before defining routes. So, right before app.route("/:list_name")............, call this: await MongoCli.init();.
Here is what my (again, really) simple server.js look like (I have separated the mongo-client code from the server):
const express = require('express');
const MongoCli = require('./mongo-cli.js');
const server = async () => {
const app = express();
await MongoCli.init();
app.route("/:list_name").get(async (req, res) => {
});
return app;
};
module.exports = server;
Now, let's start implementing what you really want from the beginning, a.k.a once a user click on a topic of tasks, it will display all the tasks on the topic he clicked:
const express = require('express');
const MongoCli = require('./mongo-cli.js');
const server = async () => {
const app = express();
await MongoCli.init();
app.route("/:list_name").get(async (req, res) => {
// we will query the collection specified by req.params.list_name
// then, .find({}) indicates we want all the results (empty filter)
// finally, we call .toArray() to transform a Cursor to a human-readable array
const tasks = await MongoCli.db.collection(req.params.list_name).find({}).toArray();
// making sure we got what we needed, you can remove the line below
console.log(tasks);
// return a HTTP 200 status code, along with the results we just queried
res.status(200).json(tasks);
});
return app;
};
module.exports = server;
Quite simple, right?
Keep in mind my server.js might not look quite as yours since there are many ways to handle this and it is to the developer to find his own preferred method, but you get the idea.
Code conception
We got our GET route going, we get the results when we call the route, everything's great! ... not quite.
What happens now if we have, say, 1500 topics of tasks? Should we really create 1500 different collections, knowing that a task consist of a description, a status, a deadline, eventually a name? Sure, we can do it, but it doesn't mean we have to.
Instead, what about creating one and only collection tasks, and adding a key topic to it?
Considering the above sentences, here's what the route would now look like:
const express = require('express');
const MongoCli = require('./mongo-cli.js');
const server = async () => {
const app = express();
await MongoCli.init();
app.route("/:topic_wanted").get(async (req, res) => {
// we now know the collection is named 'tasks'
// then, .find({topic: req.params.topic_wanted}) indicates we want all the results where the key 'topic' corresponds to req.params.topic_wanted
// finally, we call .toArray() to transform a Cursor to a human-readable array
const tasks = await MongoCli.db.collection('tasks').find({topic: req.params.topic_wanted}).toArray();
// making sure we got what we needed
console.log(tasks);
// return a HTTP 200 OK, along with the results we just queried
res.status(200).json(tasks);
});
return app;
};
module.exports = server;
Last words
I hope I'm not too off-topic and my answer could help you.
Also, I saw while writing the answer that you need to figure out how to post tasks now. Please let me know in the comments if you need further information/explanation or even help for posting tasks.
EDIT (added):
Save/Edit implementations
Seeing your implementation of creating a new task, I assume you already use mongoose. Unfortunately, when declaring a model in Mongoose, it will automatically search for (or create if it doesn't exist) the collection having the same name of your declared model, except in lowercase and pluralized (see here for more info). Meaning you can't declare a new task and assign it to a collection named "users" for example.
That's where the part 4 of this answer, "Code conception", comes into play. Otherwise, the code you edited-in has no "major" flaw.
Try this, this should work.
Changes that I made :-
MongoDb connect callback function changed to async.
Add toArray() function in the end of database.collection(req.params.list_name).find({});
And made the above function to await.
You can choose .then or async/await, it is up to you!
app.route("/:list_name").get((req, res) => {
MongoClient.connect(process.env.DB_CONNECT,async (err, db) => {
if(err) throw err;
var database = db.db("myFirstDatabase");
const todo_tasks = await database.collection(req.params.list_name).find({}).toArray(); /* add '.toArray()' */
database.listCollections().toArray((err, collections) => {
if(err) throw err;
db.close();
collections.forEach(element => {
if(element.name == req.params.list_name){
current_list = element;
current_list_name = element.name;
}
});
res.render("todo_list.ejs", { tasks: todo_tasks, lists: collections, curr_list: current_list_name });
});
});
});
After some improvements :-
app.route("/:list_name").get((req, res) => {
// Connecting to MongoDb database
MongoClient.connect(process.env.DB_CONNECT, async (err, db) => {
if (err) throw err;
// Choosing 'myFirstDatabase' database
const database = db.db("myFirstDatabase");
let todo_tasks = [];
let collections = [];
let current_list_name = "";
// Getting selected list items(todo tasks) to array
try {
todo_tasks = await database.collection(req.params.list_name).find({}).toArray(); // Change :- Add '.toArray()'
} catch (err) {
if (err) throw err;
}
// Getting collections names
try {
collections = await database.listCollections().toArray();
db.close();
} catch (err) {
if (err) throw err;
}
// Getting selected list details
collections.forEach(element => {
if (element.name === req.params.list_name) {
current_list = element; // I don't understand what this code here
current_list_name = element.name;
}
});
// Rendering front end
res.render("todo_list.ejs", {
tasks: todo_tasks,
lists: collections,
curr_list: current_list_name,
});
});
});
this my code, which works fine while uploading image but whenever I am going to upload video/files which is more than 16mb , its failed to upload , so what to do , please help , i am a beginner !
I am using formidable as a middleware , lodash and fs , But don't know to do next
const Freecoursevideo = require("../models/freecoursevideo");
const formidable = require("formidable");
const _ = require("lodash");
const fs = require("fs");
exports.createFreecoursevideo = (req, res) => {
let form = new formidable.IncomingForm();
form.keepExtensions = true;
form.parse(req, (err, fields, file) => {
if (err) {
return res.status(400).json({
error: "problem with image",
});
}
//destructure the fields
const { name, description } = fields;
if (!name || !description) {
return res.status(400).json({
error: "Please include all fields",
});
}
let freecoursevideo = new Freecoursevideo(fields);
console.log(freecoursevideo);
//handle file here
if (file.video) {
if (file.video.size > 1000000000000000000000) {
return res.status(400).json({
error: "File size too big!",
});
}
freecoursevideo.video.data = fs.readFileSync(file.video.path);
freecoursevideo.video.contentType = file.video.type;
}
// console.log(freecoursevideo);
//save to the DB
freecoursevideo.save((err, freecoursevideo) => {
if (err) {
res.status(400).json({
error: "Saving Freecoursevideo in DB failed",
});
}
res.json(freecoursevideo);
});
});
};
Mongodb has a limit of 16MB on the size of document which you can save to a collection.
Since you are trying to save videos of size greater than 16mb to your freecoursevideo collection, you are getting an error.
Solution:-
You will have to Gridfs to save documents of size greater than 16mb.
I your case i would suggest you get rid of freecoursevideo collection and save all your videos in gridfs whether it is more than 16mb or not. So that you don't have to query two places for getting your videos.
So how to save to gridfs?
To start with just try to put some videos using mongoshell. Just navigate to the bin folder of your mongo database tools.
Ensure that the bin folder has following content.
Open command prompt at this location. Run the below command by replacing , ,
<DB_NAME> and <PATH_TO_VIDEO>
mongofiles --host=<HOST> --port=<PORT> -d=<DB_NAME> put <PATH_TO_VIDEO>
Above command will save the video in fs.files and fs.chunks collections. Name of these collections are predefined by mongo and cannot be changed.
Above exercise will you the understanding of how gridfs works. Then you can go ahead look for mongoose way of doing this. Adding a link to article that may help.
https://www.freecodecamp.org/news/gridfs-making-file-uploading-to-mongodb/
I need to verify that a mongodb document exists before continuing with Azure file upload.
The form consists of a file and a text field, the logic that is needed is the following:
Form submission
Get the text field
Search in mongodb for a document with the text field data
If the item exist continue with file upload to Azure else return
File upload to Azure
Save the URL to the file in the same MongoDB document that was found in
The problem I'm facing is that I can't touch field data in form.on('part') and can't get it to work with using form.parse first.
This is my code, I'm willing to change libraries and do whatever it takes to get it working.
var form = new multiparty.Form();
var formField = "";
form.parse(req, function(err, fields, files) {
formField = fields.fieldinform[0];
});
console.log(formField); //empty async?
model
.findOne({ name: formField })
.then(obj => {
form.on("part", function(part) {
if (!part.filename) return;
var size = part.byteCount;
var name = part.filename;
var container = "test";
blobService.createBlockBlobFromStream(
container,
name,
part,
size,
function(error) {
if (error) {
console.log("failed");
}
}
);
});
})
.catch(e => {
//do not continue
});
Help would be highly appreciated!
After a lot of searching and not coming up with a proper answer I decided to go with jquery that changed the action URL of the form to /upload/textintextfield before submission and then grab that with req.params.textfield in node.
<script>
$('#fileUploadForm').submit(function() {
$('#fileUploadForm').attr('action', '/upload/addvideo/' + $('#textfield').val())
return true;
});
</script>
This may be a basic question but I've looked through the github Cloudant library and the Cloudant documentation and deleting a specific document from the database is mentioned but never thoroughly explained. It's very frustrating. The closest I've gotten to deleting a document is using an http request rather then the functions Cloudant library offers and I continuously get a "Document Update Conflict" even though I'm passing through the _rev of the document. Can anybody explain deleting a document from a Cloudant database using nodejs with an example to help sort this out. Thanks.
You can use the destroy method of nano like #JakePeyser has said, instead of using http APIs since you are using nodejs.
But since you are sending _rev and getting a "Document Update Conflict" error, it leads me to doubt if you have the latest _rev with you. "Document Update Conflict" happens mostly if the local _rev doesn't match the remote _rev. I would therefore suggest wrapping your destroy function in a get function. So an update to #JakePeyser's example would be:
var nano = require("nano")("cloudantURL"),
db = nano.db.use("yourDB");
db.get(docUniqueId, function(err, body) {
if (!err) {
var latestRev = body._rev;
db.destroy(docUniqueId, latestRev, function(err, body, header) {
if (!err) {
console.log("Successfully deleted doc", docUniqueId);
}
});
}
})
It depends what node module you are using for communicating with Cloudant. With the nano driver, you can use the destroy method to delete a document. See the following code example:
var nano = require("nano")("cloudantURL"),
db = nano.db.use("yourDB");
db.destroy(docUniqueId, docRevNum, function(err, body, header) {
if (!err) {
console.log("Successfully deleted doc", docUniqueId);
}
});
Key
cloudantURL - URL of your Cloudant instance, with username and password embedded
yourDB - your database name
docUniqueId - Unique ID of the doc you want to delete
docRevNum - Revision number of the doc you want to delete
Sample script to delete/destroy a doc from a collection "mytable" based on the value of the field "fkId".
var Cloudant = require('cloudant');
var Config = require('config-js');
var config = new Config('./settings.js');
var username = config.get('CLOUDANT_USER');
var password = config.get('CLOUDANT_PASWORD');
var cloudant = Cloudant({account:username, password:password});
var db = cloudant.db.use('mytable');
var key = 'fkId';
var value = '11234567890';
...
...
db.list({
'include_docs': true
}, function (err, body) {
/* istanbul ignore next */
if (err)
res.json({success: false, msg: 'Unable to fetch documents'});
else {
var rows = body.rows;
var items = [];
var rec_found = false;
rows.forEach(function (row) {
if (row.doc[key] === value) {
rec_found = true;
items.push(row.doc);
}
});
if (items.length === 0) {
console.log("No record found with fkId: "+ value);
res.json({success: true, msg: 'No record found with fkId: '+ value});
} else {
var docId = items[0]._id;
var docRev = items[0]._rev;
db.destroy(docId, docRev, function(err) {
if (!err) {
console.log("Successfully deleted doc with fkId: "+ value);
res.json({success: true, msg: 'Successfully deleted the item from the database.'});
} else {
res.json({success: false, msg: 'Failed to delete with fkId from the database, please try again.'});
}
});
}
}
});
I am using node-mongodb-native in my application. I send multiple POST requests to nodejs server to save/update each documents, but only one document is getting updated and all other document are not changing. The data received in the server is correct.
save : function(req,res) {
data = req.body;
if(!data._id){
data._id = new ObjectID();
}else{
data._id = ObjectID(data._id);
}
mColl(req.params.collname, function (collection,db) {
collection.save(data, {safe:true}, function(err, result) {
if(err){
res.send(err);
}
else {
res.send(result);
}
});
});
}
I am not getting the response for the request also.
For starters, don't do this:
data = req.body;
When a new request comes in, you're overwriting the (global!) data variable, and all kinds of undefined stuff can happen. So always declare a new variable:
var data = req.body;