The node mongodb docs specify to use next/each for large number of documents so as to not have everything loaded onto memory if we were to use toArray.
So, i thought my sample code should work as is. But it just returns one document.
What should be the correct way to deal with this problem?
This is my code sample :
var findAsync = function (collection,query) {
return mongodb.MongoClient.connectAsync(mongodbServerString)
.then(function (db) {
return [db.collection(collection).find(query), db];
});
};
findAsync("UserProfile",{})
.spread(function (cursor,db) {
return [cursor.project({Email:true}),db];
})
.spread(function (cursor, db) {
return cursor.eachAsync().then(function (doc) {
console.log(doc);
}).catch(function () {
db.close();
});
});
Promises represent singular values. Promises are basically like function returns, since a function cannot return multiple values - it wouldn't make sense to convert each to a promise returning function.
What you can do is either convert it to an Observable returning function and then use .forEach on that to get a promise back for the completion of the sequence or you can implement something similar manually:
function each(cursor, fn) {
return new Promise((resolve, reject) => {
cursor.forEach((err, data) => {
if(err) {
cursor.close();
return reject(err);
}
try { fn(data); } catch(e) { cursor.close(); reject(e); }
}, err => { { // finished callback
if(err) reject(err);
else resolve();
});
});
}
Which would let you write:
each(cursor, doc => console.log(doc)).then(...).catch(...)
Also note that Mongo connections are persistent, you're supposed to connect once when the server starts and then keep the connection open for as long as the server is run.
Related
I am struggling with async operations. I am trying to simply get a value from firestore and storing it in a var.
I manage to receive the value, I can even save it in the var when I do that specifically (use the var within the get function) but I don't seem to manage the await properly when trying to save this in a flexible way:
async function getValues(collectionName, docName,) {
console.log("start")
var result;
var docRef = await db.collection(collectionName).doc(docName).get()
.then(//async// (tried this as well with async) function (doc) {
if (doc.exists) {
console.log("Document data:", doc.data());
result = doc.data().text;
console.log(result);
return //await// (this as well with async) result;
} else {
// doc.data() will be undefined in this case
console.log("No such document!");
result = "No such document!";
return result;
}
console.log("end");
}).catch (function (err) {
console.log('Error getting documents', err);
});
};
helpMessage = getValues('configuration','helpMessage');
Note: doc.data().text -> "text" is the name of the field where my value is stored in. Do I have to use .value here?
The result I get in the console is: info: Document data: { text: 'The correct text from the database' }
info: The correct text from the database
But using helpMessage in my code I get {}
Image from the Telegram bot where I am trying to use the helpMessage as a response to the '/help' command.
I have checked: getting value from cloud firestore,
Firebase Firestore get() async/await, get asynchronous value from firebase firestore reference and most importantly How do I return the response from an asynchronous call?. They either deal with multiple documents (using forEach), don't address the async nature of my problem or (last case), I simply fail to understand the nature of it.
Additionally, both nodejs and firestore seems to be developing rapidly and finding good, up-to-date documentation or examples is difficult. Any pointers are much appriciated.
You have things the wrong way around. It's much easier than you think it is.
function getValues(collectionName, docName) {
return db.collection(collectionName).doc(docName).get().then(function (doc) {
if (doc.exists) return doc.data().text;
return Promise.reject("No such document");
}};
}
If a function returns a promise (like db.collection(...).doc(...).get()), return that promise. This is the "outer" return above.
In the promise handler (inside the .then() callback), return a value to indicate success, or a rejected promise to indicate an error. This is the "inner" return above. Instead of returning a rejected promise, you can also throw an error if you want to.
Now you have a promise-returning function. You can use it with .then() and .catch():
getValues('configuration','helpMessage')
.then(function (text) { console.log(text); })
.catch(function (err) { console.log("ERROR:" err); });
or await it inside an async function in a try/catch block, if you like that better:
async function doSomething() {
try {
let text = await getValues('configuration','helpMessage');
console.log(text);
} catch {
console.log("ERROR:" err);
}
}
If you want to use async/await with your getValues() function, you can:
async function getValues(collectionName, docName) {
let doc = await db.collection(collectionName).doc(docName).get();
if (doc.exists) return doc.data().text;
throw new Error("No such document");
}
Since getValues function returns a promise, you need to await getValues function while calling it.
Change getValues like so -
function getValues(collectionName, docName,) {
console.log("start")
var result;
return db.collection(collectionName).doc(docName).get()
.then(function (doc) {
if (doc.exists) {
console.log("Document data:", doc.data());
result = doc.data().text;
console.log(result);
return result;
} else {
// doc.data() will be undefined in this case
console.log("No such document!");
result = "No such document!";
return result;
}
}).catch (function (err) {
console.log('Error getting documents', err);
});
};
Then use getValues like so -
helpMessage = await getValues('configuration','helpMessage');
Explanation -
async, await are just syntactic sugar for Promises. async functions return a promise (or AsyncFunction more accurately) which needs to be resolved to use its enclosed value.
See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
Finally managed to get it working. Thanks for the input Tomalak!
getValues(help.collectionName, help.docName)
.then((text) => {
console.log(text);
help.message = text;
})
.catch((err) => { console.log("Error: ", err); });
function getValues(collectionName, docName) {
return db.collection(collectionName).doc(docName).get().then((doc) => {
if (doc.exists) {
return doc.data().text;
}
else {
return Promise.reject("No such document");
}});
}
bot.help((ctx) => ctx.reply(help.message));
Unfortunately, I can not pin-point the exact reason this worked. Some little fixes (missed comma in the console.log) and formatting definitely helped me understanding the structure though. Hope someone else finds this useful, when starting to play around with node and firebase.
I have a fairly straightforward CRUD app which renders the results of two queries onto one page. The problem that arose once I got this to "work" was that the page required a refresh in order to display the results. On first load, no results were displayed.
I came to figure out that this is a problem/symptom of Node's asynchronous nature. I've been trying to approach this problem by using async/await, and from hours of messing with things, I feel like I'm quite close to the solution, but it's just not working out - I still need a manual refresh to display/render the results on the .ejs page.
The code:
var entries = [];
var frontPageGoals = [];
app.get('/entries', async (req,res) => {
if (req.session.password) {
const entriesColl = await
db.collection('entries')
.find()
.sort({date: -1})
.toArray((err, result) => {
if (err) { console.log(err) }
else {
for (i=0; i<result.length; i++) {
entries[i] = result[i];
}
}
});
const goalsColl = await
db.collection('goals')
.find()
.toArray((err, result) => {
if (err) {console.log(err)}
else {
for (i=0; i<result.length; i++) {
frontPageGoals[i] = result[i];
}
}
});
res.render('index.ejs', {entries: entries, frontPageGoals: frontPageGoals});
}
else {
res.redirect('/');
}
});
Now, I can conceive of a few problems here, but honestly I'm just at my wits end trying to figure this out. For example, I'm sure it's problematic that the empty lists which will contain the results to be passed when the page renders are outside the actual async function. But after trying to move them a dozen different places within the async area... still no dice.
Any help would be hugely appreciated! This is basically the last big "thing" I need done for this app.
I'm not 100% sure about your database driver, but assuming that the toArray() returns a promise (which it does in the default mongodb driver), the await will actually return the value you expect in your callback, result in your case, or in case there was an error, which you expected it as err in your callback, it will be thrown, thus forcing you to use try-catch blocks, in your case, you would just use console.log(err) in the catch block, since you aren't doing any handling
Here's your code after updating :
app.get("/entries", async (req, res) => {
if (req.session.password) {
try {
const entries = await db
.collection("entries")
.find()
.sort({ date: -1 })
.toArray();
const frontPageGoals = await db
.collection("goals")
.find()
.toArray();
res.render("index.ejs", {
entries: entries,
frontPageGoals: frontPageGoals
});
} catch (err) {
console.log(err);
}
} else {
res.redirect("/");
}
});
EDIT
However, if you don't know about promises -which async/await are basically promises-, and wanna just do it using callbacks -not advised-, you would have to just send your response in the callback, and nest the 2nd query in the first query's callback, here is the code,, with some comments to hopefully help you out:
app.get("/entries", (req, res) => {
if (req.session.password) {
// First query
db.collection("entries")
.find()
.sort({ date: -1 })
.toArray((err, entryResult) => {
if (err) {
console.log(err);
} else {
// In the callback of the first query, so it will
// execute 2nd query, only when the first one is done
db.collection("goals")
.find()
.toArray((err, frontPageResult) => {
if (err) {
console.log(err);
} else {
// In the callback of the 2nd query, send the response
// here since both data are at hand
res.render("index.ejs", {
entries: entryResult,
frontPageGoals: frontPageResult
});
}
});
}
});
} else {
res.redirect("/");
}
});
I have removed the async keyword since you no longer need it
I renamed the callback arguments, instead of just result, because both callbacks would have the same argument name, and you would have had to store it in a temp variable
hello guys i got confused why this not working
here is my connection js file
function getConnection(callback) {
initPool()
mysql_pool.getConnection((err, conn) => {
if (err) {
return callback(err);
}
return callback(err, conn);
});
}
function query(queryString, callback) {
getConnection((err, connection2) => {
if (connection2 != undefined) {
if (err) {
connection2.destroy();
return callback(createDataResponseObject(err, null))
}
connection2.query(queryString, function (err, rows) {
connection2.destroy();
if (!err) {
return callback(createDataResponseObject(err, rows))
}
else {
return callback(createDataResponseObject(err, null))
}
});
}
});
}
function createDataResponseObject(error, results) {
if (error) {
logger.info(error);
}
return {
error: error,
results: results === undefined ? null : results === null ? null : results
}
}
now when i acquire this connection js in my router.js below is sample code
var conn=require("./connection")
router.post('/test',async function(res,req){
var query ="select * from users"
let x= await conn.query(result);
console.log(x)
});
in connection js file i haven't use promise then why async not working in router i.e it should still work because i m using callback can anyone tell me what i m missing or what i m doing wrong. when i tried it with return promise in connection.js file it working. i know async return promise
You can only await a promise. conn.query doesn't return a promise. It has no return statement: It returns undefined.
You need to promisify your database functions.
Async/await only works with promises, it’s a syntactic sugar on top of it. I.e. you can’t use the syntax with callbacks.
Check out this link: https://javascript.info/async-await
Async only works with promises, to get things working you have to first convert all of your function to return promises. You can use Promise.promisify() to convert all callback style function to return a promise. Once it is done you can use these function with async-await.
I want to keep a log during my integration test suite. I'm testing that every 'item' is being compiled and logging how much time it took. I'm using node 4.3.
First of all I create the log file:
before(function() {
if (!fs.existsSync("./log.csv")) {
fs.writeFile("./log.csv", "Name; Time");
}
});
Then within each it block I would do this:
for (const item of items) {
it('compiles', function() {
return item.testCompile();
});
}
And item class has these methods:
testCompile() {
return this
.buildItem()
.then(result => {
// whatever testing stuff
});
}
buildItem() {
return this
.internalLogicForCompiling()
.then(result => {
// This is not appending anything
fs.appendFile("./log.csv", `${item.name}; ${result.compileTime}`);
return result;
});
}
So far the file is created but never updated... Any clue what I'm doing wrong?
PS: I assume if the file doesn't exists, fs should throw an error, however it doesn't.
Your code is generally ignoring the fact that your fs calls are asynchronous. Promises are not magic. If you use code that is asynchronous but does not use promises, you need to do more than plop that code in a promise can call it done.
The easiest way to deal with the issue would be to use fs.writeFileSync and fs.appendFileSync instead of the calls you make. Otherwise, you should write your before like this:
before(function(done) {
if (!fs.existsSync("./log.csv")) {
fs.writeFile("./log.csv", "Name; Time", done);
}
});
I've just added the done callback.
And buildItem could be something like this so that the promise it returns won't resolve before appendFile is done doing its work:
buildItem() {
return this
.internalLogicForCompiling()
.then(result => {
return new Promise((resolve, reject) => {
fs.appendFile("./log.csv", `${item.name}; ${result.compileTime}`, (err) => {
if (err) {
reject(err);
return;
}
resolve(result);
});
});
});
}
I am trying to get Promise chaining working for me correctly.
I believe the problem boils down to understanding the difference between:
promise.then(foo).then(bar);
and:
promise.then(foo.then(bar));
In this situation I am writing both foo and bar and am trying to get the signatures right. bar does take a return value that is produced by foo.
I have the latter working, but my question is what do I need to do to get the former working?
Related to the above is the full code (below). I don't have the different logs printed in the order I am expecting (expecting log1, log2, log3, log4, log5, but getting log3, log4, log5, log1, log2). I am hoping as I figure the above I will get this working right as well.
var Promise = require('bluebird');
function listPages(queryUrl) {
var promise = Promise.resolve();
promise = promise
.then(parseFeed(queryUrl)
.then(function (items) {
items.forEach(function (item) {
promise = promise.then(processItem(transform(item)))
.then(function() { console.log('log1');})
.then(function() { console.log('log2');});
});
}).then(function() {console.log('log3')})
).then(function() {console.log('log4')})
.catch(function (error) {
console.log('error: ', error, error.stack);
});
return promise.then(function() {console.log('log5');});
};
What is the difference between promise.then(foo).then(bar); and promise.then(foo.then(bar));?
The second one is simply wrong. The then method takes a callback as its argument, not a promise. That callback might return a promise, so the first one is equivalent to
promise.then(function(x) { return foo(x).then(bar) })
(assuming that foo returns a promise as well).
Your whole code appears to be messed up a bit. It should probably read
function listPages(queryUrl) {
return parseFeed(queryUrl)
.then(function (items) {
var promise = Promise.resolve();
items.forEach(function (item) {
promise = promise.then(function() {
console.log('log1');
return processItem(transform(item));
}).then(function() {
console.log('log2');
});
});
return promise;
}).then(function() {
console.log('log3')
}, function (error) {
console.log('error: ', error, error.stack);
});
}