In my app.js I have the following to try to retrieve data from a sqlite database and pass it to one of my views:
app.get("/dynamic", function(req, res) {
var db = new sqlite3.Database(mainDatabase)
var posts = []
db.serialize(function() {
db.each("SELECT * FROM blog_posts", function(err, row) {
posts.push({title: row.post_title, date: row.post_date, text: row.post_text})
})
})
res.render("dynamic", {title: "Dynamic", posts: posts})
})
Can someone tell me what I am doing wrong here. The posts array seems to stay empty nomatter what.
EDIT
I was following a tutorial that explained that though the plugin has async, this method is not asynchronous
Here is a quote from the tutorial
Despite the callbacks and asynchronous nature of Node.js, these
transactions will run in series, allowing us to create, insert, and
query knowing that the statement prior will run before the current one
does. However, sqlite3 provides a "parallel" wrapper with the same
interface, but runs all the transactions in parallel. It just all
depends on your current circumstances.
the db calls are likely asynchronous. Which means you are rendering before they return with their data.
You need to figure out how to get one callback from your query, and render your template in that callback.
It looks like you want a second complete callback passed to db.each() (Thanks, Jonathan Lonowski, for the tip!)
var posts = [];
db.serialize(function() {
db.each("SELECT * FROM blog_posts", function(err, row) {
posts.push({title: row.post_title, date: row.post_date, text: row.post_text})
}, function() {
// All done fetching records, render response
res.render("dynamic", {title: "Dynamic", posts: posts})
})
})
The idea is the render in the last callback of any asynchronous code, that way you have everything you need.
Related
My goal is to create an abstracted POST function for an express running on Node similar to Django's inbuilt REST methods. As a part of my abstracted POST function I'm checking the database (mongo) for valid foreign keys, duplicates, etc..., which is where we are in the process here. The function below is the one that actually makes the first calls to mongo to check that the incoming foreign keys actually exist in the tables/collections that they should exist in.
In short, the inbuilt response functionality inside the native .map function seems to be causing an early return from the called/subsidiary functions, and yet still continuing on inside the called/subsidiary functions after the early return happens.
Here is the code:
const db = require(`../../models`)
const findInDB_by_id = async params => {
console.log(`querying db`)
const records = await Promise.all(Object.keys(params).map(function(table){
return db[table].find({
_id: {
$in: params[table]._ids
}
})
}))
console.log('done querying the db')
// do stuff with records
return records
}
// call await findIndDB_by_id and do other stuff
// eventually return the response object
And here are the server logs
querying db
POST <route> <status code> 49.810 ms - 9 //<- and this is the appropriate response
done querying the db
... other stuff
When I the function is modified so that the map function doesn't return anything, (a) it doesn't query the database, and (b) doesn't return the express response object early. So by modifying this:
const records = await Promise.all(Object.keys(params).map(function(table){
return db[table].find({ // going to delete the `return` command here
_id: {
$in: params[table]._ids
}
})
}))
to this
const records = await Promise.all(Object.keys(params).map(function(table){
db[table].find({ // not returning this out of map
_id: {
$in: params[table]._ids
}
})
}))
the server logs change to:
querying db
done querying the db
... other stuff
POST <route> <status code> 49.810 ms - 9 // <-appropriate reponse
But then I'm not actually building my query, so the query response is empty. I'm experiencing this behavior with the anonymous map function being an arrow function of this format .map(table => (...)) as well.
Any ideas what's going on, why, or suggestions?
Your first version of your code is working as expected and the logs are as expected.
All async function return a promise. So findInDB_by_id() will always return a promise. In fact, they return a promise as soon as the first await inside the function is encountered as the calling code continues to run. The calling code itself needs to use await or .then() to get the resolved value from that promise.
Then, some time later, when you do a return someValue in that function, then someValue becomes the resolved value of that promise and that promise will resolve, allowing the calling code to be notified via await or .then() that the final result is now ready.
await only suspends execution of the async function that it is in. It does not cause the caller to be blocked at all. The caller still has to to deal with a promise returned from the async function.
The second version of your code just runs the db queries open loop with no control and no ability to collect results or process errors. This is often referred to as "fire and forget". The rest of your code continues to execute without regard for anything happening in those db queries. It's highly unlikely that the second version of code is correct. While there are a very few situations where "fire and forget" is appropriate, I will always want to at least log errors in such a situation, but since it appears you want results, this cannot be correct
You should try something like as when it encounter first async function it start executing rest of the code
let promiseArr = []
Object.keys(params).map(function(table){
promiseArr.push(db[table].find({
_id: {
$in: params[table]._ids
}
}))
})
let [records] = await Promise.all(promiseArr)
and if you still want to use map approach
await Promise.all(Object.keys(params).map(async function(table){
return await db[table].find({
_id: {
$in: params[table]._ids
}
})
})
)
I'm trying to do some integration between two systems, and I've got some problems when it comes to synchronizing data.
I am using nodejs and the database is mongodb or firebase.
The scenario is described as below:
systemA with dbA
systemB with dbB
Do the following:
systemA sends a request (POST or PUT whatever) to systemB, the body is like this:
{
.....
fieldA1: valueA1,
fieldA2: valueA2
.....
}
then systemB needs to update several fields(fieldB1, fieldB2) in dbB according to the systemA data, like this:
fieldB1: valueA1
fieldB2: valueA2
after BOTH fieldB1 and fieldB2 are updated successfully, then execute more logic
I'm using async to control asynchronous process. My code to implement these 3 steps:
async.waterfall([
function (callback) {
//get valueA1 of fieldA1 and valueA2 of fieldA2
},
async.parallel([
function (callback) {
//set valueA1 to fieldB1
},
function (callback) {
//set valueA2 to fieldB2
}
], function (err, result) {
//more logic here
})
], function (err, result) {
//more logic here
})
Since fieldB1 and fieldB2 should be updated at the same time, either situation when failing to update fieldB1 or fieldB2 will lead to data inconsistency, which is not the correct result.
However, async.parallel cannot guarantee that any update failure will rollback or prevent the others' update right ? Is there any way to idealy keep the data consistency of BOTH when updating fieldB1 and fieldB2?
I find it difficult to map your code to the Firebase API. But what you're describing sounds like its achievable by either using transactions or multi-location updates.
I covered these type of updates in-depth in the past in: How to write denormalized data in Firebase
router.post("/application_action", function(req,res){
var Employee = req.body.Employee;
var conn = new jsforce.Connection({
oauth2 : salesforce_credential.oauth2
});
var username = salesforce_credential.username;
var password = salesforce_credential.password;
conn.login(username, password, function(err, userInfo, next) {
if (err) { return console.error(err); res.json(false);}
// I want this conn.query to execute first and then conn.sobject
conn.query("SELECT id FROM SFDC_Employee__c WHERE Auth0_Id__c = '" + req.user.id + "'" , function(err, result) {
if (err) { return console.error(err); }
Employee["Id"] = result.records[0].Id;
});
//I want this to execute after the execution of above query i.e. conn.query
conn.sobject("SFDC_Emp__c").update(Employee, function(err, ret) {
if (err || !ret.success) { return console.error(err, ret);}
console.log('Updated Successfully : ' + ret.id);
});
});
I have provided my code above. I need to modify Employee in the conn.query and use it in conn.sobject. I need to make sure that my first query executes before 2nd because I am getting value from 1st and using in the 2nd. Please do let me know if you know how to accomplish this.
New Answer Based on Edit to Question
To execute one query based on the results of the other, you put the second query inside the completion callback of the first like this:
router.post("/application_action", function (req, res) {
var Employee = req.body.Employee;
var conn = new jsforce.Connection({
oauth2: salesforce_credential.oauth2
});
var username = salesforce_credential.username;
var password = salesforce_credential.password;
conn.login(username, password, function (err, userInfo, next) {
if (err) {
return console.error(err);
res.json(false);
}
// I want this conn.query to execute first and then conn.sobject
conn.query("SELECT id FROM SFDC_Employee__c WHERE Auth0_Id__c = '" + req.user.id + "'", function (err, result) {
if (err) {
return console.error(err);
}
Employee["Id"] = result.records[0].Id;
//I want this to execute after the execution of above query i.e. conn.query
conn.sobject("SFDC_Emp__c").update(Employee, function (err, ret) {
if (err || !ret.success) {
return console.error(err, ret);
}
console.log('Updated Successfully : ' + ret.id);
});
});
});
});
The only place that the first query results are valid is inside that callback because otherwise, you have no way of knowing when those asynchronous results are actually available and valid.
Please note that your error handling is unfinished since you don't finish the response in any of the error conditions and even in the success case, you have not yet actually sent a response to finish the request.
Original Answer
First off, your code shows a route handler, not middleware. So, if you really intend to ask about middleware, you will have to show your actual middleware. Middleware that does not end the request needs to declare next as an argument and then call it when it is done with it's processing. That's how processing continues after the middleware.
Secondly, your console.log() statements are all going to show undefined because they execute BEFORE the conn.query() callback that contains the code that sets those variables.
conn.query() is an asynchronous operation. It calls its callback sometime IN THE FUTURE. Meanwhile, your console.log() statements execute immediately.
You can see the results of the console.log() by putting the statements inside the conn.query() callback, but that is probably only part of your problem. If you explain what you're really trying to accomplish, then we could probably help with a complete solution. Right now, you're just asking questions about flawed code, but not explaining the higher level problem you're trying to solve so you're making it hard for us to give you the best answer to your actual problem.
FYI:
app.locals - properties scoped to your app, available to all request handlers.
res.locals - properties scoped to a specific request, available only to middleware or request handlers involved in processing this specific request/response.
req.locals - I can't find any documentation on this in Express or HTTP module. There is discussion of this as basically serving the same purpose as res.locals, though it is not documented.
Other relevants answers:
req.locals vs. res.locals vs. res.data vs. req.data vs. app.locals in Express middleware
Express.js: app.locals vs req.locals vs req.session
You miss the basics of the asynchronous flow in javascript. All the callbacks are set to the end of event loop, so the callback of the conn.query will be executed after console.logs from the outside. Here is a good article where the the basic concepts of asynchronous programming in JavaScript are explained.
I have a mongodb Relationships collection that stores the user_id and the followee_id(person the user is following). If I query for against the user_id I can find all the the individuals the user is following. Next I need to query the Users collection against all of the returned followee ids to get their personal information. This is where I confused. How would I accomplish this?
NOTE: I know I can embed the followees in the individual user's document and use and $in operator but I do not want to go this route. I want to maintain the most flexibility I can.
You can use an $in query without denormalizing the followees on the user. You just need to do a little bit of data manipulation:
Relationship.find({user_id: user_id}, function(error, relationships) {
var followee_ids = relationships.map(function(relationship) {
return relationship.followee_id;
});
User.find({_id: { $in: followee_ids}}, function(error, users) {
// voila
});
};
if i got your problem right(i think so).
you need to query each of the "individuals the user is following".
that means to query the database multiple queries about each one and get the data.
because the queries in node.js (i assume you using mongoose) are asynchronies you need to get your code more asynchronies for this task.
if you not familier with the async module in node.js it's about time to know it.
see npm async for docs.
i made you a sample code for your query and how it needs to be.
/*array of followee_id from the last query*/
function query(followee_id_arr, callback) {
var async = require('async')
var allResults = [];
async.eachSerias(followee_id_arr, function (f_id, callback){
db.userCollection.findOne({_id : f_id},{_id : 1, personalData : 1},function(err, data){
if(err) {/*handel error*/}
else {
allResults.push(data);
callback()
}
}, function(){
callback(null, allResults);
})
})
}
you can even make all the queries in parallel (for better preformance) by using async.map
I [new to node.js and programming in general] have two mysql query results (member info and list of workshops that members can attend) and need to send them to res.render() to be presented in .jade template (Member edit page).
To do this I'm using node-mysql and mysql-queue modules. Problem is I don't know how to pass callback function to render the response before queue.execute() finishes so I made workaround and put first two queries in the queue (mysql-queue feature), executed the queue, and afterwards added third "dummy query" which has callback function that renders the template.
My question is can I use this workaround and what would be the proper way to this using this modules?
exports.memberEdit = function (req, res) {
var q = connection.createQueue();
var membersResults,
htmlDateSigned,
htmlBirthDate,
servicesResults;
q.query("SELECT * FROM members WHERE id= ?;", req.id, function (err, results) {
console.log("Članovi: " + results[0]);
membersResults = results[0];
htmlDateSigned = dater.convertDate(results[0].dateSigned);
htmlBirthDate = dater.convertDate(results[0].birthDate);
});
q.query("SELECT * FROM services", function (err, results) {
console.log("Services: " + results);
servicesResults = results;
});
q.execute();
// dummy query that processes response after all queries and callback execute
// before execute() statement
q.query("SELECT 1", function (err,result) {
res.render('memberEdit', { title: 'Edit member',
query:membersResults,
dateSigned:htmlDateSigned,
birthDate:htmlBirthDate,
services:servicesResults });
})
};
I think an alternative could be to use a transaction to wrap your queries with:
var trans = connection.startTransaction();
trans.query(...);
trans.query(...);
trans.commit(function(err, info) {
// here, the queries are done
res.render(...);
});
commit() will call execute() and it provides a callback which will be called when all query callbacks are done.
This is still a bit of a workaround though, it would make more sense if execute() would provide the option of passing a callback (but it doesn't). Alternatively, you could use a module which provides a Promise implementation, but that's still a workaround.