I am working with a NodeJS application which fetches film names and description from an Sqlite-3 file when user send a GET to /films. There seems to be a thing which I am missing while pushing the object to an array. I don't get what I am missing. The object does not get pushed to the array and always shows empty [] when I res.json() it back as response.
app.get('/films', (req, res) => {
let db = new sqlite3.Database('./data.db', sqlite3.OPEN_READWRITE, err => {
if (err) return console.error(err.message)
console.log('DB connected')
})
var films = []
db.serialize(() => {
db.each('select * from film', (err, row) => {
if (err) return console.log(err.message)
// console.log(row.name + '\t' + row.description)
films.push({
"name": row.name,
"description": row.description
})
})
})
res.json(films)
db.close(err => {
if (err) return console.error(err.message)
console.log('DB coonnection closed')
})
})
Your issue is that Express returns response object before the database retrieves any values. To fix this you’d normally move it inside the callback function.
In your case though db.each() method actually accepts 2 functions. The 1st one is callback which runs after each value is retrieved and wouldn’t be any help at all. The 2nd one is called complete in documentation and does exactly what you need.
Here’s the full reference in docs for db.each() from node-sqlite3 wiki.
With that you could write your code this way:
db.each(
'select * from film',
(err, row) => { /* Does normal stuff */ }),
(err, num) => { /* Sends response to client */
res.json(films);
console.log(`Retrieved ${num} films`); // (Just to show what the 2nd argument does)
}
);
Keep in mind that if your film database is not particularly huge docs actually recommend to use db.all method instead.
Related
I'm using node v14.2 and sqlite3: https://github.com/mapbox/node-sqlite3
I'm trying to determine if a table exists, and if it does, make a query against it. I've tried:
data = []
db.run("SELECT name FROM sqlite_master WHERE type='table' AND name='mytable';", result => {
console.log(result)
if (result) {
db.each("SELECT * FROM mytable;", (err, row) => {
data.push(row)
})
}
})
console.log(data)
However, my array data never gets pushed to, even when the table mytable exists. I've also tried testing for table existence with PRAGMA table_info('mytable');. How could I add to data once I confirm that mytable exists?
The issue I see in your code is, the console.log is outside of the callback function.
data = []
db.all("SELECT name FROM sqlite_master WHERE type='table' AND name='mytable';", function(err, rows) {
rows.forEach(function (row) {
data.push(row)
});
console.log(data)
});
What you need to understand in JS is, the callbacks are async and might get called after the next line gets executed. In your case, most likely the console.log got executed before the call back finished.
Update: Just realised the run function does not return anything.
I figured out the problem. db.run()'s second argument, result, is null if the query was successful and error if it was not. Thus, I should error if result, rather than the opposite. Here's the working code:
data = []
db.run("SELECT name FROM sqlite_master WHERE type='table' AND name='mytable';", result => {
if (result) {
console.error(result)
} else {
db.each("SELECT * FROM mytable;", (err, row) => {
data.push(row)
})
}
console.log(data)
})
How can I write a function that will delete just one row (ideally the last one) that has a given name (which may not be unique).
I have tried using row count, limits, and my own built in function (below). None of these have worked.
app.delete('/FamilyMember/:db', (req, res) => {
let db = openDB("ClientsDatabases/"+req.params.db);
let ids = [];
db.serialize(()=>{
db.each('select * from family', (err, row)=> {
if (row.name == req.body.name) {
ids.push(row.id);
}
})
db.run("DELETE FROM family WHERE id = ?",ids[ids.length-1], (err)=> {
console.log("Here is the err "+err);
if (!err) console.log('Succesful # deleting', req.body.name);
});
})
res.send();
}, () => {
db.close();
})
My expected output is for only one row with the given name to be deleted, but the table doesn't change and no errors are caught.
Any particular reason you've got single quotes (') around your ? parameter in your call to Statement#run? Nothing in the API documentation suggests that this is correct practice, could be an explanation as to why your query is mangled to the point where it doesn't delete anything, but also doesn't throw an error. I'd expect your call to .run() to look something more like the below:
db.run("DELETE FROM family WHERE id = ?",ids[ids.length-1], (err)=> {
console.log("Here is the err "+err);
if (!err) console.log('Succesful # deleting', req.body.name);
});
You may alternatively be interested in simplifying your queries into a single statement that will grab the maximum id (provided the id is incremented each time) and delete that record:
db.run("DELETE FROM family WHERE id = MAX(id)", (err)=> {
console.log("Here is the err "+err);
if (!err) console.log('Succesful # deleting', req.body.name);
});
This eliminates the need to load all the contents of family first just to grab a single id from it.
I'm using node and postgres, I'm new to writing async function, what I'm trying to do is a very simple query that will do a total count of records in the database, add one to it and return the result. The result will be visible before the DOM is generated. I don't know how to do this, since async function doesn't return value to callers (also probably I still have the synchronous mindset). Here's the function:
function generateRTA(callback){
var current_year = new Date().getFullYear();
const qry = `SELECT COUNT(date_part('year', updated_on))
FROM recruitment_process
WHERE date_part('year', updated_on) = $1;`
const value = [current_year]
pool.query(qry, value, (err, res) => {
if (err) {
console.log(err.stack)
} else {
var count = parseInt(res.rows[0].count) + 1
var rta_no = String(current_year) + '-' + count
callback(null, rta_no)
}
})
}
For the front-end I'm using pug with simple HTML form.
const rta_no = generateRTA(function (err, res){
if(err){
console.log(err)
}
else{
console.log(res)
}
})
app.get('/new_application', function(req, res){
res.render('new_application', {rta_number: rta_no})
});
I can see the rta_no in console.log but how do I pass it back to the DOM when the value is ready?
Based on the ajax call async response, it will update the div id "div1" when it gets the response from the Node js .
app.js
app.get("/webform", (req, res) => {
res.render("webform", {
title: "Render Web Form"
});
});
app.get("/new_application", (req, res) => {
// Connect to database.
var connection = getMySQLConnection();
connection.connect();
// Do the query to get data.
connection.query('SELECT count(1) as cnt FROM test ', function(err, rows, fields) {
var person;
if (err) {
res.status(500).json({"status_code": 500,"status_message": "internal server error"});
} else {
// Check if the result is found or not
if(rows.length==1) {
res.status(200).json({"count": rows[0].cnt});
} else {
// render not found page
res.status(404).json({"status_code":404, "status_message": "Not found"});
}
}
});
// Close connection
connection.end();
});
webform.pug - Via asynchronous call
html
head
script(src='https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js')
script.
$(document).ready(function(){
$.ajax({url: "/new_application", success: function(result){
$("#div1").html(result.count);
}});
});
body
div
Total count goes here :
#div1
value loading ...
That seems okay, I'm just not sure of this:
The result will be visible before the DOM is generated
This constraint defeats the purpose of async, as your DOM should wait for the returned value to be there. Instead of waiting for it you could just render the page and once the function returns and runs your callback update the value.
Also, perhaps it's worth having a look into promises
I'm currently running a stack that consists of Express and MongoClient with Mocha and Chai for testing. I'm working on writing test cases for my endpoint and am getting a random error that pops up from time to time. Below is a snippet of one of the suits I'm writing:
describe('Recipes with populated database', () => {
before((done) => {
var recipe1 = {"search_name": "mikes_mac_and_cheese", "text_friendly_name": "Mikes Mac and Cheese","ingredients": [{"name": "elbow_noodles","text_friendly_name": "elbow noodles","quantity": 12,"measurement": "oz"},{"name": "cheddar_cheese","text_friendly_name": "cheddar cheese","quantity": 6,"measurement": "oz"},{"name": "gouda_cheese","text_friendly_name": "gouda cheese","quantity": 6,"measurement": "oz"},{"name": "milk","text_friendly_name": "milk","quantity": 2,"measurement": "oz"}],"steps": ["Bring water to a boil","Cook noodels until al dente.","Add the milk and cheeses and melt down.","Stir constantly to ensure even coating and serve."],"course": ["dinner","lunch","side"],"prep_time": {"minutes": 15,"hours": 0},"cook_time":{"minutes": 25,"hours": 1},"cuisine": "italian","submitted_by": "User1","searchable": true};
db.collectionExists('recipes').then((exists) => {
if (exists) {
db.getDb().dropCollection('recipes', (err, results) => {
if (err)
{
throw err;
}
});
}
db.getDb().createCollection('recipes', (err, results) => {
if (err)
{
throw err;
}
});
db.getDb().collection('recipes').insertOne(recipe1, (err, result) => {
done();
});
});
});
The collectionExists() method simply takes in a name and returns a promise that is resolved to a true/false value. I've already done some debugging and it is working just fine. Where I am getting a problem is when I hit the section of the code where I call createCollection. I get an error about how the collection already exists thus leading to my tests failing. This appears to be happening on every third time I'm running my tests as well.
The purpose of all this is to ensure that my database collection called recipes is completely empty before I start testing so I'm not stuck with old data or in an uncontrolled environment.
You have a race condition between .createCollection and .insertOne. In other words, they start at the same time and go in parallel. There is no way to tell which will be done first.
The way .insert works in MongoDB is that if the collection is missing and you try inserting - it's going to create a collection. So if .insertOne is executed first - the collection is created and that is why you're getting the already exists error in an attempt to createCollection.
Due to the async nature of DB calls you'd have to place the subsequent calls inside the callback of a prev. one. This way there will be no parallel execution:
before((done) => {
var recipe1 = {/**/};
db.collectionExists('recipes')
.then((exists) => {
if (exists) {
// Drop the collection if it exists.
db.getDb().dropCollection('recipes', (err) => {
if (err) {
// If there's an error we want to pass it up to before.
done(err);
return;
}
// Just insert a first document into a non-existent collection.
// It's going to be created.
// Notice the done callback.
db.getDb().collection('recipes').insertOne(recipe1, done);
});
}
// If there were no such collection - simply insert the first doc to create it.
// Note that I'm passing before's done callback inside.
db.getDb().collection('recipes').insertOne(recipe1, done);
})
// We don't want to lose the error from this promise always.
.catch(err => done(err));
});
But. Actually, there is no need to drop and re-create a collection each time you run the tests. You can simply .remove all the objects in the before block. So probably the right solution would be:
before((done) => {
var recipe1 = {/**/};
const recipes = db.getDb().collection('recipes');
// Simply wipe out the data
recipes.remove({}, err => {
if (err) {
done(err);
return;
}
recipes.insertOne(recipe1, done);
});
});
I need to fetch two different MongoDB collections (db.stats and db.tables ) for the same request req.
Now, in the code below, I am nesting the queries within the callback function.
router.post('/', (req, res) => {
let season = String(req.body.year);
let resultData, resultTable;
db.stats.findOne({Year: season}, function (err, data) {
if (data) {
resultData = getResult(data);
db.tables.findOne({Year: season}, function (err, data) {
if (data) {
resultTable = getTable(data);
res.render('index.html', {
data:{
result : resultData,
message: "Working"}
});
} else {
console.log("Error in Tables");
}
});
} else {
console.log("Error in Stats");
}
});
});
This code works, but there a few things that don't seem right. So my question is:
How do I avoid this nested structure? Because it not only looks ugly but also, while I am processing these requests the client side is unresponsive and that is bad.
What you have right now is known as the callback hell in JavaScript. This is where Promises comes in handy.
Here's what you can do:
router.post('/', (req, res) => {
let season = String(req.body.year);
var queries = [
db.stats.findOne({ Year: season }),
db.tables.findOne({ Year: season })
];
Promise.all(queries)
.then(results => {
if (!results[0]) {
console.log("Error in Stats");
return; // bad response. a better way is to return status 500 here
} else if (!results[1]) {
console.log("Error in Tables");
return; // bad response. a better way is to return status 500 here
}
let resultData = getResult(results[0]);
let resultTable = getTable(results[1]);
res.render('index.html', { data: {
result : resultData,
message: "Working"
} });
})
.catch(err => {
console.log("Error in getting queries", err);
// bad response. a better way is to return status 500 here
});
});
It looks like you are using Mongoose as your ODM to access your mongo database. When you don't pass in a function as the second parameter, the value returned by the function call (e.g. db.stats.findOne({ Year: season })) will be a Promise. We will put all of these unresolved Promises in an array and call Promise.all to resolve them. By using Promise.all, you are waiting until all of your database queries get executed before moving on to render your index.html view. In this case, the results of your database function calls will be stored in the results array in the order of your queries array.
Also, I would recommend doing something like res.status(500).send("A descriptive error message here") whenever there is an error on the server side in addition to the console.log calls.
The above will solve your nested structure problem, but latter problem will still be there (i.e. client side is unresponsive when processing these requests). In order to solve this, you need to first identify your bottleneck. What function calls are taking up most of the time? Since you are using findOne, I do not think that will be the bottleneck unless the connection between your server and the database has latency issues.
I am going to assume that the POST request is not done through AJAX since you have res.render in it, so this problem shouldn't be caused by any client-sided code. I suspect that either one of getResult or getTable (or both) is taking up quite a significant amount of time, considering the fact that it causes the client side to be unresponsive. What's the size of the data when you query your database? If the size of it is so huge that it takes a significant amount of time to process, I would recommend changing the way how the request is made. You can use AJAX on the front-end to make a POST request to the back-end, which will then return the response as a JSON object. That way, the page on the browser would not need to reload, and you'll get a better user experience.
mongodb driver return a promise if you dont send a callback so you can use async await
router.post('/', async(req, res) => {
let season = String(req.body.year);
let resultData, resultTable;
try {
const [data1,data2] = await Promise.all([
db.stats.findOne({Year: season}),
db.tables.findOne({Year: season})
]);
if (data1 && data2) {
resultData = getResult(data1);
resultTable = getTable(data2);
return res.render('index.html', {
data: {
result: resultData,
message: "Working"
}
});
}
res.send('error');
console.log("Error");
} catch (err) {
res.send('error');
console.log("Error");
}
});