I am developing an Express nodeJS web app and am trying to increase my knowledge of asynchronous programming and how to access the data that I need properly.
I have two callback functions which query an SQL database. My functions are declared like so:
function CustomerList(callback) {
const db = require('../db');
db.query('SELECT * FROM users WHERE role = (?)', ['User'], (error, results, fields) => {
if (error) throw error;
callback(results);
});
}
function AdminList(callback) {
const db = require('../db');
db.query('SELECT * FROM admins WHERE role = (?)', ['Moderator'], (error, results, fields) => {
if (error) throw error;
callback(results);
});
}
I use the data retrieved from both callbacks and send it to my view engine template to be displayed in the browser like so:
CustomerList((userData) => {
AdminList((adminData) => {
res.render('adminhome', {
title: 'Admin - Home',
results: 'test',
customers: userData,
admins: adminData
});
});
This works as expected and I can manipulate the data however I please on my view template. But... This seems... 'clunky', and I feel like this method I am using is going to cause me trouble in the future if I create additional functions to retrieve more data, and then keep nesting additional callbacks to pass the data to my view template.
Essentially, I am struggling to find a suitable approach to retrieve the UserData results and AdminData results and pass them to my res.render API call.
What are better alternatives to my current approach?
Yes, it'll create callback hell if you call multiple callbacks inside the callback, use promises(async/await)
const util = require('util');
const db = require('../db');
const query = util.promisify(db.query);
const getData = async () => {
try {
const CustomerList = await query('SELECT * FROM users WHERE role = (?)', ['User']);
const AdminList = await query('SELECT * FROM admins WHERE role = (?)', ['Moderator']);
// OR Promise.all
const [CustomerList, AdminList] = await Promise.all([
query('SELECT * FROM users WHERE role = (?)', ['User']),
query('SELECT * FROM admins WHERE role = (?)', ['Moderator'])
]);
} catch (error) {
console.log(error);
}
}
Related
I am using node-pg and would like to make multiple queries within a single GET request.
For example consider I make two queries like so:
const getSomeInfo = (request, response) => {
pool.query('SELECT * FROM jobs', (error, results) => {
if (error) {
throw error
}
var jobObj = results.rows;
response.render('pages/jobinfo', {
jobObj: jobObj
});
})
pool.query('SELECT * FROM description INNER JOIN views', (error, results) => {
if (error) {
throw error
}
var descObj = results.rows;
response.render('pages/jobinfo', {
descObj: descObj
});
})
}
This code results in the following error Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client.
Is there a way to make both these queries within the same GET request so that the objects containing the results can be used on the same page?
You can render only one document, but this can contain two local variables, one from each query. To have the queries executed in parallel, use Promise.all and make use of the fact that pool.query without a callback function returns a promise.
const getSomeInfo = (request, response) => {
Promise.all([
pool.query('SELECT * FROM jobs'),
pool.query('SELECT * FROM description INNER JOIN views')
]).then(function([jobResults, descResults]) {
response.render('pages/jobinfo', {
jobObj: jobResults.rows,
descObj: descResults.rows
});
}, function(error) {
throw error;
});
}
I want a suggestion on how to perform nodejs-DB operations
Here are my options
case one : When you have to run more than one query and former one is dependent on the latter
first way
connection.query(Q1, function(error, results, fields) {
if (error) {
throw error;
}
console.log(results[0]);
//call the second query here
connection.query(Q2, function(error, results, fields) {
if (error) {
throw error;
}
console.log(results[0]);
});
});
or
define first query in a function return the result via promise and await function
call the second query in the same way
use both results
like
var res1 = await funcName1(params);
var res2 = await funcName2(params);// params deducted from res1
which one is better?
case two : when we want to call select query more than 1 times
and capture the results in an array
the last one is pretty preplexing, do i have to call the mysql async function in a loop? is it a correct way?
Can you guys redirect me in the correct way?
I would suggest using the second option, or rather a similar approach to the second option. I believe the code is much easier to read and simpler to maintain. You'll also end up with a fraction of the number of lines of code.
It's also worth nothing that many database modules such as mysql2 and mongodb return promises directly, so there's no need to create wrapper functions, that is you won't need to maintain a load of funcName1, funcName2.. etc.
for example using mysql2:
const mysql = require('mysql2/promise');
const connection = await mysql.createConnection({
host: 'localhost',
user: 'some_user',
password: '***',
database: 'test_db'
});
let [users] = await connection.query('select * from users');
let [accounts] = await connection.query('select * from accounts');
console.log( { users, accounts })
If you wish to create promisified wrapper functions for a generic db access proc, have a look at Util.promisify, this allows you to cut down on boilerplate code.
e.g.
const dbQuery = util.promisify(db.query);
Then you can use like
let result = await dbQuery(param1, param2...);
As long as db.query looks something like
db.query(query, callback(err, result))
Also, querying within loops should work very well, for example:
async function testLoopQuery() {
for(let organization of organizationList) {
let [users] = await connection.query('select * from users where org_id = ?', [organization.id]);
}
}
I have a firebase callable function that does some batch processing on documents in a collection.
The steps are
Copy document to a separate collection, archive it
Run http request to third party service based on data in document
If 2 was successful, delete document
I'm having trouble with forcing the code to run synchronously. I can't figure out the correct await syntax.
async function archiveOrders (myCollection: string) {
//get documents in array for iterating
const currentOrders = [];
console.log('getting current orders');
await db.collection(myCollection).get().then(querySnapshot => {
querySnapshot.forEach(doc => {
currentOrders.push(doc.data());
});
});
console.log(currentOrders);
//copy Orders
currentOrders.forEach (async (doc) => {
if (something about doc data is true ) {
let id = "";
id = doc.id.toString();
await db.collection(myCollection).doc(id).set(doc);
console.log('this was copied: ' + id, doc);
}
});
}
To solve the problem I made a separate function call which returns a promise that I can await for.
I also leveraged the QuerySnapshot which returns an array of all the documents in this QuerySnapshot. See here for usage.
// from inside cloud function
// using firebase node.js admin sdk
const current_orders = await db.collection("currentOrders").get();
for (let index = 0; index < current_orders.docs.length; index++) {
const order = current_orders.docs[index];
await archive(order);
}
async function archive(doc) {
let docData = await doc.data();
if (conditional logic....) {
try {
// await make third party api request
await db.collection("currentOrders").doc(id).delete();
}
catch (err) {
console.log(err)
}
} //end if
} //end archive
Now i'm not familiar with firebase so you will have to tell me if there is something wrong with how i access the data.
You can use await Promise.all() to wait for all promises to resolve before you continue the execution of the function, Promise.all() will fire all requests simultaneously and will not wait for one to finish before firing the next one.
Also although the syntax of async/await looks synchronous, things still happen asynchronously
async function archiveOrders(myCollection: string) {
console.log('getting current orders')
const querySnapshot = await db.collection(myCollection).get()
const currentOrders = querySnapshot.docs.map(doc => doc.data())
console.log(currentOrders)
await Promise.all(currentOrders.map((doc) => {
if (something something) {
return db.collection(myCollection).doc(doc.id.toString()).set(doc)
}
}))
console.log('copied orders')
}
Simple call to ec2 Describing Security groups and returning the security group ID. Using Async / await, but when logging the return value, I get undefined. I fully admit I'm coming from Python and I've tried my hardest to wrap my brain around async calls. I thought I had it nailed, but I'm obviously missing something.
'use strict';
// Load Modules
const AWS = require('aws-sdk')
//Set the region
AWS.config.update({region: 'us-west-2'});
// Call AWS Resources
const ec2 = new AWS.EC2();
// Get Security Group ID From Event
const getSgIdFromEvent = async (event) => {
var ec2params = { Filters: [{Name: 'tag:t_whitelist',Values[event['site']]}]};
await ec2.describeSecurityGroups(ec2params, function (err, response) {
if (err) {return console.error(err.message)}
else {
var sgId = response.SecurityGroups[0].GroupId;
return sgId;
};
});
};
// MAIN FUNCTION
exports.handler = (event, context) => {
getSgIdFromEvent(event)
.then(sgId => {console.log(sgId)});
}
"sgId" should return the security group ID. It does print out fine in the original function before the return.
Typically if it is an async call you want you handle it similar to this way without using a callback
// Load Modules
const AWS = require('aws-sdk')
//Set the region
AWS.config.update({ region: 'us-west-2' });
// Call AWS Resources
const ec2 = new AWS.EC2();
// Get Security Group ID From Event
const getSgIdFromEvent = async (event) => {
var ec2params = { Filters: [{ Name: 'tag:t_whitelist', Values[event['site']]}] };
try {
const securityGroupsDesc = await ec2.describeSecurityGroups(ec2params).promise();
const sgId = securityGroupsDesc.SecurityGroups[0].GroupId;
//do something with the returned result
return sgId;
}
catch (error) {
console.log('handle error');
// throw error;
}
});
};
// MAIN FUNCTION
exports.handler = (event, context) => {
getSgIdFromEvent(event)
.then(sgId => { console.log(sgId) });
}
however if it doesn't support async you just use the callback to handle the returned data or error without using async function.However Reading into AWS docs you can find that the function ec2.describeSecurityGroups() returns an AWS Request
which has a method promise() that needs to be invoked to send the request and get a promise returned.Note that the try catch here is not needed but good to have in case error occurs during the process.
As I said in the comment, chance are that describeSecurityGroups doesn't return a Promise. Try transforming it explictly in a Promise instead:
const promiseResponse = await new Promise((res, rej) => {
ec2.describeSecurityGroups(ec2params, function (err, response) {
if (err) {return rej(err.message)}
else {
var sgId = response.SecurityGroups[0].GroupId;
res(sgId);
};
})
});
// promiseResponse is now equal to sgId inside the callback
return promiseResponse; // this will work because the function is async
Note: You can drop the else keyword
Here is the code that worked using async / await. Thanks to #Cristian Traina I realized ec2.describeSecurityGroups wasn't returning a promise, it was returning an AWS.Event.
// Get Security Group ID From Event
const getSgIdFromEvent = async (event) => {
console.log('Getting Security Group ID')
var params = { Filters: [{Name: 'tag:t_whitelist', Values
[event['site']]}]};
const describeSG = await ec2.describeSecurityGroups(params).promise();
return describeSG.SecurityGroups[0].GroupId;
};
// Get Ingress Rules from Security Group
const getSgIngressRules = async (sgId) => {
console.log(`Getting SG Ingress rules for ${sgId}`)
var params = { GroupIds: [ sgId]};
try{
const ingressRules = await ec2.describeSecurityGroups(params).promise();
return ingressRules;
}
catch (error) {
console.log("Something went wrong getting Ingress Ruls");
}
};
// MAIN FUNCTION
exports.handler = (event, context) => {
getSgIdFromEvent(event)
.then(sgId => {return getSgIngressRules(sgId);})
.then(ingressRules => {console.log(ingressRules);});
}
I submitted this as the answer now since the getSgIdFromEvent function I have, is only 8 lines and still using the async/await like I was desiring.
What I was missing was the .promise() on the end of the function and returning that promise.
Thanks for all the responses!
I'm writing a cloud functions in conjunction with google's Firestore database.
I'm trying to write recursive delete more data. I can't find the syntax for accessing and deleting data in other parts of the database.
The code I have already is below.
exports.deleteProject = functions.firestore.document('{userID}/projects/easy/{projectID}').onDelete(event => {
// Get an object representing the document prior to deletion
// e.g. {'name': 'Marie', 'age': 66}
// console.log(event)
// console.log(event.data)
console.log(event.data.previous.data())
var deletedValue = event.data.previous.data();
});
I found some info here but I don't have time to check through it atm, if I find something useful I'll amend the question.
https://firebase.google.com/docs/firestore/manage-data/delete-data?authuser=0
One can use below code to delete all the documents in a collection recursively.
This code worked perfectly for me.
Make sure you have JSON file of firebase credentials and firebase-admin installed.
const admin = require('firebase-admin');
const db = admin.firestore();
const serviceAccount = require('./PATH_TO_FIREBASE_CREDENTIALS.json');
admin.initializeApp({
credential: admin.credential.cert(serviceAccount)
});
deleteCollection(db, COLLECTION_NAME, NUMBER_OF_RECORDS)
async function deleteCollection(db, collectionPath, batchSize) {
const collectionRef = db.collection(collectionPath);
const query = collectionRef.orderBy('__name__').limit(batchSize);
return new Promise((resolve, reject) => {
deleteQueryBatch(db, query, resolve).catch(reject);
});
}
async function deleteQueryBatch(db, query, resolve) {
const snapshot = await query.get();
const batchSize = snapshot.size;
if (batchSize === 0) {
// When there are no documents left, we are done
resolve();
return;
}
// Delete documents in a batch
const batch = db.batch();
snapshot.docs.forEach((doc) => {
batch.delete(doc.ref);
});
await batch.commit();
// Recurse on the next process tick, to avoid
// exploding the stack.
process.nextTick(() => {
deleteQueryBatch(db, query, resolve);
});
}
The answer is that you must write a cloud function that deletes the data on its own and is trigger by the client. There isn't an efficient way to do it with client side. The method I use is I listen in the cloud function for the first delete and then fire the recursive.
Code to delete in node js:
db.collection("cities").document("DC").delete(