why nested async/await doesn't work as intended? - node.js

I'm learning NodeJs and having some problems using async/ await. I'm using Firebase database to read/write data. Here what i'm doing. (full function in case you need it).
async getImport(reqData: any): Promise<any> {
const username = 'kdat0310';
const db = admin.database();
const userRef = db.ref('/user');
const importRef = db.ref('/import');
const refAuthentication = db.ref('/Authentication');
const keyList = [];
const providerKey = [];
const khoList = [];
let index = 0;
const providerList = [];
const isValid = await refAuthentication.once('value', function (snapshot) {
for (const val of Object.values(snapshot.val())) {
if (
Object(val).username === Object(reqData).username &&
Object(val).token === Object(reqData).token
) {
return true;
}
}
return false;
});
if (isValid) {
await userRef.once('value', function (snapshot) {
for (const value of Object.values(snapshot.val())) {
if (value) {
if (Object(value).username == username) {
for (const val of Object(value).workAt) {
if (val) khoList.push(val.khoId);
}
}
}
}
});
const typeAndColorKey = [];
const typeAndColorValue = [];
const typeAndColorRef = db.ref('/TypeAndColor');
await typeAndColorRef.once('value', function (snapshot) {
let count = 0;
for (const key in snapshot.val()) {
typeAndColorKey.push(key);
}
for (const value of snapshot.val()) {
if (value !== undefined && value != null) {
typeAndColorValue.push({
id: typeAndColorKey[count],
type: value.type,
color: value.color,
});
count = count + 1;
}
}
});
const findTypeAndColor = (id: any) => {
for (const value of typeAndColorValue) {
if (id == value.id) {
return { type: value.type, color: value.color };
}
}
};
const userKey = [];
const userList = [];
await userRef.once('value', function (snapshot) {
let count = 0;
for (const key in snapshot.val()) {
userKey.push(key);
}
for (const value of Object(snapshot.val())) {
if (value != undefined && value != null) {
userList.push({
id: userKey[count],
name: Object(value).name,
});
count++;
}
}
});
const findUserName = (userId: any) => {
const returnValue = '';
for (const value of userList) {
if (userId == Object(value).id) {
return Object(value).name;
}
}
};
const importList = [];
await importRef.once('value', async function (snapshot) {
const importKey = [];
const cloneArr = snapshot.val().map((item: any) => {
return item;
});
for (const key in snapshot.val()) {
importKey.push(key);
}
let countTemp = 0;
for (const value of Object.values(cloneArr)) {
const goodsKeyList = [];
let count = 0;
if (khoList.indexOf(Object(value).warehouseId) !== -1) {
const listGoodsList = [];
if (Object(value).listGoods) {
for (const key in Object(value).listGoods) {
goodsKeyList.push(key);
}
const refListGoods = db.ref(
'/import/' + importKey[countTemp] + '/listGoods',
);
await refListGoods.once('value', function (snapshot) {
let item: any;
for (item of Object.values(snapshot.val())) {
if (item) {
const tempItem = item.filter((n: any) => n);
listGoodsList.push({
typeAndColor: findTypeAndColor(goodsKeyList[count]),
listGoods: tempItem,
number: tempItem.length,
});
}
count++;
}
});
}
console.log('test 1', listGoodsList);
if (listGoodsList !== []) {
importList.push({
listGoods: listGoodsList,
driver: Object(value).driver,
userId: Object(value).importEmployee,
name: findUserName(Object(value).importEmployee),
orderId: Object(value).orderId,
warehouseId: Object(value).warehouseId,
time: Object(value).time,
});
}
}
countTemp++;
}
console.log('test 2', importList);
});
return importList;
}
return 'Invalid';
}
The problem show up when it came to await importRef.once When I tried to handle some data and add the Firebase once function "async" and await inside to push the data I need to the array. Then return importList; return nothing. I figure that the await refListGoods.once cause this problems. As i thought, the await inside had done its duty and I can console.log importList inside very well. But I thought that await importRef.once will finish before return too. when I delete await refListGoods.once, the return is fine but I dont get the data I need. Do I need to refactor all code as I do to findTypeAndColor and findUserName above or there's a better way to solve this problem?

If you want to use await on the Promise returned by once, you should not pass a callback function to it.
So instead of:
const isValid = await refAuthentication.once('value', function (snapshot) {
for (const val of Object.values(snapshot.val())) {
if (
Object(val).username === Object(reqData).username &&
Object(val).token === Object(reqData).token
) {
return true;
}
}
return false;
});
Do:
const snapshot = await refAuthentication.once('value');
let isValid = false;
snapshot.forEach((child) => {
const val = child.val();
if (val.username === Object(reqData).username &&
val.token === Object(reqData).token
) {
isValid = true;
}
})

Related

How to use dynamoDB batchWriteItem with nodejs sdk?

I have a use case where I need to perform a batch_write operation on dynamodb. I referred this article which has a good solution for similar use case. I implemented it with few cleanup in my code and it works as expected.
const _ = require('lodash');
// helper methods
async function asyncForEach(array, cb) {
await Promise.all(array.map(async (item) => {
await cb(item, array);
}));
}
function to(promise) {
return promise.then((data) => [null, data])
.catch((err) => [err]);
}
const call = function (params) {
return dynamoDb.batchWriteItem(params).promise();
};
async function batchWrite25(arrayOf25, tableName) {
// 25 is as many as you can write in one time
const itemsArray = [];
_.forEach(arrayOf25, (item) => {
itemsArray.push({
PutRequest: {
Item: item,
},
});
});
const params = {
RequestItems: {
[tableName]: itemsArray,
},
};
await to(call(params));
}
async function batchWrite(itemArray, tableName) {
let mainIndex = 0;
let subIndex = 0;
let arrayOf25 = [];
const arrayLength = itemArray.length;
await asyncForEach(itemArray, async (item) => {
arrayOf25.push(item);
subIndex += 1;
mainIndex += 1;
// 25 is as many as you can write in one time
if (subIndex % 25 === 0 || mainIndex === arrayLength) {
await to(batchWrite25(arrayOf25, tableName));
subIndex = 0; // reset
arrayOf25 = [];
}
});
}
module.exports = {
batchWrite,
};
However, the code looks a bit complicated here with so many callbacks involved. Is there a cleaner way of writing the same thing without using -- call or asyncForEach or to methods ?
Here's one simple way to batch the items:
const BATCH_MAX = 25;
const batchWrite = async (items, table_name) => {
const BATCHES = Math.floor((items.length + BATCH_MAX - 1) / BATCH_MAX);
for (let batch = 0; batch < BATCHES; batch++) {
const itemsArray = [];
for (let ii = 0; ii < BATCH_MAX; ii++) {
const index = batch * BATCH_MAX + ii;
if (index >= items.length) break;
itemsArray.push({
PutRequest: {
Item: items[index],
},
});
}
const params = {
RequestItems: {
[table_name]: itemsArray,
},
};
console.log("Batch", batch, "write", itemsArray.length, "items");
await dynamodb.batchWriteItem(params).promise();
}
};
To make the entire process asynchronous, you can convert this function to return an array of promises and later call Promise.all(promises) on that array. For example:
const batchWrite = (items, table_name) => {
const promises = [];
const BATCHES = Math.floor((items.length + BATCH_MAX - 1) / BATCH_MAX);
for (let batch = 0; batch < BATCHES; batch++) {
// same code as above here ...
promises.push(dynamodb.batchWriteItem(params).promise());
}
return promises;
};
A much cleaner way using lodash that worked for me is listed below. Hope this helps somone.
batchWrite=async ()=> {
const batchSplitArr=_.chunk(this.dynamoPayload,25); //dynamoPayload has the entire payload in the desired format for dynamodb insertion.
await Promise.all(
batchSplitArr.map(async (item) => {
const params = {
RequestItems: {
[this.tableName]: item,
},
};
await this.dynamoDb.batchWriteItem(params).promise();
})
);
};

Struggling to use async await within for loops with nodejs

I am trying to use multiple functions within a for loop but, one is returning undefined (or not waiting?)
Code is as below:
var arrNew = [];
var id = [];
var get1 = [];
var get2 = [];
var get3 = [];
var res = [];
var get4 = [];
for (j = 0; j < arrNew.length; j++) {
id[j] = arrNew[j];
get1[j] = await get1Fun(context, id[j]);
if (get1[j] !== false) {
get2[j] = await get2Fun(context, get1[j]);
context.log(get2[j]);
if (get2[j].length) {
get3[j] = await get3Fun(context, get2[j]);
if (get3[j] !== false && get3[j].status == 200) {
res[j] = get3[j].result;
get4[j] = await get4Fun(context, id[j], res[j]);
context.log(get4[j])
}
}
}
}
async function get1Fun(context, id) {
const querySpec = {
query: `SELECT * FROM r where r.id = "${id}"`,
};
const { resources } = await client.database(databaseId).container(containerId).items.query(querySpec).fetchAll();
if (resources.length == 1) {
return resources[0];
} else if (!resources.length) {
return false;
}
}
function get2Fun(context, getInt) {
return new Promise((resolve, reject) => {
var msg = []
// do for loop stuff with getInt and push stuff into msg
context.log(msg)
resolve(msg);
});
}
The code works up until this point where get2Fun either returns undefined or if (get2[j].length) does not wait for await get2Fun.
Can someone clarify what is going wrong please?
I have tried with async function and without the promise.
The reason I have the arrays as below is because if I structured the function as var get1 = await get1Fun(context, id) then the for loop would stop after 1 or 2 items and not do every item in arrNew.
Thanks in advance.
EDIT:
Also tried the below but the for loop stops after 1 loop (presumably an issue with each loop redefining a variable?):
for (j = 0; j < arrNew.length; j++) {
var id = arrNew[j];
var get1 = await get1Fun(context, id);
if (get1 !== false) {
var get2 = await get2Fun(context, get1);
context.log(get2);
if (get2.length) {
var get3 = await get3Fun(context, get2);
if (get3 !== false && get3.status == 200) {
var res = get3.result;
var get4 = await get4Fun(context, id, res);
context.log(get4)
}
}
}
}
async function get1Fun(context, id) {
const querySpec = {
query: `SELECT * FROM r where r.id = "${id}"`,
};
const { resources } = await client.database(databaseId).container(containerId).items.query(querySpec).fetchAll();
if (resources.length == 1) {
return resources[0];
} else if (!resources.length) {
return false;
}
}
async function get2Fun(context, getInt) {
var msg = []
// do for loop stuff with getInt and push stuff into msg
context.log(msg)
return msg;
}

AWS Lambda (NodeJS) does not log to cloudwatch

I'm trying to log my lambda app after following serverless-next.js because of the issue where I can't go to the root of my file. So basically I'm deploying nextJS app in AWS through lambda#edge, s3, and cloudfront.
I'm new to AWS so I'm not really sure how to debug this thing at all. I assume traditional console.log in my lambda where every request comes in would log it in the cloudwatch. I also made sure that I deployed my lambda to my cloud front
Here's the code:
"use strict";
var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, generator) {
function adopt(value) { return value instanceof P ? value : new P(function (resolve) { resolve(value); }); }
return new (P || (P = Promise))(function (resolve, reject) {
function fulfilled(value) { try { step(generator.next(value)); } catch (e) { reject(e); } }
function rejected(value) { try { step(generator["throw"](value)); } catch (e) { reject(e); } }
function step(result) { result.done ? resolve(result.value) : adopt(result.value).then(fulfilled, rejected); }
step((generator = generator.apply(thisArg, _arguments || [])).next());
});
};
var __importDefault = (this && this.__importDefault) || function (mod) {
return (mod && mod.__esModule) ? mod : { "default": mod };
};
Object.defineProperty(exports, "__esModule", { value: true });
exports.handler = void 0;
const prerender_manifest_json_1 = __importDefault(require("./prerender-manifest.json"));
const manifest_json_1 = __importDefault(require("./manifest.json"));
const next_aws_cloudfront_1 = __importDefault(require("#sls-next/next-aws-cloudfront"));
const addS3HostHeader = (req, s3DomainName) => {
req.headers["host"] = [{ key: "host", value: s3DomainName }];
};
const isDataRequest = (uri) => uri.startsWith("/_next/data");
const normaliseUri = (uri) => (uri === "/" ? "/index" : uri);
const normaliseS3OriginDomain = (s3Origin) => {
if (s3Origin.region === "us-east-1") {
return s3Origin.domainName;
}
if (!s3Origin.domainName.includes(s3Origin.region)) {
const regionalEndpoint = s3Origin.domainName.replace("s3.amazonaws.com", `s3.${s3Origin.region}.amazonaws.com`);
return regionalEndpoint;
}
return s3Origin.domainName;
};
const router = (manifest) => {
const { pages: { ssr, html } } = manifest;
const allDynamicRoutes = Object.assign(Object.assign({}, ssr.dynamic), html.dynamic);
return (uri) => {
let normalisedUri = uri;
if (isDataRequest(uri)) {
normalisedUri = uri
.replace(`/_next/data/${manifest.buildId}`, "")
.replace(".json", "");
}
if (ssr.nonDynamic[normalisedUri]) {
return ssr.nonDynamic[normalisedUri];
}
console.log(uri);
for (const route in allDynamicRoutes) {
const { file, regex } = allDynamicRoutes[route];
const re = new RegExp(regex, "i");
const pathMatchesRoute = re.test(normalisedUri);
if (pathMatchesRoute) {
return file;
}
}
if (html.nonDynamic["/404"] !== undefined) {
return "pages/404.html";
}
return "pages/_error.js";
};
};
exports.handler = (event) => __awaiter(void 0, void 0, void 0, function* () {
const request = event.Records[0].cf.request;
const uri = normaliseUri(request.uri);
const manifest = manifest_json_1.default;
const prerenderManifest = prerender_manifest_json_1.default;
const { pages, publicFiles } = manifest;
const isStaticPage = pages.html.nonDynamic[uri];
const isPublicFile = publicFiles[uri];
const isPrerenderedPage = prerenderManifest.routes[request.uri];
const origin = request.origin;
const s3Origin = origin.s3;
const isHTMLPage = isStaticPage || isPrerenderedPage;
const normalisedS3DomainName = normaliseS3OriginDomain(s3Origin);
s3Origin.domainName = normalisedS3DomainName;
if (isHTMLPage || isPublicFile) {
s3Origin.path = isHTMLPage ? "/static-pages" : "/public";
addS3HostHeader(request, normalisedS3DomainName);
if (isHTMLPage) {
request.uri = `${uri}.html`;
}
return request;
}
const pagePath = router(manifest)(uri);
if (pagePath.endsWith(".html")) {
s3Origin.path = "/static-pages";
request.uri = pagePath.replace("pages", "");
addS3HostHeader(request, normalisedS3DomainName);
return request;
}
const page = require(`./${pagePath}`);
const { req, res, responsePromise } = next_aws_cloudfront_1.default(event.Records[0].cf);
if (isDataRequest(uri)) {
const { renderOpts } = yield page.renderReqToHTML(req, res, "passthrough");
res.setHeader("Content-Type", "application/json");
res.end(JSON.stringify(renderOpts.pageData));
}
else {
page.render(req, res);
}
return responsePromise;
});
Permission:
Allow: logs:CreateLogGroup
Allow: logs:CreateLogStream
Allow: logs:PutLogEvents
What else should I do? Should I create a new stream or is it automatically created? I can see a log group in my cloudwatch named aws/lambda but i'm not sure how to connect them
Really appreciate any help
Cheers

Why doesn't my async function return any result?

I wrote this small program to fetch data. This however is done async. Since I nonetheless need to use the function holeVertreter(kzl) as a function in another module, I'd like to get a return value which I can eventually pass on.
Excuse my spaghetti code (I usually prettify the code when I am done with my task ...).
Credentials are stored in a file and are therefore not found in this file.
I'd like to end up with "vertreter" as a return value.
Thank you in advance.
const node = require("deasync");
const DSB = require('dsbapi');
const tabletojson = require('tabletojson');
const https = require('https');
const cred = require("./vertrCred");
const dsb = new DSB(cred["dsb"]["user"], cred["dsb"]["passw"]); //Sanitized - no Credentials here
//Stackoverflow 2332811
String.prototype.capitalize = function(lower) {
return (lower ? this.toLowerCase() : this).replace(/(?:^|\s)\S/g, function(a) { return a.toUpperCase(); });
};
function holePlan(kuerzel) {
dsb.fetch()
.then(data => {
const timetables = DSB.findMethodInData('timetable', data);
const tiles = DSB.findMethodInData('tiles', data);
var tilesStr = JSON.stringify(tiles["data"][0]["url"]);
var url = JSON.parse(tilesStr);
https.get(url, (resp) => {
let data = '';
resp.on('data', (chunk) => {
data += chunk;
});
resp.on('end',() => {
var tableasjson = tabletojson.convert(data);
var erstetab = tableasjson[0];
var zweitetab = tableasjson[1];
var drittetab = tableasjson[2];
var viertetab = tableasjson[3];
var fuenftetab = tableasjson[4];
var sechstetab = tableasjson[5];
var siebtetab = tableasjson[6];
var achtetab = tableasjson[7];
if (typeof kuerzel === "undefined")
{
var regenechse = '(Aaa|Aaa[A-Za-z?]|[A-Za-z?]Aaa)';
}
else {
var name = kuerzel.capitalize(true);
var regenechse = '('+name+'|'+name+'[A-Za-z?]|[A-Za-z?]'+name+')';
}
const regex = new RegExp(regenechse,'g');
var sammel = Object.assign(drittetab,fuenftetab);
var z= 0;
var vertreter = {}
var y = JSON.parse(JSON.stringify(sammel));
for (i=0;i<y.length;i++) {
if (typeof y[i].Vertreter =='undefined') {
}
else {
if(y[i].Vertreter.match(regex))
{
z += 1;
vertreter[z] = y[i];
}
}
}
if (z == 0) {
// console.log("Es gibt nichts zu vertreten");
}
else {
//console.log("Es werden "+z+" Stunden vertreten");
return (vertreter);
} ;
});
})
})
.catch(e => {
// An error occurred :(
console.log(e);
});
}
//Stackoverflow
function warte(promise) {
var done = 0;
var result = null;
promise.then(
function (value) {
done = 1;
result = value;
return (value);
},
function (reason) {
done = 1;
throw reason;
}
);
while (!done)
node.runLoopOnce();
return (result);
}
function holeVertretung(kzl) {
var aufgabe = new Promise((resolve,reject) => {
setTimeout(resolve,1000,holePlan(kzl));
});
var ergebnis = warte(aufgabe);
if (typeof ergebnis === "undefined") {
console.log("Mist");
}
else {
console.log(ergebnis);
}
return ergebnis;
}
holeVertretung("Aaa");
That's not the right way to work with promises. If you do such infinite loop, it beats the whole purpose of using promises. Instead, return value from the promise, and use async-await like this:
function warte(promise) {
var done = 0;
var result = null;
return promise.then(
...
}
async function holeVertretung(kzl) {
var aufgabe = new Promise((resolve, reject) => {
setTimeout(resolve, 1000, holePlan(kzl));
});
var ergebnis = await warte(aufgabe);
...
If async-await does not work for some reason, use then clause:
warte(aufgabe).then(value => {
var ergebnis = value;
});

is there a way to avoid set a timeout

I'm trying to collect all values from a mysql table with all the values of the referenced_table_name for each index of the table.
How avoid set a random time out while waiting for a promise
To collect the expected information i need to set a random time out, otherwise my object is undefined...
module.exports = {
getTable: async (req, res) => {
const tablename = req.params.table,
dbName = req.params.dbName;
let jsonResult = {};
getTableValues(dbName, tablename)
.then(tableValues => {
getTableIndexedCol(dbName, tablename)
.then(indexedColumns => {
let indexedArr = {};
for (let index = 0; index < indexedColumns.length; index++) {
const element = indexedColumns[index],
column = element.column_name,
referencedTable = element.referenced_table_name;
let allValuesRefTable = new Array();
getTableValues(dbName, referencedTable)
.then(referencedTableValues => {
for (let i = 0; i < referencedTableValues.length; i++) {
const el = referencedTableValues[i];
allValuesRefTable.push(el.name);
}
})
.catch(err => console.log(err));
/*IF NO TIMEOUT DOESN'T WORK*/
setTimeout(function(){
indexedArr[column] = allValuesRefTable;
}, 100);
}
setTimeout(function(){
jsonResult = {
name: tablename,
rows : tableValues,
rowIndexed : indexedArr
}
res.json(jsonResult);
}, 5000);
})
.catch(err => console.log(err));
})
.catch(err => console.log(err));
}
};
Is there a way to don't use setTimeout? or how can I 'wait' that the promise is resolved?
Here is my function getTableIndexedCol for example:
async function getTableIndexedCol(dbName, tablename) {
const sqlRefTable = SELECT...;
return new Promise (async function(resolve, reject){
try{
[refTable, refTableFields] = await promisePool.query(sqlRefTable)
}
catch(err){
reject(err)
}
setTimeout(function () {
resolve(refTable);
}, 500);
})
If you are already using async/await you can use it all the way and avoid the "Promise Hell" (nested .then calls):
module.exports = {
getTable: async (req, res) => {
try {
const tablename = req.params.table,
dbName = req.params.dbName;
const tableValues = await getTableValues(dbName, tablename);
const indexedColumns = await getTableIndexedCol(dbName, tablename);
let indexedArr = {};
for (let index = 0; index < indexedColumns.length; index++) {
const element = indexedColumns[index],
column = element.column_name,
referencedTable = element.referenced_table_name;
let allValuesRefTable = new Array();
const referencedTableValues = await getTableValues(dbName, referencedTable);
for (let i = 0; i < referencedTableValues.length; i++) {
const el = referencedTableValues[i];
allValuesRefTable.push(el.name);
}
indexedArr[column] = allValuesRefTable;
}
const = jsonResult = {
name: tablename,
rows: tableValues,
rowIndexed: indexedArr
}
res.json(jsonResult);
} catch (err) {
console.log(err);
}
}
};

Resources