I am trying to call an async function in app.js to initialize parameters (fetch configuration from AWS, connect to DB, etc.)
This is the init.js that does the initialization:
const fs = require('fs');
var params_path = process.argv.slice(2);
const paramdata = JSON.parse(fs.readFileSync(params_path[0]));
module.exports.initApp = async function()
{
// Retrieve the configuration parameters from AWS
const aws = await require('./util/configparams').AWS(paramdata);
}
This is configparams.js:
async function AWS(params)
{
awscredentials = new AWSCredentials(params);
await awscredentials.initWalletParams();
await awscredentials.initDbParams();
return awscredentials;
}
initWalletParams and initDbParams are async functions that have await statements.
Then this is my app.js where I call the init:
(async() => {
init = require('./init');
await init.initApp();
})();
The execution reaches initApp() and goes to await awscredentials.initWalletParams(); function but it does not wait there.
When I run init.js by itself the execution happens sequentially as expected but calling in app.js await is not waiting as expected.
Could someone please help figure this out?
Thank you.
Related
I am working on an aws lambda that requires a puppeteer browser to be launched for each new s3 object in a bucket. The browser launch code was taking a very long time on the initial invocation, so I thought I would put the launch code outside the handler and use Provisioned Concurrency to have the browser ready to go when a new file in inserted into the bucket.
It does seem to call the promise because before any actual invocations are made, I'm getting logs saying "Getting executable path from the provisioned concurrency instances. However, it never outputs the message "Launching browser" until an actual invocation of the lambda is made. Why would the promise chromium.executablePath not complete until an invocation is made if it is outside the handler?
let startTime = Date.now();
const chromium = require("chrome-aws-lambda");
const AWS = require("aws-sdk");
const s3 = new AWS.S3();
const { createSSRApp } = require("vue");
const { renderToString } = require("vue/server-renderer");
const path = require("path");
const fs = require("fs");
const manifest = require("../../compiled/ssr-manifest.json");
console.log("Load packages: " + (Date.now() - startTime));
const browserPromise = new Promise((res) => {
const browserStartTime = Date.now();
console.log("Getting executable path");
chromium.executablePath.then((executablePath) => {
console.log("Launching browser");
chromium.puppeteer
.launch({
args: chromium.args,
defaultViewport: chromium.defaultViewport,
executablePath: executablePath,
headless: true,
})
.then((browser) => {
res(browser);
console.log("Start headless browser: " + (Date.now() - browserStartTime));
});
});
});
browserPromise.then(() => console.log("Started Headless Browser"));
/**
* A Lambda function that logs the payload received from S3.
*/
exports.handler = async (event, context) => {
const bucketName = event.Records[0].s3.bucket.name;
const objectKey = event.Records[0].s3.object.key;
const browser = await browserPromise;
... //use browser code
}
If I require this file locally in another node file it runs the promise fine without calling the handler function, so it must be some lambda environment specific thing I'm not understanding. Does anyone have any insight into this? Thanks in advance.
The issue you describe results from poor control of the async code.
Delay instantiation of browser promise
When you instantiate a Promise using the new keyword, execution of the function you provide starts immediately.
const x = new Promise(res => console.log('test'))
This will print 'test' immediately without needing a .then or await. This is why your code prints 'Getting executable path' right away, vs. waiting for a request event from lambda.
To solve this, don't instantiate this promise until a request actually happens. Move your promise construction to a function that you can call from the handler when a request occurs.
async function startBrowser () {
// code to start browser
return browser
}
exports.handler = async (event, context) => {
const bucketName = event.Records[0].s3.bucket.name;
const objectKey = event.Records[0].s3.object.key;
const browser = await startBrowser();
// use browser
}
Fixing async return flow
Secondly, you need to make your startBrowser function actually return a browser. Because you haven't awaited any of the promises created inside your browserPromise, it will trigger the code to start chromium but resolve immediately. It will take some time for the browser to start, which is why you don't see 'Launching browser' until much later.
To fix this, make sure your browser promise doesn't resolve until the browser is ready, and then return the browser object so it can be used.
function startBrowser () {
const browserStartTime = Date.now();
console.log("Getting executable path");
// await this promise
const browser = await chromium.executablePath.then((executablePath) => {
console.log("Launching browser");
// return browser promise
return chromium.puppeteer
.launch({
args: chromium.args,
defaultViewport: chromium.defaultViewport,
executablePath: executablePath,
headless: true,
})
console.log("Start headless browser: " + (Date.now() - browserStartTime));
return browser
}
exports.handler = async (event, context) => {
const bucketName = event.Records[0].s3.bucket.name;
const objectKey = event.Records[0].s3.object.key;
const browser = await startBrowser();
// use browser
}
Improving performance by sharing browser across requests
You can make further performance improvements by saving the browser in a singleton so that a new one doesn't need to be instantiated every request cycle.
let browser; // singleton/global browser object
// start a browser to be used for all requests
// this will take a little time, so hold a reference to the promise so we
// can know when it is ready to use
const browserPromise = startBrowser.then(newBrowser => { browser = newBrowser }
exports.handler = async (event, context) => {
const bucketName = event.Records[0].s3.bucket.name;
const objectKey = event.Records[0].s3.object.key;
// if the first request comes before the browser is ready, we should
// wait for the promise to resolve
if (!browser) await browserPromise
// use browser
}
Notes about memory usage
Browsers can use a lot of memory and can also leak memory. If fixing your async code still does not make the lambda work, consider that loading a webpage can consume gigabytes of memory especially for large and complex pages (Google Maps for example). I'm not experienced with lambda but you may find yourself running into memory limits.
I've initialised a basic WDIO project using mocha framework.
I have a class below
const axios = require('axios');
class Joke{
async getJoke(){
const response = axios.get('https://api.chucknorris.io/jokes/random');
return response;
}
}
module.exports = new Joke();
This is my spec file:
const Joke = require("./Joke");
describe("GET random jokes", () => {
it("responds with json", async () => {
const res = browser.call(async() => await Joke.getJoke());
console.log(res);
});
});
The console shows that my test has passed however there's no output of the response. Been trying to crack my head at this, could anyone point out what the issue is?
Aren't you missing an await here before browser.call?
const res = **await** browser.call(async() => await Joke.getJoke());
If you are using Webdriver in sync mode, there is no need to await browser.call as suggested in the previous comment, since browser.call is used to wrap async code to perform like sync code. However, the await is missing when calling axios.get.
const response = await axios.get('https://api.chucknorris.io/jokes/random');
I have created a saperate js file in node js and I am trying to connect to postgres sql using below code but it never worked. always promise is pending. code is not waiting at this line (await client.connect()). I Could not understand what the issue is. can any one please help me on this.
const pg = require('pg');
async function PostgresConnection(query,config) {
const client = new pg.Client(config);
await client.connect(); //not waiting for the connection to succeed and returning to caller and exit
const pgData = await client.query(query);
await client.end();
return pgData;
}
async function uploadDataToPostgres(config) {
var query="select * from firm_data";
await PostgresConnection(query, config);
}
module.exports = {
uploadDataToPostgres
}
I am trying to call above method from other page
function ProcessData()
{
var result = uploadDataToPostgres(config);
}
When you invoke an async function from outside an async function, you need to treat it as a Promise. Change your invoker function to look like this:
function ProcessData() {
uploadDataToPostgres(config)
.then(function success(result) {
/* do something useful with your result */
console.log(result)
})
.catch(function error(err) {
console.error('uploadDataToPostgres failure', err)
})
}
Your code, var result = uploadDataToPostgres(config), stashes a Promise object in result. But the code behind the promise (the code in your async function) doesn't run until you invoke .then() or .else() on it.
It's a bit confusing until you completely grasp the similarity between async functions and Promises: async functions are Promise functions with nice syntax.
It's possible your uploadDataToPostgres() function throws an error (failure to connect to the db?) and the .catch() will catch that.
I can't figure out why this app keeps running. I've tried using the why-is-node-running package but I'm not perfectly sure how to read the output properly. Here's the first output of it:
There are 30 handle(s) keeping the process running
# TCPWRAP
/node_modules/mongodb/lib/core/connection/connect.js:269 - socket = tls.connect(parseSslOptions(family, options));
/node_modules/mongodb/lib/core/connection/connect.js:29 - makeConnection(family, options, cancellationToken, (err, socket) => {
/node_modules/mongodb/lib/core/sdam/monitor.js:182 - connect(monitor.connectOptions, monitor[kCancellationToken], (err, conn) => {
/node_modules/mongodb/lib/core/sdam/monitor.js:206 - checkServer(monitor, e0 => {
/node_modules/mongodb/lib/core/sdam/monitor.js:92 - monitorServer(this);
My guess is it has something to do with MongoDB not closing properly. Although, when I removed all of the other functions between opening the client and closing it, it opened and closed perfectly.
Adding process.exit() at the end closes program properly, but I'd like to figure out why it isn't closing.
A summary of the app is that it is getting data from MongoDB, cleaning it, and then writing it into Firestore - so a lot of async actions going on, but I didn't see Firestore-related stuff pop up in the why-is-node-running logs.
const GrabStuffFromDBToCalculate = require("./helpers/GrabStuffFromDBToCalculate");
const SendToFirestore = require("./helpers/SendToFirestore");
const log = require("why-is-node-running");
const { MongoClient } = require("mongodb");
require("dotenv").config();
const main = async () => {
try {
const client = await MongoClient.connect(process.env.MONGODB_URI, {
useNewUrlParser: true,
useUnifiedTopology: true
});
const collection = await client.db("test").collection("testcollection");
const trip_object = await GrabStuffFromDBToCalculate(collection);
SendToFirestore(trip_object);
client.close();
log(); // "There are 30 handle(s) keeping the process running including node_modules/mongodb/lib/core/connection/connect.js:269 - socket = tls.connect(parseSslOptions(family, options));"
// process.exit() // this closes everything but I'd rather not have to use this
} catch (err) {
console.log(err);
client.close();
}
};
const runAsync = async () => {
await main(); // this exists because I'm usually running multiple main() functions
};
runAsync();
SendToFirestore code:
const firebase = require("firebase");
const firebaseConfig = require("../config");
module.exports = SendToFirestore = trip_object => {
if (!firebase.apps.length) {
firebase.initializeApp(firebaseConfig);
}
const db = firebase.firestore();
db.doc(`hello/${object._id}`).set({
objectid:object._id
});
};
GrabStuffFromDBToCalculate code (way simplified):
module.exports = GrabStuffFromDBToCalculate = async collection => {
const cursor = await collection
.aggregate([
// does a bunch of stuff here
])
.toArray();
const newObj = cursor[0];
return newObj;
};
Making my comment into an answer since it led to the missing piece.
Node does not shut down because you have an open Firestore connection. You will have to call terminate to allow the SDK to shut down and release resources:
db.terminate();
Which is relevant for allowing nodejs to shut itself down automatically.
Also, I'm not sure you understood that I was suggesting that you use await as in
await client.close()
before calling log() so you are sure that the client connection has been closed before you do the logging. client.close() is an asynchronous method so your original code would log() before that close was complete.
I have a middleware that load some JSON data from a REST API.
loadDataStartup.js
'use strict';
const db_contants = require('./db_contants');
async function loadData () {
console.log("*** loadData() called");
var equipmentCategoryArr = await db_contants.getEquipmentCategory();
return equipmentCategoryArr;
}
module.exports = {loadData};
In my app.js, I need to set the data to an Application-level variable by app.set such that it can be later used by other Routers.
app.js
const { loadData } = require('./db/loadDataStartup');
var app = express();
app.set('dataOnStartup', async loadData());
When I ran npm start, it threw the following error:
app.set('dataOnStartup', async loadData());
^^^^^
SyntaxError: missing ) after argument list
I am with node.js version 8.11.3.
First of all, you don't do async callAsyncFn(), but await callAsyncFn(). So async loadData() is a syntax error. What you probably wanted to do was await loadData(). But you also can't do this in the entry-level app file, because this statement is not in an async function (you can only await inside async functions).
The workaround would be using an async IIFE (Immediately-Invoked Function Expression), like this:
const { loadData } = require("./db/loadDataStartup");
var app = express();
(async function() {
app.set("dataOnStartup", await loadData());
// The rest of your app startup logic
})();