How to properly await createWriteStream? - node.js

Async programming in node.js is usually done with callbacks. I find callback-based code hard to read and reason about, which is why I'm using async & await whenever I can. This almost always works well and leads to robust code. However, rarely I'm wondering whether I'm making things more difficult than necessary. For example, how do you create a stream such that you can await its creation? More specifically, the result of the await should of course be the stream itself when things go well. When they don't, an appropriate exception should be thrown.
The best I could come up with is the function below. It feels very clunky and I'm wondering whether there is an easier way to do this?
import type { EventEmitter } from "events";
export const createStreamAsync = async <T extends EventEmitter>(create: () => T): Promise<T> => {
const result = create();
let onOpen = (): void => void 0;
let onError = (reason?: unknown): void => void reason;
try {
await new Promise<void>((resolve, reject) => {
onOpen = resolve;
onError = reject;
result.on("open", onOpen).on("error", onError);
});
return result;
} finally {
result.removeListener("open", onOpen);
result.removeListener("error", onError);
}
};
You'd use the function as follows:
import { createWriteStream } from "fs";
import { createStreamAsync } from "./createStreamAsync.js";
try {
const stream = await createStreamAsync(() => createWriteStream("~/whatever.txt"));
// Use stream ...
} catch (error: unknown) {
// Handle error
}

Readable streams have an AsyncIterator symbol, so they can be processed with for await ... of:
const readable = fs.createReadStream('file.txt');
for await (const chunk of readable) {
console.log(chunk);
}
You can listen to both an event and catch the error event (if any) with events.once:
const { once } = require('node:events');
const writable = fs.createWriteStream('file.txt');
try {
await once(writable, 'open');
// If the event emits data, `once` returns it as an array:
// const x = await once(...);
} catch (err) {
console.error('Could not open file', err);
}
This will automatically remove the listener, just like EventEmitter.once does.
event.on returns an AsyncIterator to handle multiple events:
const { on } = require('node:events');
const { exec } = require('node:child_process');
const child = exec(...);
try {
for await (const event of on(child, 'message')) {
// `event` is an array of 0 or more values.
console.log(event);
}
} catch (err) {
console.error(err);
}

Related

async/await troubles in a recursive Redis function

Ima rookie using async/await but must now to use Redis-om. NN_walkd walks through a Redis database looking for loop-chains and does this by recursion. So the 2 questions I have is:
Am I calling the inner recursive NN_walkd calls correctly via async/await?
At runtime, the compSearchM proc is called first and seems to work (it gets 5 entries so it has to call NN_walkd 5 times). A NN_walkd is then recursively called, and then when it loops the 1st time it then calls compSearchK where the problems are. It seems to sit on the first Redis call in compSearchK (.search). Yet the code for compSearchK and compSearchM look basically identical.
main call
NN_walk = async function(req, db, cnode, pnode, chain, cb) {
var vegas, sneaker;
req.session.walk = [];
await NN_walkd(req, cnode, pnode, [], 1);
req.session.walk = null;
console.log('~~~~~~~~~~~~ Out of Walk ~~~~~~~~~~~~~~~');
cb();
};
redis.mjs
export class RedisDB {
constructor() {
...
this._companyRepo = ...
}
compSearchK(ckey) { // doesn't matter if I have a async or not here
return new Promise(async (resolve) => {
const sckey = await this._companyRepo.search()
.where('COMPANYKEY').equals(ckey)
.return.all();
if (sckey.length) {
const ttrr = await this._companyRepo.fetch(sckey[0].entityId);
resolve(ttrr.toJSON());
} else
resolve(null);
});
}
compSearchM(mkey) {
var tArr=[];
return new Promise(async (resolve) => {
const smkey = await this._companyRepo.search()
.where('MASTERKEY').equals(mkey)
.and('TBLNUM').equals(10)
.return.all();
if (smkey.length) {
for (var spot in smkey) {
const ttrr = await this._companyRepo.fetch(smkey[spot].entityId);
tArr.push(ttrr.toJSON());
}
resolve(tArr);
} else {
resolve(null);
}
});
}
walk.js
NN_walkd = async function(req, cnode, pnode, chain, lvl) {
...
if (cnode[1]) {
const sObj = await req.app.get('redis').compSearchK(cnode[1]);
if (sObj) {
int1 = (sObj.TBLNUM==1) ? null : sObj.CLIENTKEY;
(async () => await NN_walkd(req, [sObj.COMPANYKEY,int1], cnode, Array.from(chain), tlvl))()
}
} else {
const sArr = await req.app.get('redis').compSearchM(cnode[0]);
if (sArr.length) {
for (sneaker in sArr) {
(async () => await NN_walkd(req, [sArr[sneaker].COMPANYKEY,sArr[sneaker].CLIENTKEY], cnode, Array.from(chain), tlvl))()
}
} else {
console.log('no more links on this chain: ',cnode);
}
}
}
"doesn't matter if i have async or not here"
compSearchK(ckey) { // doesn't matter if I have a async or not here
return new Promise(async (resolve) => {
const sckey = await this._companyRepo.search()
.where('COMPANYKEY').equals(ckey)
.return.all();
if (sckey.length) {
const ttrr = await this._companyRepo.fetch(sckey[0].entityId);
resolve(ttrr.toJSON());
} else
resolve(null);
});
}
Of course it doesn't matter, because you're not using await inside of compSearchK!
You are using the explicit promise contructor anti-pattern. You should avoid it as it demonstrates lack of understanding. Here is compSearchK rewritten without the anti-pattern -
async compSearchK(ckey) {
// await key
const sckey =
await this._companyRepo.search()
.where('COMPANYKEY').equals(ckey)
.return.all();
// return null if key is not found
if (sckey.length == 0) return null;
// otherwise get ttrr
const ttrr = await this._companyRepo.fetch(sckey[0].entityId);
// return ttrr as json
return ttrr.toJSON();
}

How to build an eventual return value though multiple promises

How do I chain some promises (NodeJS) together to continually enhance and object with new data and return the final object at the end of the chain. ie:
return getOrder().then((order) => {
return StatusMachine.getStatus(order.orderId);
}.then((status) => {
// get status response
// how do I access the "Order" in the previous then
// so I can do something like
order.status = status;
return order;
}).then((order) => {
return Shipping.getTrackingNumber(order.orderId);
}).then((tracking) => {
order.trackingNumber = tracking;
return order
});
Goal here is to load an order, then query its status and add it to the order, then query its tracking number and add it to the order. Currently, I'm setting Order as a global variable so that I can access it from every then() function and then in the final then(), I'm simply returning the global Order variable. However, that doesn't feel right.
To get order & status you need to wrap it with async function like this.
getOrder()
.then(async (order) => { // <--
const status = await StatusMachine.getStatus(order.orderId) // <-- await for status
return { order, status }
})
.then(({ status, order }) => {
order.status = status
return Promise.resolve(order) // if you are going to chain `.then` again then you have to return promise (converting this to async function will also works)
})
//.....
But I high recommend going full async, its much more readable and error handling becomes super easy
const getOrderWithMetadata = async () => {
const order = await getOrder()
const status = await StatusMachine.getStatus(order.orderId)
order.status = status
const tracking = await Shipping.getTrackingNumber(order.orderId)
order.trackingNumber = tracking
return order
}
or with Promise.all
const getOrderWithMetadataBetter = async () => {
const order = await getOrder()
const [status, tracking] = await Promsie.all([ // waits for both to complete, throws error if anyone of them fails.
StatusMachine.getStatus(order.orderId),
Shipping.getTrackingNumber(order.orderId),
])
order.status = status
order.trackingNumber = tracking
return order
}
Edit: It depends on how you want to handle error.
case 1: Ignore errors
const getOrderWithMetadata = async () => {
try {
// ...
} catch (err) {
// ...
}
}
case 2: return data and error (like go lang does.)
const getOrderWithMetadata = async () => {
try {
// ...
return [value, null] // [result, error]
} catch (err) {
// ...
return [null, err] // [result, error]
}
}
const [data, err] = await getOrderWithMetadata()
case 3: handle error outside using try/catch
const getOrderWithMetadata = async () => {
//...
}
try {
// ...
await getOrderWithMetadata()
//...
} catch (err) {
// ...
}
case 4: handle error outside using promise
const getOrderWithMetadata = async () => {
//...
}
getOrderWithMetadata().then(() => /* ... */).catch(() => /* ... */)

From Node.js event handler to generator

I have this monitor object that raises an event every time data is found, by means of an event handler:
monitor.on("data", data => { /* do something */ })
I would like to replace this pattern by using a generator:
for await(const data of monitor.iterate()) { /* do something */ }
I know I can do this:
async function monitorWrapper() {
const allData = await new Promise( resolve => {
const _allData = []
monitor.on("data", d => _allData.push(d))
monitor.on("end", () => resolve(_allData))
} )
yield *allData
}
and call it this way:
for await (const data of monitorWrapper()) { /* do something */ }
But this negates the whole of point of using yield since I have to wait for all elements to be available before processing.
In this context, my question is : Is there any pattern that allows to yield as the data events are triggered ?
Just ran into this problem myself.
const stream = new class { // mock event stream
async start() {
while (true) {
const ms = Math.random() * 100;
await new Promise(resolve => setTimeout(resolve, ms));
if (this.handler) this.handler(ms);
}
}
on(handler) { this.handler = handler; }
}
class Monitor {
events = []; // store events
resolve = () => {}; // resolves current outstanding promise
constructor(handler) {
handler(event => { // binds event listener
this.events.push(event); // add event
this.resolve(); // call resolve
});
}
async *stream() { // generator
while (true) {
let event;
while (event = this.events.shift()) yield event; // remove and yield
await new Promise(r => this.resolve = r); // wait for next
}
}
}
const monitor = new Monitor(handler => stream.on(handler)); // bind events
stream.start();
for await (const event of monitor.stream()) { console.log({ event }); }
I am not sure, but I think you can try this and let me know what happened.
//start is a thunk creator
function start(cb){
var c = cb();
c.next(); //start up!
return function(data){
c.next(data);
};
}
monitor.on('data', start(function(message){
var m = yield message;
console.log(m);
}));

Calling asynchronous recursive functions in Node.js with Promise

at first i need to say im pretty new to electron and node.js.
What i am doing is im trying to send an array with some directory data to my browser and this asynchronous (Error: array is empty).
The Problem should be my function 'scanDirectories(path)'.
If i make it non recursive (scanDirectories(res) -> only return res) it works pretty fine for 1 level directories, but recursive it doesnt.
So i think i need to do it like my function above with returning a new promise?
I tried it but can´t figure out how it works or my syntax should be.
Hope you can help me.
Simon
main.js:
calling function from ipcmain
ipcMain.on('fileTree', (event, arg) => {
let fileDirectory = helperFile.getDirectories(arg);
fileDirectory.then(function(result) {
event.reply('fileTree', result);
}, function(err) {
console.log(err);
})
})
files.js
const { readdir } = require('fs').promises;
const resolvePath = require('path').resolve;
module.exports = {
getDirectories: async function(path) {
return new Promise(function (resolve, reject) {
try {
resolve(scanDirectories(path));
} catch {
reject('Error');
}
});
}
};
async function scanDirectories(path) {
const dirents = await readdir(path, {withFileTypes: true});
const files = dirents.map((dirent) => {
const res = resolvePath(path, dirent.name);
return dirent.isDirectory() ? scanDirectories(res) : res;
});
return Array.prototype.concat(...files);
}
You can try something like this where you generate an array of promises:
files.js
const { readdir } = require('fs').promises;
const resolvePath = require('path').resolve;
module.exports = {
// This is an async function so don’t need internal promise
getDirectories: async function(path) {
try {
const dirs = await scanDirectories(path);
return dirs;
}
catch {
throw new Error('Error');
}
}
};
async function scanDirectories(path) {
const dirents = await readdir(path, {withFileTypes: true});
// Generate array of promises
const promises = dirents.map(dirent => {
const res = resolvePath(path, dirent.name);
return dirent.isDirectory()
? scanDirectories(res)
: Promise.resolve(res);
});
// Await all promises
const files = await Promise.all(promises);
return Array.prototype.concat(...files);
}
If you call an async function without await, you receive a promise.
Your event handler handles this sort-of-OK with then (it has trouble with error handling), but your recursive call to scanDirectories does not.
The simplest way to wait for an async function to resolve is to use await.
So this change makes the recursive call properly:
return dirent.isDirectory() ? (await scanDirectories(res)) : res;
Note the addition of await.
However "Array.map" is not designed for use in async functions. It will call them synchronously and create an array of promises. Not what you want.
In addition, this code is unnecessarily complicated, wrapping a promise in a promise and using try/catch in a way that won't work:
getDirectories: async function(path) {
return new Promise(function (resolve, reject) {
try {
resolve(scanDirectories(path));
} catch {
reject('Error');
}
});
}
Just call scanDirectories directly from your original event handler, and make the event handler an async function, so a lot of code just goes away.
In general: if you have to deal with async stuff, write an async function, and always await it in the function that calls it, even if that function is itself. You may write async functions anywhere, even if they are event handlers or Express routes or other contexts where the promise they resolve to won't be consumed.
Here is your original code simplified and corrected but working basically the same way:
ipcMain.on('fileTree', async (event, arg) => {
try {
event.reply('fileTree', await helperFile.scanDirectories(arg);
} catch (e) {
console.log(e);
}
});
// files.js
const { readdir } = require('fs').promises;
const resolvePath = require('path').resolve;
module.exports = {
scanDirectories: async function(path) {
const dirents = await readdir(path, { withFileTypes: true });
const files = [];
for (const dirent of dirents) {
const res = resolvePath(path, dirent.name);
if (dirent.isDirectory()) {
files = files.concat(await scanDirectories(res));
} else {
files.push(res);
}
});
return files;
}
}

(async) JSON RPC API performance optimization

I have to make a JSON RPC API that will have to support a big traffic and manage a postgreSQL database.
To do it I chose 'http' for the server and pg-promise for the database.
My problem is that I have some difficulty understanding and using the promises and async/wait, so I'm not sure I did it correctly
I put some code below
What I did
./server/server.js create an http server with requestHandler() as request handler. It does some checks and then calls async requestProcessor() to execute the method
The methods are defined in the repos (here a transaction in devices.js) as async and in my example below use await to wait for the required results
Some questions :
I have to define as async only the methods that should use await ?
In my SystemRepository, do I need to define 'InsertOneSystem' as async ?
How can I do a simple test script to test the load ? Like requests per second, ... ?
Thanks in advance !
A bit of code
server.js
const http = require('http');
const Database = require('../db');
const path = '/api/v1', port = 9000;
const methods = Database.methods;
/* hidden for brevity */
function sendResponse(res, response) {
if (response) {
const responseStr = JSON.stringify(response);
res.setHeader('Content-Type', 'application/json');
res.setHeader('Content-Length', responseStr.length);
res.write(responseStr);
} else {
/* hidden for brevity */
}
res.end();
}
const requestHandler = (req, res) => {
/* some checks, hidden for brevity */
const body = [];
req.on('data', (chunk) => {
body.push(chunk);
}).on('end', () => {
const bodyStr = Buffer.concat(body).toString();
// parse body en JSON
let request = JSON.parse(bodyStr);
requestProcessor(request).then((response) => {
sendResponse(res, response);
});
});
}
async function requestProcessor(request) {
let response = {
id: request.id,
jsonrpc: '2.0',
};
try {
response.result = await Promise.resolve(methods[request.method](request.params));
} catch (err) {
/* hidden for brevity */
}
return response;
}
const server = http.createServer(requestHandler);
server.listen(port, (err) => { /* hidden for brevity */ });
devices.js
'use strict';
/* hidden for brevity */
async function InsertOne(params) {
return Database.tx('Insert-New-Device', async function(transaction) {
let system = null, disks = null, cpus = null;
const query = pgp.helpers.insert(params.data.device, Collections.insert) + " RETURNING *";
let device = await transaction.one(query);
// if a system is present, insert with diviceId and return
if(params.data.system) {
params.data.system.deviceid = device.deviceid;
system = transaction.systems.InsertOne(params);
}
// same as system
if(params.data.disks) {
params.data.disks.deviceid = device.deviceid;
disks = transaction.disks.InsertOne(params);
}
// same as system
if(params.data.cpus) {
params.data.cpus.deviceid = device.deviceid;
cpus = transaction.cpus.InsertOne(params);
}
return {
device: device,
system: await system,
disks: await disks,
cpus: await cpus
}
})
.then(data => {
return data;
})
.catch(ex => {
console.log(ex)
throw new Error(ex);
});
}
/* hidden for brevity */
const DevicesRepository = {
InsertOne: InsertOne
};
module.exports = (db, pgpLib) => {
/* hidden for brevity */
return DevicesRepository;
}
systems.js
'use strict';
/* hidden for brevity */
async function InsertOneSystem(params) {
var system = params.data.system;
system.archid=2;
system.distributionid=3;
var query = pgp.helpers.insert(system, Collections.insert);
if(params.return) query += " RETURNING *";
return Database.one(query)
.then(data => {
return data;
})
.catch(ex => {
throw new Error(ex);
});
}
/* hidden for brevity */
const SystemsRepository = {
InsertOne: InsertOneSystem
};
module.exports = (db, pgpLib) => {
/* hidden for brevity */
return SystemsRepository;
}
I have to define as async only the methods that should use await ?
Have to - yes. But you should use async on all methods that return a promise, it's just a nice coding style, especially in TypeScript.
In my SystemRepository, do I need to define InsertOneSystem as async ?
You don't have to, but the same as above, it's a good coding style ;)
How can I do a simple test script to test the load ? Like requests per second, ... ?
I'm not answering that right now, as it is a whole separate area that deserves a separate question. You should investigate it yourself, how to test HTTP services load.
A little code improving, as you have plenty of redundancies:
async function InsertOne(params) {
return Database.tx('Insert-New-Device', async t => {
let system = null, disks = null, cpus = null;
const query = pgp.helpers.insert(params.data.device, Collections.insert) + " RETURNING *";
let device = await t.one(query);
// if a system is present, insert with diviceId and return
if(params.data.system) {
params.data.system.deviceid = device.deviceid;
system = await t.systems.InsertOne(params);
}
// same as system
if(params.data.disks) {
params.data.disks.deviceid = device.deviceid;
disks = await t.disks.InsertOne(params);
}
// same as system
if(params.data.cpus) {
params.data.cpus.deviceid = device.deviceid;
cpus = await t.cpus.InsertOne(params);
}
return {device, system, disks, cpus};
})
.catch(ex => {
console.log(ex); // it is better use "pg-monitor", or handle events globally
throw ex;
});
}

Resources