How to execute queries during a stream using pg-promise? - node.js

Question

I'm trying to execute an insert query for each row of a query stream using pg-promise with pg-query-stream. With the approach I have, memory usage increases with each query executed.
I've also narrowed the problem down to just executing any query during the stream, not just inserts. I currently listen for 'data' events on the stream, pause the stream, execute a query, and resume the stream. I've also tried piping the query stream into a writeable stream that executes the query, but I get the error that the db connection is already closed.
let count = 0;
const startTime = new Date();
const qs = new QueryStream('SELECT 1 FROM GENERATE_SERIES(1, 1000000)');
db.stream(qs, stream => {
stream.on('data', async () => {
count++;
stream.pause();
await db.one('SELECT 1');
if (count % 10000 === 0) {
const duration = Math.round((new Date() - startTime) / 1000);
const mb = Math.round(process.memoryUsage().heapUsed/1024/1024);
console.log(`row ${count}, ${mb}MB, ${duration} seconds`);
}
stream.resume();
});
});
I expected the memory usage to hover around a constant value, but the output looks like the following:
row 10000, 105MB, 4 seconds
row 20000, 191MB, 6 seconds
row 30000, 278MB, 9 seconds
row 40000, 370MB, 10 seconds
row 50000, 458MB, 14 seconds
It takes over 10 minutes to reach row 60000.
UPDATE:
I edited the code above to include async/await to wait for the inner query to finish and I increased the series to 10,000,000. I ran the node process with 512MB of memory and the program slows significantly when approaching that limit but doesn't crash. This problem occurred with v10 and not v11+ of node.

Answer 1

This is due to invalid use of promises / asynchronous code.
Line db.one('SELECT 1'); isn't chained to anything, spawning loose promises at a fast rate, which in turn pollutes memory.
You need to chain it either with .then.catch or with await.