Error when do Promise.all (close connection / hang up) - node.js

Question

I have working with google-spreadsheet#2.0.7 package, I have many data to be exported to Google Sheet,
for now I use this code
const insertDataToSheet = async (data, sheet, msg) => {
let query = []
try {
data.map(async item => {
query.push(promisify(sheet.addRow)(item))
})
const result = await Promise.all(query)
if (result) return result
throw new Error(`${msg} Unkown Error`)
} catch (e) {
throw new Error(`${msg} Failed: ${e.message}`)
}
}
This code is working with 100 data or less, but if I use 150+ data the connection not support it.
Error List
- Client network socket disconnected before secure TLS connection was established
- Socket hang up
- Error: HTTP error 429 (Too Many Requests)
Is there any limitation for Promise.all.?
or
Is there any better solution to export batch / bulk data to Google
Spreadsheet?

Answer 1

Promise.all will throw if one of the promise throw. If you want to proceed even if one promise fails, you do not want to rethrow it as in your code above.
you can re-add it to pending queue and try it again.
also, i may consider batching it. divide them into chunks and upload it.
example:
create a pool of worker (number of work = number of cpu cores (default))
run uploading logic with the worker pool
simulate error / retry with Math.random
process.js file
const path = require('path')
const _ = require('lodash')
const Pool = require('piscina')
const BB = require('bluebird')
const workerPool = new Pool({
filename: path.resolve(__dirname, 'worker.js'),
})
const generateData = (numItems = 5) => {
return Array.from({ length: numItems }, (v, idx) => 'item ' + idx)
}
const CHUNK_SIZE = 10
const data = generateData(100)
const chunks = _.chunk(data, CHUNK_SIZE)
BB.map(
chunks,
(chunk) => {
workerPool.runTask(chunk)
},
{ concurrency: 1 /* 1 chunk at a time */ }
)
worker.js file
const retry = require('p-retry')
// your upload logic here
function process(data) {
if (Math.random() > 0.5) {
console.log('processing ', data)
} else {
console.log('fail => retry ', data)
throw new Error('process failed' + data)
}
}
module.exports = (data) => {
return retry(() => process(data), { retries: 10 })
}
run with node process.js

Answer 2

In final i work on this, and find out there is new version of the package google-spreadsheet#3.0.11.
it's change from Google Drive API to Google Sheets API.
It has many changes, but in my case now I can Batch / Bulk insert just with single line.
this is my code now.
const insertDataToSheet = async (data, sheet, msg) => {
try {
const result = await sheet.addRows(data)
if (result) return result
throw new Error(`${msg} Unkown Error`)
} catch (e) {
throw new Error(`${msg} Failed: ${e.message}`)
}
}
I just use sheet.addRows and tada it's working.
My Problem is solved, but with promise I still need to learn,
Thanks for all of your suggestion / attention.