Send batches to web API - node.js

Question

I have a mongodb and NodeJS setup on expressJS. What this API basically does is storing e-mail adresses and other information about users.
These are called personas and are stored in a MongoDB database. What I'm trying to do now is calling a url in my app, which sends all personas to the Mailchimp API.
However, as the amount of personas that are stored is quite high (144.000), I can not send them in one batch to the Mailchimp API. What I'm trying to do is send them in batches, without much luck.
How would I go about to set this up? Currently I'm using the Async package to limit the simultaneous sends to the Mailchimp API. But I'm not sure if this is the correct way to go.
I guess the code below is not working, as the personas-array I collect is too big to fit in the memory. But I'm not sure how to chunk it up in a correct way.
//This is a model function which searches the database to collect all personas
Persona.getAllSubscriptions(function(err, personas) {
//Loop send each persona to mailchimp
var i = 1;
//This is the async module I'm using to limit the simultaneous requests to Mailchimp
async.forEachLimit(personas, 10, function (persona, callback) {
//This is the function to send one item to mailchimp
mailchimpHelper.sendToMailchimp(persona, mailchimpMergefields, function(err,body){
if(err) {
callback(err);
} else if(!body) {
callback(new Error("No response from Mailchimp"));
} else {
console.log(i);
i++;
callback();
}
});
}, function(err) {
if (err) console.log(err);
//Set a success message
res.json({error: false, message: "All personas updated"});
});
});

Answer 1

I ran into a similar problem with a query to a collection that could return more than 170,000 documents. I ended up using the "stream" API to build batches to be processed. You could do something similar to "build" batches to send to MailChimp.
Here's an example.
var stream = db.collection.find().stream(); //be sure find is returning a cursor
var batch = []
this.stream.on('data', function(data){
batch.push(data);
if(batch.length >= maxBatchSize){
stream.pause();
// send batch to mail chimp
}
});
this.stream.on('pause', function(){
// send batch to mailChimp
// when mailChimp has finished
stream.resume();
});
this.stream.on('end', ()=>{
// data finished
});
You can look at the documentation for cursor and stream here
Hope this helps.
Cheers.

Answer 2

There seem to be some things that I wouldn't do so like you described. You are trying to quite heavy processing inside the node server. The trigger by url could cause you a lot of problems if you do not secure it.
Also, this is a heavy process which is better to be implemented as queue-worker approach separated from the server. This would give you more control over the process, some of the email sendings might fail or error might occur on the mailchimp side(API is down etc). So instead of triggering directly sending, just trigger worker and process emails as chunks as #jackfrster described.
Make sure you have checked the Mailchimp API limits. Do you have considered alternatives like creating campaign and send out the campaign so you would not need to sending for each person in list ?