How to properly throttle message sending with nodemailer SES transport? - node.js

The nodemailer documentation says:
If you use rate or connection limiting then you can also use helper
methods to detect if the sending queue is full or not. This would help
to avoid buffering up too many messages.
It also provides an example:
let transporter = nodemailer.createTransport({
SES: new aws.SES({
apiVersion: '2010-12-01'
}),
sendingRate: 1 // max 1 messages/second
});
// Push next messages to Nodemailer
transporter.on('idle', () => {
while (transporter.isIdle()) {
transporter.sendMail(...);
}
});
Unfortunately, this is rather cryptic to me. Does sendingRate: 1 only provides a helper, or does it handle throttling ?
Also this piece of code looks to me like it would loop infinitely as soon as sendMail(...) is executed. Am I missing something here ?
Is there any example or recommendation on how to use this feature ?
Thanks a lot !

From the documentation:
SES can tolerate short spikes but you can’t really flush all your emails at once and expect these to be delivered. To overcome this you can set a rate limiting value and let Nodemailer handle everything – if too many messages are being delivered then Nodemailer buffers these until there is an opportunity to do the actual delivery.
I don't think listening for the idle event is mandatory, it's only needed if you want to avoid Nodemailer buffering messages. I have an SES send rate of 15 messages per second and regularly throw 250 emails at once at Nodemailer and don't hit any throttling issues.

You are right, the while loop only appears to be there for testing sending rate. Once you remove the while loop the code in documentation should work fine.
transporter.on('idle', () => {
transporter.sendMail(...);
});

You don't need the while loop or the on idle handler. Just set the sendingRate and then use sendMail as normal.
transporter = nodemailer.createTransport({
SES: { ses, aws },
sendingRate: 14,
});
const params = {
from: 'EMAIL',
to: 'EMAIL',
subject: 'Message',
html: 'I hope this <b>message</b> gets sent!',
text: 'I hope this message gets sent!',
// attachments: [{ filename: 'card.pdf', content: data, contentType: 'application/pdf' }],
};
transporter.sendMail(params, (err, info) => {
if (err) {
console.log(JSON.stringify(err));
}
console.log(info.envelope);
console.log(info.messageId);
});
Important thing to note here, nodemailer waits for the next second to continue with the next batch of throttled emails and the next and so on. So if you are running a script that exits immediately after calling the last sendMail(), the throttled emails will never get sent. Make sure the process runs until all emails are sent by listening to on idle or use settimeout.

Related

Send a lot of SQS messages in a lambda without getting timed out

I have an array, lets say with a length of 9999. Now I want to send a message to SQS with the contents of each object in the array.
Here is how I want to send the messages:
const promises = tokens.map((token) => {
const params = {
Body: JSON.stringify(data),
};
return sqs.sendMessage({
MessageBody: JSON.stringify(params),
QueueUrl: 'my-queue-url',
}).promise();
});
await Promise.all(promises);
Now, the problem is that I trigger this function via API Gateway, which times out after 30 seconds. How can I avoid this? Is there a better way to do this?
Based on your response in the comments I suggest you do two things:
Use the SendMessageBatch API to reduce the number of API calls and speed up the process as #Marc B suggests
Set up asynchronous invocation of the backend Lambda function. This means the API Gateway will just send the Event to your Lambda and not wait for the response. That means the Lambda functions is free to run up to 15 minutes. If you have more messages than can be handled in that period of time, you may want to look into StepFunctions.

Node.js/NodeMailer/Express/Outlook smtp host - Concurrent connections limit exceeded

Hope you are well. I am in the middle of working on an application that uses express and nodemailer. My application sends emails successfully, but the issue is, is that I cannot send the emails off one at a time in a manner that I'd like. I do not want to put an array of addresses into the 'to' field, I'd like each e-mail out individually.
I have succeeded in this, however there is an issue. It seems Microsoft has written some kind of limit that prevents applications from having more than a certain number of connections at a time. (link with explanation at end of document of post.)
I have tried to get around this by a number of expedients. Not all of which I'll trouble you with. The majority of them involve setInterval() and either map or forEach. I do not intend to send all that many e-mails - certainly not any to flirt with any kind of standard. I do not even want any HTML in my emails. Just plain text. When my application sends out 2/3 e-mails however, I encounter their error message (response code 432).
Below you will see my code.
As you can see, I'm at the point where I've even been willing to try adding my incrementer into setInterval, as if changing the interval the e-mails fire at will actually help.
Right now, this is sending out some e-mails, but I'm eventually encountering that block. This usually happens around 2/3 e-mails. It is strangely inconsistent however.
This is the first relevant section of my code.
db.query(sqlEmailGetQuery, param)
.then(result => {
handleEmail(result, response);
}).catch(error => {
console.error(error);
response.status(500).json({ error: 'an unexpected error occured.' });
});
});
This is the second section of it.
function handleEmail(result, response) {
const email = result.rows[0];
let i = 0;
email.json_agg.map(contact => {
const msg = {
from: process.env.EMAIL_USER,
to: email.json_agg[i].email,
subject: email.subject,
text: email.emailBody + ' ' + i
};
i++;
return new Promise((resolve, reject) => {
setInterval(() => {
transporter.sendMail(msg, function (error, info) {
if (error) {
return console.log(error);
} else {
response.status(200).json(msg);
transporter.close();
}
});
}, 5000 + i);
});
});
}
I originally tried a simple for loop over the contacts iterable email.json_agg[i].email, but obviously as soon as I hit the connection limit this stopped working.
I have come onto stackoverflow and reviewed questions that are similar in nature. For example, this question was close to being similar, but this guy has over 8000 connections and if you read the rule I posted by microsoft below, they implemented the connection rule after he made that post.
I have tried setInterval with forEach and an await as it connects with each promise, but as this was not the source of the issue, this did not work either.
I have tried similar code to what you see above, except I have set the interval to as long as 20 seconds.
As my understanding of the issue has grown, I can see that I either have to figure out a way to wait long enough so I can send another e-mail - without the connection timing out or break off the connections every time I send an e-mail, so that when I send the next e-mail I have a fresh connection. It seems to me that if the latter were possible though, everyone would be doing it and violating Microsofts policy.
Is there a way for me to get around this issue, and send 3 emails every say 3 seconds, and then wait, and send another three? The volume of e-mails is such that, I can wait ten seconds if necessary. Is there a different smtp host that is less restrictive?
Please let me know your thoughts. My transport config is below if that helps.
const transporter = nodemailer.createTransport({
pool: true,
host: 'smtp-mail.outlook.com',
secureConnection: false,
maxConnections: 1,
port: 587,
secure: false,
tls: { ciphers: 'SSLv3' },
auth: {
user: process.env.EMAIL_USER,
pass: process.env.EMAIL_PASS
}
});
https://learn.microsoft.com/en-us/exchange/troubleshoot/send-emails/smtp-submission-improvements#new-throttling-limit-for-concurrent-connections-that-submitmessages
First off, the most efficient way to send the same email to lots of users is to send it to yourself and BCC all the recipients. This will let you send one email to the SMTP server and then it will distribute that email to all the recipients with no recipient being able to see the email address of any individual recipient.
Second, you cannot use timers to reliably control how many requests are running at once because timers are not connected to how long a given requests takes to complete so timers are just a guess at an average time for a request and they may work in some conditions and not work when things are responding slower. Instead, you have to actually use the completion of one request to know its OK to send the next.
If you still want to send separate emails and send your emails serially, one after the other to avoid having too many in process at a time, you can do something like this:
async function handleEmail(result) {
const email = result.rows[0];
for (let [i, contact] of email.json_agg.entries()) {
const msg = {
from: process.env.EMAIL_USER,
to: contact.email,
subject: email.subject,
text: email.emailBody + ' ' + i
};
await transporter.sendMail(msg);
}
}
If you don't pass transporter.sendMail() the callback, then it will return a promise that you can use directly - no need to wrap it in your own promise.
Note, this code does not send a response to your http request as that should be the responsibility of the calling code and your previous code was trying to send a response for each of the emails when you can only send one response and the previous code was not sending any response if there was an error.
This code relies on the returned promise back to the caller to communicate whether it was successful or encountered an error and the caller can then decide what to do with that situation.
You also probably shouldn't pass result to this function, but should instead just pass email since there's no reason for this code to know it has to reach into some database query result to get the value it needs. That should be the responsibility of the caller. Then, this function is much more generic.
If, instead of sending one email at a time, you want to instead send N emails at a time, you can use something like mapConcurrent() to do that. It iterates an array and keeps a max of N requests in flight at the same time.

Messages order of smooch - whatsapp

I have a bot and I use smooch to run the bot on whatsapp.
I use 'smooch-core' npm for that.
When I send a lot of messages one after the other sometimes the messages are displayed in reverse order in whatsapp.
Here is the code for sending messages:
for (const dataMessage of data) {
await sendMessage(dataMessage);
}
function sendMessage(dataMessage) {
return new Promise((resolve, reject) => {
smoochClient.appUsers.sendMessage({
appId: xxxx,
userId: userId,
message: dataMessage
}).then((response) => {
console.log('response: ' + JSON.stringify(response), 'green');
resolve();
}).catch(err => {
console.log('error: ' + JSON.stringify(err), 'red');
reject(err);
});
});
All dataMessage looks like this:
{
role: "appMaker",
type: "text",
text: txt
}
I tried to see how I could arrange it, and I saw that there was an option to get the message status from the webhook, and then wait for each message to arrive for the appropriate status. And only then send the following message.
But I would like to know is there something simpler? Is there a parameter that can add to the message itself to say what its order is? Or is there something in the npm that gives information about the message and its status?
In the doc below, Whatsapp mentions that they do not guarantee message ordering.
https://developers.facebook.com/docs/whatsapp/faq#faq_173242556636267
The same limitation applies to any async messaging platform (and most of them are), so backend processing times and other random factors can impact individual message processing/delivery times and thus ordering on the user device (e.g. backend congestion, attachments, message size, etc.).
You can try to add a small [type-dependent] delay between sending each message to reduce the frequency of mis-ordered messages (longer delay for messages with attachments, etc.).
The fool-proof way (with much more complexity) is to queue messages by appUser on your end, only sending the next message after receiving the message:delivery:user webhook event for the previous message.

Twilio Function to send VoiceMail or Recording to Email

Twilio Stuido does not provide a native way to email a recording therefore this needs to be coded. In this example, a voicemail has been taken of a users call. Now that we have the recording a Twilio Function is created (NodeJS) to send this recording to email.
In the sample code below the function is failing however there is no available debug tool inside the Twilio Console for dealing with functions. NodeJS is fairly new to us so this maybe easy for someone to spot.
Possible other errors may be however:
Incorrect gmail authentication (although the details are correct in terms of username and password, perhaps this is not how you configure google apps to send the email)
Inability to import or incorrectly importation of the nodemailer dependency into Twilio
Bad NodeJS skills
The input variables are:
attachment
{{widgets.Voicemail_Recording.RecordingUrl}}
- which contains the URL to the voicemail recording.
lang
- the language of the caller (based on IVR choices earlier).
phone_number
{{trigger.call.From}}
NodeJS Twilio Function
var mailer = require('nodemailer');
mailer.SMTP = {
host: 'smtp.gmail.com',
port:587,
use_authentication: true,
user: 'info#example.com',
pass: '*********'
};
exports.handler = function(context, event, callback) {
mailer.send_mail({
sender: event.phone_number + '<info#example.com>',
to: 'info#example.com',
subject: 'Voicemail (' + event.lang + ')',
body: 'mail content...',
attachments: [{'filename': event.attachment, 'content': data}]
}), function(err, success) {
if (err) {
}
}
};
Sam here from Twilio's support team - I just released a blog post that should help with this. It shows how to forward voicemails with Studio, Functions, and SendGrid. Check it out here: https://www.twilio.com/blog/forward-voicemail-recordings-to-email

Waiting for email to arrive using node-imap

I'm using node-imap as a mail solution, but I need a way of waiting until an email arrives. In this post, someone referenced using the IMAP IDLE command to do this. I was wondering if anybody has had success with this, and how you would suggest incorporating that into the example code provided in the node-imap readme?
I decided to go with the inbox module. This provides a clear and quick solution through use of the call, client.on("new", function(message){.
I think good starting point is to research how is this method created in https://www.npmjs.com/package/inbox#wait-for-new-messages module.
Looks like this code emits the event of new.
As far as i understood the code of this module, they call the fetch command with intervals
For node-imap, "mail" event will be emitted when new mail arrives in the currently open mailbox.
You can listen to new mail event like this:
imap.on('mail', function(numNewMsgs) {
// Fetch new mail
});
You need to open a mailbox, which you want to listen for new messages. Here an example:
function listerBox() {
imap.once("error", console.error);
imap.on("ready", () => {
imap.openBox("INBOX", true, (error, box) => {
if (error) throw error;
console.log('Connected!')
imap.on("mail", () => {
console.log("New one!")
})
});
});
imap.connect();
}
listerBox()

Resources