I am developing an email sending service, probably for sending bulk emails using sendgrid web API, but I am not able to figure out best practice for scalable system. I wish to keep a record of all those emails which failed to deliver and retry sending to those failed emails after all emails have been sent. I am using NodeJs, so just wanted to know if there is any way to speed up my process(something like sending multiple emails at the same time)
There are multiple ways to handle this, I will suggest two which seems obvious to me.
(Recommended - Easy) Use Async module's control flow option called queue Async Documentation. You can feed in all the request in form of an array of object request and then change concurrency setting to let's say 100, it'll run concurrent 100 workers at one time and to log errors make a separate mechanism and once all the values have been run through handle it separately.
Spawn multiple workers using node.js native approach.
Sendgrid offers an npm package for node.js integration, so you don't have to reinvent the wheel. It accepts messages at a high rate, so you shouldn't have problems delivering yours to sendgrid. You just dump your messages into sendgrid.
Email, being a store-and-forward system, is inherently asynchronous. That means it operates far from real time. Some messages are delivered in a few seconds, and others take hours (when they get soft--"retry later"--rejections from destination servers, for example).
Sendgrid handles this issue with a "bounces" API. (And with "bounces" features in their web back end application). Many bounces are "hard" bounces, meaning you must avoid trying to send messages to that address again. You can use the bounces API to retrieve a list of bounced messages. You should remove those addresses from your email list, and not try to send them again. (Sendgrid bans users who repeatedly send mailings with a high undeliverable rate.)
They also have an "invalid emails" API. This works like "bounces" and returns lists of addresses that are ill-formed or, if sendgrid can tell, not present on the destination server. Again, you should remove these addresses from your email list. If they're invalid now, they will be invalid tomorrow.
Sendgrid offers all sorts of tutorials on this subject.
Related
can I send an email through a PHP mailer with my localhost without using Gmail? or any other email service like Hotmail, outlook using my localhost?
Yes, and this is actually the best way to send using PHPMailer. There are two things you need:
A hosting provider that permits outbound SMTP. A large number of providers block SMTP by default, but you may find that they will remove the block on request (scaleway.com), provide an alternative service (AWS), which may also be acceptable.
A locally installed mail server. If you're on Linux, something like postfix is a good choice.
The way it then works is this:
Write your PHPMailer script as normal, but delivering to localhost over SMTP (not via mail())
Configure your mail server to act as a "full" mail server, where it attempts to deliver messages directly, or to act as a "smarthost" to relay through some other service, such as gmail or mailgun.
The main advantages of this approach are:
Perceived performance: your message submissions will appear to be instant, so it's great for things like contact forms
Reliability: if your message can't be delivered immediately, the local mail server will automatically take care of queuing and redelivery attempts, in a way that is far more efficient than you would ever achieve if you handled that within your own application.
As with many things, with great power comes great responsibility, and you will have to deal with bounce handling, blacklisting, inbound filtering (or just block it), etc. It's probably not worth doing all this for a simple contact form, but if you have an application that generates a reasonable amount of email traffic (signups, password resets, notifications, etc), it's a great way to go.
I'm able to successfully load test my bot server by getting the proper auth token from Microsofts auth URL (basically through this page)
I was wondering if this was a valid test on the service considering that we're not actually hitting the bot frameworks endpoint (which has rate limiting)
Is there another way to load test a bot service wherein i can replicate the bot frameworks throttling/rate limits?
I ended up with using load test with Visual Studio and Visual Studio Team Services.
The reason why I used this approach is that you can setup full path of load tests. Azure Bot Service can be either Web App or Function App with endpoint prepared for receiving messages - using HTTP POST so in the end is just web service.
You can setup load tests for different endpoints including number of hits to selected endpoint. In case of Bots you can for instance setup test with 100 fake messages sent to the bot to see the performance.
You can read more under these two links below:
Load test your app in the cloud using Visual Studio and VSTS
Quickstart: Create a load test project
Unfortunately as stated in the documentation you linked, the rates are not publicly available due to how often they are adjusted.
Regarding user-side throttling- this should not actually have an effect either way as long as you simulate reasonable traffic, but even if you go a bit overboard, an individual user hitting rate-limiting would be functionally equivalent to just having a bit more traffic. The single user sending more messages to the bot is the same as three users sending the same amount of messages slightly slower and there's no limit for your bot in terms of how many customers you might have. That said, a user getting a message, reading it, and typing up a response should not put themselves into a situation where they are rate-limited.
However, regarding bot side throttling it is useful to know if your bot is sending messages too fast for the system. If you are only ever replying directly to messages from users, this will not be an issue, as the system is built with replying to each user message in mind. The only area you might run into trouble is if you are sending additional (or unsolicited) messages, however even here as long as you are within reasonable limits you should be OK. (i.e. if you aren't sending several messages back to a user as fast as possible for each message they send you, you will probably not have problems.) You can set a threshold for bot replies within your channel at some reasonable-sounding limit to test this.
If you would like to see how your bot responds in cases where throttling is occurring (and not necessarily forcing it into tripping the throttling threshold), consider setting your custom channel to send 429 errors to your bot every so often so that it has to retry sending the message.
I am developing a node.js app. In my app, I need to send blast emails and SMS to users satisfying a particular criteria. I use Gmail SMTP for emails and a third party vendor for SMS. I'm assuming firing the API's for email and sms services in a loop is dangerous. What's the right way do it?
The time spent is obviously proportional to N, being N the size of your set. As much users you have, longer it takes. Keep in mind that requests over the network are not blocking in any case.
Anyway, unless it risks N to be thousands or millions of items, you can do it in a loop and attach a proper callback to handle responses/errors.
Otherwise, you can send an email/sms and schedule the same operation for the next element using nextTick (see nodejs documentation for further details).
This way you'll spread all the activities over several iterations of the event loop.
I'm relatively new to nodejs and was wondering what the best approach would be in setting up a automated job between set hours (lets say 8am-10pm every hour dependent on the customers timezone), to perform a task then based on results send users emails?
The task you describe is not so simple at first:
You need to have some queue implementation, if you want something simple you can use Kue.js which uses Redis for storing the jobs. For more scaled solution would be using ZMQ or RabbitMQ.
The job processor, is the part where you have to interact with database and give you out the list of emails. Depending on the size of your email list, it's better if it could be splitted into chunks like 200 emails and then create "child-jobs"
Email sending, there are different providers and possible solutions. First your own SMTP server, Amazon SES, Mailchimp and other similar services. For the sending library there is already great, Nodemailer.
This is not the full concept, but should give you the idea. I am sorry if this is too abstract concept, but the question is not specific either.
I'd like to implement push notification server using node.js. The basic scenario is:
Some applications sends notification messages to the server.
Notification server receives the request and forwards the message to uesr's mail or IM client based on user's preference.
In step 1, which protocol (e.g. REST, socket, HTTP/XML and so on.) would you recommend from the performance perspective?
Also in step 2, I have a plan to use node-xmpp module for IM client but for mail, which way is the best to implement? For example,
Just use SMTP. (But I think this might occur performance degradation because SMTP is an expensive communication and performance depends on SMTP server capacity.
use queue mechanism, in order to avoid drawbacks from the above. node.js app simply puts the message into the queue, and smtp server pulls the message.
other solutions...
Thanks in advance.
With regards to what to use as a protocol, i would go for a REST interface, whereby the application posting sends a POST request to a resource associated with the USER. something along the lines of "http://example.com/rest/v1/{userID}/notifications
I personally would use json as the data/content of the rest request and have node.js write this information to a message queue. (as a json string).
You can than have xmpp readers for each user, as well as an SMTP handler reading from this queue as fast as the SMTP server allows it to go.
However, this full post is what i would do in your situation, rather than a factual response on what is best. I know JMS fairly well and i've been working a lot with rest interfaces lately, therefore this is the way i would do it.