I have to send email to 10 users in my app but I have to send them email separately . I am using loopback framework and for sending email and rsmq library
I have two approaches in mind for sending email
Approach 1
I should pass an array of emails to one message in the queue producer and in the queue listener I should iterate that array of email and send email one by one
Approach 2
I should pass separate message in the queue producer for every user to send email
Which approach is better and why ?
In Message Queuing, it is good to send one processable entry as one message to queue.
Reason Why:
In case you split the email sender into muliple functions, say for example after sending email, you need to update some log, update email count or anything (other example may suit well to explain), then every email needs to be proceeded independently by different functions.
Reason Why Not:
In case of batch processing, you may need to maintain no of emails per message and it exceed you need to write logic to split into batchee
Failure in processing of one message in middle may fail sending the rest of emails in same message.
Related
I'm trying to get some information from the twilio queue. And in order to get that information i have to send requests every time i want to know whether or not the queue is full. Is there a way to kind of "watch" with the help of sockets to see if the queue is full or not?
Here is a blog post to derive the message queue based on message time stamps.
How to Calculate Your Message Queue Length
I'm using google's latest python libraries to send/receive e-mails to self (i.e., to and from fields are the same as the user account being used to send) to collect network performance metrics.
When I send a message, a 'msgId' is created which can be used to track the message that was sent. I can then subsequently delete (or trash) that 'msgId'. However, from time to time, AFTER I delete the msgId, the same message seems to "arrive" but is given a different msgId. If I wait sufficiently long enough period (10-15seconds) between sending and deleting, I only ever see the same msgId as returned by the user.message.create(), the delete is successful and only ever see the single msgId.
I believe there may be some kind of race condition whereby when sending within GMAIL (or perhaps only when sending to self), it uses the same message instance for both the sent and received message...and if the sent message is deleted before it is actually received, a new msgId is created for the received msg.
Question: Is there a way to determine whether a sent message (to self) has been received and can be deleted without risk of the message being received later and a new msgId generated?
I've tried moving the sent message to TRASH and then poll the message to receive the label for 'INBOX' without avail.
Alert is originally generated by external system and it is routed to Twilio (incoming SMS message). Twilio provides smart, prioritized SMS alerting for group of receivers. I have used Twilio Studio to implement the basic flow.
In the beginning, there is only a few numbers on receiver list and each of them is receiving an SMS, that is send by Twilio flow (I have used Twilio functions to send these outbound SMS messages) . If any receiver accepts to handle alert root cause (by pressing 1) the system logs the data, distributes that information to other on receiver list and stops repeating alerts. If receiver skips the alert (by pressing 2), the system logs the information and does not send any more alerts for this specific receiver. If there is no reply at all or if answering receivers have only selected 2 (skipping), the system increases new numbers to group of receivers and repeats the alert distribution (except numbers that have skipped). This is done a few times in configurable interval e.g. 10mins or until alert is accepted by some of the receivers. Group of receivers may be increased in all rounds. In the end, summary message is sent to all receivers (e.g. alert not accepted/number XXX accepted).
So my problems are 1.) how to collect answers (accept/skipped) from received SMS messages and 2.) how to implement that 10min pause to Twilio flow.
For problems, I'm thinking of following options:
By using Twilio Functions call from Studio flow, read all inbound message resources from Programmable SMS API. "inbound messages received during last hour, to my TwilioNumber, and from receiver that belong to list of receivers" and figure out then message by message, what belong to this flow? Is there anywhere an example how that is done?
Maybe it would be possible to pass each outbound SMS message through REST API back to Studio Flow and use "Send and Wait for Reply" widget - If this would be possible it would solve also the next problem pausing/delaying the studio flow.
How to stop/delay studio flow execution after each SMS group message has been send, e.g. for 10 minutes (or until someone accepts the alert by replying with "1"). Can I do it inside Studio flow, is there pause/wait function somewhere. Or should the delay be implemented otherwise. Any examples, how it is done.
Thanks in advance
Quite new to RabbitMQ and I'm trying to see if I can achieve what I need with it.
I am looking for the Worker Queues pattern but with one caveat. I want to have only a single worker running concurrently per routing key.
An example for clarification:
If i send the following messages with routing keys by order: a, a, b, c, I want to have only 3 workers running concurrently. When the first a message is received a worker picks it up and handles it.
When the next a message is received and the previous a message is still handled (not acknowledged) the new a message should wait in queue. When the b and c messages are received they each get a worker handling them. When the first a message is acknowledged any worker can pick up the next a message.
Would that pattern be possible using RabbitMQ in a natural way (without writing any application code on my side to handle the locking and stuff...)
Edit:
Another clarification. All workers can and should handle all messages, and I don't want to have a queue per Worker as I want to share the load between them, and the Publisher doesn't know which Worker should process the message. But I do want to make sure that no 2 Workers are working on messages sharing the same key at the same time.
For example, if I have a Publisher publishing messages with a userId field, I want to make sure no 2 Workers are handling messages with the same userId at the same time.
Edit 2
Expanding on the userId example. Let's say I have a single Publisher and 3 Workers. The publisher publishes messages like these: { userId: 1, text: 'Hello' }, with varying userIds. My 3 Workers all do the same thing to this messages, so I can have any of them handle the messages coming in. But what I'm trying to achieve is to have only a single worker processing a message from a certain user at the same time. If a Worker has received a message with userId 1 and is still processing it, and another message with userId 1 is received I want to make sure no other Worker picks up that message. But other messages coming in with different userIds should be processed by other available Workers.
userIds are not known beforehand, and the publisher doesn't know how many workers are or anything specific about them, he just wants to schedule the messages for processing.
what your asking is not possible with routing keys, but is built into queues with a few settings.
if you define "queue_a" for a messages, "queue_b" for b messages, etc, you can then have as many consumers connect to it as you want.
RabbitMQ will only deliver a given message to a single consumer of a given queue.
The way it works with multiple consumers on a single queue is basic round-robin style dispatch of the messages. that is, the first message will be delivered to one of the consumers, and the next message (assuming the first consumer is still busy) will be delivered to the next consumer.
So, that should satisfy the need to deliver the message to any given consumer of the queue.
To ensure your messages have an equal chance of getting to any of the consumer (and are not all delivered to the same consumer all the time), there are a few other settings you should put in place.
First, make sure to set the message consumer no ack setting to false (sometimes called "auto ack"). This will force you to ack the message from your code.
Lastly, set the "consumer prefetch" limit of the consumer to 1.
With this combination of settings, a single consumer will retrieve a single message and begin working on it. While that consumer is working, any message waiting in the queue will be delivered to other consumers if any are available. If there are none available, the message will wait in the queue until a consumer is available.
With this, you should be able to achieve the behavior you are wanting, on a given queue.
...
Keep in mind this only applies to queues, though. routing keys cannot be managed this way. all matched routing keys from an exchange will cause a copy of the message to be sent to the destination queue.
I am using Node.js with node-amqp to create a simple message queue. Most examples that I see do the following:
Create a connection
Create an exchange
Creat a Queue and Bind it to the Exchange
Publish via the Exchange
In my code, I omit the queue (step 3) since it is not used for publishing.
var _connection = amqp.createConnection(_options);
_connection.on('ready', function() {
_connection.exchange('myexchange', { type: 'direct', autoDelete: false }, function(ex) {
ex.publish({hello:'world'});
});
});
Is this ok? or is there a reason for the queue?
There is nothing wrong with the code that you have. This is a good example of how you can keep your message producer nice and simple / clean.
However, the code you've shown is only half of the messaging solution. You need both a message producer, as shown, and a message consumer.
A Message Consumer
The message consumer is the code that does the real work. It receives a message from a queue to which it is subscribed, and processes that message however you tell it to.
That's the key, here - a message consumer will consume a message from a queue. If you want to send a message and have it be processed, then, you must have a queue for the message.
The Postal System Analogy
Think of this like this:
When you write a letter (pen and paper), you put it in an envelope. Then you write an address on the envelope and send it through your postal system. The postal system knows what the address means, sends it through various trucks and mail processing centers, and eventually puts it in a mailbox for the recipient.
It's the same thing with messaging in RabbitMQ.
You are delivering a letter to a destination. You write an "address" (exchange name, and routing key) on the message and RabbitMQ figures out how to deliver it to the appropriate place.
With physical mail, your letter is put in a mailbox for someone to read. With RabbitMQ and messaging, your message it put in a queue for some software to read.
You need a queue for the software to receive the message and process it.
...
P.S. If you're in need of some ground-up materials on RabbitMQ and NodeJS, check out my RabbitMQ For Developers package. It will get you up and running in no time, with the most common RMQ questions and patterns.
Queues are explicitly created and bound to exchange to make sure published message will not be lost in case no queue(s) and bindings previously exists.
In RabbitMQ most operations about entities creations (exchanges, queues, bindings) are idempotent. It means that if you call them more than once with the same arguments, they will provide the same result as called once.
In case of exchange, you can't publish to nonexistent exchange (channel-level AMQP exception occurs), but if no proper queues and bindings exist for particular message, it will be lost (or dead-lettered, see Dead Letter Exchanges and Alternate Exchanges for more).