I am in the process of moving an application from c# to node.js. I am a node.js newbie, coming from a .net background. I am looking to incorporate domain driven design patterns into the app. development which led me to the concept of bounded contexts and micro services. I would like to use aws as my cloud provider but am having a problem in determining which tool I should use for handlling command and event processing? Azure has the service bus which seems to work pretty good for this.
Is there an equivalent to the service bus for aws or should I just look to use SQS?
There is no direct equivalent to Azure Service Bus, but it can be replaced by combining SQS and SNS. Let's take a look. Azure Service Bus consists of two parts:
Queues. In most cases, SQS (Simple Queue Service) will provide an adequate replacement, but keep in mind that Azure Service Bus queues are First In-First Out (FIFO), while SQS queues do not guarantee the order of messages.
Update 2018-01-09: SQS now allows to create FIFO queues. (see https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html)
Topics and subscriptions. This is used in PubSub (publish/subscribe) event-driven design, when you need to deliver the same message to several consumers. SQS can't do that, but SNS (Simple Notification Service) is exactly that type of service.
Update 2018-08-01: on November 28th 2017 Amazon introduced Amazon MQ, which is Apache Active MQ in Amazon cloud. Amazon MQ has both queues and topics (for publish/subscribe usage model), so it can be seen as a full-featured replacement for Azure Service Bus.
July 2019 Update: Amazon introduced EventBridge service bus service that enables event-driven architectures for custom applications, as well as integration with AWS services and other SaaS hosted on its platform, see https://aws.amazon.com/about-aws/whats-new/2019/07/introducing-amazon-eventbridge/
If SQS suits your needs then it is well integrated with the platform:
https://aws.amazon.com/sqs/
Amazon Simple Queue Service (SQS) is a fast, reliable, scalable, fully managed message queuing service. Amazon SQS makes it simple and cost-effective to decouple the components of a cloud application. You can use Amazon SQS to transmit any volume of data, without losing messages or requiring other services to be always available. Amazon SQS includes standard queues with high throughput and at-least-once processing, and FIFO queues that provide FIFO (first-in, first-out) delivery and exactly-once processing.
Also, there's Enterprise Service Bus (HVM) in the marketplace but is seems like a Windows only thing.
But you don't have to use only solutions that are directly integrated into your hosting provider's platform. You can run anything on AWS. For example you can use tools like Redis, RabbitMQ, ZeroMQ, ActiveMQ, NSQ etc.
See:
https://redis.io/
https://www.rabbitmq.com/
http://zeromq.org/
http://activemq.apache.org/
http://nsq.io/
Related
I have a local machine that reads RabbitMQ queue messages.
I wish to move it to cloud. Which Azure service can be used in this case?
I went through event hubs, but I am not sure, if it would read messages from rabbitMQ continuously.
Any suggestions for the service that should be put to use.
You should take a look at Azure Service Bus. It has got FIFO queues as well as publish/subscribe capabilities. However if using Azure managed service is not a strict requirement you can use RabbitMQ on a VM (or a cluster for high availability) as well.
UPDATE: Your response means you want a managed service. There are 2 options - if you want to go with RabbitMQ but do not want to manage the infrastructure you can go for 3rd party service provider like CloudAMQP who will manage it on your behalf. The other option is to go for Cloud native messaging - meaning if you are on Azure you change your messaging service to Azure Service Bus. This would mean changing you code as well.
We use Azure Service Bus to send messages from one service to another. Producer produces quite huge amount of messages(a couple of millions) during 1-2 hours. As a result our Service Bus(we use Premium Azure Service Bus) is throttled and we receive errors on producer and consumer sides. I wonder if we could check somehow the load of our ASB by using Azure SDK(we use ASB Java SDK) and if it is high we can slow down the services that sends messages into the queue/topic.
I also understand that we can add more Premium units, but it is the last option we will take.
What we use:
Azure Service Bus Java SDK
Java 9 and Spring Boot 2.0
Azure Service Bus Premium version
Do you have any recommendations for my case? Any recommendation - patterns, frameworks, ASB SDK features would be great.
What you usually can do is to check from the portal your usage (from the overview panel), there is no feature in the SDK itself that can show you the load on your service bus namespace.
Since you're using premium messaging, there is no certain threshold that can determine if you've exceeded your messaging unit capacity or not.
you will get throttling errors as shown in the documentation:
Also be aware that you can scale up and down according to your usage with how many messaging units you want.
I have implemented a Proof of Concept in Azure Service fabric that uses Azure Service Bus as a message queue. I'm using nServiceBus within the application to send and respond to messages which makes it very easy to change from one queuing technology to another.
I'd like to know if it's possible to use MSMQ instead of Service Bus in Azure Service Fabric as the nodes that are created are just windows 2016 servers and I'm not sure I need something like Service Bus. It's a question I want to answer with my POC.
Does anyone know whether MSMQ is included in an Azure Service Fabric node or how I could turn it on and if it's a viable solution?
Short answer - MSMQ is not suitable for Azure Service Fabric.
MSMQ is store and forward technology. It's using local file system to persist messages and then forward to another machine. When Service Fabric is going to move service from one node to another, it will not move the file system along. Meaning you'll lose messages.
I would recommend to stay with Azure Service Bus unless there's a good reason you're looking for an alternative.
We currently have 2 Azure Windows VMs with all of our applications installed on each instance. These are then load balanced.
For now this works well for us financially as we have multiple .NET core APIs, Angular clients and .NET full framework apps running all with relatively low loads at the moment so there's no need to separate them on to individual instances (although we will likely do this in the future).
We've recently developed a service bus integration using .NET core, rabbitmq and MassTransit to handle email and report generation (as these take a while to generate and slow the apps down). This all works great locally and I'm ready to deploy.
I wanted to use Azure Service Bus so we have a centralised and cost effective message broker but at the moment MassTransit doesn't support .NET core integrations with Azure Service Bus (and I don't want to switch our producer or consumers to full .NET). I believe this should be available in the near future.
So until MassTransit supports ASB I want to use rabbitmq and my plan was to install rabbitmq on each of our 2 VMs (not linked in anyway) and point the producers and consumers on those machines to the local version of rabbitmq (because if the VM is down the producer, consumer and broker would all be down anyway).
Is this the right approach or is there a better way for me to configure rabbitmq for this scenario without paying for another instance/hosted service?
The same as Azure SB, RabbitMQ is the message broker, which means you need one centralised instance. All your services will speak with this instance.
You can install it on any of your VMs, it does not take much RAM and you need just as much disk space as your queues require.
I'm simply trying to work out how best to retrieve messages as quickly as possible from an Azure Service Bus Queue.
I was shocked that there wasn't some way to properly subscribe to the queue for notifications and that I'm going to have to poll. (unless I'm wrong in which case the documentation is terrible).
I got long polling working, but checking a single message every 60 seconds looks like it'll cost around £900 per month (again, unless I've misunderstood that). And if I add a redundant/second service to poll it'll double.
So I'm wondering what the best/most cost efficient way of doing it is.
Essentially I just want to take a message from the queue, perform an API lookup on some internally held data (perhaps using hybrid services?) and then perhaps post a message back to a different queue with some additional information .
I looked at worker roles(?) -- is that something that could do it?
I should mention that I've been looking at doing this with node.js.
Check out these videos from Scott Hanselman and Mark Simms on Azure Queues.
It's C# but you get the idea.
https://channel9.msdn.com/Search?term=azure%20queues%20simms#ch9Search
Touches on:
Storage Queues vs. Service Bus Queues
Grabbing messages in bulk vs. one by one (chunky vs. chatty)
Dealing with poison messages (bad actors)
Misc implementation details
Much more stuff i can't remember now
As for your compute, you can either do a VM, a Worker Role (Cloud Services), App Service Webjobs, or Azure Functions.
The Webjobs SDK and Azure Functions bot have a way to subscribe to Queue events (notify on message).
(Listed from IaaS to PaaS to FaaS - Azure Functions - if such a thing exists).
Azure Functions already has sample code provided as templates to do all that with Node. Just make a new Function and follow the wizard.
If you need to touch data on-prem you either need to look at integrating with a VNET that has site-to-site connectivity back to your prem, or Hybrid Connections (App Service only!). Azure Functions can't do that yet, but every other compute is a go.
https://azure.microsoft.com/en-us/documentation/articles/web-sites-hybrid-connection-get-started/
(That tutorial is Windows only but you can pull data from any OS. The Hybrid Connection Manager has to live on a Windows box, but then it acts as a reverse proxy to any host on your network).
To deal with Azure ServiceBus Queue easily, the best option seems to be Azure Webjob.
There is a ServiceBusTrigger that allows you to get messages from an Azure ServiceBus queue.
For node.js integration, you should have a look at Azure Function. It is built on top of the webjob SDK and have node.js integration :
Azure Functions NodeJS developer reference
Azure Functions Service Bus triggers and bindings for queues and topics
In the second article, there is an example on how get messages from a queue using Azure Function and nodejs :
module.exports = function(context, myQueueItem) {
context.log('Node.js ServiceBus queue trigger function processed message', myQueueItem);
context.done();
};