We continue to see instability in Azure Service Bus and are looking for alternatives. Ideally we would like something we can run locally in a Windows 2012 R2 domain and has many of the same feature sets as Azure SB. We have taken a look at the Service Bus For Windows 1.1 however that product has not been updated in a while and we are not sure about its future. We use C# so ideally there would be a client API/wrapper that would make the integration with our existing application relatively easy. Open source and free software is totally acceptable. :-)
Background
In the couple of weeks we have had SB go down in the datacenter we have chosen to host our queues. We have moved to another datacenter and it too has issues every once in a while. The issues typically last anywhere from 2 min to almost an entire day and eventually Microsoft gets them running again.
I would recommend exploring RabbitMQ with MassTransit as an API, or NServiceBus (which is commercial)
RabbitMQ is a message broker that supports multiple transport layers (such as HTTP, AMQP, and STOMP among others) has feature parity with Azure service bus, however it is not cloud based, and you will need to host it on some infrastructure and support it, but it is robust and supports clustering and federation.
RabbitMQ is open source. Both free and commercial versions available. Has paid support.
Related
I have 40+ micro-services using Windows Service Bus 1.1 with lots of Queues/Topics/Subscriptions and messages, and I am going to use Azure Service Bus instead.
How can I move all the information and the farm on-premises to Azure?
Not sure you can "move" anything off on-premises into Azure. What you will need to do is to transition your solution. And that's where it's getting a bit hairy.
First, answer the question if you can stop your system for a massive redeployment w/o impacting the business. If you are (which would be rare), you're in a luck as you could take the system offline and "transition" to the new topology on the Azure Service Bus. But that is highly unpropable situation.
A more realistic scenario is when you cannot turn down the sytem. An approach to take is to transition gradually. 40 microservices you've mentioned operate on the same WSSB. You could attempt to take one by one on the Azure Service Bus, but then other services need to know how to communicate over ASB and WSSB as well. Potentially, having a middleware infrastructure that knows to send and recieve to/from both WSSB and ASB until you can disable the WSSB completely. The devil is in details, which for a clear reason cannot be shared here.
And there are also complications such as messages in flight that are sent in the future. Those need to be accounted for. I would recommend to turn to Microsoft support for some pointers, but be aware that the product is already out of support and they technically are not necessarily have to provide any assistence.
We currently have 2 Azure Windows VMs with all of our applications installed on each instance. These are then load balanced.
For now this works well for us financially as we have multiple .NET core APIs, Angular clients and .NET full framework apps running all with relatively low loads at the moment so there's no need to separate them on to individual instances (although we will likely do this in the future).
We've recently developed a service bus integration using .NET core, rabbitmq and MassTransit to handle email and report generation (as these take a while to generate and slow the apps down). This all works great locally and I'm ready to deploy.
I wanted to use Azure Service Bus so we have a centralised and cost effective message broker but at the moment MassTransit doesn't support .NET core integrations with Azure Service Bus (and I don't want to switch our producer or consumers to full .NET). I believe this should be available in the near future.
So until MassTransit supports ASB I want to use rabbitmq and my plan was to install rabbitmq on each of our 2 VMs (not linked in anyway) and point the producers and consumers on those machines to the local version of rabbitmq (because if the VM is down the producer, consumer and broker would all be down anyway).
Is this the right approach or is there a better way for me to configure rabbitmq for this scenario without paying for another instance/hosted service?
The same as Azure SB, RabbitMQ is the message broker, which means you need one centralised instance. All your services will speak with this instance.
You can install it on any of your VMs, it does not take much RAM and you need just as much disk space as your queues require.
I am working with Azure Service Bus Topics and Subscriptions. It's being used to send control messages across the application. The message listeners (subscribers) are running in a worker role and they are picking up the messages and processing the request. Each message in the bus can be picked up by only once, even if there are multiple listeners running simultaneously.
There is no issue in using the Service Bus; however we are facing some issues while debugging/testing the application in local. We have 2 service bus, one for the cloud and one for local debugging. Now if multiple people are debugging the application simultaneously, the message is being picked by only one of the system (at random). This is the intended behavior, but it's causing a nuisance while debugging.
Is there any way how I can use a local emulator for the Service Bus? I did some research but I couldn't find any reliable solution for this. Is there any way how we debug the application in isolation?
Unfortunately, there is no Azure SB local emulator. That was asked before, and you can try to use Service Bus for Windows Server but it is slightly behind the cloud service in terms of features support/functionality, etc. Still, it supports Azure SDK, for example. MSDN link for SB for WS.
Azure Service Bus is a broker with competing consumers. Having multiple developers debugging using the same namespace will be be tough (message lock duration expired and another developer that happened to be debugging got that message).
I would suggest to look into a namespace per developer. With MSDN license you're given enough Azure credit to have each developer work in a "sandboxed" namespace. As to how to get it working, you can read from a configuration file, environment variable, etc.
On the ASB for Windows Server - currently it's on version 1.1 where Azure SB is on 3+. Hosted version will be always ahead of the on-premises. Something to consider.
The problem should probably be solved by abstracting the service bus implementation so that it can be swapped for something that you actually can run locally (or in memory). I would suggest not to re-invent that wheel though and choose a library like Masstransit https://masstransit-project.com.
We ended up using a different topic name for each developer while debugging which is working pretty good.
We use the devlopers machine name in the path for alot of dev things. for example a queueName like "TheEventMessageQueue" can be "TheEventMessageQueue-Machine123" and we use a dev subscription so we dont make a mess :)
By using the machine name its easy to se who the que belongs to if its going out of control
I am looking for a solution for real-time data integration between few on-premise databases. There is no much transformation of data involved.
I am evaluating various ESBs available. I am thinking that data integration using Azure Service Bus as quick to develop a solution. Is it advisable to use Azure service bus for integration of all on-premise databases?
Unless there is extraordinary complexity in the integration, BizTalk is probably not the right tool for the job here. On the other hand, sending data out to the cloud just to transform it back to another database (on the same LAN?) is also not the right approach - this will introduce latency and traffic cost.
(Near) Real Time integration of databases sounds like a job for something like:
SQL Server Integration Services (SSIS)
If DB's are Sql-Server and the schemas are similar, Sql Server Replication
And similar technologies exist for other RDMBS technologies , e.g. Oracle Streams
If you really want to build a service bus, either build a local AMQP based bus as Sam suggests (e.g. Windows Service Bus or Rabbit), or buy an existing product (NServiceBus etc).
If you have all your applications 'on prem', you introduce an extra risk by moving your integration layer to the cloud (suddenly your internet connection could bring down your integration layer)
but the good news is that you can use Service Bus for Windows Server, that you run locally (even with Windows Azure pack!)
The same programming model, similar messaging features, so that might be a good option.
Comparing with BizTalk... Service Bus is light weight, messaging only. BizTalk provides much more rich features (transformations, pipelines, BAM, Business Rules, adapters).
Good luck
If you are only looking to integrate between a few on premise databases then you might consider using Sql Server's Service Broker (http://msdn.microsoft.com/en-gb/library/bb522893.aspx).
It provides a reliable asynchronous way of passing data between databases in a real time way. It can manage the message order and can have numerous conversations running at the same time on the same queue, each being processed by its own instance of the receiver.
There's a good overview here...
http://technet.microsoft.com/en-us/library/ms166104(v=sql.105).aspx
Can I use the same service bus Azure uses and host it in my own environment? If not what service buses are simple and effective and work well with low latency/bandwidth?
Microsoft also has Service Bus for Windows Server http://msdn.microsoft.com/en-us/library/windowsazure/jj193022(v=azure.10).aspx.
Clearly the architecture of Azure is slightly different then the average Windows Server installation; so as to whether they're the "same", I don't know.
In terms of the API, they seem to have parity (identical) as far as I've been able to tell. http://msdn.microsoft.com/en-us/library/windowsazure/jj542433(v=azure.10).aspx I have had a problem trying to get a client running on Windows 8, so I haven't taken it much further than a simple test.
Depending on what you need, alternatives might be NServiceBus, MassTransit, message queues like RabbitMQ, ZeroMQ, etc.
Just to add to Peter's answer, the Service Bus for Windows Server currently only focusses on the messaging part of Azure Service Bus (queues & topics), not the actual Relay Service that is in the cloud