I am looking for a solution for real-time data integration between few on-premise databases. There is no much transformation of data involved.
I am evaluating various ESBs available. I am thinking that data integration using Azure Service Bus as quick to develop a solution. Is it advisable to use Azure service bus for integration of all on-premise databases?
Unless there is extraordinary complexity in the integration, BizTalk is probably not the right tool for the job here. On the other hand, sending data out to the cloud just to transform it back to another database (on the same LAN?) is also not the right approach - this will introduce latency and traffic cost.
(Near) Real Time integration of databases sounds like a job for something like:
SQL Server Integration Services (SSIS)
If DB's are Sql-Server and the schemas are similar, Sql Server Replication
And similar technologies exist for other RDMBS technologies , e.g. Oracle Streams
If you really want to build a service bus, either build a local AMQP based bus as Sam suggests (e.g. Windows Service Bus or Rabbit), or buy an existing product (NServiceBus etc).
If you have all your applications 'on prem', you introduce an extra risk by moving your integration layer to the cloud (suddenly your internet connection could bring down your integration layer)
but the good news is that you can use Service Bus for Windows Server, that you run locally (even with Windows Azure pack!)
The same programming model, similar messaging features, so that might be a good option.
Comparing with BizTalk... Service Bus is light weight, messaging only. BizTalk provides much more rich features (transformations, pipelines, BAM, Business Rules, adapters).
Good luck
If you are only looking to integrate between a few on premise databases then you might consider using Sql Server's Service Broker (http://msdn.microsoft.com/en-gb/library/bb522893.aspx).
It provides a reliable asynchronous way of passing data between databases in a real time way. It can manage the message order and can have numerous conversations running at the same time on the same queue, each being processed by its own instance of the receiver.
There's a good overview here...
http://technet.microsoft.com/en-us/library/ms166104(v=sql.105).aspx
Related
I have 40+ micro-services using Windows Service Bus 1.1 with lots of Queues/Topics/Subscriptions and messages, and I am going to use Azure Service Bus instead.
How can I move all the information and the farm on-premises to Azure?
Not sure you can "move" anything off on-premises into Azure. What you will need to do is to transition your solution. And that's where it's getting a bit hairy.
First, answer the question if you can stop your system for a massive redeployment w/o impacting the business. If you are (which would be rare), you're in a luck as you could take the system offline and "transition" to the new topology on the Azure Service Bus. But that is highly unpropable situation.
A more realistic scenario is when you cannot turn down the sytem. An approach to take is to transition gradually. 40 microservices you've mentioned operate on the same WSSB. You could attempt to take one by one on the Azure Service Bus, but then other services need to know how to communicate over ASB and WSSB as well. Potentially, having a middleware infrastructure that knows to send and recieve to/from both WSSB and ASB until you can disable the WSSB completely. The devil is in details, which for a clear reason cannot be shared here.
And there are also complications such as messages in flight that are sent in the future. Those need to be accounted for. I would recommend to turn to Microsoft support for some pointers, but be aware that the product is already out of support and they technically are not necessarily have to provide any assistence.
I am using Azure Table storage to store the data of my application. I need to build an API to retrieve the data from Azure table storage. There are like a million records in the table. In the initial stage, my API would be getting approximately 100-1000 hits per day. What would be the best choice to develop that API on, API apps or Service Fabric?
It depends on a lot of aspects:
How do you want to maintain it?
What performance requirements do you have? Do all those 100-1000 hits per day happen at the same time or are they distributed across the day?
How do you want to handle scaling?
As described by #feranto the main difference is the architectural pattern. The decision will have a lot of implications for you, not only when it comes to development, but you will have to consider how to maintain your application.
Only based on your short description, it sounds like a relatively straightforward solution and not very complex nor very high performance/scale requirements. Given that, it might be easier for you to go for a more basic API App. Service Fabric is an excellent framework and architecture that allows us to build very flexible, highly scalable solutions, but it should be noted that that also comes with a lot more complexity when it comes to handling, monitoring and maintaining both the underlying service cluster and it's applications.
Setting up an API App, connecting to an Azure Storage table and control that with for instance Application Insights is relatively straight forward and you can find a lot of documentation and samples for that. As a contrast, Service Fabric is fairly nascent in documentation and samples, and you would have to read up on a lot to get a basic stable solution running if you are new to it.
The main difference between Azure API and Azure Service Fabric is the architecture pattern each one follows:
Azure API APPs supports a monolithic architecture. You have all your code running as one single package. You can develop this using Java, PHP, nodejs, python or .net.
Service Fabric follows a microservice architecture. Every service is deployed on a separate node, which can scale independently, and you can monitor the health of every service in every node. You can develop this using Java or C# mainly.
Here you can see advantages and disadvantages between microservices and a monolithic architecture.
We continue to see instability in Azure Service Bus and are looking for alternatives. Ideally we would like something we can run locally in a Windows 2012 R2 domain and has many of the same feature sets as Azure SB. We have taken a look at the Service Bus For Windows 1.1 however that product has not been updated in a while and we are not sure about its future. We use C# so ideally there would be a client API/wrapper that would make the integration with our existing application relatively easy. Open source and free software is totally acceptable. :-)
Background
In the couple of weeks we have had SB go down in the datacenter we have chosen to host our queues. We have moved to another datacenter and it too has issues every once in a while. The issues typically last anywhere from 2 min to almost an entire day and eventually Microsoft gets them running again.
I would recommend exploring RabbitMQ with MassTransit as an API, or NServiceBus (which is commercial)
RabbitMQ is a message broker that supports multiple transport layers (such as HTTP, AMQP, and STOMP among others) has feature parity with Azure service bus, however it is not cloud based, and you will need to host it on some infrastructure and support it, but it is robust and supports clustering and federation.
RabbitMQ is open source. Both free and commercial versions available. Has paid support.
I'm building a game for Windows Phone 8 and would like to use Windows Azure SQL Database for storing my users' data (mostly scores and rankings).
I have been reading Azure's documentation on SQL Database and found this link which describes just the scenario I'm looking for (it's Scenario B in the picture): I want my clients (the game running in a user's windows phone) to get data from an SQL Server through a middle application also hosted on Windows Azure.
By reading further the documentation (personally I think it's really messy and hard to find what you're looking for in there), I've learned that I could use Cloud Services for this middle application, however I'm not sure if I should use a background worker which provides an HTTP API or a worker with a Service Bus Relay (I discovered that I can use service bus in WP8 in this link).
I've got a few questions that I couldn't find an answer to:
1) What would be the "standard" way to go in this case?
2) If both ways are acceptable, are there other advantages to using a Service Bus other than an easier way to connect and send messages to my middle application? What are the disadvantages?
3) Is a cloud service really what I'm looking for (and not just a VM with the middle application code running in it)?
Its difficult to answer these sort of question as there are lots of considerations. I don't believe there is a necessarily 'standard way'.
The Service Bus' relay service's purpose is to help traverse firewalls and NATs, not something that directly relates to your scenario, I suspect.
The Service Bus, though, also includes a messaging capability which provides queues, topics and subscriptions to use to exchange messages between clients or client/server.
You could use the phone client to write and read messages to/from queues. you would then have a worker role hosting your application logic and accessing the database as needed.
Some of the advantages of using messaging include being load leveller, helping handling peaks in traffic (at the expense of latency), helping separating concerns and allowing you to accept requests from the clients when the backend is down as so can help with resiliency.
In theory they can also help you deliver messages to the client in the same fashion, by using a queue or subscription per client, but for a large number of clients this may become a management issue.
On the downside you would have to work with what is a proprietary protocol, and will need to understand the characteristics and limitations of the service bus. you will need to manage the queues and topics over time. there will also be some increased latency, although typically not an issue and, finally, you will have to implement asynchronous messaging on the client side which has advantages but is also harder to implement.
I would imagine that many architectures follow the WEB API route by using a web role cloud service exposing the API. The web role can then perform any business logic and connect to the database in the background.
A third option, which you didn't mention, is to use Windows Azure Mobile Services and implement your business logic as a service API there
I am having a hard time understanding Windows Azure service bus and access control concepts. In layman's terms, what are they? What are they used for?
The Service Bus component of Windows Azure is meant to handle the problems arising from services that are living in multiple networks. Basically, a service bus just makes it appear as if your code is running on a single machine, while in reality it could be running anywhere within the Azure datacenters.
Access Control lets you use "federated authentication for your service based on a claim-based RESTful model. (Sorry, copy&Paste from an O'Reilly book about Azure!)
Basically, when you create an Azure site, application or service, it could be running on any of the thousands of systems within the datacenter. And each of those systems has it's own IP address, it's own network, memory, processor and whatever more. To let them collaborate and to appear as a single system, these two services have been created.
If you want to learn more about Azure, this would be a good moment to buy a book! :-)
Azure is quite complex and service buses and access control are a bit more advanced topics.
Service Bus is a solution for the integration between multiple applications whether they are hosted on the same infrastructure or even spread along multiple infrastructure or/and Cloud Computing provider. If you search more in the internet you might find a lot about EAI (Enterprise application integration) here is my blog post about this topic:
http://hhaggan.wordpress.com/2013/03/07/introduction-to-enterprise-application-integration-eai/
and here another that I hope that helps you understand better what is the service bus:
http://hhaggan.wordpress.com/2013/03/09/introducing-service-bus/
in another words, it is a messaging platform that helps you communicate with multiple applications, softwares or services no matter what programming language they are written with or on which os or platform they are hosted on. you will feel its effect specially when you work on connecting multiple nodes together, I don't mean 5 or 6 nodes but 10 and above.
Certainly there are several types of service bus, whether they are based on relayed messaging service or brokered messaging service, each one of them has several uses, its purpose and way of working.
For the Access control, this is so easy, it is a way of authentication and authorization for your application using third parties, It is a claim based identity that you can do the required authentication through the third party database. you wont need to build everything from scratch in your database. this helps a lot during development and I believe that this can help a lot in social media marketing and branding because of the use of facebook, twitter during the authentication.