We produce an enterprise DLT (Blockchain) application for big banks, FMIs, exchanges etc..
Being distributed each instance of the application is installed on-prem for each customer, as they must remain the sovereign owner of their private keys.
We want to integrate with a SaaS application that is widely used in the banking sector. We intend to achieve this by writing a "connector" which will also run on-prem and be able to communicate and marshal data between the SaaS system and our on-prem system.
Events occur in the SaaS app, which must then trigger something to happen in our on-prem app.
The SaaS app has a RESTful API as well as webhooks. So there are 2 options in my eyes:
Poll the RESTful API
Con: This is inefficient as most traffic will simply be "any new events?" "no"
Con: There will be some latency between the event occurring on the SaaS system and our on-prem app being triggered
Pro: This is stable. If the connector (the thing doing the polling) goes down, it will pick up any missed "events" from the SaaS system when it comes back up and process them
Pro: There is no requirement to allow internet traffic into the firewall - the comms are all outbound.
Use the webhooks
Pro: Very efficient
Pro: Get events in near real-time
Con: What happens if the connector is down and we miss a webhook? Does the SaaS system need a retry mechanism? We need to ensure that we only process messages exactly once. (this is important because the action we perform moves large amounts of funds so double processing would be extremely bad!)
Con: The bank would need to punch a hole in the firewall to allow the SaaS app to communicate into the connector - the bank's security teams won't like this IMO.
Is there a common, enterprise ready, security policy friendly way to do deal with this?
I think here you can use RESTful API with an enterprise ready-solution for API management. I would recommend that you explore APIGEE and see if it fits your usecase.
APIGEE is a platform for developing and managing API proxies.
An API proxy is an interface to developers that want to use backend services. Rather than having them consume those services directly, they access an Edge API proxy that you create. You can have it on cloud and also on-premises.
Here, you will solve your two main issues which are events management and latency.
Related
New to Azure and would like to know what the difference between Azure API Management and Logic Apps is.
Some pros and cons would be nice. Also what the costing for each of these is like.
Thanks
Seems you want to know about Azure API Management and Azure Logic Apps
Well let me help you what that actually are...
Azure API Management
Azure API Management allows organizations to publish APIs more securely, reliably, and at scale. Use API Management to drive API consumption among internal teams, partners, and developers while benefiting from business and log analytics available in the admin portal
Pros are
Consistent and modern API gateways for existing back-end services
Verifies API keys, JWT tokens, certificates, and other credentials
Help you to publish APIs to external, partner, and internal
developers to unlock the potential of their data and services
High availability Responds to requests to perform operations at
least 99.9% of the time.
Analytics, metrics and many more
For further details you could have a look on official docs
Pricing
There are different categories of pricing available based on type and uses. See the below screen shot:
You even can read more details about pricing here
Logic Apps:
Azure Logic Apps is a cloud service that helps you schedule, automate, and orchestrate tasks, business processes, and workflows when you need to integrate apps, data, systems, and services across enterprises or organizations. Logic Apps simplifies how you design and build scalable solutions for app integration, data integration, system integration
Pros are
Maximum automation with hassle free service
Process and route orders across on-premises systems and cloud
services
Can move uploaded files from an SFTP or FTP server to Azure Storage
Help you connect legacy, modern, and cutting-edge systems more
easily and quickly by providing prebuilt APIs as Microsoft-managed
connectors
Pricing
Usually two categories of pricing you may have Consumption pricing model which depends on how much you consume and have to pay as per your consumption.
another one Fixed pricing model. See the screen shot:
You can have a look more details on official docs
Hope this would help you.
I am looking for a solution for real-time data integration between few on-premise databases. There is no much transformation of data involved.
I am evaluating various ESBs available. I am thinking that data integration using Azure Service Bus as quick to develop a solution. Is it advisable to use Azure service bus for integration of all on-premise databases?
Unless there is extraordinary complexity in the integration, BizTalk is probably not the right tool for the job here. On the other hand, sending data out to the cloud just to transform it back to another database (on the same LAN?) is also not the right approach - this will introduce latency and traffic cost.
(Near) Real Time integration of databases sounds like a job for something like:
SQL Server Integration Services (SSIS)
If DB's are Sql-Server and the schemas are similar, Sql Server Replication
And similar technologies exist for other RDMBS technologies , e.g. Oracle Streams
If you really want to build a service bus, either build a local AMQP based bus as Sam suggests (e.g. Windows Service Bus or Rabbit), or buy an existing product (NServiceBus etc).
If you have all your applications 'on prem', you introduce an extra risk by moving your integration layer to the cloud (suddenly your internet connection could bring down your integration layer)
but the good news is that you can use Service Bus for Windows Server, that you run locally (even with Windows Azure pack!)
The same programming model, similar messaging features, so that might be a good option.
Comparing with BizTalk... Service Bus is light weight, messaging only. BizTalk provides much more rich features (transformations, pipelines, BAM, Business Rules, adapters).
Good luck
If you are only looking to integrate between a few on premise databases then you might consider using Sql Server's Service Broker (http://msdn.microsoft.com/en-gb/library/bb522893.aspx).
It provides a reliable asynchronous way of passing data between databases in a real time way. It can manage the message order and can have numerous conversations running at the same time on the same queue, each being processed by its own instance of the receiver.
There's a good overview here...
http://technet.microsoft.com/en-us/library/ms166104(v=sql.105).aspx
I'm building a game for Windows Phone 8 and would like to use Windows Azure SQL Database for storing my users' data (mostly scores and rankings).
I have been reading Azure's documentation on SQL Database and found this link which describes just the scenario I'm looking for (it's Scenario B in the picture): I want my clients (the game running in a user's windows phone) to get data from an SQL Server through a middle application also hosted on Windows Azure.
By reading further the documentation (personally I think it's really messy and hard to find what you're looking for in there), I've learned that I could use Cloud Services for this middle application, however I'm not sure if I should use a background worker which provides an HTTP API or a worker with a Service Bus Relay (I discovered that I can use service bus in WP8 in this link).
I've got a few questions that I couldn't find an answer to:
1) What would be the "standard" way to go in this case?
2) If both ways are acceptable, are there other advantages to using a Service Bus other than an easier way to connect and send messages to my middle application? What are the disadvantages?
3) Is a cloud service really what I'm looking for (and not just a VM with the middle application code running in it)?
Its difficult to answer these sort of question as there are lots of considerations. I don't believe there is a necessarily 'standard way'.
The Service Bus' relay service's purpose is to help traverse firewalls and NATs, not something that directly relates to your scenario, I suspect.
The Service Bus, though, also includes a messaging capability which provides queues, topics and subscriptions to use to exchange messages between clients or client/server.
You could use the phone client to write and read messages to/from queues. you would then have a worker role hosting your application logic and accessing the database as needed.
Some of the advantages of using messaging include being load leveller, helping handling peaks in traffic (at the expense of latency), helping separating concerns and allowing you to accept requests from the clients when the backend is down as so can help with resiliency.
In theory they can also help you deliver messages to the client in the same fashion, by using a queue or subscription per client, but for a large number of clients this may become a management issue.
On the downside you would have to work with what is a proprietary protocol, and will need to understand the characteristics and limitations of the service bus. you will need to manage the queues and topics over time. there will also be some increased latency, although typically not an issue and, finally, you will have to implement asynchronous messaging on the client side which has advantages but is also harder to implement.
I would imagine that many architectures follow the WEB API route by using a web role cloud service exposing the API. The web role can then perform any business logic and connect to the database in the background.
A third option, which you didn't mention, is to use Windows Azure Mobile Services and implement your business logic as a service API there
We are looking to use Windows Azure to host our existing SaaS platform and extend our functionality and capability. WE will be taking adavantage of both the data storage and application and web service functionality of Azure.
My question is as follows:
Some of our clients will not want Public CLoud access. Since our datastore stores sensitive client data many of them will require our whole system to be hosted internally on their own network and servers.
If we setup a full Azure setup of database and connected applications and processes how difficult is it to be able to duplicate that system for a specific client on their own servers and network using existing Microsoft technologies?
I know its a vague question and I also have a liminted understanding of Azure so whatever information you can provide here would be most appreciated.
Thank you
It sounds like you need the flexibility of a hybrid cloud/on-prem solution. Likely the best solution is the Windows Azure Service Bus. Essentially, you configure a WCF web service in the cloud (SOAP, REST, etc) that performs asynchronous brokered messaging between your on-premise application and your web application. This can be performed using queue messages, for example:
The web application (cloud) requests resources from the brokering service (cloud) by sending a queue message
The service handles the queue message and makes it available to the consuming (on-prem) service
On-prem service checks for new messages from the brokering service, gets the request for data, and returns desired data from DB
On-prem service sends message to brokering service with desired data
Web app (cloud) checks for new messages from the brokering service, then uses the data from on-prem service
Service bus is secure, asynchronous, fault-tolerant, and ensures that both components are decoupled.
Another method is to use Windows Azure Connect, which is a VPN solution that sets up network-level connnectivity. I recommend Service Bus because it promotes a more robust and scalable architecture, and fault-tolerance is high.
I am having a hard time understanding Windows Azure service bus and access control concepts. In layman's terms, what are they? What are they used for?
The Service Bus component of Windows Azure is meant to handle the problems arising from services that are living in multiple networks. Basically, a service bus just makes it appear as if your code is running on a single machine, while in reality it could be running anywhere within the Azure datacenters.
Access Control lets you use "federated authentication for your service based on a claim-based RESTful model. (Sorry, copy&Paste from an O'Reilly book about Azure!)
Basically, when you create an Azure site, application or service, it could be running on any of the thousands of systems within the datacenter. And each of those systems has it's own IP address, it's own network, memory, processor and whatever more. To let them collaborate and to appear as a single system, these two services have been created.
If you want to learn more about Azure, this would be a good moment to buy a book! :-)
Azure is quite complex and service buses and access control are a bit more advanced topics.
Service Bus is a solution for the integration between multiple applications whether they are hosted on the same infrastructure or even spread along multiple infrastructure or/and Cloud Computing provider. If you search more in the internet you might find a lot about EAI (Enterprise application integration) here is my blog post about this topic:
http://hhaggan.wordpress.com/2013/03/07/introduction-to-enterprise-application-integration-eai/
and here another that I hope that helps you understand better what is the service bus:
http://hhaggan.wordpress.com/2013/03/09/introducing-service-bus/
in another words, it is a messaging platform that helps you communicate with multiple applications, softwares or services no matter what programming language they are written with or on which os or platform they are hosted on. you will feel its effect specially when you work on connecting multiple nodes together, I don't mean 5 or 6 nodes but 10 and above.
Certainly there are several types of service bus, whether they are based on relayed messaging service or brokered messaging service, each one of them has several uses, its purpose and way of working.
For the Access control, this is so easy, it is a way of authentication and authorization for your application using third parties, It is a claim based identity that you can do the required authentication through the third party database. you wont need to build everything from scratch in your database. this helps a lot during development and I believe that this can help a lot in social media marketing and branding because of the use of facebook, twitter during the authentication.