I recently started looking at spring cloud streams. Is there a streams module that would allow us to talk directly through Amazon SNS?
There's not currently a module, but Spring Integration has an extension project that supports SNS so it shouldn't be too hard to create a custom module.
Consider contributing.
The inbound adapter is available and there is a pull request for an outbound adapter that's not merged yet.
EDIT
The oubound adapter is now on master and available in the 1.0.0.BUILD-SNAPSHOT - we will create a milestone release shortly.
Related
I want to implement spring-integration-aws to send and receive messages with SQS. I am looking at localstack and would like to know the recommendation of the spring team.
Which tool/api should I use for local setup of spring integration flows for SQS inbound and outbound adapters?
Also, will there be examples of AWS in spring-integration-samples in future? I am looking for an example with xml config that reads the aws config from credentials and send and receive messages via outbound adapters.
Not sure what recommendation you expect from us, but I see an answer in your own question - Localstack: https://github.com/localstack/localstack.
In the project test we indeed use this tool over a docker container:
https://github.com/spring-projects/spring-integration-aws/blob/master/src/test/java/org/springframework/integration/aws/lock/DynamoDbLockRegistryTests.java#L62
We don't have such a test against SQS, but the configuration technique is similar.
I recall I heard that Testcontainers project can be used for testing AWS services locally as well: https://www.testcontainers.org/modules/localstack/
We don't have resources to write samples for this project.
I am new to spring integration , i am trying to achieve the following through spring integration:
there are 3 micro services A,B,C , based on some event i would want to publish a message to spring integration channel to which services B and C should have subscribed.
i am unable to figure out a way to make an external service as subscriber.
please help.
thanks in advance.
You need to use a message broker such as RabbitMQ, Kafka, JMS etc. to communicate between external services.
Or, you could connect them explicitly using TCP channel adapters or gateways.
Read the reference manual.
See the samples.
I am trying azure functions to connect to websocket to get the data and send to locally installed kafka in my laptop. but unable to send to kafka?.Is there any way to send data from azure functions to locally installed kafka?
There is nothing in Azure Function that will do that for you, so you need to send your messages to Kafka the same you would do from any other .NET code.
The steps:
Make sure Kafka is accessible from the internet as #Dhinesh suggested.
Pick a Kafka client library to use (the official one is here, but there are others too).
Write a simple console app which would send some sample messages to your Kafka and make sure it works.
Use exactly the same code inside your Azure Function body to send real messages to Kafka.
If you need something more elaborate than this basic guide, feel free to adjust your question.
Azure AppService supports Virtual Networks (vNets). Functions run on top of an AppService so this means that your Functions can securely access your on-prem Kafka instance over VPN. For this to work you would need to:
Create a site to site or a site to point VPN using the VNET integration -instructions here: https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-site-to-site-create
Ensure your function and app service is inside this vNET - instructions here: https://learn.microsoft.com/en-us/azure/app-service-web/web-sites-integrate-with-vnet
I hope this helps
Thinking to use Spring Integration for dynamic Inbound / Outbound configuration. There can be 'n' number of customers using our system, each of them will have their own inbound FTP / webservice configuration to pull files for processing into our system. Likewise after processing each of those customers can have outbound FTP / webservice configuration where the reports (end result) need to be pushed. Is spring integration appropriate for such cases? if yes, can you please suggest or point to any of the examples. Webservice includes both REST and SOAP.
Also, instead of polling at fixed interval, can each customer have their own cron expression as to when to pull the files during Inbound?
Your use-case looks fully as an appropriate for the Spring Cloud Data Flow.
Each customer will configure a stream with desired options for each module.
Otherwise your question looks very broad and it has a big chance to be closed here.
See the dynamic ftp sample (XML configuration). This is outbound; see the readme for some links about doing it the same way for inbound.
See this answer - mail inbound adapters using Java Config
And the Dynamic TCP example using the Java DSL dynamic flow registration.
Dynamic WebService inbound is a bit more tricky because you have to register the new endpoints
I am in the process of moving an application from c# to node.js. I am a node.js newbie, coming from a .net background. I am looking to incorporate domain driven design patterns into the app. development which led me to the concept of bounded contexts and micro services. I would like to use aws as my cloud provider but am having a problem in determining which tool I should use for handlling command and event processing? Azure has the service bus which seems to work pretty good for this.
Is there an equivalent to the service bus for aws or should I just look to use SQS?
There is no direct equivalent to Azure Service Bus, but it can be replaced by combining SQS and SNS. Let's take a look. Azure Service Bus consists of two parts:
Queues. In most cases, SQS (Simple Queue Service) will provide an adequate replacement, but keep in mind that Azure Service Bus queues are First In-First Out (FIFO), while SQS queues do not guarantee the order of messages.
Update 2018-01-09: SQS now allows to create FIFO queues. (see https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html)
Topics and subscriptions. This is used in PubSub (publish/subscribe) event-driven design, when you need to deliver the same message to several consumers. SQS can't do that, but SNS (Simple Notification Service) is exactly that type of service.
Update 2018-08-01: on November 28th 2017 Amazon introduced Amazon MQ, which is Apache Active MQ in Amazon cloud. Amazon MQ has both queues and topics (for publish/subscribe usage model), so it can be seen as a full-featured replacement for Azure Service Bus.
July 2019 Update: Amazon introduced EventBridge service bus service that enables event-driven architectures for custom applications, as well as integration with AWS services and other SaaS hosted on its platform, see https://aws.amazon.com/about-aws/whats-new/2019/07/introducing-amazon-eventbridge/
If SQS suits your needs then it is well integrated with the platform:
https://aws.amazon.com/sqs/
Amazon Simple Queue Service (SQS) is a fast, reliable, scalable, fully managed message queuing service. Amazon SQS makes it simple and cost-effective to decouple the components of a cloud application. You can use Amazon SQS to transmit any volume of data, without losing messages or requiring other services to be always available. Amazon SQS includes standard queues with high throughput and at-least-once processing, and FIFO queues that provide FIFO (first-in, first-out) delivery and exactly-once processing.
Also, there's Enterprise Service Bus (HVM) in the marketplace but is seems like a Windows only thing.
But you don't have to use only solutions that are directly integrated into your hosting provider's platform. You can run anything on AWS. For example you can use tools like Redis, RabbitMQ, ZeroMQ, ActiveMQ, NSQ etc.
See:
https://redis.io/
https://www.rabbitmq.com/
http://zeromq.org/
http://activemq.apache.org/
http://nsq.io/