Our application needs to read from Kafka and use Scatter-Gather/Aggregate pattern to do processing and finally writes to datastore. We are using Spring integration for couple of years and are really comfortable with it. However, we want to leverage Kafka Streams and have questions around Spring integration, spring cloud stream and spring cloud data flow
· Does spring-integration-kafka supports kafka streams or do i need to use spring-cloud-stream for using them? https://github.com/spring-projects/spring-integration-kafka . Are there any code examples?
· Is Scatter-Gather/Aggregator pattern supported in spring cloud stream and data flow? Would really appreciate if someone can share examples of how it can be implemented
· Can we mix and match spring-cloud-stream and spring-integration in one flow? If both the above are not possible, I am thinking more like use steam listener and then pass it to integration flow for using other patterns. Is that advisable?
·
Would really appreciate the guidance from experts. Thanks in advance.
Related
we are planning to use Spring cloud-stream with a Kafka binder as the eventing base for our application. But we are currently unable to make it work with Spring-security, which we use for HTTP request security. One example I see is with Spring-web-flux and with reactive method security.
Currently we don't use reactive/web flux.
Is this the only way for the same, or is there another way with spring-web and spring-security.
Any help in this regard is highly appreciated as I'm unable to find any info for the same.
I am a little bit confused on the roles played by Kafka and SignalR with regards to real time communication. Can somebody help me by providing insights whether Kafka can be used as a like to like replacement for SignalR or are they complementary?
Thanks and Regards,
Nagarajan B P
SignalR is a library that simplifies the process of adding real-time web functionality to applications using WebSockets.
Kafka is an open source software which provides a framework for storing, reading and analysing streaming data.
You can use both if you want, using kafka as the entry point for events, and SignalR to notify web/mobile apps in real time.
Kafka aims to provide streaming data between many different programs. SignalR aims to provide real time communication between a client and server. So if you have a web app, thanks to cross scripting prevention, that page can only talk to its own server. But if you have many of your own custom services that don't all need to go through one point, Kafka would be more appropriate.
I came across "Amazon SNS" it is also a message sending service having complete structure and components like in "Spring integration"(topic, endpoints, pub/sub, synchronous). But haven't found out are they similar things so one can choose as an alternative solution or are they working together to build messaging system?
I know Spring integration is a Spring framework component and work with only Spring Framework and seems like Amazon SNS can be used with Spring Framework or simple Servlet/JSP technology to achieve messaging.
Please shed light on my knowledge. Thanks
I have too many emails. I should write scheduler in order to send messages to them. Messages are different. I use spring framework 4.x.
I can write simple class, which connects to SMTP server. But in this case I should write my thread library too in order to send emails parallel.
Do spring have already written library which give me more flexible way to do this tasks? I do not want to use threads. It will be nice if spring already have this functionality.
Do I need Spring integration for this?
Best regards,
Yes, you definitely can do that with Spring Integration, because there is an ExecutorChannel implementation with can be supplied with an TaskExecutor from the Spring Core:
<channel id="sendEmailChannel">
<dispatcher task-executor="threadPoolTaskExecutor"/>
</channel>
<int-mail:outbound-channel-adapter channel="sendEmailChannel" mail-sender="mailSender"/>
But anyway you should keep in mind that all Spring Integration components are based on the Java and that ExecutorService is used on the background.
From other side if you need only the mail sending stuff from the Spring Integration, it would be an overhead and can simply use Core Spring Framework legacy like JavaMailSender as a bean and #Async for the sendMail method to achieve your parallel requirement.
UPDATE
could you tell me whether I need JMS for this situation?
I don't see any JMS-related stuff here. You don't have (or at least don't show) any real integration points in your solution. The same I can say even about Spring Integration just for email sending. However with the Spring Boot your SI config will be enough short. From other side if you'll study Spring Integration better eventually you'll get more gain to rely on the Integration components for your systems, as internally, as well as externally with other systems through JMS, AMQP, Kafka etc.
To be honest: a lot of years ago my first acquaintance with Spring Integration was due the requirement to get files from the FTP and have ability to pick up new files automatically. I found the solution only in the Spring Integration 1.0.0.M1. After that short XML config for the <int-ftp:inbound-channel-adapter> I loved Spring Integration and since that time it became as a part of my life. :-)
So, it's up to you to go ahead with Spring Integration in your simple app, or just follow with more formal solution with JavaMailSender direct usage.
You should use java executors framework. For example you can write something like the code below:
ExecutorService executor = Executors.newWorkStealingPool();
executor.execute(() -> mailSender.send(mail));
Spring integration is used to make the communication between two systems easier.
So does that mean the if two systems are talking using JMS queus, then we can remove queues and integrate two systems using Spring Integration Framework?
Looks like you should study more on the matter. I mean EIP-book, Spring Integration in Action or, at least the Reference Manual from Spring IO site.
The main Spring Integration goal is integration and JMS is only one way to do that.
If your two system can get deal with JMS, there is no stops to integrate them using Spring Integration: just provide JMS adapters for boht of them.
As per my understanding,
Spring Integration solution a framework level solutions to Design Patterns listed in http://eaipatterns.com/
I might be an alternate to ESB which is like one big tunnel & everything passes through that.
Spring Integration provides end to end message between different de-coupled, disparate connecting points.
On the other hand JMS is an API in the Java EE spec which can be used in conjunction with Spring-Integration.
You might as well want to read about AMQP which is a messaging protocol.