In my former life I worked on a few Apache Camel projects, so I'm not entirely new to EIPs, but I am now trying to learn & understand Spring Integration. I have (what I think is) a small snippet of code for a "flow" that:
Defines a control bus for managing & monitoring the flow
Flow starts by fetching PNG images out of a folder (polling for new ones once a day); then
Uploads them to a directory on an FTP server
FileReadingMessageSource fileSource = new FileReadingMessageSource();
fileSource.setBeanName("fileMessageSource");
fileSource.setDirectory(new File("C:/DestDir"));
fileSource.setAutoCreateDirectory(true);
DefaultFtpSessionFactory ftpSessionFactory = new DefaultFtpSessionFactory();
IntegrationFlow flow = IntegrationFlows.from(fileSource, configurer -> configurer.poller(Pollers.cron("0 0 * * *")))
.filter("*.png") // only allow PNG files through
.controlBus() // add a control bus
.handle(Ftp.outboundAdapter(ftpSessionFactory, FileExistsMode.FAIL)
.useTemporaryFileName(false)
.remoteDirectory("uploadDir"))
.get();
Although admittedly I am a little unsure of the differences between "flows" and "channels" in Spring Integration parlance (I believe a flow is a composition of channels, and channels connect individual endpoints, maybe?), I am not understanding how, given my code above, the control bus can be leverage to turn the fileSource input endpoint on/off.
I know that with control buses, you send SPeL messages to channels and the control bus takes those SPeL messages and uses them to figure out which beans/methods to invoke, but above I am starting my flow from a FileReadingMessageSource. So what is the exact message I would need to send to which channel so that it stops/pauses or starts/restarts the FileReadingMessageSource endpoint?
The idea would be that if I used the control bus to turn the FileReadingMessageSource off, then days and days could pass and no PNG files would ever be uploaded to the FTP server, until I used the control bus to turn it back on.
Thanks for any-and-all help here!
The control bus needs to be in its own flow, not a component in another flow; you send control messages to it, to control other endpoints.
See a recent example in this question and answer.
Using Control Bus EIP in Spring Integration to start/stop channels dynamically
Related
***strong text***I want to use 2 aws outbound adaptor in a single integration flow. One is outbound S3 adaptor and one outbound SQS adaptor. I have to move file from smb share to S3 bucket then use the transformer and then use the SQS adaptor to send transformed message to SQS queue. I can achieve this with 2 integration flow but I want to achieve this with only one integration flow. If I add both outbound adaptor part of one flow only one of them is working
That’s true, two Outbound channel Adapters can’t work one after another. Just because one-way component doesn’t return anything to send as a payload into the output channel.
You might need to make yourself familiar with the publish-subscribe pattern: https://www.enterpriseintegrationpatterns.com/patterns/messaging/PublishSubscribeChannel.html.
In the IntegrationFlow you just need to configure a publishSubscribeChannel with two subscribers as sub-flows for your different Channel Adapters.
See docs for more info: https://docs.spring.io/spring-integration/docs/5.1.7.RELEASE/reference/html/#java-dsl-subflows
UPDATE
Since you already have an .enrichHeaders(h->h.header("bucketName”,”mybucket”)) before publishSubscribeChannel(), it is a fact that this one is going to be available for both subscribers downstream.
To make access into it from the s3MessageHandler, you must configure it like this:
public MessageHandler s3MessageHandler() {
S3MessageHandler handler = new S3MessageHandler(amazonS3,
new FunctionExpression<>(m -> m.getHeaders().get(”mybucket”)));
return handler;
}
To make access to that header for your next SQS subscriber part, you need to change your transform() method signature to accept the whole Message<>, so you get access to headers again for building some custom message for SQS.
Spring Cloud Stream is based on At least once method,This means that in some rare cases a duplicate message can arrive at an endpoint.
Does Spring Cloud Stream keep a buffer of already received messages?
The IdempotentReceiver in Enterprise Integration Patterns book suggests :
Design a receiver to be an Idempotent Receiver,one that can safely receive the same message multiple times.
Does Spring Cloud Stream control duplicate messages in consumers?
Update:
A paragraph from Spring Cloud Stream says :
4.5.1. Durability
Consistent with the opinionated application model of Spring Cloud Stream, consumer group subscriptions are durable. That is, a binder implementation ensures that group subscriptions are persistent and that, once at least one subscription for a group has been created, the group receives messages, even if they are sent while all applications in the group are stopped.
Anonymous subscriptions are non-durable by nature. For some binder implementations (such as RabbitMQ), it is possible to have non-durable group subscriptions.
In general, it is preferable to always specify a consumer group when binding an application to a given destination. When scaling up a Spring Cloud Stream application, you must specify a consumer group for each of its input bindings. Doing so prevents the application’s instances from receiving duplicate messages (unless that behavior is desired, which is unusual).
I think your assumption on the responsibility of the spring-cloud-stream framework are incorrect.
Spring-cloud-stream in a nutshell is a framework responsible for connecting and adapting producers/consumers provided by the developer to the message broker(s) exposed by the spring-cloud-stream binder (e.g., Kafka, Rabbit, Kinesis etc).
So connecting to a broker, receiving message from the broker, deserialising it, invoking user code, serialising message and sending it back to the broker is in the scope of framework responsibility. So you can look at it as purely infrastructure.
What you're describing is more of an application concern since the actual receiver is something that user would develop as part of the spring-cloud-stream development experience, hence responsibility for idempotence would reside with such user.
Also, on top of that most brokers already handle idempotency (in a way) by ensuring that a particular message has been delivered only once. That said, if someone sends identical message to such broker, it will have no idea that it is duplicate so the requirement for idempotency and/or deduplication is still valid, but as you can see it is not as straight forward given the amount of factor that are in play where your understanding of idempotence could be different from mine, hence our approaches could be different as well.
One last thing (partially to prove my last point): can safely receive the same message multiple times. - That is all it states, but what does safely really mean to you vs. me vs. some other person?
If you are concerned about a case where the application receives and processes message from the broker but crashes before it acknowledges the message, that can happen. Spring cloud stream app starters provides support for auto-configuration of a persistent message metadata store which backs Spring Integration's IdempotentReceiverInterceptor. An example of this is in the SFTP source app starter. By default, the sftp source uses an in-memory metadata store, so it would not survive a restart, but can be customized to use a persistent store.
AFAIK the Spring Cloud Stream project is based on Spring Integration. Hence I was wondering if there is a nice way to resequence a subset of inbound messages before the StreamListener handler is triggered? Or do I need to assemble the whole IntegrationFlow from scratch using XML or Java DSL config from Spring Integration?
My use case is as follows. Most of the time I process inbound messages on a Kafka topic as they come. However, a few events have to be resequenced based on CORRELATION_ID, SEQUENCE_NUMBER, and SEQUENCE_SIZE headers. In other words I'd like to keep using StreamListener as much as possible and simply plug in resequencing strategy for some events.
Yes, you would need to use Spring Integration for it. In fact Spring Cloud Stream is effectively a binding framework only. It binds message handlers to the message brokers via binders. The message handlers themselves are provided by the users.
The #StreamListener annotation is pretty much an equivalent of Spring Integration's #ServiceActivator with few extra features (e.g., conditional routing), but other then it is just a message handler.
Now, as you eluded to, you are aware that you can use Spring Integration (SI) to implement a message handler or an internal SI flow, and that is normal and recommended for complex cases.
That said, we do provide out of the box apps that implements certain EIP components and we do have, for example, and aggregator app which you can use as a starting point in implementing resequencer. Further more, given that we have an aggregator app and not resequencer, we would be glad to accept a contribution for it if you're interested.
I hope this answers you question.
I would like process the message programmatically using message-driven-channel-adapter. Here is scenario which I have to implement:
My application during the startup read the configuration from a service. The configuration provides information about the queues which will contain the messages. Hence I would like to create a message-driven-channel-adapter for each queue to listen to messages asynchronously.
Any example which initializes all the spring integration context programatically instead of using XML will be useful.
If you are going to do everything programmatically, I'd suggest you bypass Spring Integration magic and just use DefaultMessageListenerContainer directly.
Afterwards you can send messages to an existing MessageChannel directly from the MessageListener implementation or using Messaging Gateway.
Please, be careful with programmatic configuration with that do not miss important attributes like ApplicationContext or invocation for afterPropertiesSet().
We previously used to have a Spring Integration flow (XML configuration-based) where we would do an update in a database after sending a message to a JMS queue. To achieve this, the SI flow was configured with a publish-subscribe queue channel as an input to a JMS Outbound Channel Adapter (order 0) and a Service Activator (order 1). The idea here being that after a successful JMS send, the service activator would be called thus, updating the data in the database.
We are now in the process of updating our flows to work with spring-integration:4.0.x APIs and wanted to use this opportunity to see if the described flow pattern is still a good/recommended way of doing a database update after a successful JMS send or if there is now a simpler/better way of achieving this? As a side note, our flows are now being implemented using spring-integration-java-dsl:1.0.0.M3 APIs.
Thanks in advance for any input on this,
PM.
publish-subscribe queue channel
There's no such thing as a pub-sub queue channel; by definition, it's a subscribable channel; so I assume that's what you mean.
It is one of the ways to do what you need, and perfectly fine; you can also achieve the same result with a RecipientListRouter. The dsl syntax is quite nice, especially with Java 8; see the SpringOne demo app for an example.