Using filter with a discard channel in Spring Integration DSL - spring-integration

I don't know if this question is about spring-integration, spring-integration-dsl or both, so I just added the 2 tags...
I spend a considerable amount of time today, first doing a simple flow with a filter
StandardIntegrationFlow flow = IntegrationFlows.from(...)
.filter(messagingFilter)
.transform(transformer)
.handle((m) -> {
(...)
})
.get();
The messagingFilter being a very simple implementation of a MessageSelector. So far so good, no much time spent. But then I wanted to log a message in case the MessageSelector returned false, and here is where I got stuck.
After quite some time I ended up with this:
StandardIntegrationFlow flow = IntegrationFlows.from(...)
.filter(messagingFilters, fs -> fs.discardFlow( i -> i.channel(discardChannel()))
.transform(transformer)
.handle((m) -> {
(...)
})
.get();
(...)
public MessageChannel discardChannel() {
MessageChannel channel = new MessageChannel(){
#Override
public boolean send(Message<?> message) {
log.warn((String) message.getPayload().get("msg-failure"));
return true;
}
#Override
public boolean send(Message<?> message, long timeout) {
return this.send(message);
}
};
return channel;
}
This is both ugly and verbose, so the question is, what have I done wrong here and how should I have done it in a better, cleaner, more elegant solution?
Cheers.

Your problem that you don't see that Filter is a EI Pattern implementation and the maximum it can do is to send discarded message to some channel. It isn't going to log anything because that approach won't be Messaging-based already.
The simplest way you need for your use-case is like:
.discardFlow(df -> df
.handle(message -> log.warn((String) message.getPayload().get("msg-failure")))))
That your logic to just log. Some other people might do more complicated logic. So, eventually you'll get to used to with channel abstraction between endpoints.
I agree that new MessageChannel() {} approach is wrong. The logging indeed should be done in the MessageHandler instead. That is the level of the service responsibility. Also don't forget that there is LoggingHandler, which via Java DSL can be achieved as:
.filter(messagingFilters, fs -> fs.discardFlow( i -> i.log(message -> (String) message.getPayload().get("msg-failure"))))

Related

the right way to return a Single from a CompletionStage

I'm playing around with reactive flows using RxJava2, Micronaut and Cassandra. I'm new to rxjava and not sure what is the correct way to return a of List Person in the best async manner?
data is coming from a Cassandra Dao interface
public interface PersonDAO {
#Query("SELECT * FROM cass_drop.person;")
CompletionStage<MappedAsyncPagingIterable<Person>> getAll();
}
that gets injected into a micronaut controller
return Single.just(personDAO.getAll().toCompletableFuture().get().currentPage())
.subscribeOn(Schedulers.io())
.map(people -> HttpResponse.ok(people));
OR
return Single.just(HttpResponse.ok())
.subscribeOn(Schedulers.io())
.map(it -> it.body(personDAO.getAll().toCompletableFuture().get().currentPage()));
OR switch to RxJava3
return Single.fromCompletionStage(personDAO.getAll())
.map(page -> HttpResponse.ok(page.currentPage()))
.onErrorReturn(throwable -> HttpResponse.ok(Collections.emptyList()));
Not a pro of RxJava nor Cassandra :
In your first and second example, you are blocking the thread executing the CompletionStage with get, even if you are doing it in the IO thread, I would not recommand doing so.
You are also using a Single wich can emit, only one value, or an error. Since you want to return a List, I would sugest to go for at least an Observable.
Third point, the result from Cassandra is paginated, I don't know if it's intentionnaly but you list only the first page, and miss the others.
I would try a solution like the one below, I kept using the IO thread (the operation may be costly in IO) and I iterate over the pages Cassandra fetch :
/* the main method of your controller */
#Get()
public Observable<Person> listPersons() {
return next(personDAO.getAll()).subscribeOn(Schedulers.io());
}
private Observable<Person> next(CompletionStage<MappedAsyncPagingIterable<Person>> pageStage) {
return Single.fromFuture(pageStage.toCompletableFuture())
.flatMapObservable(personsPage -> {
var o = Observable.fromIterable(personsPage.currentPage());
if (!personsPage.hasMorePages()) {
return o;
}
return o.concatWith(next(personsPage.fetchNextPage()));
});
}
If you ever plan to use reactor instead of RxJava, then you can give cassandra-java-driver-reactive-mapper a try.
The syntax is fairly simple and works in compile-time only.

Spring Integration: Switch routing dynamically

A spring integration based converter consumes the messages from one system, checks, converts and sends it to the other one.
Should the target system be down, we stop the inbound adapters, but would also like to persist locally or forward the currently "in-flight" converted messages. For that would simply like to reroute the messages from the normal output channel to some "backup"-channel dynamically.
In the docs I have found only the option to route the messages based on their headers ( so on some step before in flow I would have to add those dynamically once the targer system is not availbale), or based on the payload type, which is not really my case. The case with adding dynamically some header, and then filtering it out down the pipe, or during de-/serializing still seems not the best approach for me. I would like rather to be able to turn a switch(on some internal Event) that would then reroute those "in-flight" messages to the "backup"-channel.
What would be a best SI approach to achive this? Thanks!
The router could not only be based on the the payload type or some header. You really can have a general POJO method invocation to return a channel, its name or some routing key which is mapped. That POJO method indeed can check some internal system state and produce this or that routing key.
So, you may have something like this in the router configuration:
.route(myRouter())
where your myRouter is something like this:
#Bean
MyRouter myRouter() {
return;
}
and its internal code might be like this:
public class MyRouter {
#Autowired
private SystemState systemState;
String route(Object payload) {
return this.systemState.isActive() ? "successChannel" : "backupChannel";
}
}
The same can be achieved a simple lambda definition:
.<Object, Boolean>route(p -> systemState().isActive(),
m -> m.channelMapping(true, "sucessChannel")
.channelMapping(false, "backupChannel"))
Also...
private final AtomicBoolean switcher = new AtomicBoolean();
#Bean
public IntegrationFlow flow() {
return IntegrationFlows.from(() -> "foo", e -> e.poller(Pollers.fixedDelay(Duration.ofSeconds(5))))
.route(s -> switcher.get() ? "foo" : "bar")
.get();
}

spring-integration: how to deliver deferred details as SSE

I have a list of items which I want to retrieve and return as fast as possible.
For each item I also need to retrieve details, they may be returned a few seconds later.
I could of course create two different routes with HTTP gateways and request first the list, then the details. However, I then have to wait until all details have arrived. I want to send back the list immediately and then the details as soon as I get them.
UPDATE
Following Artem Bilan's advice my flow returns a Flux as payload which merges the list of items as a Mono and the processed items as a Flux.
Note that the example below simulates detail processing of the items by calling toUpperCase; my real use case requires routing and outgoing calls to get the details for each item:
#Bean
public IntegrationFlow sseFlow() {
return IntegrationFlows
.from(WebFlux.inboundGateway("/strings/sse")
.requestMapping(m -> m.produces(MediaType.TEXT_EVENT_STREAM_VALUE))
.mappedResponseHeaders("*"))
.enrichHeaders(Collections.singletonMap("aHeader", new String[]{"foo", "bar"}))
.transform("headers.aHeader")
.<String[]>handle((p, h) -> {
return Flux.merge(
Mono.just(p),
Flux.fromArray(p)
.map(t -> {
return t.toUpperCase();
// return detailsResolver.resolveDetail(t);
}));
})
.get();
}
That comes closer to my goal. When I request data from this flow using curl, I get the list of items immediately and the processed items slightly later:
λ curl http://localhost:8080/strings/sse
data:["foo","bar"]
data:FOO
data:BAR
While simply converting the string to uppercase works fine, I have difficulty to make an outgoing call for details using WebFlux.outboundGateway. The detailsResolver in the commented out code above is defined as follows:
#MessagingGateway
public interface DetailsResolver {
#Gateway(requestChannel = "itemDetailsFlow.input")
Object resolveDetail(String item);
}
#Bean
IntegrationFlow itemDetailsFlow() {
return f -> f.handle(WebFlux.<String>outboundGateway(m ->
UriComponentsBuilder.fromUriString("http://localhost:3003/rest/path/")
.path(m.getPayload())
.build()
.toUri())
.httpMethod(HttpMethod.GET)
.expectedResponseType(JsonNode.class)
.replyPayloadToFlux(false));
}
When I comment in the detailsResolver call and comment out t.toUpperCase, the outboundGateway seems to be set up properly (the log says Subscriber present, Demand signaled) but never gets a response (doesn't reach a breakpoint in ExchangeFunctions.exchange#91).
I have ensured that the DetailsResolver itself is working by getting it as a bean from the context and invoking its method - that gives me a JsonNode response.
What can be the reason?
Yes, I wouldn't use toReactivePublsiher() there because you have a context of the current request. You need fluxes per request. I would use something like Flux.merge(Publisher<? extends I>... sources), where the first Flux is for items and the second is for details per item (something like Tuple2).
For this purpose you really can use something like this:
IntegrationFlows
.from(WebFlux.inboundGateway("/sse")
.requestMapping(m -> m.produces(MediaType.TEXT_EVENT_STREAM_VALUE)))
And your downstream flow should produce Flux as a payload for reply.
I have a sample like this in test cases:
#Bean
public IntegrationFlow sseFlow() {
return IntegrationFlows
.from(WebFlux.inboundGateway("/sse")
.requestMapping(m -> m.produces(MediaType.TEXT_EVENT_STREAM_VALUE))
.mappedResponseHeaders("*"))
.enrichHeaders(Collections.singletonMap("aHeader", new String[] { "foo", "bar", "baz" }))
.handle((p, h) -> Flux.fromArray((String[]) h.get("aHeader")))
.get();
}

Integration flow accessing a paged http resource

I'm trying to consume entirely a paged resource as follow, however my aproach is raising a StackOverflowException.
Any clue abount this? Or a different aproach?
Example: https://gist.github.com/daniel-frank/a88fa4553ed34c348528f51d33c3733b
OK. I see now. Let me simplify your recursive code to show the problem:
private IntegrationFlow getPageFlow() {
return f -> f
.publishSubscribeChannel(ps -> ps
.subscribe(this.nextPageFlow())
);
}
private IntegrationFlow nextPageFlow() {
return f -> f
.publishSubscribeChannel(ps -> ps
.subscribe(this.getPageFlow())
);
}
So, technically we have this structure in the memory:
getPageFlow
nextPageFlow
getPageFlow
nextPageFlow
getPageFlow
and so on.
Another problem here that each .subscribe(this.nextPageFlow()) creates a new instance of the IntegrationFlow meanwhile logically you expect only one.
I understand that you can't declare beans in the IntegrationFlowAdapter impl, but that won't have with the StackOverflowException anyway.
What I see as a problem in your approach is a lack of the MessageChannel abstraction.
You use publishSubscribeChannel everywhere, meanwhile you could just distinguish the logic by the explicit channel definition in your flow.
To break the recursion and keep the code as closer to your solution as possible I'd make like this:
private IntegrationFlow getPageFlow() {
return f -> f
.channel("pageServiceChannel")
.handle(Http
.outboundGateway("https://jobs.github.com/positions.json?description={description}&page={page}")
...
private IntegrationFlow nextPageFlow() {
return f -> f
.filter("!payload.isEmpty()")
.enrichHeaders(e -> e.headerExpression("page", "headers.getOrDefault('page', 0) + 1", true))
.channel("pageServiceChannel");
}
Of course you still have a recursion, but that will be already at run time, logical.

Enriching in parallel after a split

This is a continuation of the shopping cart sample, where we have an external API that allows checkout from a shopping cart. To recap, we have a flow where we create an empty shopping, add line item(s) and finally checkout. All the operations above, happen as enrichments through HTTP calls to an external service. We would like to add line items concurrently (as part of the add line items) call. Our current configuration looks like this:
#Bean
public IntegrationFlow fullCheckoutFlow() {
return f -> f.channel("inputChannel")
.transform(fromJson(ShoppingCart.class))
.enrich(e -> e.requestChannel(SHOPPING_CART_CHANNEL))
.split(ShoppingCart.class, ShoppingCart::getLineItems)
.enrich(e -> e.requestChannel(ADD_LINE_ITEM_CHANNEL))
.aggregate(aggregator -> aggregator
.outputProcessor(g -> g.getMessages()
.stream()
.map(m -> (LineItem) m.getPayload())
.map(LineItem::getName)
.collect(joining(", "))))
.enrich(e -> e.requestChannel(CHECKOUT_CHANNEL))
.<String>handle((p, h) -> Message.called("We have " + p + " line items!!"));
}
#Bean
public IntegrationFlow addLineItem(Executor executor) {
return f -> f.channel(MessageChannels.executor(ADD_LINE_ITEM_CHANNEL, executor).get())
.handle(outboundGateway("http://localhost:8080/api/add-line-item", restTemplate())
.httpMethod(POST)
.expectedResponseType(String.class));
}
#Bean
public Executor executor(Tracer tracer, TraceKeys traceKeys, SpanNamer spanNamer) {
return new TraceableExecutorService(newFixedThreadPool(10), tracer, traceKeys, spanNamer);
}
To add line items in parallel, we are using an executor channel. However, they still seem to be getting processed sequentially when seen in zipkin:
What are we doing wrong? The source for the whole project is on github for reference.
Thanks!
First of all the main feature of Spring Integration is MessageChannel, but it still isn't clear to me why people are missing .channel() operator in between endpoint definitions.
I mean that for your case it must be like:
.split(ShoppingCart.class, ShoppingCart::getLineItems)
.channel(c -> c.executor(executor()))
.enrich(e -> e.requestChannel(ADD_LINE_ITEM_CHANNEL))
Now about your particular problem.
Look, ContentEnricher (.enrich()) is request-reply component: http://docs.spring.io/spring-integration/reference/html/messaging-transformation-chapter.html#payload-enricher.
Therefore it sends request to its requestChannel and waits for reply. And it is done independently of the requestChannel type.
I raw Java we can demonstrate such a behavior with this code snippet:
for (Object item: items) {
Data data = sendAndReceive(item);
}
where you should see that ADD_LINE_ITEM_CHANNEL as an ExecutorChannel doesn't have much value because we are blocked within loop for the reply anyway.
A .split() does exactly similar loop, but since by default it is with the DirectChannel, an iteration is done in the same thread. Therefore each next item waits for the reply for the previous.
That's why you definitely should parallel exactly as an input for the .enrich(), just after .split().

Resources