Integration flow accessing a paged http resource - spring-integration

Question

I'm trying to consume entirely a paged resource as follow, however my aproach is raising a StackOverflowException.
Any clue abount this? Or a different aproach?
Example: https://gist.github.com/daniel-frank/a88fa4553ed34c348528f51d33c3733b

Answer 1

OK. I see now. Let me simplify your recursive code to show the problem:
private IntegrationFlow getPageFlow() {
return f -> f
.publishSubscribeChannel(ps -> ps
.subscribe(this.nextPageFlow())
);
}
private IntegrationFlow nextPageFlow() {
return f -> f
.publishSubscribeChannel(ps -> ps
.subscribe(this.getPageFlow())
);
}
So, technically we have this structure in the memory:
getPageFlow
nextPageFlow
getPageFlow
nextPageFlow
getPageFlow
and so on.
Another problem here that each .subscribe(this.nextPageFlow()) creates a new instance of the IntegrationFlow meanwhile logically you expect only one.
I understand that you can't declare beans in the IntegrationFlowAdapter impl, but that won't have with the StackOverflowException anyway.
What I see as a problem in your approach is a lack of the MessageChannel abstraction.
You use publishSubscribeChannel everywhere, meanwhile you could just distinguish the logic by the explicit channel definition in your flow.
To break the recursion and keep the code as closer to your solution as possible I'd make like this:
private IntegrationFlow getPageFlow() {
return f -> f
.channel("pageServiceChannel")
.handle(Http
.outboundGateway("https://jobs.github.com/positions.json?description={description}&page={page}")
...
private IntegrationFlow nextPageFlow() {
return f -> f
.filter("!payload.isEmpty()")
.enrichHeaders(e -> e.headerExpression("page", "headers.getOrDefault('page', 0) + 1", true))
.channel("pageServiceChannel");
}
Of course you still have a recursion, but that will be already at run time, logical.