Correct way to call commit offsets - spring-integration

Question

I am using Spring integration framework, where input channel is kafka and output is jdbc.
I want to manually commit kafka offsets, only after jdbcMessageHandler successfully processed each kafka message.
#Bean
#ServiceActivator(inputChannel = "outChannel")
public MessageHandler jdbcMessageHandler() {
JdbcMessageHandler jdbcMessageHandler = new JdbcMessageHandler(getDataSource(), getSql());
jdbcMessageHandler.setPreparedStatementSetter((ps, message) -> {
Item item = ((Item) message.getPayload());
ps.setString(1, item.getName());
Acknowledgment ack = (Acknowledgment) message.getHeaders().get(KafkaHeaders.ACKNOWLEDGMENT);
ack.acknowledge();
}
return jdbcMessageHandler;
}
#Bean
public ConsumerFactory<String, String> consumerFactory() {
...
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
...
return new DefaultKafkaConsumerFactory<>(props);
}
#Bean
public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, String>> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL);
...
return factory;
}
I tried , as can be seen above:
Acknowledgment ack = (Acknowledgment) message.getHeaders().get(KafkaHeaders.ACKNOWLEDGMENT);
ack.acknowledge();
But it yields unwanted effect:
each kafka message can have n - Items, therefore transformer returns a List of items , so ack.acknowledge() (which is commit) will be called for each item, n times!
And I want to call commit once only , after all items of the message are handled.
Update
After applying the recommended from answer.
I set on
ConcurrentKafkaListenerContainerFactory
.setErrorHandler(new SeekToCurrentErrorHandler(new FixedBackOff(1000L, 99L)));
I also have
#ServiceActivator(inputChannel = "errorChannel")
public void onError(ErrorMessage message) {
}
What happens: in JdbcMessageHandler error occurs,
onError is triggered once. no retries, kafka offset committed.
I need to prevent committing offsets.
Update 2
The flow:
1)
kafkainput -> PublishSubscribeChannel , attached to KafkaMessageDrivenChannelAdapter -> KafkaListenerContainerFactory -> KafkaMessageListenerContainer
also attempted to set listener.setErrorHandler(...
2)
subscribers:
#Transformer(inputChannel = "kafkainput", outputChannel = "aggregator")
#ServiceActivator(inputChannel = "kafkainput")
aggregator -> PublishSubscribeChannel
subscriber:
`#ServiceActivator(inputChannel = "aggregator")`
public FactoryBean<MessageHandler> aggregatorFactoryBean(..
AggregatorFactoryBean aggregatorFactoryBean =
aggregatorFactoryBean.setOutputChannel(outputChannel);
outputChannel -> DirectChannel
subscriber:
#ServiceActivator(inputChannel = "outputChannel")
public MessageHandler jdbcMessageHandler() {
The error occurs in jdbc.
Then only onError(..) is triggered
Update 3
Did a lot of changes, got rid of aggregator and instead using:
= new KafkaMessageDrivenChannelAdapter<>(container, KafkaMessageDrivenChannelAdapter.ListenerMode.batch)
kafkaMessageDrivenChannelAdapter.setBatchMessageConverter(new BatchMessagingMessageConverter(converter()));
kafkaMessageDrivenChannelAdapter.setErrorChannelName("error");
kafkaMessageDrivenChannelAdapter.setOutputChannelName("splitter");
set in KafkaListenerContainerFactory
factory.setErrorHandler(new SeekToCurrentErrorHandler(new FixedBackOff(500,10000)));
set in `KafkaMessageListenerContainer'
.setAckMode(ContainerProperties.AckMode.BATCH);
and I have splitter:
#Splitter(inputChannel = "splitter", outputChannel = "outputChannel")
In kafka I put corrupted message , so error occurs in splitter (I am throwing MessagingException, then `onError' is triggered once only, and kafka offsets are commited!
#ServiceActivator(inputChannel = "error")
public void onError(ErrorMessage message) {
}
why its not retrying number of times that it was configured and why does it commits offsets right away?

Answer 1

You need to consider to make yourself familiar with Publish-Subscribe pattern. For example a PublishSubscribeChannel can have a several subscribers to handle the same message. So, along side with your transformer to produce a list of items, you also can have a service activator which would call that ack.acknowledge() only once, when all the items in the batch are processed by JDBC channel adapter. But, of course, the input channel of your transformer must be that PublishSubscribeChannel. You also may consider to specify an order option explicitly for your subscribers to be sure that they are called in a proper order.
Another way is a RecipientListRouter.
See docs for more info:
https://docs.spring.io/spring-integration/reference/html/core.html#channel-implementations-publishsubscribechannel
https://docs.spring.io/spring-integration/reference/html/message-routing.html#router-implementations-recipientlistrouter
and of course an #Order annotation JavaDocs.