Messages "in delivery" status not released on ActiveMQ Artemis broker - spring-integration

Question

I have setup two ActiveMQ Artemis brokers (version 2.17) on a master-slave configuration with a shared file system for HA. During testing with
heavy traffic I would stop master broker and see that the slave takes over and starts forwarding messages to consumers. After a while
I can witness that a number of messages seem to have been stuck in queues. The "stuck" messages are considered as "in delivery" as I can
see from the Artemis UI when queried.
My issue is that even when I restart the master broker these messages are not delivered to the consumer and remain stuck even if more
messages are still populating the same queue and the queue has consumers. My assumption was that it had to do with previous connections
setup by consumers still remaining active because there were not acknowledged.
So I did try to setup <connection-ttl-override> on broker, or on client connection string
(tcp://host1:61616,tcp://host2:61616)?ha=true&connectionTtl=60000&reconnectAttempts=-1) but that did not seemed to have any effect
since the connection would not be closed and the messages were not released.
For consuming messages I am using Artemis JMS Spring client with a CachingConnectionFactory but also tried JmsPoolConnectionFactory
to no avail.
<bean id="connectionFactory"
class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory">
<bean class="org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory">
<property name="user" value="${spring.artemis.user}" />
<property name="password" value="${spring.artemis.password}" />
<property name="brokerURL" value="${spring.artemis.brokerUrl}" />
</bean>
</property>
<property name="cacheConsumers" value="false"/>
</bean>
<int-jms:message-driven-channel-adapter
connection-factory="connectionFactory"
destination="myQueue"
message-converter="messageConverter"
channel="inputChannel"
concurrent-consumers="${processing.poolsize}"
max-concurrent-consumers="${max.processing.poolsize}"
error-channel="errorChannel"
acknowledge="transacted"
/>
The only remedy to this problem seems to be to restart consumer app which unblocks messages but that is not the desirable option.
How can I resolve this issue? Is there a way to release messages without manual intervention or restarting consumer app?

Answer 1

Using variable concurrency with the CachingConnectionFactory could be the issue.
When an "idle" consumer is returned to the cache, the broker doesn't know the consumer is not active and still sends messages to it.
The caching factory is really only needed on the producer side (JmsTemplate) to avoid creating a connection and session for each send.
It's best to not use the CCF if you use variable concurrency; or configure it to not cache consumers, or don't use variable concurrency.