I have an scenario where I must send messages in order to a rest service and I plan to use a resequencer. The behaviour of this resequencer must be:
Order messages by time in day (hh:mm:ss): data on the message
Release messages only after they stay a period of time in the bus (p.e. 2 minutes)
As the default Resequencer didn't serve for this purpose I decided to develop a custom one changing the ResequencerMessageGroupProcessor for a CustomResequencerMessageGroupProcessor.
I succeded using a service activator but I had to explictly define the output-channel as a property. Isn't there a way to use the output-channel attribute on the xml declaration?
When I use the output-channel attribute the following error occurs:
Caused by: java.lang.IllegalArgumentException: no outputChannel or replyChannel header available
at org.springframework.util.Assert.notNull(Assert.java:112)
at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.sendReplies(AbstractCorrelatingMessageHandler.java:616)
at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.completeGroup(AbstractCorrelatingMessageHandler.java:597)
at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.handleMessageInternal(AbstractCorrelatingMessageHandler.java:405)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
... 46 more
here's my example:
<int:channel id="resequencerChannel"/>
<int:service-activator id="customResequencer" ref="resequencingMessageHandler"
input-channel="resequencerChannel" />
<int:channel id="aggregatedMessageChannel" />
<bean id="resequencingMessageHandler" class="org.springframework.integration.aggregator.ResequencingMessageHandler">
<constructor-arg name="releaseStrategy" ref="timeoutReleaseStrategy"/>
<constructor-arg name="processor" ref="customResequencerMessageGroupProcessor"/>
<constructor-arg name="store" ref="redisMessageStore"/>
<constructor-arg name="correlationStrategy" ref="customCorrelationStrategy"/>
<property name="outputChannel" ref="aggregatedMessageChannel"/>
<property name="sendPartialResultOnExpiry" value="true"></property>
</bean>
<bean id="customResequencerMessageGroupProcessor" class="test.resequencer.CustomResequencerMessageGroupProcessor">
<constructor-arg name="timeout" value="10000"/>
</bean>
<bean id="timeoutReleaseStrategy" class="org.springframework.integration.aggregator.TimeoutCountSequenceSizeReleaseStrategy" >
<constructor-arg name="threshold" value="100000"></constructor-arg>
<constructor-arg name="timeout" value="10000"/>
</bean>
<bean id="customCorrelationStrategy" class="org.springframework.integration.aggregator.HeaderAttributeCorrelationStrategy" >
<constructor-arg name="attributeName" value="correlationId"/>
Also, if you think there is a better way to do this, please, I would apreciatte telling so
Thanks in advance!
Regards
Guzman
When referencing (ref) a MessageHandler from a <service-activator/> the XML output-channel is only applied if the referenced handler is an AbstractReplyProducingMessageHandler (ARPMH).
Components such as routers, aggregators, resequencers, are not considered to be ARPMHs because they sometimes produce a reply, sometimes don't and, in the case of a router, might produce multiple "replies" which doesn't fit the service activator model.
We could probably refactor the aggregator/resequencer to be ARPMHs because they only produce 0 or 1 "reply". We could also add some smarts to the ServiceActivatorFactoryBean to inject the output channel if the reference is an AbstractCorrelatingMessageHandler. Feel free to open an Improvement JIRA Issue.
In the meantime, your solution is the correct work-around.
Related
Referring to my earlier question at URL - Spring integration multithreading requirement - I think I may have figured out the root cause of the issue.
My requirement in brief -
Poll the database after a fixed delay of 1 sec and then publish very limited data to Tibco EMS queue. Now from this EMS queue I have to do the following tasks all in multithreaded fashion :- i) consume the messages, ii) fetch the full data now from the database and iii) converting this data into json format.
My design -
`<int:channel id="dbchannel"/>
<int-jdbc:inbound-channel-adapter id="dbchanneladapter"
channel="dbchannel" data-source="datasource"
query="${selectquery}" update="${updatequery}"
max-rows-per-poll="1000">
<int:poller id="dbchanneladapterpoller"
fixed-delay="1000">
<int:transactional transaction-manager="transactionmanager" />
</int:poller>
</int-jdbc:inbound-channel-adapter>
<int:service-activator input-channel="dbchannel"
output-channel="publishchannel" ref="jdbcmessagehandler" method="handleJdbcMessage" />
<bean id="jdbcmessagehandler" class="com.citigroup.handler.JdbcMessageHandler" />
<int:publish-subscribe-channel id="publishchannel"/>
<int-jms:outbound-channel-adapter id="publishchanneladapter"
channel="publishchannel" jms-template="publishrealtimefeedinternaljmstemplate" />
<int:channel id="subscribechannel"/>
<int-jms:message-driven-channel-adapter
id="subscribechanneladapter" destination="subscriberealtimeinternalqueue"
connection-factory="authenticationconnectionfactory" channel="subscribechannel"
concurrent-consumers="5" max-concurrent-consumers="5" />
<int:service-activator input-channel="subscribechannel"
ref="subscribemessagehandler" method="logJMSMessage" />
<bean id="subscribemessagehandler" class="com.citigroup.handler.SubscribeJMSMessageHandler" />
</beans>
<bean id="authenticationconnectionfactory"
class="org.springframework.jms.connection.UserCredentialsConnectionFactoryAdapter">
<property name="targetConnectionFactory" ref="connectionFactory" />
<property name="username" value="test" />
<property name="password" value="test123" />
</bean>
<bean id="connectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiTemplate">
<ref bean="jndiTemplate" />
</property>
<property name="jndiName" value="app.jndi.testCF" />
</bean>
<bean id="subscriberealtimeinternalqueue" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiTemplate">
<ref bean="jndiTemplate" />
</property>
<property name="jndiName"
value="app.queue.testQueue" />
</bean>
<bean id="jndiTemplate" class="org.springframework.jndi.JndiTemplate">
<property name="environment">
<props>
<prop key="java.naming.factory.initial">com.tibco.tibjms.naming.TibjmsInitialContextFactory
</prop>
<prop key="java.naming.provider.url">tibjmsnaming://test01d.nam.nsroot.net:7222</prop>
</props>
</property>
</bean>`
Issue -
Using message-driven-channel with concurrent consumers value set to 5. However, it looks like just one consumer thread (container-2) is created and is picking up the messages from EMS queue. Please find below the log4j log -
16 Aug 2018 11:31:12,077 INFO SubscribeJMSMessageHandler [subscribechanneladapter.container-2][]: Total count of records read from Queue at this moment is 387
record#1:: [ID=7694066395] record#2:: [ID=7694066423] .. .. .. record#387:: [ID=6147457333]
Probable root cause here -
May be its the first step in the configuration where I am polling the database to fetch the data after a fixed-delay that's causing this multithreading issue. Referring to the logs above, my assumption here is since the number of records fetched is 387 and all these are bundled into a List object (List> message), it is being considered as just 1 message/payload instead of 387 different messages and that's why just one thread/container/consumer is picking up this bundled message. Reason for this assumption is the logs below -
GenericMessage [payload=[{"ID":7694066395},{"ID":7694066423},{"ID":6147457333}], headers={json__ContentTypeId__=class org.springframework.util.LinkedCaseInsensitiveMap, jms_redelivered=false, json__TypeId__=class java.util.ArrayList, jms_destination=Queue[app.queue.testQueue], id=e034ba73-7781-b62c-0307-170099263068, priority=4, jms_timestamp=1534820792064, contentType=application/json, jms_messageId=ID:test.21415B667C051:40C149C0, timestamp=1534820792481}]
Question -
Is my understanding of the root cause correct? If yes then what can be done to treat these 387 messages as individual messages (and not one List object of messages) and publish them one by one without impacting the transaction management??
I had discussed this issue with https://stackoverflow.com/users/2756547/artem-bilan in my earlier post on stackoverflow and I had to check this design by replacing Tibco EMS with ActiveMQ. However, ActiveMQ infrastructure is is still being analysed by our architecture team and so can't be used till its approved.
Oh! Now I see what is your problem. The int-jdbc:inbound-channel-Adapter indeed returns a list of records it could select from the DB. And this whole list is sent as a single message to the JMS. That’s the reason how you see only one thread in the consumer side: there is just only one message to get from the queue.
If you would like to have separate messages for each pulled record, you need to consider to use a <splitter> in between JDBC polling operation and sending to JMS.
I have a http gateway call that's occasionally returning 503 errors. I'd like to configure retry advice around that call, but I don't want to do it for every error, just the 503s.
<int-http:outbound-gateway ... errorHandler="...">
<int-http:request-handler-advice-chain>
<int:retry-advice max-attempts="3" />
</int-http:request-handler-advice-chain>
</int-http:outbound-gateway>
I already have a custom error handler configured that filters statuses (ex: 404) that I don't want to treat as errors, but I don't see an obvious way to control how the advice is applied based on what I can do in the error handler. This question deals with the same issue, but the answer doesn't explain how to control the advice behavior or reissue the request at the error handler level. Is there a specific exception type to throw?
edit: Example based on answer:
<bean id="spelParser" class="org.springframework.expression.spel.standard.SpelExpressionParser" />
<int-http:outbound-gateway ...>
<int-http:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.RequestHandlerRetryAdvice">
<property name="retryTemplate">
<bean class="org.springframework.retry.support.RetryTemplate">
<property name="retryPolicy">
<bean class="org.springframework.retry.policy.ExpressionRetryPolicy">
<constructor-arg index="0" type="org.springframework.expression.Expression" value="#{spelParser.parseExpression('cause.statusCode.value() == 503')}" />
<property name="maxAttempts" value="3" />
</bean>
</property>
<property name="backOffPolicy">
<bean class="org.springframework.retry.backoff.ExponentialBackOffPolicy">
<property name="initialInterval" value="1000" />
<property name="multiplier" value="2" />
</bean>
</property>
</bean>
</property>
</bean>
</int-http:request-handler-advice-chain>
</int-http:outbound-gateway>
Well, for the case when you have a HttpServerErrorException but would like to distinguish it by the statusCode from others and don't retry, I would suggest to take a look into the:
* Subclass of {#link SimpleRetryPolicy} that delegates to super.canRetry() and,
* if true, further evaluates an expression against the last thrown exception.
*
* #author Gary Russell
* #since 1.2
*
*/
#SuppressWarnings("serial")
public class ExpressionRetryPolicy extends SimpleRetryPolicy implements BeanFactoryAware {
Where your expression can be like:
expression="statusCode.value() == 503"
UPDATE
Ah! I see. Since ExpressionRetryPolicy uses TemplateParserContext your expression definitely must be like #{statusCode.value() == 503}. But at the same time it is going to be evaluate during bean factory initialization. I suggest you to do something like this:
<bean id="spelParser" class="org.springframework.expression.spel.standard.SpelExpressionParser"/>
and in the ExpressionRetryPolicy bean definition do:
<constructor-arg index="0" type="org.springframework.expression.Expression"
value="#{spelParser.parseExpression('statusCode.value() == 503')}" />
To overcome the collision.
3 of the webservices that I am working on uses Springs, SimpleMessageStore for storing the messages. For some reason it is causing memory leak in production env and I am unable to reproduce it in the lower environments. I am new to spring integration and need help in understanding what might be causing this.
the spring config code looks like this:
<!-- MESSAGE STORES -->
<bean id="monitoringHeaderRequestMsgStore" class="org.springframework.integration.store.SimpleMessageStore"/>
<bean id="gbqHeaderRequestMsgStore" class="org.springframework.integration.store.SimpleMessageStore"/>
<bean id="bondAgreementResponseMsgStore" class="org.springframework.integration.store.SimpleMessageStore"/>
<bean id="bondWIthRulesRequestMsgStore" class="org.springframework.integration.store.SimpleMessageStore"/>
<bean id="ProcessVariableMessageStores" class="com.aviva.uklife.investment.impl.ProcessVariableMessageStores">
<property name="_monitoringHeaderRequestMsgStore" ref="monitoringHeaderRequestMsgStore"/>
<property name="_gbqHeaderRequestMsgStore" ref="gbqHeaderRequestMsgStore"/>
<property name="_bondWIthRulesRequestMsgStore" ref="bondWIthRulesRequestMsgStore"/>
<property name="_bondAgreementResponseMsgStore" ref="bondAgreementResponseMsgStore"/>
</bean>
<!-- Retrieve stored MonitoringHeaderRequest -->
<int:transformer expression="headers.get('#{T(.....Constants).MONITORING_HEADER_REQUEST_CLAIM_CHECK_ID}')"/>
<int:claim-check-out message-store="monitoringHeaderRequestMsgStore" remove-message="false"/>
<!-- Store HeaderRequest -->
<int:gateway request-channel="header-req-store-channel"/>
<!-- PROCESS VARIABLES STORAGE IN STORE CHANNELS WITH KEY OR CLAIMCHECK ID -->
<int:chain input-channel="monitoring-header-req-store-channel">
<int:claim-check-in message-store="monitoringHeaderRequestMsgStore"/>
<int:header-enricher>
<int:header name="#{T(....Constants).MONITORING_HEADER_REQUEST_CLAIM_CHECK_ID}" expression="payload"/>
</int:header-enricher>
<int:claim-check-out message-store="monitoringHeaderRequestMsgStore" remove-message="false"/>
</int:chain>
thank you
To be honest, it isn't recommended to use SimpleMessageStore in the production environment. That's because of memory-leak, as you noticed. If you don't clear the MessageStore periodically.
Right, there are might be some cases, when you need to keep messages in the MessageStore for the long time. So consider to replace SimpleMessageStore with some persistent MessageStore.
From other side we need to have more info on the matter to provide better help.
Maybe you just have several aggregators and don't use expire-groups-upon-completion = "true"...
I have an application relying on Spring Integration (4.0.4.RELEASE) and RabbitMQ. My flow is as follow:
Messages are put in queue via a process (they do not expect any answer):
Gateway -> Channel -> RabbitMQ
And then drained by another process:
RabbitMQ --1--> inbound-channel-adapter A --2--> chain B --3--> aggregator C --4--> service-activator D --5--> final service-activator E
Explanations & context
The specific thing is that nowhere in my application I am using a splitter: aggregator C just waits for enough messages to come, or for a timeout to expire, and then forwards the batch to service D. Messages can get stuck in aggregator C for quite a long time, and should NOT be considered as consumed there. They should only be consumed once service D successfully completes. Therefore, I am using MANUAL acknowledgement on inbound-channel-adapter A and service E is in charge of acknowledging the batch.
Custom aggregator
I solved the acknowledgement issue I had when set to AUTO by redefining the aggregator. Indeed, messages are acknowledged immediately if any asynchronous process occurs in the flow (see question here). Therefore, I switched to MANUAL acknowledgement and implemented the aggregator like this:
<bean class="org.springframework.integration.config.ConsumerEndpointFactoryBean">
<property name="inputChannel" ref="channel3"/>
<property name="handler">
<bean class="org.springframework.integration.aggregator.AggregatingMessageHandler">
<constructor-arg name="processor">
<bean class="com.test.AMQPAggregator"/>
</constructor-arg>
<property name="correlationStrategy">
<bean class="com.test.AggregatorDefaultCorrelationStrategy" />
</property>
<property name="releaseStrategy">
<bean class="com.test.AggregatorMongoReleaseStrategy" />
</property>
<property name="messageStore" ref="messageStoreBean"/>
<property name="expireGroupsUponCompletion" value="true"/>
<property name="sendPartialResultOnExpiry" value="true"/>
<property name="outputChannel" ref="channel4"/>
</bean>
</property>
</bean>
<bean id="messageStoreBean" class="org.springframework.integration.store.SimpleMessageStore"/>
<bean id="messageStoreReaperBean" class="org.springframework.integration.store.MessageGroupStoreReaper">
<property name="messageGroupStore" ref="messageStore" />
<property name="timeout" value="${myapp.timeout}" />
</bean>
<task:scheduled-tasks>
<task:scheduled ref="messageStoreReaperBean" method="run" fixed-rate="2000" />
</task:scheduled-tasks>
I wanted indeed to aggregate the headers in a different way, and keep the highest value of all the amqp_deliveryTag for later multi-acknoledgement in service E (see this thread). This works great so far, apart from the fact that it is far more verbose than the typical aggregator namespace (see this old Jira ticket).
Services
I am just using basic configurations:
chain-B
<int:chain input-channel="channel2" output-channel="channel3">
<int:header-enricher>
<int:error-channel ref="errorChannel" /> // Probably useless
</int:header-enricher>
<int:json-to-object-transformer/>
<int:transformer ref="serviceABean"
method="doThis" />
<int:transformer ref="serviceBBean"
method="doThat" />
</int:chain>
service-D
<int:service-activator ref="serviceDBean"
method="doSomething"
input-channel="channel4"
output-channel="channel5" />
Error management
As I rely on MANUAL acknowledgement, I need to manually reject messages as well in case an exception occurs. I have the following definition for inbound-channel-adapter A:
<int-amqp:inbound-channel-adapter channel="channel2"
queue-names="si.queue1"
error-channel="errorChannel"
mapped-request-headers="*"
acknowledge-mode="MANUAL"
prefetch-count="${properties.prefetch_count}"
connection-factory="rabbitConnectionFactory"/>
I use the following definition for errorChannel:
<int:chain input-channel="errorChannel">
<int:transformer ref="errorUnwrapperBean" method="unwrap" />
<int:service-activator ref="amqpAcknowledgerBean" method="rejectMessage" />
</int:chain>
ErrorUnwrapper is based on this code and the whole exception detection and message rejection works well until messages reach aggregator C.
Problem
If an exception is raised while processing the messages in service-activator D, then I see this exception but errorChannel does not seem to receive any message, and my ErrorUnwrapper unwrap() method is not called. The tailored stack traces I see when an Exception("ahahah") is thrown are as follow:
2014-09-23 16:41:18,725 ERROR o.s.i.s.SimpleMessageStore:174: Exception in expiry callback
org.springframework.messaging.MessageHandlingException: java.lang.Exception: ahahaha
at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:78)
at org.springframework.integration.handler.ServiceActivatingHandler.handleRequestMessage(ServiceActivatingHandler.java:71)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:170)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
(...)
Caused by: java.lang.Exception: ahahaha
at com.myapp.ServiceD.doSomething(ServiceD.java:153)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
(...)
2014-09-23 16:41:18,733 ERROR o.s.s.s.TaskUtils$LoggingErrorHandler:95: Unexpected error occurred in scheduled task.
org.springframework.messaging.MessageHandlingException: java.lang.Exception: ahahaha
(...)
Question
How can one tell the services that process messages coming from such an aggregator to publish errors to errorChannel? I tried to specify in the header via a header-enricher the error-channel with no luck. I am using the default errorChannel definition, but I tried as well to change its name and redefine it. I am clueless here, and even though I found this and that, I have not managed to get it to work. Thanks in advance for your help!
As you see by StackTrace your process is started from the MessageGroupStoreReaper Thread, which is initiated from the default ThreadPoolTaskScheduler.
So, you must provide a custom bean for that:
<bean id="scheduler" class="org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler">
<property name="errorHandler">
<bean class="org.springframework.integration.channel.MessagePublishingErrorHandler">
<property name="defaultErrorChannel" ref="errorChannel"/>
</bean>
</property>
</bean>
<task:scheduled-tasks scheduler="scheduler">
<task:scheduled ref="messageStoreReaperBean" method="run" fixed-rate="2000" />
</task:scheduled-tasks>
However I see the benefits from having the error-channel on the <aggregator>, where we really have several points from different detached Threads, with wich we can't get deal normally.
I've got for a JPA Outbound-channel-adapter both transactional and request-handler-advice-chain. In the advice-chain I try to log the Exception, when it happens.
It iss not logged, but I know that it actually happend since the Message was sent to failover clickDbFailoverChannel . What can be a problem with it? Is it a bug in Spring Integration?
<int:channel id="clickDbWithFailoverChannelSite-1">
<int:dispatcher load-balancer="none" task-executor="clickDbSiteRouterExecutor"/>
</int:channel>
<int:bridge input-channel="clickDbWithFailoverChannelSite-1"
output-channel="jpaOutboundChannelSite-1" order="1" send-timeout="100" />
<int:bridge input-channel="clickDbWithFailoverChannelSite-1"
output-channel="clickDbFailoverChannel" order="2" />
<int-jpa:outbound-channel-adapter id="jpaOutboundChannelSite-1"
persist-mode="PERSIST" flush-size="100" entity-manager-factory="emfSite-1">
<int-jpa:transactional transaction-manager="transactionManagerSite-1" />
<int-jpa:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="failureChannel" ref="clickDbFailureLogger"/>
<property name="onFailureExpression" value="#exception"/>
</bean>
</int-jpa:request-handler-advice-chain>
</int-jpa:outbound-channel-adapter>
OK. I can guess where is your issue. The real exception to rollback the transaction is caused before an internal logic, where <request-handler-advice-chain> does the stuff. That's why your ExpressionEvaluatingRequestHandlerAdvice doesn't get a failure message.
To workaround your rollback issue, you should replace <int-jpa:transactional> with <tx:advice> within <int-jpa:request-handler-advice-chain>.
You should understand here that <int-jpa:transactional> is for entire MessageHandler.handleMessage, but <int-jpa:request-handler-advice-chain> is just for its part - AbstractReplyProducingMessageHandler.handleRequestMessage.
UPDATE
TX Advice should be like this:
<tx:advice transaction-manager="transactionManagerSite-1"/>
<tx:attributes>
<tx:method name="*"/>
</tx:attributes>
</tx:advice>