WARN JdbcChannelMessageStore Message with id was not deleted - spring-integration

I have a queue channel backed by a JdbcChannelMessageStore. I have two instances of this application and with high concurrency I have this warning:
2020-03-13 19:25:38,209 task-scheduler-5 WARN JdbcChannelMessageStore:652 - Message with id '06b73eab-727a-780f-d0fa-1b0e0dd1ea20' was not deleted.
Is there a way to remove them?
As far as my understanding, messages are being read twice, am I correct?
I am using SI 4.3.19.RELEASE. Here is my spring flow
<int:channel id="channel">
<int:queue message-store="messageStoreBean"/>
</int:channel>
<int:header-value-router input-channel="channel
header-name="name" >
<int:poller max-messages-per-poll="2" fixed-rate="500" >
<int:transactional />
</int:poller>
...
</int:header-value-router>
<bean id="storeQueryProviderBean" class="org.springframework.integration.jdbc.store.channel.PostgresChannelMessageStoreQueryProvider" />
<bean id="messageStoreBean" class="org.springframework.integration.jdbc.store.JdbcChannelMessageStore">
<property name="dataSource" ref="messageStoreDataSource" />
<property name="channelMessageStoreQueryProvider" ref="storeQueryProviderBean" />
<property name="region" value="region" />
</bean>

It looks like PostgreSQL doesn't guarantee exclusive reading with transactions and LIMIT 1 FOR UPDATE.
Anyway that WARN is just a note that some other process has removed the message. Nothing is duplicated if other process is similar to that poller:
public Message<?> pollMessageFromGroup(Object groupId) {
final String key = getKey(groupId);
final Message<?> polledMessage = this.doPollForMessage(key);
if (polledMessage != null) {
if (!this.doRemoveMessageFromGroup(groupId, polledMessage)) {
return null;
}
}
return polledMessage;
}
You see if message was not removed, we return null therefore nothing to poll at the moment.
You can turn off the warning level for the org.springframework.integration.jdbc.store.JdbcChannelMessageStore to avoid that message specifying a category level as ERROR in your logging config.

Related

handling exceptions for Http rest endpoints in spring integration

I need to throw an exception to the end user, can you please assist with this?
<int-http:request-handler-advice-chain>
<!--<bean class="org.springframework.integration.handler.advice.RequestHandlerRetryAdvice">
<property name="recoveryCallback">
<bean class="org.springframework.integration.handler.advice.ErrorMessageSendingRecoverer">
<constructor-arg ref="retryErrorChannel" />
</bean>
</property>
<property name="retryTemplate" ref="retryLoadTemplate" />
</bean>-->
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="successChannel" ref="afterSuccessFetchChannel" />
<property name="failureChannel" ref="afterFailFetchChannel" />
</bean>
</int-http:request-handler-advice-chain>
</int-http:outbound-gateway>
<int:transformer ref="testTransformer" method="processDetails" />
<int:transformer input-channel="afterSuccessFetchChannel" output-channel="goodResultChannel"
expression="'Fetching load id: ' + payload + ' details was successful'" />
<int:transformer input-channel="afterFailFetchChannel" output-channel="badResultChannel"
expression="payload + ' was bad, with reason: ' + payload.cause.message" />
<int:logging-channel-adapter id="badResultChannel" level="ERROR"/>
<int:logging-channel-adapter id="goodResultChannel" level="INFO" />
How and where to handle custom exceptions and how to re throw them?
My api is throwing 404 Not found exception, need to catch and re throw it to the user?
for now, I am just using logging channel, I don't really know the way to deal with exceptions in spring integration, can you please help?
So, instead of that <int:logging-channel-adapter id="badResultChannel" level="ERROR"/> you just need to write some POJO method which would throw some other exception after its handling. The ExpressionEvaluatingRequestHandlerAdvice has this property returnFailureExpressionResult to be set to true.
The logic is like this:
try {
evalResult = this.onFailureExpression.getValue(prepareEvaluationContextToUse(exception), message);
}
catch (Exception e) {
evalResult = e;
logger.error("Failure expression evaluation failed for " + message + ": " + e.getMessage());
}
and then:
if (this.onFailureExpression != null) {
Object evalResult = evaluateFailureExpression(message, actualException);
if (this.returnFailureExpressionResult) {
return evalResult;
}
}
So, your custom thrown exception is going to be sent as a reply message payload back to the caller. And there probably an MVC layer can simply convert it to respective HTTP response for the client.

Reuse of service-activator for several gateway methods with Splitter

does somebody happen to know if it is valid to reuse a service activator and so also the output-channel using several methods (inbound) especially with a splitter and aggregator.
--> Always with result on the gateway.
In several tests it seems to work fine. As soon I added a splitter with an aggregator I get wrongs result routed to the gateway which then fails with a conversion exception (here in my case it cannot convert boolean to integer).
Thanks,
Paul
Flow
Spring Integration Config
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int="http://www.springframework.org/schema/integration"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd">
<int:gateway service-interface="activity.BulbMessageGateway">
<int:method name="sendToBulb" request-channel="bulbMessages" reply-channel="bulbSendResult"></int:method>
<int:method name="updateHomeLightStatus" request-channel="homeBulbEntity" reply-channel="homeBulbEntityResult">
</int:method>
<int:method name="updateLightStatus" request-channel="bulbEntities" reply-channel="bulbSendResult">
<int:header name="homeId" expression="#args[0].homeId"/>
<int:header name="bulbId" expression="#args[0].strongId"/>
</int:method>
</int:gateway>
<int:channel id="bulbMessages" />
<int:channel id="bulbSendResult" />
<int:channel id="bulbEntities" />
<int:channel id="homeBulbEntity" />
<int:channel id="homeBulbEntityResult" />
<int:chain input-channel="homeBulbEntity" output-channel="bulbEntities">
<int:splitter expression="payload.bulbs" />
<int:header-enricher>
<int:header name="bulbId" expression="payload.strongId"/>
<int:header name="homeId" expression="payload.homeId"/>
</int:header-enricher>
</int:chain>
<int:transformer method="bulbToLightStatus" input-channel="bulbEntities" output-channel="bulbMessages">
<bean class="util.BulbTransformer"></bean>
</int:transformer>
<int:aggregator input-channel="bulbSendResult" output-channel="homeBulbEntityResult" method="aggregate">
<bean class="util.BooleanAggregator" />
</int:aggregator>
<int:service-activator input-channel="bulbMessages" output-channel="bulbSendResult" method="send">
<bean class="activity.BulbWebsocketMessageSenderBA" />
</int:service-activator>
</beans>
Unit test
#Test
public void sendMessageNoReceiver() {
assertFalse(gateway.sendToBulb(new HomeId("1"), new BulbId("1"), BulbMessageBuilder.restart("foo")));
}
#Test
public void sendMessageWithReceiver() {
MockSession<BulbId, BulbBE> bulbSession = new MockSession<BulbId, BulbBE>(new BulbBE(HomeId.of("1"), BulbId.of("1"), "bulb", "pass"));
registry.addBulbSession(bulbSession);
assertTrue(gateway.sendToBulb(new HomeId("1"), new BulbId("1"), BulbMessageBuilder.restart("foo")));
assertEquals(1, bulbSession.receivedMessages());
}
#Test
public void updateBulbStatus() {
final MockSession<BulbId, BulbBE> bulbSession1 = new MockSession<BulbId, BulbBE>(new BulbBE(HomeId.of("1"), BulbId.of("1"), "bulb", "pass"));
assertFalse(gateway.updateLightStatus(bulbSession1.getIdentity()));
registry.addBulbSession(bulbSession1);
assertTrue(gateway.updateLightStatus(bulbSession1.getIdentity()));
assertEquals(1, bulbSession1.receivedMessages());
final MockSession<BulbId, BulbBE> bulbSession2 = new MockSession<BulbId, BulbBE>(new BulbBE(HomeId.of("1"), BulbId.of("2"), "bulb", "pass"));
assertFalse(gateway.updateLightStatus(bulbSession2.getIdentity()));
registry.addBulbSession(bulbSession2);
assertTrue(gateway.updateLightStatus(bulbSession2.getIdentity()));
assertTrue(gateway.updateLightStatus(bulbSession2.getIdentity()));
assertEquals(2, bulbSession2.receivedMessages());
assertEquals(1, bulbSession1.receivedMessages());
}
#Test
public void updateHomeBulbStatus() {
final HomeBE home = new HomeBE();
home.setId(new ObjectId());
final MockSession<BulbId, BulbBE> bulbSession1 = new MockSession<BulbId, BulbBE>(new BulbBE(home.getStrongId(), BulbId.of("1"), "bulb", "pass"));
registry.addBulbSession(bulbSession1);
final MockSession<BulbId, BulbBE> bulbSession2 = new MockSession<BulbId, BulbBE>(new BulbBE(home.getStrongId(), BulbId.of("2"), "bulb", "pass"));
registry.addBulbSession(bulbSession2);
home.addBulb(bulbSession1.getIdentity());
assertEquals(1, gateway.updateHomeLightStatus(home));
assertEquals(1, bulbSession1.receivedMessages());
assertEquals(0, bulbSession2.receivedMessages());
home.addBulb(bulbSession2.getIdentity());
assertEquals(2, gateway.updateHomeLightStatus(home));
assertEquals(2, bulbSession1.receivedMessages());
assertEquals(1, bulbSession2.receivedMessages());
}
The last test fails if it is executed together with the other tests. It passes if it is executed alone.
The error is that the last method (using the splitter) receives now a boolean, which seems to be a result of the other two methods registered. The result of these methods is a boolean.
Please, share config on the matter.
And according your graph it would be better if you'd minimize the config as much as possible to isolate the problem.
From other side, please, be more specific: your question is fully unclear.
That is your own service. How can we be sure that it is safe to be used in different places? Only you, as an author, can determine that.
UPDATE
Sorry for the delay. Was busy with the release.
And thank for sharing the config for your use-case.
Now I see the problem.
You use everywhere on the gateway's methods a reply-channel. See the documentation on the matter when you need that:
Typically you don’t have to specify the default-reply-channel, since a Gateway will auto-create a temporary, anonymous reply channel, where it will listen for the reply. However, there are some cases which may prompt you to define a default-reply-channel (or reply-channel with adapter gateways such as HTTP, JMS, etc.).
Since you use the same bulbSendResult in different places the behavior is really unpredictable. Moreoever that channel is DirectChannel, so the round-robin balancer are on the scene.
You should get rid of those reply-channel's at all and just rely from your downstream components on the replyChannel header. Therefore you should remove those output-channels in the components which are intended to return replies to your gateway.
For example the last service-activator should be just like this:
<int:service-activator input-channel="bulbMessages" method="send">
<bean class="activity.BulbWebsocketMessageSenderBA"/>
</int:service-activator>
Since you general question how to reuse this service-activator, I'm answering to the question having the config from you:
<int:chain input-channel="homeBulbEntity">
<int:splitter expression="payload.bulbs"/>
<int:header-enricher>
<int:header name="bulbId" expression="payload.strongId"/>
<int:header name="homeId" expression="payload.homeId"/>
</int:header-enricher>
<int:transformer method="bulbToLightStatus">
<bean class="util.BulbTransformer"/>
</int:transformer>
<int:gateway request-channel="bulbMessages"/>
<int:aggregator method="aggregate">
<bean class="util.BooleanAggregator"/>
</int:aggregator>
</int:chain>
Pay attention to the absent output-channel for the <chain>. Therefore it sends reply directly to the replyChannel from headers and as a return to your updateHomeLightStatus gateway's method.
Another trick is that <int:gateway request-channel="bulbMessages"/> which sends messages in the middle of the <chain> flow to your <service-activator> and wait for the reply from there exactly the same way as a top-level gateway - via replyChannel header. For the <service-activator> without an output-channel it is a black-box where to send the reply. It uses just replyChannel from headers!
After receiving the reply gateway in the <chain> push the message to the <aggregator>.
When aggregator will do its logic, the result will be send to the top-level gateway as an output from the <chain>.
That's all.
Let me know what else isn't clear here.

How to extract the data from a kafka inbound message with spring

I am building a Kafka consumer with spring. My configuration seems to be pretty straightforward. Messages are being consumed and saved in files. However, the payload is cryptic and I can't get the data (short "hello world" messages).
This is what I get when I access the payload (e.g. when I set up a transformer bean btween in inbound kafka and the outboud file:
{test-topic={0=[[B#713c9d72, [B#7d656f90, [B#26bb8c83, [B#4b959d83 [B#5ed74e8e]}}
My question is: How do I access the actual payload (the "hellow world" string")?
My configuration is :
<int:channel id="inputFromKafka">
<int:queue />
</int:channel>
<int:poller
max-messages-per-poll="5" default = "true" fixed-delay="10" time-unit="MILLISECONDS"/>
<int-kafka:inbound-channel-adapter
id="kafkaInboundChannelAdapter" kafka-consumer-context-ref="consumerContext"
auto-startup="true" channel="inputFromKafka">
<int-kafka:consumer-context id="consumerContext"
consumer-timeout="40000" zookeeper-connect="zookeeperConnect">
<int-kafka:consumer-configurations>
<int-kafka:consumer-configuration
group-id="group12" max-messages="5">
<int-kafka:topic id="test-topic" streams="1" />
</int-kafka:consumer-configuration>
</int-kafka:consumer-configurations>
</int-kafka:consumer-context>
<int-kafka:zookeeper-connect id="zookeeperConnect"
zk-connect="localhost:2181" zk-connection-timeout="6000"
zk-session-timeout="6000" zk-sync-time="2000" />
<file:outbound-channel-adapter id="filesOut"
directory="/tmp/fromKafka">
</file:outbound-channel-adapter>
You are seeing raw byte[].
Add...
<bean id="decoder"
class="org.springframework.integration.kafka.serializer.common.StringDecoder" />
and
<int-kafka:consumer-configuration
value-decoder="decoder"
...

Duplicate call in select-sql-parameter-source

I'm working with a dynamic query, using select-sql-parameter-source to search the information that I need.
This is my configuration:
<int-jdbc:inbound-channel-adapter query="SELECT * FROM CUSTOMER WHERE CUSTOMER.LASTUPDATE_ACTIVE < TO_DATE(:last_process_date,'YYYY-MM-DD HH24:Mi:SS') "
channel="headerEnricher.customerBR01"
update=""
row-mapper="customerRowMapper"
data-source="jdbcTemplate"
max-rows-per-poll="0"
select-sql-parameter-source="parameterSource.customerBR01">
<!-- Cron Time -->
<int:poller fixed-rate="50" time-unit="SECONDS">
</int:poller>
</int-jdbc:inbound-channel-adapter>
<!-- This is to get last process date -->
<bean id="parameterSource.customerBR01" factory-bean="parameterSourceFactory.customerBR01" factory-method="createParameterSourceNoCache">
<constructor-arg value="" />
</bean>
<bean id="parameterSourceFactory.customerBR01" class="org.springframework.integration.jdbc.ExpressionEvaluatingSqlParameterSourceFactory">
<property name="parameterExpressions">
<map>
<!-- Here we get the last process date -->
<entry key="last_process_date" value="#hsqlHistoricProcessServiceDateDAO.getLastProcessDate(3,1,'CUSTOMER')" />
</map>
</property>
</bean>
I was looking that loggin appeared twice, so I changed my code in this function :
hsqlHistoricProcessServiceDateDAO.getLastProcessDate
To return only an account variable.
Code of function hsqlHistoricProcessServiceDateDAO.getLastProcessDate is the following:
private int contador = 0;
public String getLastProcessDate(Integer country, Integer business, String tableName) {
contador++;
System.out.println("Contador "+ contador);
return Integer.toString(contador);
}
And its result is :
Contador 1
Contador 2
So, this method is called twice, and I need only one call, because in the "real code" I have all logging twice for that.
For your use case, you don't need to disable the cache; use this instead...
<bean id="parameterSource.customerBR01"
factory-bean="parameterSourceFactory.customerBR01"
factory-method="createParameterSource">
<constructor-arg value="" />
</bean>
The ...NoCache version is needed when the same key is used in multiple parameters and you want each one to be re-evaluated.
Disabling the cache has this additional side-effect because the getValue() method is called twice for each use of the key. One call is from NamedParameterUtils.substituteNamedParameters(); the second is from NamedParameterUtils.buildValueArray().

File inbound-channel-adapter spring integration for Multiple Files aggregation into one master File for Job processing

I have written a code to combined multiple files into one single Master file.
The issue is with int-transformer where I am getting one file at a time although I have aggregated List of File in composite Filter of File inbound-channel-adapter. The List of File size in composite filter is correct but in Transformer bean the List of File size is always one and not getting the correct list size aggregated file by the filter.
Here is my config:
<!-- Auto Wiring -->
<context:component-scan base-package="com.nt.na21.nam.integration.*" />
<!-- intercept and log every message -->
<int:logging-channel-adapter id="logger"
level="DEBUG" />
<int:wire-tap channel="logger" />
<!-- Aggregating the processed Output for OSS processing -->
<int:channel id="networkData" />
<int:channel id="requests" />
<int-file:inbound-channel-adapter id="pollProcessedNetworkData"
directory="file:${processing.files.directory}" filter="compositeProcessedFileFilter"
channel="networkData">
<int:poller default="true" cron="*/20 * * * * *" />
</int-file:inbound-channel-adapter>
<bean id="compositeProcessedFileFilter"
class="com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine" />
<int:transformer id="aggregateNetworkData"
input-channel="networkData" output-channel="requests">
<bean id="networkData" class="com.nt.na21.nam.integration.helper.CSVFileAggregator">
</bean>
</int:transformer>
CompositeFileListFilterForBaseLine:
public class CompositeFileListFilterForBaseLine implements FileListFilter<File> {
private final static Logger LOG = Logger
.getLogger(CompositeFileListFilterForBaseLine.class);
#Override
public List<File> filterFiles(File[] files) {
List<File> filteredFile = new ArrayList<File>();
int index;
String fetchedFileName = null;
String fileCreatedDate = null;
String todayDate = DateHelper.toddMM(new Date());
LOG.debug("Date - dd-MM: " + todayDate);
for (File f : files) {
fetchedFileName = StringUtils.removeEnd(f.getName(), ".csv");
index = fetchedFileName.indexOf("_");
// Add plus one to index to skip underscore
fileCreatedDate = fetchedFileName.substring(index + 1);
// Format the created file date
fileCreatedDate = DateHelper.formatFileNameDateForAggregation(fileCreatedDate);
LOG.debug("file created date: " + fileCreatedDate + " today Date: "
+ todayDate);
if (fileCreatedDate.equalsIgnoreCase(todayDate)) {
filteredFile.add(f);
LOG.debug("File added to List of File: " + f.getAbsolutePath());
}
}
LOG.debug("SIZE: " + filteredFile.size());
LOG.debug("filterFiles method end.");
return filteredFile;
}
}
The Class file for CSVFileAggregator
public class CSVFileAggregator {
private final static Logger LOG = Logger.getLogger(CSVFileAggregator.class);
private int snePostion;
protected String masterFileSourcePath=null;
public File handleAggregateFiles(List<File> files) throws IOException {
LOG.debug("materFileSourcePath: " + masterFileSourcePath);
LinkedHashSet<String> allAttributes = null;
Map<String, LinkedHashSet<String>> allAttrBase = null;
Map<String, LinkedHashSet<String>> allAttrDelta = null;
LOG.info("Aggregator releasing [" + files.size() + "] files");
}
}
Log Output:
INFO : com.nt.na21.nam.integration.aggregator.NetFileAggregatorClient - NetFileAggregator context initialized. Polling input folder...
INFO : com.nt.na21.nam.integration.aggregator.NetFileAggregatorClient - Input directory is: D:\Projects\csv\processing
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - Date - dd-MM: 0103
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - file created date: 0103 today Date: 0103
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - File added to List of File: D:\Projects\NA21\NAMworkspace\na21_nam_integration\csv\processing\file1_base_0103.csv
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - file created date: 0103 today Date: 0103
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - File added to List of File: D:\Projects\NA21\NAMworkspace\na21_nam_integration\csv\processing\file2_base_0103.csv
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - **SIZE: 2**
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - filterFiles method end.
DEBUG: org.springframework.integration.file.FileReadingMessageSource - Added to queue: [csv\processing\file1_base_0103.csv, csv\processing\file2_base_0103.csv]
INFO : org.springframework.integration.file.FileReadingMessageSource - Created message: [GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]]
DEBUG: org.springframework.integration.endpoint.SourcePollingChannelAdapter - Poll resulted in Message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'networkData', message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: org.springframework.integration.handler.LoggingHandler - org.springframework.integration.handler.LoggingHandler#0 received message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: org.springframework.integration.handler.LoggingHandler - csv\processing\file2_base_0103.csv
DEBUG: org.springframework.integration.channel.DirectChannel - postSend (sent=true) on channel 'logger', message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: org.springframework.integration.transformer.MessageTransformingHandler - org.springframework.integration.transformer.MessageTransformingHandler#606f8b2b received message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: com.nt.na21.nam.integration.helper.CSVFileAggregator - materFileSourcePath: null
INFO : com.nt.na21.nam.integration.helper.CSVFileAggregator - **Aggregator releasing [1] files**
Can some one help me here in identifying the issue with Filter and same is not collecting for transformation?
Thanks in advance.
The issue is with int:aggregator as I am not sure how to invoke. I have used this earlier in my design but it didn't get executed at all. Thanks for the quick response.
For this problem I have written a FileScaner utility which will scan all the files in Folder inside and aggregation is working perfectly.
Please find the config with Aggregator which didn't works, hence I splited the design by two poller first produced all the CSV file(s) and second collect it and aggregate it.
<!-- Auto Wiring -->
<context:component-scan base-package="com.bt.na21.nam.integration.*" />
<!-- intercept and log every message -->
<int:logging-channel-adapter id="logger" level="DEBUG" />
<int:wire-tap channel = "logger" />
<int:channel id="fileInputChannel" datatype="java.io.File" />
<int:channel id="error" />
<int:channel id="requestsCSVInput" />
<int-file:inbound-channel-adapter id="pollNetworkFile"
directory="file:${input.files.directory}" channel="fileInputChannel"
filter="compositeFileFilter" prevent-duplicates="true">
<int:poller default="true" cron="*/20 * * * * *"
error-channel="error" />
</int-file:inbound-channel-adapter>
<bean id="compositeFileFilter"
class="com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForTodayFiles" />
<int:transformer id="transformInputZipCSVFileIntoCSV"
input-channel="fileInputChannel" output-channel="requestsCSVInput">
<bean id="transformZipFile"
class="com.nt.na21.nam.integration.file.net.NetRecordFileTransformation" />
</int:transformer>
<int:router ref="docTypeRouter" input-channel="requestsCSVInput"
method="resolveObjectTypeChannel">
</int:router>
<int:channel id="Vlan" />
<int:channel id="VlanShaper" />
<int:channel id="TdmPwe" />
<bean id="docTypeRouter"
class="com.nt.na21.nam.integration.file.net.DocumentTypeMessageRouter" />
<int:service-activator ref="vLanMessageHandler" output-channel="newContentItemNotification" input-channel="Vlan" method="handleFile" />
<bean id="vLanMessageHandler" class="com.nt.na21.nam.integration.file.handler.VLanRecordsHandler" />
<int:service-activator ref="VlanShaperMessageHandler" output-channel="newContentItemNotification" input-channel="VlanShaper" method="handleFile" />
<bean id="VlanShaperMessageHandler" class="com.nt.na21.nam.integration.file.handler.VlanShaperRecordsHandler" />
<int:service-activator ref="PweMessageHandler" output-channel="newContentItemNotification" input-channel="TdmPwe" method="handleFile" />
<bean id="PweMessageHandler" class="com.nt.na21.nam.integration.file.handler.PseudoWireRecordsHandler" />
<int:channel id="newContentItemNotification" />
<!-- Adding for aggregating the records in one place for OSS output -->
<int:aggregator input-channel="newContentItemNotification" method="aggregate"
ref="netRecordsResultAggregator" output-channel="net-records-aggregated-reply"
message-store="netRecordsResultMessageStore"
send-partial-result-on-expiry="true">
</int:aggregator>
<int:channel id="net-records-aggregated-reply" />
<bean id="netRecordsResultAggregator" class="com.nt.na21.nam.integration.aggregator.NetRecordsResultAggregator" />
<!-- Define a store for our network records results and set up a reaper that will
periodically expire those results. -->
<bean id="netRecordsResultMessageStore" class="org.springframework.integration.store.SimpleMessageStore" />
<int-file:outbound-channel-adapter id="filesOut"
directory="file:${output.files.directory}"
delete-source-files="true">
</int-file:outbound-channel-adapter>
The code is working fine till the routed to all the channel below:
<int:channel id="Vlan" />
<int:channel id="VlanShaper" />
<int:channel id="TdmPwe" />
I am trying to return LinkedHashSet from the Process of the above channel which contains CSV data and I need to aggregate all the merge
LinkedHashSet vAllAttributes to get the master output CSV file.
List<String> masterList = new ArrayList<String>(vAllAttributes);
Collections.sort(masterList);
Well, looks like you misunderstood a bit <int-file:inbound-channel-adapter> behaviour. Its nature is producing one file per message to the channel. It doesn't depend on the logic of the FileListFilter. The is like:
The FileReadingMessageSource uses DirectoryScanner to retrieve files from the provided directory to an internal toBeReceived Queue
Since we scan the directory for the files the design for the DirectoryScanner looks like List<File> listFiles(File directory). I guess this has led you astray.
After that the filter is applied to the original file list and returns only appropriate files.
They are stored to the toBeReceived Queue.
And only after that the FileReadingMessageSource polls an item from the queue to build message for the output channel.
To achieve your aggregation requirements you really should use an <aggregator> between <int-file:inbound-channel-adapter> and your <int:transformer>.
You can mark the <poller> of the <int-file:inbound-channel-adapter> with max-messages-per-poll="-1" to really poll all your files during the single scheduled task. But anyway there will as much messages as your filter returns files.
After that you must accept some tricks for the <aggregator>:
correlationKey - to allow your file messages to be combined to the single MessageGroup for release a single message for the further <transformer>. Since we don't have any context from <int-file:inbound-channel-adapter>, but we know that all messages are provided by the single polling task and withing scheduled Thread (you don't use task-executor on the <poller>), hence we can simply use correlationKey as:
correlation-strategy-expression="T(Thread).currentThread().id"
But the is not enough, because we should produce somehow the single message in the end anyway. Unfortunately we don't know the number of files (however you can do that via the ThreadLocal from your custom FileListFilter) to allow the ReleaseStrategy to return true for the aggregate phase. Hence we never have the normal group completion. But we can forceRelease uncompleted groups from the aggregator to use the MessageGroupStoreReaper or group-timeout on the <aggregator>.
In addition to the previous clause you should supply these options on the <aggegator>:
send-partial-result-on-expiry="true"
expire-groups-upon-completion="true"
And that's all. There is no reason to provide any custom aggregation function (ref/method or expression), because the default on just build a single message with the List of payloads from all messages in group. And that is appropriate for your CSVFileAggregator. Although you can avoid that <transformer> and this CSVFileAggregator for the aggregation function.
Hope I ma clear

Resources