i have a code
<int-jpa:updating-outbound-gateway
request-channel="nativeQlChannel" auto-startup="true"
native-query="update Transactions t set t.transaction_Status = :transactionStatus where t.bank_Reference_Number = :bankReferenceNumber "
entity-manager="entityManager" persist-mode="PERSIST" reply-channel="nativeQlChannelOne"
use-payload-as-parameter-source="false">
after this execution it calls
<int-jpa:outbound-channel-adapter
channel="nativeQlChannel" entity-class="org.ncb.quickpay.grs.persistence.entities.PartnerResponseDetails"
persist-mode="PERSIST" entity-manager="entityManager" >
<int-jpa:transactional/>
</int-jpa:outbound-channel-adapter>
<int-jpa:outbound-channel-adapter
channel="nativeQlChannelOne"
native-query="update Transactions t set t.transaction_Status = :transactionStatus where t.bank_Reference_Number = :bankReferenceNumber "
entity-manager="entityManager">
<int-jpa:transactional />
<int-jpa:parameter name="transactionStatus"
expression="payload['transactionStatus']" />
<int-jpa:parameter name="bankReferenceNumber"
expression="payload['bankReferenceNumber']" />
</int-jpa:outbound-channel-adapter>
but i am not able to get any parameter here , as it has 1 as value in payload which is the status of the previous query execution , how can i share the payload so that i can get all parameter which are available to previous query.
A common pattern is to store a reference to the payload in a header; then restore it...
<int:header-enricher ...> <!-- before the first gateway -->
<int:header name="savePayload" expression="payload">
</int:header-enricher>
<int:transformer ... expression="headers.savePayload"> <!-- after the first gateway -->
Related
I have a flow that uses a pub/sub channel based gateway with 2 subscribers to send message to 2 SQS queues. In between I have chain to do message transformation. An aggregator is used to summarize the report of every gateway invocation. Happy path works fine, but when my transformer throws error I get this message Reply message received but the receiving thread has already received a reply and the aggregator gets invoked but future.get in the Runner never returns. Sample Config and code to test is as follows:
Config
<!-- Gateway to Publish Data to SQS -->
<task:executor id="dataExecutor" pool-size="10"/>
<int:publish-subscribe-channel id="dataChannel" task-executor="dataExecutor" apply-sequence="true"/>
<int:publish-subscribe-channel id="sqsResultChannel"/>
<int:publish-subscribe-channel id="gatewayErrorChannel"/>
<int:publish-subscribe-channel id="gatewayReplyChannel"/>
<int:gateway service-interface="com.abc.DataPublishGateway" id="dataPublishGateway"
error-channel="gatewayErrorChannel">
<int:method name="publishToDataService"
payload-expression="#args[0]"
request-channel="dataChannel"
reply-channel="gatewayReplyChannel">
</int:method>
</int:gateway>
<int:chain input-channel="gatewayErrorChannel" output-channel="sqsResultChannel">
<int:header-enricher>
<int:correlation-id expression="payload.failedMessage.headers.correlationId" />
<int:header name="sequenceSize" expression="payload.failedMessage.headers.sequenceSize" />
<int:header name="sequenceNumber" expression="payload.failedMessage.headers.sequenceNumber" />
</int:header-enricher>
</int:chain>
<!-- Route to system-a SQS -->
<int:channel id="sqsSystemAPublishChannel" />
<int:chain input-channel="dataChannel" output-channel="sqsSystemAPublishChannel">
<int:header-enricher>
<int:header name="targetSystem" value="system-a" />
</int:header-enricher>
<int:transformer expression="payload/2"/> <!-- Simulate transformer error -->
</int:chain>
<int-aws:sqs-outbound-channel-adapter sqs="amazonSQS"
queue="a-queue"
channel="sqsSystemAPublishChannel"
success-channel="sqsResultChannel"
failure-channel="gatewayErrorChannel"/>
<!-- Route to system-b SQS -->
<int:channel id="sqsSystemBPublishChannel" />
<int:chain input-channel="dataChannel" output-channel="sqsSystemBPublishChannel">
<int:header-enricher>
<int:header name="targetSystem" value="system-b" />
</int:header-enricher>
<int:transformer expression="payload.toLowerCase() "/>
</int:chain>
<int-aws:sqs-outbound-channel-adapter sqs="amazonSQS"
queue="b-queue"
channel="sqsSystemBPublishChannel"
success-channel="sqsResultChannel"
failure-channel="gatewayErrorChannel"/>
<bean class="com.abc.DataResponseAggregator" id="responseAggregator" />
<int:aggregator input-channel="sqsResultChannel" output-channel="gatewayReplyChannel" ref="responseAggregator"/>
<!-- Generic Error Channel Logger -->
<int:logging-channel-adapter log-full-message="true"
logger-name="errorLogger"
level="ERROR"
channel="errorChannel"
id="globalErrorLoggingAdapter"/>
Aggregator
public class DataResponseAggregator {
public Map<String, String> aggregate(List<Message> responses) {
Map<String, String> resultMap = new HashMap<>();
responses.forEach(message -> {
if (message instanceof ErrorMessage) {
String exceptionMessage = ((ErrorMessage) message).getPayload().getCause().getMessage();
String targetSystem = ((MessagingException) message.getPayload()).getFailedMessage().getHeaders()
.get("targetSystem").toString();
resultMap.put(targetSystem, exceptionMessage);
}
else {
String targetSystem = message.getHeaders().get("targetSystem").toString();
resultMap.put(targetSystem, "Ack -> " + message.getHeaders().get("aws_messageId").toString());
}
});
return resultMap;
}
}
Gateway
public interface DataPublishGateway {
Future<Map<String, String>> publishToDataService(String message);
}
Runner
#Bean
CommandLineRunner runner(DataPublishGateway dataPublishGateway) {
return args -> {
String[] messages = new String[]{"Message 1", "Message 2"};
List<Future<Map<String, String>>> futureList = new ArrayList<>();
Arrays.stream(messages).forEach(s -> {
futureList.add(dataPublishGateway.publishToDataService(s));
});
System.out.println("Processing Futures and Printing Results...");
futureList.forEach(mapFuture -> {
try {
mapFuture.get().entrySet().forEach(entry -> {
System.out.println(entry.getKey() + " - " + entry.getValue());
});
} catch (Exception e) {
e.printStackTrace();
}
});
};
}
Logs
14:47:21.650 INFO 91449 --- [pool-1-thread-1] o.s.i.h.s.MessagingMethodInvokerHelper : Overriding default instance of MessageHandlerMethodFactory with provided one.
2020-09-07 14:47:21.651 INFO 91449 --- [pool-1-thread-2] o.s.i.h.s.MessagingMethodInvokerHelper : Overriding default instance of MessageHandlerMethodFactory with provided one.
2020-09-07 14:47:21.655 WARN 91449 --- [pool-1-thread-2] cMessagingTemplate$TemporaryReplyChannel : Reply message received but the receiving thread has already received a reply: GenericMessage [payload={system-a=Expression evaluation failed: payload/2; nested exception is org.springframework.expression.spel.SpelEvaluationException: EL1030E: The operator 'DIVIDE' is not supported between objects of type 'java.lang.String' and 'java.lang.Integer', system-b=Ack -> 90b62dff-f3c3-4288-b5e3-8178e410f60d}, headers={aws_messageId=90b62dff-f3c3-4288-b5e3-8178e410f60d, replyChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#1b519afe, sequenceNumber=2, errorChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#1b519afe, sequenceSize=2, correlationId=dc914588-f9a9-537f-0623-94938f84cec4, aws_serviceResult={MD5OfMessageBody: 83b2330607fe8f817ce6d24249dea373,MD5OfMessageAttributes: 5f1f442c363809afbd334ff00232c834,MessageId: 90b62dff-f3c3-4288-b5e3-8178e410f60d,}, id=9a6ad209-5291-6216-100b-502b4c37eb01, targetSystem=system-b, timestamp=1599482841654}]
2020-09-07 14:47:21.655 WARN 91449 --- [pool-1-thread-1] cMessagingTemplate$TemporaryReplyChannel : Reply message received but the receiving thread has already received a reply: GenericMessage [payload={system-a=Expression evaluation failed: payload/2; nested exception is org.springframework.expression.spel.SpelEvaluationException: EL1030E: The operator 'DIVIDE' is not supported between objects of type 'java.lang.String' and 'java.lang.Integer', system-b=Ack -> 3c8a4479-22e1-48d8-aca6-a86c252e90c1}, headers={aws_messageId=3c8a4479-22e1-48d8-aca6-a86c252e90c1, replyChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#f5a99c3, sequenceNumber=2, errorChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#f5a99c3, sequenceSize=2, correlationId=74ba49e0-fef8-02e6-bc4b-b2ccdd7cd13b, aws_serviceResult={MD5OfMessageBody: 1db65a6a0a818fd39655b95e33ada11d,MD5OfMessageAttributes: cd4eae624b7c115f948034aafaba01bc,MessageId: 3c8a4479-22e1-48d8-aca6-a86c252e90c1,}, id=3e3de3f3-e8b0-470e-e62f-31815e662857, targetSystem=system-b, timestamp=1599482841654}]
What am I missing? I would expect the error from the transformer to follow the flow : gatewayErrorChannel -> sqsResultChannel -> aggregator -> gatewayReplyChannel which it does since the aggregator output appears in the WARN message. But why does future.get never return and also it seems like gatewayReplyChannel receives aggregator output twice?
With some logging and most importantly the understanding of error and reply channels set on Gateway messages from this post, I am able to solve the issue.
What was the issue here?
The errorChannel was not set on the messages coming into the chain.
In case when both chains throw exceptions, the error messages coming to the aggregator did not have a reply channel set on them to return the aggregated message to the gateway's reply channel. The final config looks as below:
<!-- Gateway to Publish Data to SQS -->
<task:executor id="dataExecutor" pool-size="10"/>
<int:publish-subscribe-channel id="dataChannel" task-executor="dataExecutor" apply-sequence="true"/>
<int:publish-subscribe-channel id="sqsResultChannel"/>
<int:publish-subscribe-channel id="gatewayErrorChannel"/>
<int:publish-subscribe-channel id="gatewayReplyChannel"/>
<int:gateway service-interface="com.abc.DataPublishGateway" id="dataPublishGateway"
error-channel="gatewayErrorChannel">
<int:method name="publishToDataService"
payload-expression="#args[0]"
request-channel="dataChannel"
reply-channel="gatewayReplyChannel">
</int:method>
</int:gateway>
<int:chain input-channel="gatewayErrorChannel" output-channel="sqsResultChannel">
<int:header-enricher>
<int:correlation-id expression="payload.failedMessage.headers.correlationId" />
<int:header name="sequenceSize" expression="payload.failedMessage.headers.sequenceSize" />
<int:header name="sequenceNumber" expression="payload.failedMessage.headers.sequenceNumber" />
<int:reply-channel expression="payload.failedMessage.headers.replyChannel" />
</int:header-enricher>
</int:chain>
<!-- Route to system-a SQS -->
<int:channel id="sqsSystemAPublishChannel" />
<int:chain input-channel="dataChannel" output-channel="sqsSystemAPublishChannel">
<int:header-enricher>
<int:header name="targetSystem" value="system-a" />
<int:error-channel ref="gatewayErrorChannel" overwrite="true" />
</int:header-enricher>
<int:transformer expression="payload/2"/> <!-- Simulate transformer error -->
</int:chain>
<int-aws:sqs-outbound-channel-adapter sqs="amazonSQS"
queue="a-queue"
channel="sqsSystemAPublishChannel"
success-channel="sqsResultChannel"
failure-channel="gatewayErrorChannel"/>
<!-- Route to system-b SQS -->
<int:channel id="sqsSystemBPublishChannel" />
<int:chain input-channel="dataChannel" output-channel="sqsSystemBPublishChannel">
<int:header-enricher>
<int:header name="targetSystem" value="system-b" />
<int:error-channel ref="gatewayErrorChannel" overwrite="true" />
</int:header-enricher>
<int:transformer expression="payload.toLowerCase() "/>
</int:chain>
<int-aws:sqs-outbound-channel-adapter sqs="amazonSQS"
queue="b-queue"
channel="sqsSystemBPublishChannel"
success-channel="sqsResultChannel"
failure-channel="gatewayErrorChannel"/>
<bean class="com.abc.DataResponseAggregator" id="responseAggregator" />
<int:aggregator input-channel="sqsResultChannel" output-channel="gatewayReplyChannel" ref="responseAggregator"/>
<!-- Generic Error Channel Logger -->
<int:logging-channel-adapter log-full-message="true"
logger-name="errorLogger"
level="ERROR"
channel="errorChannel"
id="globalErrorLoggingAdapter"/>
In current model, we have a REST endpoint, which gets requestbody, based on which a jms text message is created and sent to JMS queue,
TextMessage outMessage = session.createTextMessage(messagePayloadText);
..
outMessage.setStringProperty("clientType", clientType);
outMessage.setStringProperty("DYNAMIC", dynaHeader);
In above code DYNAMIC is required to help me in creating our url
<int:chain input-channel="gCStatusInChannel" output-channel="headerFilterChannel">
<int:header-enricher>
<int:header name="Api-Key" value="B8872853E8B"></int:header>
<int:header name="Accept" value="application/json" />
<int:header name="Content-Type" value="application/json" />
</int:header-enricher>
<int-http:outbound-gateway
url="https://i-zaie.sr13.tst.bst/ia-zaaie/rest/search/v2/cReference/{cref}"
http-method="PUT"
header-mapper="headerMapper"
expected-response-type="java.lang.String"
encode-uri="false"
request-factory="sslFactory">
<int-http:uri-variable name="cref" expression="headers['DYNAMIC']" />
</int-http:outbound-gateway>
<int:object-to-string-transformer></int:object-to-string-transformer>
</int:chain>
Everything works in this model. Now I want to use gateway instead of JMS
New code:
<int:gateway id="gService"
service-interface="n.d.lr.eai.gw.GGateway"
default-reply-channel="dest-channel"
default-request-timeout="5000" default-reply-timeout="5000">
<int:method name="vCreateSignal" request-channel="vCreateSignalInChannel"/> ...
Question:
can i have method in gateway as below?
public String vCreateSignal(String caseDat, String dynamic);
what should I do to enable
<int:chain input-channel="gCStatusInChannel"...
..>
to get headers['DYNAMIC'] value and continue.
Yes, you can do that. What you just need is to add a #Header("DYNAMIC") into that dynamic parameter:
public String vCreateSignal(String caseDat, #Header("DYNAMIC") String dynamic);
And when you call this gateway's method you just specify an argument and it will be mapped to an appropriate header and that all: https://docs.spring.io/spring-integration/docs/current/reference/html/messaging-endpoints-chapter.html#gateway-mapping
I am running the client portion of the Spring Integration TCP Multiplex example. I was trying to see how many requests it could handle at once and around 1000, I started to get this error: ChannelResolutionException: no output-channel or replyChannel header available
Everything is fine below about 1000 calls.
<beans:description>
Uses conversion service and collaborating channel adapters.
</beans:description>
<context:property-placeholder />
<converter>
<beans:bean class="org.springframework.integration.samples.tcpclientserver.ByteArrayToStringConverter" />
</converter>
<!-- Fastest Wire Protocol - takes a byte array with its length definied in the first x bytes-->
<beans:bean id="fastestWireFormatSerializer" class="org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer">
<beans:constructor-arg value="1" />
</beans:bean>
<!-- Client side -->
<gateway id="gw"
service-interface="org.springframework.integration.samples.tcpclientserver.SimpleGateway"
default-request-channel="input" />
<ip:tcp-connection-factory id="client"
type="client"
host="localhost"
port="${availableServerSocket}"
single-use="false"
serializer="fastestWireFormatSerializer"
deserializer="fastestWireFormatSerializer"
so-timeout="10000" />
<publish-subscribe-channel id="input" />
<!-- scheduler - Thread used to restablish connection so the other threads aren't starved while waiting to re-establish connection -->
<!-- client-mode - Automatically re-establishes the connection if lost -->
<ip:tcp-outbound-channel-adapter id="outAdapter.client"
order="2"
channel="input"
client-mode="true"
connection-factory="client" /> <!-- Collaborator -->
<!-- Also send a copy to the custom aggregator for correlation and
so this message's replyChannel will be transferred to the
aggregated message.
The order ensures this gets to the aggregator first -->
<bridge input-channel="input" output-channel="toAggregator.client"
order="1"/>
<!-- Asynch receive reply -->
<ip:tcp-inbound-channel-adapter id="inAdapter.client"
channel="toAggregator.client"
connection-factory="client" /> <!-- Collaborator -->
<!-- dataType attribute invokes the conversion service, if necessary -->
<channel id="toAggregator.client" datatype="java.lang.String" />
<aggregator input-channel="toAggregator.client"
output-channel="toTransformer.client"
correlation-strategy-expression="payload.substring(0,3)"
release-strategy-expression="size() == 2"
expire-groups-upon-completion="true" />
<transformer input-channel="toTransformer.client"
expression="payload.get(1)"/> <!-- The response is always second -->
<task:scheduler id="reconnectScheduler" pool-size="10"/>
And the code used to test:
TaskExecutor executor = new SimpleAsyncTaskExecutor();
final CountDownLatch latch = new CountDownLatch(100);
final Set<Integer> results = new HashSet<Integer>();
for (int i = 100; i < 1050; i++) {
results.add(i);
final int j = i;
executor.execute(new Runnable() {
public void run() {
String result = gateway.send(j + "Hello world!"); // first 3 bytes is correlationid
System.out.println("Test Result: " + result);
results.remove(j);
latch.countDown();
}});
}
I haven't figured out entirely why you are getting that exception, but there are several problems with your test.
The countdown latch needs to be initialized at 950
Since you are exceeding 999, we need to change the correlation:
payload.substring(0,4)
With those changes, it works for me.
I'll try to figure out why we're getting that exception when I get a bit more time.
EDIT
The issue is indeed caused by the conflicting correlation ids.
The last 50 messages all have correlation id 100 which means messages are released in an indeterminate fashion (given the release is based on size). In some cases two input messages are released (causing the wrong reply to the test case). When 2 replies are released; there is no output channel.
I am building a Kafka consumer with spring. My configuration seems to be pretty straightforward. Messages are being consumed and saved in files. However, the payload is cryptic and I can't get the data (short "hello world" messages).
This is what I get when I access the payload (e.g. when I set up a transformer bean btween in inbound kafka and the outboud file:
{test-topic={0=[[B#713c9d72, [B#7d656f90, [B#26bb8c83, [B#4b959d83 [B#5ed74e8e]}}
My question is: How do I access the actual payload (the "hellow world" string")?
My configuration is :
<int:channel id="inputFromKafka">
<int:queue />
</int:channel>
<int:poller
max-messages-per-poll="5" default = "true" fixed-delay="10" time-unit="MILLISECONDS"/>
<int-kafka:inbound-channel-adapter
id="kafkaInboundChannelAdapter" kafka-consumer-context-ref="consumerContext"
auto-startup="true" channel="inputFromKafka">
<int-kafka:consumer-context id="consumerContext"
consumer-timeout="40000" zookeeper-connect="zookeeperConnect">
<int-kafka:consumer-configurations>
<int-kafka:consumer-configuration
group-id="group12" max-messages="5">
<int-kafka:topic id="test-topic" streams="1" />
</int-kafka:consumer-configuration>
</int-kafka:consumer-configurations>
</int-kafka:consumer-context>
<int-kafka:zookeeper-connect id="zookeeperConnect"
zk-connect="localhost:2181" zk-connection-timeout="6000"
zk-session-timeout="6000" zk-sync-time="2000" />
<file:outbound-channel-adapter id="filesOut"
directory="/tmp/fromKafka">
</file:outbound-channel-adapter>
You are seeing raw byte[].
Add...
<bean id="decoder"
class="org.springframework.integration.kafka.serializer.common.StringDecoder" />
and
<int-kafka:consumer-configuration
value-decoder="decoder"
...
I have written a code to combined multiple files into one single Master file.
The issue is with int-transformer where I am getting one file at a time although I have aggregated List of File in composite Filter of File inbound-channel-adapter. The List of File size in composite filter is correct but in Transformer bean the List of File size is always one and not getting the correct list size aggregated file by the filter.
Here is my config:
<!-- Auto Wiring -->
<context:component-scan base-package="com.nt.na21.nam.integration.*" />
<!-- intercept and log every message -->
<int:logging-channel-adapter id="logger"
level="DEBUG" />
<int:wire-tap channel="logger" />
<!-- Aggregating the processed Output for OSS processing -->
<int:channel id="networkData" />
<int:channel id="requests" />
<int-file:inbound-channel-adapter id="pollProcessedNetworkData"
directory="file:${processing.files.directory}" filter="compositeProcessedFileFilter"
channel="networkData">
<int:poller default="true" cron="*/20 * * * * *" />
</int-file:inbound-channel-adapter>
<bean id="compositeProcessedFileFilter"
class="com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine" />
<int:transformer id="aggregateNetworkData"
input-channel="networkData" output-channel="requests">
<bean id="networkData" class="com.nt.na21.nam.integration.helper.CSVFileAggregator">
</bean>
</int:transformer>
CompositeFileListFilterForBaseLine:
public class CompositeFileListFilterForBaseLine implements FileListFilter<File> {
private final static Logger LOG = Logger
.getLogger(CompositeFileListFilterForBaseLine.class);
#Override
public List<File> filterFiles(File[] files) {
List<File> filteredFile = new ArrayList<File>();
int index;
String fetchedFileName = null;
String fileCreatedDate = null;
String todayDate = DateHelper.toddMM(new Date());
LOG.debug("Date - dd-MM: " + todayDate);
for (File f : files) {
fetchedFileName = StringUtils.removeEnd(f.getName(), ".csv");
index = fetchedFileName.indexOf("_");
// Add plus one to index to skip underscore
fileCreatedDate = fetchedFileName.substring(index + 1);
// Format the created file date
fileCreatedDate = DateHelper.formatFileNameDateForAggregation(fileCreatedDate);
LOG.debug("file created date: " + fileCreatedDate + " today Date: "
+ todayDate);
if (fileCreatedDate.equalsIgnoreCase(todayDate)) {
filteredFile.add(f);
LOG.debug("File added to List of File: " + f.getAbsolutePath());
}
}
LOG.debug("SIZE: " + filteredFile.size());
LOG.debug("filterFiles method end.");
return filteredFile;
}
}
The Class file for CSVFileAggregator
public class CSVFileAggregator {
private final static Logger LOG = Logger.getLogger(CSVFileAggregator.class);
private int snePostion;
protected String masterFileSourcePath=null;
public File handleAggregateFiles(List<File> files) throws IOException {
LOG.debug("materFileSourcePath: " + masterFileSourcePath);
LinkedHashSet<String> allAttributes = null;
Map<String, LinkedHashSet<String>> allAttrBase = null;
Map<String, LinkedHashSet<String>> allAttrDelta = null;
LOG.info("Aggregator releasing [" + files.size() + "] files");
}
}
Log Output:
INFO : com.nt.na21.nam.integration.aggregator.NetFileAggregatorClient - NetFileAggregator context initialized. Polling input folder...
INFO : com.nt.na21.nam.integration.aggregator.NetFileAggregatorClient - Input directory is: D:\Projects\csv\processing
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - Date - dd-MM: 0103
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - file created date: 0103 today Date: 0103
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - File added to List of File: D:\Projects\NA21\NAMworkspace\na21_nam_integration\csv\processing\file1_base_0103.csv
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - file created date: 0103 today Date: 0103
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - File added to List of File: D:\Projects\NA21\NAMworkspace\na21_nam_integration\csv\processing\file2_base_0103.csv
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - **SIZE: 2**
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - filterFiles method end.
DEBUG: org.springframework.integration.file.FileReadingMessageSource - Added to queue: [csv\processing\file1_base_0103.csv, csv\processing\file2_base_0103.csv]
INFO : org.springframework.integration.file.FileReadingMessageSource - Created message: [GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]]
DEBUG: org.springframework.integration.endpoint.SourcePollingChannelAdapter - Poll resulted in Message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'networkData', message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: org.springframework.integration.handler.LoggingHandler - org.springframework.integration.handler.LoggingHandler#0 received message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: org.springframework.integration.handler.LoggingHandler - csv\processing\file2_base_0103.csv
DEBUG: org.springframework.integration.channel.DirectChannel - postSend (sent=true) on channel 'logger', message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: org.springframework.integration.transformer.MessageTransformingHandler - org.springframework.integration.transformer.MessageTransformingHandler#606f8b2b received message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: com.nt.na21.nam.integration.helper.CSVFileAggregator - materFileSourcePath: null
INFO : com.nt.na21.nam.integration.helper.CSVFileAggregator - **Aggregator releasing [1] files**
Can some one help me here in identifying the issue with Filter and same is not collecting for transformation?
Thanks in advance.
The issue is with int:aggregator as I am not sure how to invoke. I have used this earlier in my design but it didn't get executed at all. Thanks for the quick response.
For this problem I have written a FileScaner utility which will scan all the files in Folder inside and aggregation is working perfectly.
Please find the config with Aggregator which didn't works, hence I splited the design by two poller first produced all the CSV file(s) and second collect it and aggregate it.
<!-- Auto Wiring -->
<context:component-scan base-package="com.bt.na21.nam.integration.*" />
<!-- intercept and log every message -->
<int:logging-channel-adapter id="logger" level="DEBUG" />
<int:wire-tap channel = "logger" />
<int:channel id="fileInputChannel" datatype="java.io.File" />
<int:channel id="error" />
<int:channel id="requestsCSVInput" />
<int-file:inbound-channel-adapter id="pollNetworkFile"
directory="file:${input.files.directory}" channel="fileInputChannel"
filter="compositeFileFilter" prevent-duplicates="true">
<int:poller default="true" cron="*/20 * * * * *"
error-channel="error" />
</int-file:inbound-channel-adapter>
<bean id="compositeFileFilter"
class="com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForTodayFiles" />
<int:transformer id="transformInputZipCSVFileIntoCSV"
input-channel="fileInputChannel" output-channel="requestsCSVInput">
<bean id="transformZipFile"
class="com.nt.na21.nam.integration.file.net.NetRecordFileTransformation" />
</int:transformer>
<int:router ref="docTypeRouter" input-channel="requestsCSVInput"
method="resolveObjectTypeChannel">
</int:router>
<int:channel id="Vlan" />
<int:channel id="VlanShaper" />
<int:channel id="TdmPwe" />
<bean id="docTypeRouter"
class="com.nt.na21.nam.integration.file.net.DocumentTypeMessageRouter" />
<int:service-activator ref="vLanMessageHandler" output-channel="newContentItemNotification" input-channel="Vlan" method="handleFile" />
<bean id="vLanMessageHandler" class="com.nt.na21.nam.integration.file.handler.VLanRecordsHandler" />
<int:service-activator ref="VlanShaperMessageHandler" output-channel="newContentItemNotification" input-channel="VlanShaper" method="handleFile" />
<bean id="VlanShaperMessageHandler" class="com.nt.na21.nam.integration.file.handler.VlanShaperRecordsHandler" />
<int:service-activator ref="PweMessageHandler" output-channel="newContentItemNotification" input-channel="TdmPwe" method="handleFile" />
<bean id="PweMessageHandler" class="com.nt.na21.nam.integration.file.handler.PseudoWireRecordsHandler" />
<int:channel id="newContentItemNotification" />
<!-- Adding for aggregating the records in one place for OSS output -->
<int:aggregator input-channel="newContentItemNotification" method="aggregate"
ref="netRecordsResultAggregator" output-channel="net-records-aggregated-reply"
message-store="netRecordsResultMessageStore"
send-partial-result-on-expiry="true">
</int:aggregator>
<int:channel id="net-records-aggregated-reply" />
<bean id="netRecordsResultAggregator" class="com.nt.na21.nam.integration.aggregator.NetRecordsResultAggregator" />
<!-- Define a store for our network records results and set up a reaper that will
periodically expire those results. -->
<bean id="netRecordsResultMessageStore" class="org.springframework.integration.store.SimpleMessageStore" />
<int-file:outbound-channel-adapter id="filesOut"
directory="file:${output.files.directory}"
delete-source-files="true">
</int-file:outbound-channel-adapter>
The code is working fine till the routed to all the channel below:
<int:channel id="Vlan" />
<int:channel id="VlanShaper" />
<int:channel id="TdmPwe" />
I am trying to return LinkedHashSet from the Process of the above channel which contains CSV data and I need to aggregate all the merge
LinkedHashSet vAllAttributes to get the master output CSV file.
List<String> masterList = new ArrayList<String>(vAllAttributes);
Collections.sort(masterList);
Well, looks like you misunderstood a bit <int-file:inbound-channel-adapter> behaviour. Its nature is producing one file per message to the channel. It doesn't depend on the logic of the FileListFilter. The is like:
The FileReadingMessageSource uses DirectoryScanner to retrieve files from the provided directory to an internal toBeReceived Queue
Since we scan the directory for the files the design for the DirectoryScanner looks like List<File> listFiles(File directory). I guess this has led you astray.
After that the filter is applied to the original file list and returns only appropriate files.
They are stored to the toBeReceived Queue.
And only after that the FileReadingMessageSource polls an item from the queue to build message for the output channel.
To achieve your aggregation requirements you really should use an <aggregator> between <int-file:inbound-channel-adapter> and your <int:transformer>.
You can mark the <poller> of the <int-file:inbound-channel-adapter> with max-messages-per-poll="-1" to really poll all your files during the single scheduled task. But anyway there will as much messages as your filter returns files.
After that you must accept some tricks for the <aggregator>:
correlationKey - to allow your file messages to be combined to the single MessageGroup for release a single message for the further <transformer>. Since we don't have any context from <int-file:inbound-channel-adapter>, but we know that all messages are provided by the single polling task and withing scheduled Thread (you don't use task-executor on the <poller>), hence we can simply use correlationKey as:
correlation-strategy-expression="T(Thread).currentThread().id"
But the is not enough, because we should produce somehow the single message in the end anyway. Unfortunately we don't know the number of files (however you can do that via the ThreadLocal from your custom FileListFilter) to allow the ReleaseStrategy to return true for the aggregate phase. Hence we never have the normal group completion. But we can forceRelease uncompleted groups from the aggregator to use the MessageGroupStoreReaper or group-timeout on the <aggregator>.
In addition to the previous clause you should supply these options on the <aggegator>:
send-partial-result-on-expiry="true"
expire-groups-upon-completion="true"
And that's all. There is no reason to provide any custom aggregation function (ref/method or expression), because the default on just build a single message with the List of payloads from all messages in group. And that is appropriate for your CSVFileAggregator. Although you can avoid that <transformer> and this CSVFileAggregator for the aggregation function.
Hope I ma clear