Spring Integration - Scheduling Job from configuration file - spring-integration

I'm using Spring Integration to parse XML file and i will need to create a thread (and each one have a different rate) for each tag.
Right now (with the help of many users here :)) i'm able to split XML by tag and then route it to the appropiate service-activator.
This works great but i'm not able to redirect to a channel that create "a thread" and then execute the operations. Right now i have the following configuration and in my mind (that i dont know if it is correct...)
Split tag -> Route to the appropiate channel -> Start a thread(from tag configuration) -> Execute the operation
This is my actual configuration that split tag and redirect to the channel.
The router should redirect not toward a channel directly, but schedule them.
In first instance will be enought to redirect it in a pool with fixed rate and later i will use XPATH to get the attribute and then replace this "fixed" rate with the correct value.
I've tried many solutions to create this flow but each one fails or do not compile :(
<context:component-scan base-package="it.mypkg" />
<si:channel id="rootChannel" />
<si-xml:xpath-splitter id="mySplitter" input-channel="rootChannel" output-channel="routerChannel" create-documents="true">
<si-xml:xpath-expression expression="//service" />
</si-xml:xpath-splitter>
<si-xml:xpath-router id="router" input-channel="routerChannel" evaluate-as-string="true">
<si-xml:xpath-expression expression="concat(name(./node()), 'Channel')" />
</si-xml:xpath-router>
<si:service-activator input-channel="serviceChannel" output-channel="endChannel">
<bean class="it.mypkg.Service" />
</si:service-activator>
UPDATE:
Using this configuration for the service this should run a task every 10 seconds (the id=service1) and every 5 seconds the other (the id=service2). In the same way i can have another tag that is handle by another class (because this will have another behaviour)
<root>
<service id="service1" interval="10000" />
<service id="service2" interval="5000" />
<activity id="activity1" interval="50000" />
<root>
I will have a classe (Service) that is general to handle Service tag and this complete some operation and then "return me" the value so i can redirect to another channel.
public class Service {
public int execute() {
// Execute the task and return the value to continue the "chain"
}
}

It's not at all clear what you mean; you split a tag; route it but want to "schedule" it at a rate in the XML. It's not clear what you mean by "schedule" here - normally each message is processed once not multiple times on a schedule.
As I said, I don't understand what you need to do, but a smart poller might be suitable.
Another possibility is the delayer where the amount of the delay can be derived from the message.
EDIT
Since your "services" don't seem to take any input data, it looks like you simply need to configure/start an <inbound-channel-adapter/> for each service, and then start it, based on the arguments in the XML.
<int:inbound-channel-adapter id="service1" channel="foo"
auto-startup="false"
ref="service1Bean" method="execute">
<poller fixed-delay="1000" />
</int:inbound-channel-adapter/>
Note auto-startup="false".
Now, in the code that receives the split
#Autowired
SourcePollingChannelAdapter service1;
...
public void startService1(Node node) {
...
service1.setTrigger(new PeridicTrigger(...));
service1.start();
...
}

I dont know if this is the right way to implement the flow, but i've write the follow code:
applicationContext.xml
<context:component-scan base-package="it.mypkg" />
<!-- Expression to extract interval from XML tag -->
<si-xml:xpath-expression id="selectIntervalXpath" expression="//*/#interval" />
<si:channel id="rootChannel" />
<!-- Split each tag to redirect on router -->
<si-xml:xpath-splitter id="mySplitter" input-channel="rootChannel" output-channel="routerChannel" create-documents="true">
<si-xml:xpath-expression expression="//service|//activity" />
</si-xml:xpath-splitter>
<!-- Route each tag to the appropiate channel -->
<si-xml:xpath-router id="router" input-channel="routerChannel" evaluate-as-string="true">
<si-xml:xpath-expression expression="concat(name(./node()), 'Channel')" />
</si-xml:xpath-router>
<!-- Activator for Service Tag -->
<si:service-activator input-channel="serviceChannel" method="schedule">
<bean class="it.mypkg.Service" />
</si:service-activator>
<!-- Activator for Activity Tag -->
<si:service-activator input-channel="activityChannel" method="schedule">
<bean class="it.mypkg.Activity" />
</si:service-activator>
<!-- Task scheduler -->
<task:scheduler id="taskScheduler" pool-size="10"/>
Each tag will extend an Operation class (to avoid code duplication on bean injection)
Operation.java
public abstract class Operation {
protected TaskScheduler taskScheduler;
protected XPathExpression selectIntervalXpath;
abstract public void schedule(Node document);
#Autowired
public void setTaskScheduler(TaskScheduler taskScheduler) {
this.taskScheduler= taskScheduler;
}
public TaskScheduler getTaskScheduler() {
return this.taskScheduler;
}
#Autowired
public void setSelectIntervalXpath(XPathExpression selectIntervalXpath) {
this.selectIntervalXpath = selectIntervalXpath;
}
public XPathExpression getSelectIntervalXPath() {
return this.selectIntervalXpath;
}
}
And an example of Service class (that handle all tags service provided on .xml)
public class Service extends Operation {
private static final Logger log = Logger.getLogger(Service.class);
#Override
public void schedule(Node document) {
log.debug("Scheduling Service");
long interval = Long.parseLong(this.selectIntervalXpath.evaluateAsString(document));
this.taskScheduler.scheduleAtFixedRate(new ServiceRunner(), interval);
}
private class ServiceRunner implements Runnable {
public void run() {
log.debug("Running...");
}
}
}
Now to continue my flow i will need to find a way to redirect the output of each job to Spring Integration (applicationContext.xml).

Related

SAP Hybris E-commerce : Send Email in CronJob

I have created a CronJob that works perfectly.
But I want to generate the sending of an email within this Cronjob. I followed a tutorial on the internet.
I start with the creation itemType of ProductsApprovedEmailProcess.
then I created productsApprovedEmailProcess to define the steps be executed by the Process Engine as follow
Then I have added an EmailContext to holds the data to be passed to the email template as follow
public class ProductsApprovedEmailContext extends CustomerEmailContext
{
private String message;
#Override
public void init(final StoreFrontCustomerProcessModel processModel, final EmailPageModel emailPageModel)
{
super.init(processModel, emailPageModel);
if (processModel instanceof ProductsApprovedEmailProcessModel)
{
setMessage(((ProductsApprovedEmailProcessModel) processModel).getMessage());
}
}
public String getMessage()
{
return message;
}
public void setMessage(final String message)
{
this.message = message;
}
}
And I had register ProductsApprovedEmailContext as a bean in Spring as follow
<bean id="productsApprovedEmailContext" class="com.hybris.training.facades.process.email.context.ProductsApprovedEmailContext"
parent="abstractEmailContext"
scope="prototype" >
</bean>
Then I created 2 Velocity templates, one for the email Subject and the other for the Body email-productsapproved-subject.vm and email-productsapproved-body.vm
And the following impex allows you to create RendererTemplates for the Subject and the Body, and attach them to an EmailPageTemplate as follow
$contentCatalog=electronicsContentCatalog
$contentCV=catalogVersion(CatalogVersion.catalog(Catalog.id[default=$contentCatalog]),CatalogVersion.version[default=Online])[default=$contentCatalog:Online]
UPDATE GenericItem[processor=de.hybris.platform.commerceservices.impex.impl.ConfigPropertyImportProcessor];pk[unique=true]
$emailResource=$config-emailResourceValue
$emailPackageName=$config-emailContextPackageName
$lang=en
INSERT_UPDATE RendererTemplate ;code[unique=true] ;contextClass ;templateScript[lang=en,translator=de.hybris.platform.commerceservices.impex.impl.FileLoaderValueTranslator];rendererType(code)[default='velocity']
;email-productsapproved-body ;$emailPackageName.ProductsApprovedEmailContext ;$emailResource/email-productsapproved-body.vm
;email-productsapproved-subject ;$emailPackageName.ProductsApprovedEmailContext ;$emailResource/email-productsapproved-subject.vm
INSERT_UPDATE EmailPage ;$contentCV[unique=true];uid[unique=true] ;masterTemplate(uid,$contentCV) ;approvalStatus(code)[default='approved']
; ;ProductApprovedEmail ;ProductApprovedEmailTemplate ;
And in the Cronjob I added this code !
final ProductsApprovedEmailProcessModel productsApprovedEmailProcessModel = (ProductsApprovedEmailProcessModel) businessProcessService
.createProcess("productsApprovedEmailProcess" + "-" + System.currentTimeMillis(), "productsApprovedEmailProcess");
productsApprovedEmailProcessModel.setMessage("Products approved in csv file");
productsApprovedEmailProcessModel.setSite(baseSiteService.getBaseSiteForUID("electronics"));
productsApprovedEmailProcessModel.setLanguage(CommerceCommonI18NService.getCurrentLanguage());
modelService.save(productsApprovedEmailProcessModel);
businessProcessService.startProcess(productsApprovedEmailProcessModel);
But a acheive this error when I'm strating CronJob using HMC Interface :
Error executing ActionNode with ID [generateProductsApprovedEmail]: HtmlTemplate associated with MasterTemplate of EmailPageModel cannot be null
UPDATE :
Here is my business process :
<process xmlns="http://www.hybris.de/xsd/processdefinition"
start="generateProductsApprovedEmail"
name="productsApprovedEmailProcess"
processClass="com.hybris.training.core.model.process.ProductsApprovedEmailProcessModel"
onError="error">
<action id="generateProductsApprovedEmail" bean="generateProductsApprovedEmail">
<transition name="OK" to="sendEmail"/>
<transition name="NOK" to="error"/>
</action>
<action id="sendEmail" bean="sendEmail">
<transition name="OK" to="removeSentEmail"/>
<transition name="NOK" to="failed"/>
</action>
<action id="removeSentEmail" bean="removeSentEmail">
<transition name="OK" to="success"/>
<transition name="NOK" to="error"/>
</action>
<end id="error" state="ERROR">Something went wrong.</end>
<end id="failed" state="FAILED">Could not send products approved in csv File email.</end>
<end id="success" state="SUCCEEDED">Sent file in email.</end>
After declaring ProductApprovedEmailTemplate (EmailPageTemplate) i got this warn and the mail is not generated :
WARN [TaskExecutor-master-264-ProcessTask [8796715713462]] [GenerateEmailAction] Could not retrieve email page model for ProductApprovedEmail and Electronics Content Catalog:Online, cannot generate email content
Look like, the blog you have followed, it has mentioned each step correctly, but you might be missed something.
Make sure you have followed the below steps correctly.
e.g.
frontendTemplateName should be matched with EmailPageTemplate one
<bean id="generateProductApprovedEmail" parent="abstractGenerateEmailAction">
<property name="frontendTemplateName" value="ProductApprovedEmail"/>
</bean>
Create Email page Template
INSERT_UPDATE EmailPageTemplate ;$contentCV[unique=true];uid[unique=true] ;active ;frontendTemplateName ;subject(code) ;htmlTemplate(code) ;restrictedPageTypes(code)
; ;ProductApprovedEmailTemplate ;true ;ProductApprovedEmail ;email-productsapproved-subject ;email-productsapproved-body ;EmailPage
Create Email Page
INSERT_UPDATE EmailPage ;$contentCV[unique=true];uid[unique=true] ;masterTemplate(uid,$contentCV);approvalStatus(code)[default='approved']
; ;ProductApprovedEmail ;ProductApprovedEmailTemplate ;

Reuse of service-activator for several gateway methods with Splitter

does somebody happen to know if it is valid to reuse a service activator and so also the output-channel using several methods (inbound) especially with a splitter and aggregator.
--> Always with result on the gateway.
In several tests it seems to work fine. As soon I added a splitter with an aggregator I get wrongs result routed to the gateway which then fails with a conversion exception (here in my case it cannot convert boolean to integer).
Thanks,
Paul
Flow
Spring Integration Config
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int="http://www.springframework.org/schema/integration"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd">
<int:gateway service-interface="activity.BulbMessageGateway">
<int:method name="sendToBulb" request-channel="bulbMessages" reply-channel="bulbSendResult"></int:method>
<int:method name="updateHomeLightStatus" request-channel="homeBulbEntity" reply-channel="homeBulbEntityResult">
</int:method>
<int:method name="updateLightStatus" request-channel="bulbEntities" reply-channel="bulbSendResult">
<int:header name="homeId" expression="#args[0].homeId"/>
<int:header name="bulbId" expression="#args[0].strongId"/>
</int:method>
</int:gateway>
<int:channel id="bulbMessages" />
<int:channel id="bulbSendResult" />
<int:channel id="bulbEntities" />
<int:channel id="homeBulbEntity" />
<int:channel id="homeBulbEntityResult" />
<int:chain input-channel="homeBulbEntity" output-channel="bulbEntities">
<int:splitter expression="payload.bulbs" />
<int:header-enricher>
<int:header name="bulbId" expression="payload.strongId"/>
<int:header name="homeId" expression="payload.homeId"/>
</int:header-enricher>
</int:chain>
<int:transformer method="bulbToLightStatus" input-channel="bulbEntities" output-channel="bulbMessages">
<bean class="util.BulbTransformer"></bean>
</int:transformer>
<int:aggregator input-channel="bulbSendResult" output-channel="homeBulbEntityResult" method="aggregate">
<bean class="util.BooleanAggregator" />
</int:aggregator>
<int:service-activator input-channel="bulbMessages" output-channel="bulbSendResult" method="send">
<bean class="activity.BulbWebsocketMessageSenderBA" />
</int:service-activator>
</beans>
Unit test
#Test
public void sendMessageNoReceiver() {
assertFalse(gateway.sendToBulb(new HomeId("1"), new BulbId("1"), BulbMessageBuilder.restart("foo")));
}
#Test
public void sendMessageWithReceiver() {
MockSession<BulbId, BulbBE> bulbSession = new MockSession<BulbId, BulbBE>(new BulbBE(HomeId.of("1"), BulbId.of("1"), "bulb", "pass"));
registry.addBulbSession(bulbSession);
assertTrue(gateway.sendToBulb(new HomeId("1"), new BulbId("1"), BulbMessageBuilder.restart("foo")));
assertEquals(1, bulbSession.receivedMessages());
}
#Test
public void updateBulbStatus() {
final MockSession<BulbId, BulbBE> bulbSession1 = new MockSession<BulbId, BulbBE>(new BulbBE(HomeId.of("1"), BulbId.of("1"), "bulb", "pass"));
assertFalse(gateway.updateLightStatus(bulbSession1.getIdentity()));
registry.addBulbSession(bulbSession1);
assertTrue(gateway.updateLightStatus(bulbSession1.getIdentity()));
assertEquals(1, bulbSession1.receivedMessages());
final MockSession<BulbId, BulbBE> bulbSession2 = new MockSession<BulbId, BulbBE>(new BulbBE(HomeId.of("1"), BulbId.of("2"), "bulb", "pass"));
assertFalse(gateway.updateLightStatus(bulbSession2.getIdentity()));
registry.addBulbSession(bulbSession2);
assertTrue(gateway.updateLightStatus(bulbSession2.getIdentity()));
assertTrue(gateway.updateLightStatus(bulbSession2.getIdentity()));
assertEquals(2, bulbSession2.receivedMessages());
assertEquals(1, bulbSession1.receivedMessages());
}
#Test
public void updateHomeBulbStatus() {
final HomeBE home = new HomeBE();
home.setId(new ObjectId());
final MockSession<BulbId, BulbBE> bulbSession1 = new MockSession<BulbId, BulbBE>(new BulbBE(home.getStrongId(), BulbId.of("1"), "bulb", "pass"));
registry.addBulbSession(bulbSession1);
final MockSession<BulbId, BulbBE> bulbSession2 = new MockSession<BulbId, BulbBE>(new BulbBE(home.getStrongId(), BulbId.of("2"), "bulb", "pass"));
registry.addBulbSession(bulbSession2);
home.addBulb(bulbSession1.getIdentity());
assertEquals(1, gateway.updateHomeLightStatus(home));
assertEquals(1, bulbSession1.receivedMessages());
assertEquals(0, bulbSession2.receivedMessages());
home.addBulb(bulbSession2.getIdentity());
assertEquals(2, gateway.updateHomeLightStatus(home));
assertEquals(2, bulbSession1.receivedMessages());
assertEquals(1, bulbSession2.receivedMessages());
}
The last test fails if it is executed together with the other tests. It passes if it is executed alone.
The error is that the last method (using the splitter) receives now a boolean, which seems to be a result of the other two methods registered. The result of these methods is a boolean.
Please, share config on the matter.
And according your graph it would be better if you'd minimize the config as much as possible to isolate the problem.
From other side, please, be more specific: your question is fully unclear.
That is your own service. How can we be sure that it is safe to be used in different places? Only you, as an author, can determine that.
UPDATE
Sorry for the delay. Was busy with the release.
And thank for sharing the config for your use-case.
Now I see the problem.
You use everywhere on the gateway's methods a reply-channel. See the documentation on the matter when you need that:
Typically you don’t have to specify the default-reply-channel, since a Gateway will auto-create a temporary, anonymous reply channel, where it will listen for the reply. However, there are some cases which may prompt you to define a default-reply-channel (or reply-channel with adapter gateways such as HTTP, JMS, etc.).
Since you use the same bulbSendResult in different places the behavior is really unpredictable. Moreoever that channel is DirectChannel, so the round-robin balancer are on the scene.
You should get rid of those reply-channel's at all and just rely from your downstream components on the replyChannel header. Therefore you should remove those output-channels in the components which are intended to return replies to your gateway.
For example the last service-activator should be just like this:
<int:service-activator input-channel="bulbMessages" method="send">
<bean class="activity.BulbWebsocketMessageSenderBA"/>
</int:service-activator>
Since you general question how to reuse this service-activator, I'm answering to the question having the config from you:
<int:chain input-channel="homeBulbEntity">
<int:splitter expression="payload.bulbs"/>
<int:header-enricher>
<int:header name="bulbId" expression="payload.strongId"/>
<int:header name="homeId" expression="payload.homeId"/>
</int:header-enricher>
<int:transformer method="bulbToLightStatus">
<bean class="util.BulbTransformer"/>
</int:transformer>
<int:gateway request-channel="bulbMessages"/>
<int:aggregator method="aggregate">
<bean class="util.BooleanAggregator"/>
</int:aggregator>
</int:chain>
Pay attention to the absent output-channel for the <chain>. Therefore it sends reply directly to the replyChannel from headers and as a return to your updateHomeLightStatus gateway's method.
Another trick is that <int:gateway request-channel="bulbMessages"/> which sends messages in the middle of the <chain> flow to your <service-activator> and wait for the reply from there exactly the same way as a top-level gateway - via replyChannel header. For the <service-activator> without an output-channel it is a black-box where to send the reply. It uses just replyChannel from headers!
After receiving the reply gateway in the <chain> push the message to the <aggregator>.
When aggregator will do its logic, the result will be send to the top-level gateway as an output from the <chain>.
That's all.
Let me know what else isn't clear here.

ChannelResolutionException: no output-channel or replyChannel header available - Only with many requests

I am running the client portion of the Spring Integration TCP Multiplex example. I was trying to see how many requests it could handle at once and around 1000, I started to get this error: ChannelResolutionException: no output-channel or replyChannel header available
Everything is fine below about 1000 calls.
<beans:description>
Uses conversion service and collaborating channel adapters.
</beans:description>
<context:property-placeholder />
<converter>
<beans:bean class="org.springframework.integration.samples.tcpclientserver.ByteArrayToStringConverter" />
</converter>
<!-- Fastest Wire Protocol - takes a byte array with its length definied in the first x bytes-->
<beans:bean id="fastestWireFormatSerializer" class="org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer">
<beans:constructor-arg value="1" />
</beans:bean>
<!-- Client side -->
<gateway id="gw"
service-interface="org.springframework.integration.samples.tcpclientserver.SimpleGateway"
default-request-channel="input" />
<ip:tcp-connection-factory id="client"
type="client"
host="localhost"
port="${availableServerSocket}"
single-use="false"
serializer="fastestWireFormatSerializer"
deserializer="fastestWireFormatSerializer"
so-timeout="10000" />
<publish-subscribe-channel id="input" />
<!-- scheduler - Thread used to restablish connection so the other threads aren't starved while waiting to re-establish connection -->
<!-- client-mode - Automatically re-establishes the connection if lost -->
<ip:tcp-outbound-channel-adapter id="outAdapter.client"
order="2"
channel="input"
client-mode="true"
connection-factory="client" /> <!-- Collaborator -->
<!-- Also send a copy to the custom aggregator for correlation and
so this message's replyChannel will be transferred to the
aggregated message.
The order ensures this gets to the aggregator first -->
<bridge input-channel="input" output-channel="toAggregator.client"
order="1"/>
<!-- Asynch receive reply -->
<ip:tcp-inbound-channel-adapter id="inAdapter.client"
channel="toAggregator.client"
connection-factory="client" /> <!-- Collaborator -->
<!-- dataType attribute invokes the conversion service, if necessary -->
<channel id="toAggregator.client" datatype="java.lang.String" />
<aggregator input-channel="toAggregator.client"
output-channel="toTransformer.client"
correlation-strategy-expression="payload.substring(0,3)"
release-strategy-expression="size() == 2"
expire-groups-upon-completion="true" />
<transformer input-channel="toTransformer.client"
expression="payload.get(1)"/> <!-- The response is always second -->
<task:scheduler id="reconnectScheduler" pool-size="10"/>
And the code used to test:
TaskExecutor executor = new SimpleAsyncTaskExecutor();
final CountDownLatch latch = new CountDownLatch(100);
final Set<Integer> results = new HashSet<Integer>();
for (int i = 100; i < 1050; i++) {
results.add(i);
final int j = i;
executor.execute(new Runnable() {
public void run() {
String result = gateway.send(j + "Hello world!"); // first 3 bytes is correlationid
System.out.println("Test Result: " + result);
results.remove(j);
latch.countDown();
}});
}
I haven't figured out entirely why you are getting that exception, but there are several problems with your test.
The countdown latch needs to be initialized at 950
Since you are exceeding 999, we need to change the correlation:
payload.substring(0,4)
With those changes, it works for me.
I'll try to figure out why we're getting that exception when I get a bit more time.
EDIT
The issue is indeed caused by the conflicting correlation ids.
The last 50 messages all have correlation id 100 which means messages are released in an indeterminate fashion (given the release is based on size). In some cases two input messages are released (causing the wrong reply to the test case). When 2 replies are released; there is no output channel.

Spring integration no files picked

The following is my configuration. I was able to poll files until recently. Now the filter always gets an empty list of files. The only change i have made is install kaspersky antivirus. Hopefully that is not a problem. I can successfully access the ftp from command prompt as well as from the browser.
The conf:
<int:channel id="ftpChannel"/>
<int-ftp:inbound-channel-adapter id="ftpInbound1"
channel="ftpChannel"
session-factory="ftpClientFactory"
charset="UTF-8"
local-directory="file:${paths.root}"
delete-remote-files="false"
temporary-file-suffix=".writing"
remote-directory="${file.ftpfolder}"
preserve-timestamp="true"
auto-startup="true"
filter="compositeFilterLocal"
>
<int:poller max-messages-per-poll="10000" fixed-rate="1000" error-channel="errorChannel"/>
</int-ftp:inbound-channel-adapter>
<int-ftp:outbound-channel-adapter id="ftpOutbound"
channel="ftpChannel"
session-factory="ftpClientFactory"
charset="UTF-8"
remote-file-separator="/"
auto-create-directory="true"
remote-directory="DMS"
use-temporary-file-name="true"
temporary-file-suffix=".writing">
</int-ftp:outbound-channel-adapter>
<!-- <bean id="acceptAllFilter" class="org.springframework.integration.file.filters.AcceptAllFileListFilter" /> -->
<bean id="compositeFilterLocal" class="org.springframework.integration.file.filters.CompositeFileListFilter">
<constructor-arg>
<list>
<!-- Ensures that the file is whole before processing it -->
<bean class="org.springframework.integration.file.filters.AcceptAllFileListFilter" />
<bean class="com.polling.util.CustomFileFilterLocal"/>
<!-- Ensures files are picked up only once from the directory -->
</list>
</constructor-arg>
</bean>
Please tell me if anything should be changed in it...Thanks
Please let me know if anymore information is needed!
EDIT:: Update
If I use the Apache commons-net-3.3 to retrieve the same file from the same folder, it is working fine and allowing me to take the file and download it. So this has nothing to do with jvm access to the ftp site.
EDIT::Code for the filter is simple. Currently I am only using it for pattern matching.
#Override
public List<File> filterFiles(File[] files)
{
List<File> ret = new ArrayList<File>();
Pattern pattern = Pattern.compile(".*?~.*?");//(".*?#.*?#.*?");
DocumentFile documentFile;
Matcher matcher;
for (File file : files)
{
matcher = pattern.matcher(file.getName());
if(matcher.find())// matching the input file name pattern
{
//get key and documentfile
//create sha key to check file existance
String key = EncodeUtil.generateKey(file);
documentFile = documentDaoImpl.getDocumentFile(key,Constants.INPROGRESS);
if (documentFile != null)
{
ret.add(file);
}
}/*else
{
file.delete();
}*/
}
return ret;
}
I have been successfully working with this for atleast a couple of months and now suddenly I am getting no files!!
Currently I am in process of using a timer cron expression and will do the ftp using apache commons-net within the triggered class. Seems such a waste having to do the ftp inspite of having the spring ftp tag.
I have built a project with the configuration you have used and everything seems to work fine.
There are some pieces in your code (not posted here) that might lead to discard the files in the filter and that you will have to check (adding log messages will help):
if (matcher.find())// matching the input file name pattern
{
// get key and documentfile
// create sha key to check file existance
// TODO: does this call throw any exception? return null?
String key = EncodeUtil.generateKey(file);
documentFile = documentDaoImpl.getDocumentFile(key, Constants.INPROGRESS);
if (documentFile != null) {
ret.add(file);
}
else {
// TODO: Log here that your DAO implementation did not return anything for this specific file
}
}
else {
// TODO: Log here that the file does not meet the naming convention
}

File inbound-channel-adapter spring integration for Multiple Files aggregation into one master File for Job processing

I have written a code to combined multiple files into one single Master file.
The issue is with int-transformer where I am getting one file at a time although I have aggregated List of File in composite Filter of File inbound-channel-adapter. The List of File size in composite filter is correct but in Transformer bean the List of File size is always one and not getting the correct list size aggregated file by the filter.
Here is my config:
<!-- Auto Wiring -->
<context:component-scan base-package="com.nt.na21.nam.integration.*" />
<!-- intercept and log every message -->
<int:logging-channel-adapter id="logger"
level="DEBUG" />
<int:wire-tap channel="logger" />
<!-- Aggregating the processed Output for OSS processing -->
<int:channel id="networkData" />
<int:channel id="requests" />
<int-file:inbound-channel-adapter id="pollProcessedNetworkData"
directory="file:${processing.files.directory}" filter="compositeProcessedFileFilter"
channel="networkData">
<int:poller default="true" cron="*/20 * * * * *" />
</int-file:inbound-channel-adapter>
<bean id="compositeProcessedFileFilter"
class="com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine" />
<int:transformer id="aggregateNetworkData"
input-channel="networkData" output-channel="requests">
<bean id="networkData" class="com.nt.na21.nam.integration.helper.CSVFileAggregator">
</bean>
</int:transformer>
CompositeFileListFilterForBaseLine:
public class CompositeFileListFilterForBaseLine implements FileListFilter<File> {
private final static Logger LOG = Logger
.getLogger(CompositeFileListFilterForBaseLine.class);
#Override
public List<File> filterFiles(File[] files) {
List<File> filteredFile = new ArrayList<File>();
int index;
String fetchedFileName = null;
String fileCreatedDate = null;
String todayDate = DateHelper.toddMM(new Date());
LOG.debug("Date - dd-MM: " + todayDate);
for (File f : files) {
fetchedFileName = StringUtils.removeEnd(f.getName(), ".csv");
index = fetchedFileName.indexOf("_");
// Add plus one to index to skip underscore
fileCreatedDate = fetchedFileName.substring(index + 1);
// Format the created file date
fileCreatedDate = DateHelper.formatFileNameDateForAggregation(fileCreatedDate);
LOG.debug("file created date: " + fileCreatedDate + " today Date: "
+ todayDate);
if (fileCreatedDate.equalsIgnoreCase(todayDate)) {
filteredFile.add(f);
LOG.debug("File added to List of File: " + f.getAbsolutePath());
}
}
LOG.debug("SIZE: " + filteredFile.size());
LOG.debug("filterFiles method end.");
return filteredFile;
}
}
The Class file for CSVFileAggregator
public class CSVFileAggregator {
private final static Logger LOG = Logger.getLogger(CSVFileAggregator.class);
private int snePostion;
protected String masterFileSourcePath=null;
public File handleAggregateFiles(List<File> files) throws IOException {
LOG.debug("materFileSourcePath: " + masterFileSourcePath);
LinkedHashSet<String> allAttributes = null;
Map<String, LinkedHashSet<String>> allAttrBase = null;
Map<String, LinkedHashSet<String>> allAttrDelta = null;
LOG.info("Aggregator releasing [" + files.size() + "] files");
}
}
Log Output:
INFO : com.nt.na21.nam.integration.aggregator.NetFileAggregatorClient - NetFileAggregator context initialized. Polling input folder...
INFO : com.nt.na21.nam.integration.aggregator.NetFileAggregatorClient - Input directory is: D:\Projects\csv\processing
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - Date - dd-MM: 0103
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - file created date: 0103 today Date: 0103
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - File added to List of File: D:\Projects\NA21\NAMworkspace\na21_nam_integration\csv\processing\file1_base_0103.csv
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - file created date: 0103 today Date: 0103
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - File added to List of File: D:\Projects\NA21\NAMworkspace\na21_nam_integration\csv\processing\file2_base_0103.csv
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - **SIZE: 2**
DEBUG: com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForBaseLine - filterFiles method end.
DEBUG: org.springframework.integration.file.FileReadingMessageSource - Added to queue: [csv\processing\file1_base_0103.csv, csv\processing\file2_base_0103.csv]
INFO : org.springframework.integration.file.FileReadingMessageSource - Created message: [GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]]
DEBUG: org.springframework.integration.endpoint.SourcePollingChannelAdapter - Poll resulted in Message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'networkData', message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: org.springframework.integration.handler.LoggingHandler - org.springframework.integration.handler.LoggingHandler#0 received message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: org.springframework.integration.handler.LoggingHandler - csv\processing\file2_base_0103.csv
DEBUG: org.springframework.integration.channel.DirectChannel - postSend (sent=true) on channel 'logger', message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: org.springframework.integration.transformer.MessageTransformingHandler - org.springframework.integration.transformer.MessageTransformingHandler#606f8b2b received message: GenericMessage [payload=csv\processing\file2_base_0103.csv, headers={timestamp=1425158920029, id=cb3c8505-0ee5-7476-5b06-01d14380e24a}]
DEBUG: com.nt.na21.nam.integration.helper.CSVFileAggregator - materFileSourcePath: null
INFO : com.nt.na21.nam.integration.helper.CSVFileAggregator - **Aggregator releasing [1] files**
Can some one help me here in identifying the issue with Filter and same is not collecting for transformation?
Thanks in advance.
The issue is with int:aggregator as I am not sure how to invoke. I have used this earlier in my design but it didn't get executed at all. Thanks for the quick response.
For this problem I have written a FileScaner utility which will scan all the files in Folder inside and aggregation is working perfectly.
Please find the config with Aggregator which didn't works, hence I splited the design by two poller first produced all the CSV file(s) and second collect it and aggregate it.
<!-- Auto Wiring -->
<context:component-scan base-package="com.bt.na21.nam.integration.*" />
<!-- intercept and log every message -->
<int:logging-channel-adapter id="logger" level="DEBUG" />
<int:wire-tap channel = "logger" />
<int:channel id="fileInputChannel" datatype="java.io.File" />
<int:channel id="error" />
<int:channel id="requestsCSVInput" />
<int-file:inbound-channel-adapter id="pollNetworkFile"
directory="file:${input.files.directory}" channel="fileInputChannel"
filter="compositeFileFilter" prevent-duplicates="true">
<int:poller default="true" cron="*/20 * * * * *"
error-channel="error" />
</int-file:inbound-channel-adapter>
<bean id="compositeFileFilter"
class="com.nt.na21.nam.integration.file.filter.CompositeFileListFilterForTodayFiles" />
<int:transformer id="transformInputZipCSVFileIntoCSV"
input-channel="fileInputChannel" output-channel="requestsCSVInput">
<bean id="transformZipFile"
class="com.nt.na21.nam.integration.file.net.NetRecordFileTransformation" />
</int:transformer>
<int:router ref="docTypeRouter" input-channel="requestsCSVInput"
method="resolveObjectTypeChannel">
</int:router>
<int:channel id="Vlan" />
<int:channel id="VlanShaper" />
<int:channel id="TdmPwe" />
<bean id="docTypeRouter"
class="com.nt.na21.nam.integration.file.net.DocumentTypeMessageRouter" />
<int:service-activator ref="vLanMessageHandler" output-channel="newContentItemNotification" input-channel="Vlan" method="handleFile" />
<bean id="vLanMessageHandler" class="com.nt.na21.nam.integration.file.handler.VLanRecordsHandler" />
<int:service-activator ref="VlanShaperMessageHandler" output-channel="newContentItemNotification" input-channel="VlanShaper" method="handleFile" />
<bean id="VlanShaperMessageHandler" class="com.nt.na21.nam.integration.file.handler.VlanShaperRecordsHandler" />
<int:service-activator ref="PweMessageHandler" output-channel="newContentItemNotification" input-channel="TdmPwe" method="handleFile" />
<bean id="PweMessageHandler" class="com.nt.na21.nam.integration.file.handler.PseudoWireRecordsHandler" />
<int:channel id="newContentItemNotification" />
<!-- Adding for aggregating the records in one place for OSS output -->
<int:aggregator input-channel="newContentItemNotification" method="aggregate"
ref="netRecordsResultAggregator" output-channel="net-records-aggregated-reply"
message-store="netRecordsResultMessageStore"
send-partial-result-on-expiry="true">
</int:aggregator>
<int:channel id="net-records-aggregated-reply" />
<bean id="netRecordsResultAggregator" class="com.nt.na21.nam.integration.aggregator.NetRecordsResultAggregator" />
<!-- Define a store for our network records results and set up a reaper that will
periodically expire those results. -->
<bean id="netRecordsResultMessageStore" class="org.springframework.integration.store.SimpleMessageStore" />
<int-file:outbound-channel-adapter id="filesOut"
directory="file:${output.files.directory}"
delete-source-files="true">
</int-file:outbound-channel-adapter>
The code is working fine till the routed to all the channel below:
<int:channel id="Vlan" />
<int:channel id="VlanShaper" />
<int:channel id="TdmPwe" />
I am trying to return LinkedHashSet from the Process of the above channel which contains CSV data and I need to aggregate all the merge
LinkedHashSet vAllAttributes to get the master output CSV file.
List<String> masterList = new ArrayList<String>(vAllAttributes);
Collections.sort(masterList);
Well, looks like you misunderstood a bit <int-file:inbound-channel-adapter> behaviour. Its nature is producing one file per message to the channel. It doesn't depend on the logic of the FileListFilter. The is like:
The FileReadingMessageSource uses DirectoryScanner to retrieve files from the provided directory to an internal toBeReceived Queue
Since we scan the directory for the files the design for the DirectoryScanner looks like List<File> listFiles(File directory). I guess this has led you astray.
After that the filter is applied to the original file list and returns only appropriate files.
They are stored to the toBeReceived Queue.
And only after that the FileReadingMessageSource polls an item from the queue to build message for the output channel.
To achieve your aggregation requirements you really should use an <aggregator> between <int-file:inbound-channel-adapter> and your <int:transformer>.
You can mark the <poller> of the <int-file:inbound-channel-adapter> with max-messages-per-poll="-1" to really poll all your files during the single scheduled task. But anyway there will as much messages as your filter returns files.
After that you must accept some tricks for the <aggregator>:
correlationKey - to allow your file messages to be combined to the single MessageGroup for release a single message for the further <transformer>. Since we don't have any context from <int-file:inbound-channel-adapter>, but we know that all messages are provided by the single polling task and withing scheduled Thread (you don't use task-executor on the <poller>), hence we can simply use correlationKey as:
correlation-strategy-expression="T(Thread).currentThread().id"
But the is not enough, because we should produce somehow the single message in the end anyway. Unfortunately we don't know the number of files (however you can do that via the ThreadLocal from your custom FileListFilter) to allow the ReleaseStrategy to return true for the aggregate phase. Hence we never have the normal group completion. But we can forceRelease uncompleted groups from the aggregator to use the MessageGroupStoreReaper or group-timeout on the <aggregator>.
In addition to the previous clause you should supply these options on the <aggegator>:
send-partial-result-on-expiry="true"
expire-groups-upon-completion="true"
And that's all. There is no reason to provide any custom aggregation function (ref/method or expression), because the default on just build a single message with the List of payloads from all messages in group. And that is appropriate for your CSVFileAggregator. Although you can avoid that <transformer> and this CSVFileAggregator for the aggregation function.
Hope I ma clear

Resources