I have to copy files A and B sequentially to a remote folder. It is important that B is sent only after A has been sent, ot at least at the same time, but not before.
I've read the doc, but it's not clear. My idea is to put 2 messages into the same channel. But I don't know if the files linked to these 2 messages will be sent sequentially.
#Component
public class JobExportExecutionsRouter {
...
#Autowired
private MessageChannel sftpIncrExportChannel;
...
#Router
public List<String> routeJobExecution(JobExecution jobExecution) {
final List<String> routeToChannels = new ArrayList<String>();
...
sftpIncrExportChannel.send(MessageBuilder.withPayload(fileA).build());
sftpIncrExportChannel.send(MessageBuilder.withPayload(fileB).build());
routeToChannels.add("sftpIncrExportChannel");
return routeToChannels;
}
}
My XML configuration contains:
<int:channel id="sftpIncrExportChannel">
<int:queue/>
</int:channel>
...
<int-sftp:outbound-channel-adapter session-factory="sftpSessionFactory" channel="sftpIncrExportChannel" charset="UTF8" remote-directory="${export.incr.sftp.dir}" />
...
<bean id="sftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="${export.incr.sftp.dir}"/>
<property name="user" value="${export.incr.sftp.user}"/>
<property name="password" value="${export.incr.sftp.password}"/>
</bean>
Do you have suggestions?
If you remove the <queue/> from the channel, they will run sequentially on your calling thread.
If you use a queue channel; you need a poller but, as long as the poller does not have a task-executor, the messages will be sent sequentially on the poller thread. The next poll doesn't happen until the current one completes.
Related
I am trying to use ftp-inbound-adapter to poll files based on the the current date.And my inbound adapter uses a filter which refers to bean myfilter .The problem here is the current date is initailized at the startup and is not been processed dynamically..I would like to get the current date for every new message
<int-ftp:inbound-channel-adapter id="ftpInbound"
channel="ftpChannel"
session-factory="ftpsClientFactory"
filter="myFilter"
</int-ftp:inbound-channel-adapter>
<bean id="myFilter" class="org.springframework.integration.ftp.filters.FtpRegexPatternFileListFilter" scope="prototype">
<constructor-arg value="#{T(java.time.LocalDateTime).now().format(T(java.time.format.DateTimeFormatter).ofPattern('MMddyy'))}.(xls|xlsx)"/>
<aop:scoped-proxy/>
</bean>
UPDATE
I changed from this
<bean id="currentDate" class="java.util.Date" factory-bean="fastDateFormat"
scope="prototype" factory-method="format" >
<aop:scoped-proxy/>
<constructor-arg>
<bean class="java.util.Date" />
</constructor-arg>
</bean>
<bean id="myFilter" class="org.springframework.integration.ftp.filters.FtpRegexPatternFileListFilter" scope="prototype">
<constructor-arg value="#{currentDate}.(xls|xlsx)"/>
</bean>
And my inbound adapter uses a filter which refers to bean myFilter .. The problem here is the current date is initailized at the startup and is not been processed dynamically..I would like to get the current date for every new message
This impossible with your current configuration because that filter is just a singleton bean created only once at start up pulling that your currentDate for injection only once as well.
You may try to add <aop:scoped-proxy/> into your currentDate bean definition, though: https://docs.spring.io/spring/docs/5.1.3.RELEASE/spring-framework-reference/core.html#beans-factory-scopes-other-injection, but I would suggest to inject BeanFactorty into your filter and call getBean("currentDate", Date.class) every time you need a new instance of that prototype.
UPDATE
You inject BeanFactory into your filter instead of that currentDate bean. And then when filter logic is called you do Date currentDate = this.beanFactory.getBean("currentDate", Date.class);.
UPDATE2
Here is what I think should work for you:
public class DynamicRegexPatternFilter extends AbstractFileListFilter<File> {
#Autowired
private BeanFactory beanFactory;
#Override
public boolean accept(File file) {
return Pattern.compile(this.beanFactory.getBean("currentDate", String.class) + ".(xls|xlsx)")
.matcher(file.getName())
.matches();
}
}
You can make custom filter. Implement FileListFilter and override filterFiles method which takes input argument as array of LsEntry. LsEntry is nothing but a small metadata object of a File present at (S)FTP. It comprises of File Name and the date on which it was modified.
#Override
public List<LsEntry> filterFiles(LsEntry[] files) {
List<LsEntry> result = new ArrayList<LsEntry>();
Vector<LsEntry> list = new Vector<LsEntry>();
Collections.addAll(list, files);
ChannelSftp.LsEntry lastModifiedEntry = Collections.max(list,
(Comparator.comparingInt(entry -> entry.getAttrs().getMTime())));
result.add(lastModifiedEntry);
return result;
}
Reference: https://stackoverflow.com/a/60635859/11325128
I am building Docker Containers which have a war file running on Jetty, and I have been alternating a few settings to see if performance improves but nothing so far. Per container it has been achieving 7 tps.
The settings are
<bean id="cachingConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="MQConnectionFactory" />
<property name="sessionCacheSize" value="10"/>
</bean>
<bean id="requestQueue" class="com.ibm.mq.jms.MQQueue">
<constructor-arg index="0" value="${queuemanager}"/>
<constructor-arg index="1" value="${incoming.queue}"/>
</bean>
<integration:poller id="poller" default="true" fixed-delay="1000" error-channel="errorChannel"/>
How can I improve the number of threads processing over here?
Also, my connection factory details are as shown below
#Bean(name="DefaultJmsListenerContainerFactory")
public DefaultJmsListenerContainerFactory provideJmsListenerContainerFactory(PlatformTransactionManager transactionManager) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(connectionFactory());
factory.setTransactionManager(transactionManager);
factory.setConcurrency(jmsConcurrency);
factory.setCacheLevel(jmsCacheLevel);
factory.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
factory.setSessionTransacted(true);
return factory;
}
#Bean(name = "txManager")
public PlatformTransactionManager provideTransactionManager() {
return new JmsTransactionManager(connectionFactory());
}
#Bean(name = "JmsTemplate")
public JmsTemplate provideJmsTemplate() {
JmsTemplate jmsTemplate = new JmsTemplate(connectionFactory());
jmsTemplate.setReceiveTimeout(Long.parseLong(env.getRequiredProperty(RECEIVE_TIMEOUT)));
return jmsTemplate;
}
#Bean(name="MQConnectionFactory")
public ConnectionFactory connectionFactory() {
if (factory == null) {
factory = new MQXAConnectionFactory();
try {
factory.setHostName(env.getRequiredProperty(HOST));
factory.setPort(Integer.parseInt(env.getRequiredProperty(PORT)));
factory.setQueueManager(env.getRequiredProperty(QUEUE_MANAGER));
factory.setChannel(env.getRequiredProperty(CHANNEL));
factory.setTransportType(WMQConstants.WMQ_CM_CLIENT);
} catch (JMSException e) {
throw new RuntimeException(e);
}
}
return factory;
}
The initial setting for the concurrency was '1-2' and I changed that to '10-15'. Did not affect performance.
The jmsCache was set to 3 (Consumer cache), but no change there yet either.
Any help is much appreciated.
Cheers
Kris
Answering my own post here. What we found out was that the problem was actually with our Database pooling not setup correctly in the first place.
But in order to increase the Listener count, I had to change my Spring integration adapter settings
<jms:message-driven-channel-adapter id="jmsIn"
destination="requestQueue"
channel="inputJsonConversionChannel"
connection-factory="cachingConnectionFactory"
error-channel="errorChannel"
concurrent-consumers="${jms_adapter_concurrent_consumers}" />
Only when the concurrent-consumers is varied, does the number of listeners on the queue increase.
I have some performance problem with spring integration tcp factory.
my application have about 70 clients which trying to send data through tcp connection. i used below configuration for tcp server using spring integration but in server i receive data every 5 seconds. but when i implement tcp socket manually without using spring integration i receive about 5 connections in every second. any idea about my problem ? i really want to use spring integration but i don't know how can i increase my performance.
<int:poller id="defaultPoller" default="true" tast-executor="defaultTaskExecutor" fixed-delay="500" />
<task:executor id="defaultTaskExecutor" pool-size="5-20" queue-capacity="50"/>
<bean id="CustomeSerializerDeserializer"
class="CustomeSerializerDeserializer" />
<task:executor id="tcpFactoryTaskExecutor" pool-size="5-20"
queue-capacity="20000" />
<int-ip:tcp-connection-factory id="tcpConnectionFactory"
type="server" port="5423"
single-use="false" so-timeout="5000" task-executor="tcpFactoryTaskExecutor"
serializer="CustomeSerializerDeserializer" deserializer="CustomeSerializerDeserializer" />
<int-ip:tcp-inbound-channel-adapter
id="tcpInboundAdapter" channel="requestChannel" connection-factory="tcpConnectionFactory" />
<int:channel id="requestChannel">
<int:queue capacity="50" />
</int:channel>
<int:service-activator input-channel="requestChannel"
output-channel="responseChannel" ref="MessageHandler" method="parse" />
<bean id="MessageHandler"
class="TCPMessageHandler" />
<int:channel id="responseChannel">
<int:queue capacity="50" />
<int:channel />
<int-ip:tcp-outbound-channel-adapter
id="tcpOutboundAdapter" channel="responseChannel" connection-factory="tcpConnectionFactory" />
UPDATE1: here is my custom serialize/deserializer class:
public class SerializerDeserializer extends AbstractByteArraySerializer{
#Override
public void serialize(byte[] object, OutputStream outputStream)
throws IOException {
if (object != null && object.length != 0) {
outputStream.write(object);
outputStream.flush();
}
}
#Override
public byte[] deserialize(InputStream inputStream) throws IOException {
int c = inputStream.read();
if (c!=0){
// 2 byte
byte[] configMessage = BinaryUtil.readNByteArrayFromStream(inputStream, c)/*(inputStream , c)*/;
return configMessage;
}
int d = inputStream.read();
if (d==0){
// 253 byte
byte[] dataMessage = BinaryUtil.readNByteArrayFromStream(inputStream,253);
return dataMessage;
}
// 15 byte
byte[] hanshakeMessage = BinaryUtil.readNByteArrayFromStream(inputStream,d);
return hanshakeMessage;
}
}
I suspect a problem with your custom deserializer 5 seconds is suspicious since your timeout is also 5 seconds - show the code and explain the protocol.
If the deserializer doesn't receive a full message it will time out.
Also turn on TRACE level logging for org.springframework.integration to debug this - if you can't figure it out from the trace, post the log file somewhere like pastebin and we'll take a look.
I need to handle certain error conditions within my spring integration flow. My flow is using a message store and setting the error channel on the poller. I had thought that if I handled the message in the error handler that the rollback would not occur, but the messageStore remove (delete) is being rolled back before the error flow is even executed.
Here is a pseudo-flow that duplicates my issue.
<int:channel id="rollbackTestInput" >
<int:queue message-store="messageStore"/>
</int:channel>
<int:bridge input-channel="rollbackTestInput" output-channel="createException" >
<int:poller fixed-rate="50"
error-channel="myErrorChannel">
<int:transactional />
</int:poller>
</int:bridge>
<int:transformer input-channel="createException" output-channel="infoLogger"
expression="T(GarbageToForceException).doesNotExist()" />
<int:channel id="myErrorChannel">
<int:queue/>
</int:channel>
<!-- JDBC Message Store -->
<bean id="messageStore" class="org.springframework.integration.jdbc.JdbcMessageStore">
<property name="dataSource">
<ref bean="dataSource" />
</property>
</bean>
<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager" >
<property name="dataSource" ref="dataSource"/>
This flow will result in an infinite rollback/poll loop. How can I handle the error and not rollback?
That's correct behavior. The error-channel logic is accepted around the TX advice for the polling task.
The code looks like:
#Override
public void run() {
taskExecutor.execute(new Runnable() {
#Override
public void run() {
int count = 0;
while (initialized && (maxMessagesPerPoll <= 0 || count < maxMessagesPerPoll)) {
try {
if (!pollingTask.call()) {
break;
}
count++;
}
catch (Exception e) {
if (e instanceof RuntimeException) {
throw (RuntimeException) e;
}
else {
throw new MessageHandlingException(new ErrorMessage(e), e);
}
}
}
}
});
}
Where TX Advice is on the pollingTask.call(), but error handling is done from the taskExecutor:
this.taskExecutor = new ErrorHandlingTaskExecutor(this.taskExecutor, this.errorHandler);
Where your error-channel is configured on that errorHandler as MessagePublishingErrorHandler.
To reach your requirements you can try to follow with synchronization-factory on the <poller>:
<int:transaction-synchronization-factory id="txSyncFactory">
<int:after-rollback channel="myErrorChannel" />
</int:transaction-synchronization-factory>
Or supply your <int:transformer> with <request-handler-advice-chain>:
<int:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="onFailureExpression" value="#exception" />
<property name="failureChannel" value="myErrorChannel" />
<property name="trapException" value="true" />
</bean>
</int:request-handler-advice-chain>
I have found a different solution that meets my requirements and is less invasive.
Instead of using transactional on the poller, I can use an advice chain that that specifies which exceptions I should rollback and which I shouldn't. For my case, I do not want to rollback most exceptions. So I rollback Throwable and list any specific exceptions I want to rollback.
<int:bridge input-channel="rollbackTestInput"
output-channel="createException">
<int:poller fixed-rate="50"
error-channel="myErrorChannel">
<int:advice-chain>
<int:ref bean="txAdvice"/>
</int:advice-chain>
</int:poller>
</int:bridge>
<tx:advice id="txAdvice" transaction-manager="transactionManager">
<tx:attributes>
<tx:method name="*" rollback-for="javax.jms.JMSException"
no-rollback-for="java.lang.Throwable" />
</tx:attributes>
</tx:advice>
I want to use read files in one directory and move to another directory, I am using spring-integration for this. I want to execute the task(move files to output directory) when request coming from another system. I do not want to run file mover repeatedly, Is there a way to do this in Spring Integration?
Thank you in advance,
Udeshika
There are some tricks you can do with pollers, such as the FireOnceTrigger I mentioned in this answer. But in this case, probably the simplest solution is, instead of using an inbound adapter, define a <bean/> of type FileReadingMessageSource in your context; get a reference to it in your main() (context.getBean(FileReadingMessagesource.class)). Keep calling receive() and send the message received to the first channel in your flow (or use a <gateway/>.
When receive() returns null, exit.
Example for read specified file from samba share without poller:
FileService.java bean:
import org.springframework.context.ApplicationContext;
import org.springframework.integration.smb.session.SmbSession;
import org.springframework.integration.smb.session.SmbSessionFactory;
...
#Component
public class FileService {
private #Autowired ApplicationContext appContext;
...
private byte[] getFileContent( final String filename ) {
final SmbSessionFactory smbSessionFactory = (SmbSessionFactory) appContext.getBean("MySmbSession");
final SmbSession smbSession = smbSessionFactory.getSession();
final ByteArrayOutputStream output = new ByteArrayOutputStream();
try {
smbSession.read("mysharefolder\" + filename, output);
} catch (final IOException e) {
throw new IllegalStateException(e);
}
return output.toByteArray();
}
}
spring_integration.xml config file:
...
<!-- smb configuration for samba smb://[[[domain;]username[:password]#]server[:port]/[[share/[dir/]file]]][?[param=value[param2=value2[...]]] -->
<bean id="MySmbSession" class="org.springframework.integration.smb.session.SmbSessionFactory">
<property name="host" value="${samba.host}"/>
<property name="port" value="${samba.port}"/>
<property name="username" value="${samba.username}"/>
<property name="password" value="${samba.password}"/>
<property name="shareAndDir" value="${samba.shareAndDir}"/>
<property name="replaceFile" value="true"/>
</bean>
...
application.yml config file:
samba:
host : '${SAMBA_HOST:172.16.0.7}'
hostAlias : '${SAMBA_HOST_ALIAS:mywinsrv}'
port : '${SAMBA_PORT:445}'
username : '${SAMBA_USER_NAME:DMZ\myusername}'
password : '${SAMBA_USER_PASS:mypassword}'
shareAndDir : '${SAMBA_SHARE_AND_DIR:myDocs$\myproject\}'