FTP/FTPS Adapters custom trigger/poller - spring-integration

First, I have just started looking at Spring Integration today, so I have very little experience. I already have a basic scheduled ftp file parser setup using spring integration:
<int:channel id="ftpIn" />
<int-ftp:inbound-channel-adapter
channel="ftpIn"
session-factory="ftpClientFactory"
filename-pattern="*.xml"
local-directory="${TEMP_DIR}">
<int:poller fixed-rate="${ftp.polling.rate}" />
</int-ftp:inbound-channel-adapter>
<bean id="ftpClientFactory" class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="${ftp.host}" />
<property name="port" value="${ftp.port}" />
<property name="username" value="${ftp.username}" />
<property name="password" value="${ftp.password}" />
</bean>
<int:service-activator
input-channel="ftpIn"
method="handle"
ref="ftpInHandler" />
<bean id="ftpInHandler" class="thanks.for.looking.FtpInHandler" />
This works; however, I want to add additional functionality that checks (at a fixed-rate) if the system is ready before starting the scheduled (fixed-rate) ftp adapter. I am stuck on the best way to implement this. Any help or guidance is appreciated.
Best Regards,
Jared

<poller> has an option like <advice-chain>.
So you just need to write some custom Advice:
public class CheckSystemInterceptor implements MethodInterceptor {
Object invoke(MethodInvocation invocation) throws Throwable {
return mySystem.isOK() ? invocation.proceed() : null;
}
}
Configure it as a <bean> with your system checker and inject it into that <advice-chain>.
It will be invoked on each poll.

Related

JMS MQ Channels keep increasing and exhausted

I have configured queueconnection factory in the WebSphere admin console and using jndi lookup to use it. MQ Queue Connection Factories in WebSphere Application Server has settings. Connection pool Max size to 30 and session pool Max size to 20
These are than used in several jms:message-driven-channel-adapter or jms channels or jms:inbound-channel-adapter as part of various spring integration workflows that I have in my application.
Over a period we see that the connection count on MQ channel keeps increasing upto maximum allowed (about 1800).Once we bounce the server the count comes back to normal below 50.
Is there any setting missing ?
How can I be sure if the JMS session pools are being closed/released ?
Any help is appreciated
<jee:jndi-lookup id="queueConnectionFactory" jndi-name="$env{Queue.ConnectionFactory}" />
<si-jms:message-driven-channel-adapter
id="messageDrivenAdapter" channel="routingChannel"
container="messageListenerContainer" />
<bean id="messageListenerContainer"
class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="queueConnectionFactory" />
<property name="destination" ref="inQueue" />
<property name="transactionManager" ref="txManager" />
<property name="taskExecutor" ref="MQExecutor" />
</bean>
<si-jms:channel id="regChannel" queue="regQueue" connection-factory="queueConnectionFactory" transaction-manager="txManager" task-executor="regtaskExecutor" />
In my old days when we use Spring Integration on WebSphere we had a bean like this:
<bean id="connectionFactory" class="org.springframework.jms.connection.DelegatingConnectionFactory">
<property name="targetConnectionFactory">
<jee:jndi-lookup jndi-name="someConnectionFactory"/>
</property>
<property name="shouldStopConnections" value="true"/>
</bean>
Let's see if that can help you!

Spring Integration with Oracle AQ

I'm trying to learn Spring Integration and for this i would like to create an application like this:
From Oracle i send messages (on Oracle Queue), this message will be intercepted from a Java application (build with Spring Integration) and the application will send an email based on message received. The message will contain To: - Cc: and the text to send.
To make this kind of communication i've decided to use JMS (i think in Oracle this is made with Oracle AQ).
In the database i've already created the Queue and now i'm trying to create a simple applicationContext.xml to start this handshake.
Looking on the net i've found really few articles about this (Spring Integration + Oracle AQ) and i'm getting some error. The main error is this: java.lang.ClassNotFoundException: oracle.jms.AQjmsFactory
Right now this is my applicationContext.xml
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:int="http://www.springframework.org/schema/integration"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:orcl="http://www.springframework.org/schema/data/orcl"
xmlns:int-jms="http://www.springframework.org/schema/integration/jms"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/integration/jms http://www.springframework.org/schema/integration/jms/spring-integration-jms.xsd
http://www.springframework.org/schema/data/orcl http://www.springframework.org/schema/data/orcl/spring-data-orcl-1.0.xsd">
<int:channel id="inbound" />
<int:channel id="outbound" />
<bean id="simpleMessageListener" class="it.dirimo.SimpleMessageListener" />
<int-jms:inbound-channel-adapter
channel="inbound"
connection-factory="connectionFactory"
destination-name="Q1">
<int:poller fixed-delay="1000" />
</int-jms:inbound-channel-adapter>
<int:service-activator input-channel="inbound" output-channel="outbound" ref="simpleMessageListener" method="onMessage" />
<int-jms:outbound-channel-adapter id="jmsOut"
channel="outbound"
connection-factory="connectionFactory"
destination-name="sampleQueue" />
<int:poller id="poller" default="true" fixed-delay="10000" />
<orcl:aq-jms-connection-factory id="connectionFactory"
data-source="dataSource"
use-local-data-source-transaction="true"/>
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" lazy-init="false">
<property name="driverClassName" value="oracle.jdbc.driver.OracleDriver" />
<property name="url" value="jdbc:oracle:thin:#localhost:1521:ORCL" />
<property name="username" value="user" />
<property name="password" value="password" />
</bean>
</beans>
Maybe i'm using "old" technologies (for example i've seen for the first time this org.apache.commons.dbcp.BasicDataSource)
Unfortunally i'm so new about Spring Integration and i've seen for the first time Oracle Queue (i'm using Oracle for work but never used any kind of Queue).
Some advice of how to proceed will be apreciated :)
EDIT 1
To solve the problem about the AQjmsFactory need to include aqapi.jar
java.lang.ClassNotFoundException: oracle.jms.AQjmsFactory
This simply means you are missing the jar that contains that class from the classpath.
Oracle, typically, requires that you manually download their jars from them directly.

Spring Integration is not triggering int-http:outbound-gateway URL

I have the following spring integration configuration which starts from reading a file from drive, transforming it into java object and sending http GET request to a REST APT. The first 2 steps are working fine in my chain, but when it comes to int-http:outbound-gateway step, the URL is never triggered nor it is displaying any error message. The application remains running state without showing any error message. It never goes to kbbCvsReadResponseTransformer class where I can check the response received from this REST service. What could be the reason. I am using spring-integration-4.1.2
<int-file:inbound-channel-adapter id="kbbFileInbound"
channel="kbbInboundFileChannel"
directory="file:/ftpguest/kbb-gm-rem/data"
filename-pattern="GM_Remarketing_Pricing_Res_*.csv"
auto-startup="true"
prevent-duplicates="true">
<int:poller fixed-rate="5000"/>
</int-file:inbound-channel-adapter>
<int:chain input-channel="kbbInboundFileChannel" output-channel="kbbCvsReadRequest">
<int:transformer ref="kbbInputFileDataTransformer" />
</int:chain>
<int:chain input-channel="kbbCvsReadRequest" output-channel="cvsVehicleReadRequest">
<int:transformer ref="kbbCvsUpdateRequestTransformer" />
</int:chain>
<int:chain input-channel="cvsVehicleReadRequest" output-channel="cvsVehicleReadResponse">
<int:header-enricher>
<int:header name="Content-Type" expression="'application/json'" />
</int:header-enricher>
<int-http:outbound-gateway http-method="GET" expected-response-type="com.fasterxml.jackson.databind.JsonNode"
charset="UTF-8" request-factory="clientHttpRequestFactory" url="http://services.dev-sea.cobaltgroup.com/inventoryWebApp/rest/v1.0/vehicles/search?vin={vin};inventoryOwner={inventoryOwner}">
<int-http:uri-variable name="vin" expression="payload.getVin()"/>
<int-http:uri-variable name="inventoryOwner" expression="payload.getInventoryOwner()"/>
<int-http:request-handler-advice-chain>
<ref bean="retrier" />
</int-http:request-handler-advice-chain>
</int-http:outbound-gateway>
</int:chain>
<int:chain input-channel="cvsVehicleReadResponse">
<int:transformer ref="kbbCvsReadResponseTransformer" />
</int:chain>
<bean id="clientHttpRequestFactory"
class="org.springframework.http.client.HttpComponentsClientHttpRequestFactory">
<property name="httpClient" value="#{httpComponentsMessageSender.httpClient}" />
</bean>
<bean id="kbbInputFileDataTransformer" class="com.cdk.dmg.kbb.integration.transformer.KbbInputFileDataTransformer"/>
<bean id="kbbCvsUpdateRequestTransformer" class="com.cdk.dmg.kbb.integration.transformer.KbbCVSUpdateRequestTransformer" />
<bean id="kbbCvsReadResponseTransformer" class="com.cdk.dmg.kbb.integration.transformer.KbbCVSReadResponseTransformer" />
<!-- Need to use it from integration-context.xml, there are lot of issues with this file right now, so temporerly copied it here -->
<int:handler-retry-advice id="retrier" max-attempts="5" recovery-channel="errorChannel">
<int:exponential-back-off initial="1000" multiplier="5.0" maximum="60000" />
</int:handler-retry-advice>
We can't help you here without any logs confirmation.
Please, share DEBUG logs for the org.springframework.integration to have a picture how your messages are traveling.
From other side looks like you have some error there and just send it to the errorChannel according to the <int:handler-retry-advice> configuration.
And that's why you don't get any reply to the cvsVehicleReadResponse after that...

Spring Integration Kafka threading config

I'm using spring-integration-kafka 1.1.0 with the following config. I don't quite understand about the streams config. When I increase this, does Spring automatically spawn more threads to handle the messages? e.g. when I have streams=2, does the correlated transformer and service-activator all run in 2 threads? I feel like missing some thread-executor configurations, but not sure how. Any hint is appreciated. Thanks.
<int:poller default="true" fixed-delay="10"/>
<int:channel id="tag.track">
</int:channel>
<int-kafka:inbound-channel-adapter id="kafkaInboundChannelAdapterForTagTrack" kafka-consumer-context-ref="consumerContextForTagTrack" auto-startup="true" channel="tag.track">
</int-kafka:inbound-channel-adapter>
<int-kafka:consumer-context id="consumerContextForTagTrack"
consumer-timeout="${kafka.consumer.timeout}" zookeeper-connect="zookeeperConnect">
<int-kafka:consumer-configurations>
<int-kafka:consumer-configuration group-id="${kafka.consumer.group.track}" max-messages="200">
<int-kafka:topic id="tag.track" streams="2" />
</int-kafka:consumer-configuration>
</int-kafka:consumer-configurations>
</int-kafka:consumer-context>
<int:channel id="tag.track.transformed">
<int:interceptors>
<int:wire-tap channel="event.logging" />
</int:interceptors>
</int:channel>
<int:transformer id="kafkaMessageTransformerForTagTrack"
ref="kafkaMessageTransformer" input-channel="tag.track" method="transform"
output-channel="tag.track.transformed" />
<int:service-activator input-channel="tag.track.transformed" ref="tagTrackMessageHandler" method="handleTagMessage">
<int:request-handler-advice-chain>
<ref bean="userTagRetryAdvice" />
</int:request-handler-advice-chain>
</int:service-activator>
Tried message-driven-channel-adapter, but can't get it work, the following config doesn't pick up any message. Also tried the org.springframework.integration.kafka.listener.KafkaTopicOffsetManager , it complains Offset management topic cannot have more than one partition. Also, in this adapter, how to configure the consumer group?
Is there any detailed example on how to use the message-driven-channel-adapter? The instruction on the project page is pretty high level.
<int:channel id="tag.track">
<int:queue capacity="100"/>
</int:channel>
<bean id="kafkaConfiguration" class="org.springframework.integration.kafka.core.ZookeeperConfiguration">
<constructor-arg ref="zookeeperConnect"/>
</bean>
<bean id="connectionFactory" class="org.springframework.integration.kafka.core.DefaultConnectionFactory">
<constructor-arg ref="kafkaConfiguration"/>
</bean>
<bean id="decoder" class="org.springframework.integration.kafka.serializer.common.StringDecoder"/>
<int-kafka:message-driven-channel-adapter
id="adapter"
channel="tag.track"
connection-factory="connectionFactory"
key-decoder="decoder"
payload-decoder="decoder"
max-fetch="100"
topics="tag.track"
auto-startup="true"
/>
The streams property has nothing to do with Spring itself; it's simply passed to Kafka when invoking ConsumerConnector.createMessageStreams() (each topic/streams entry is passed in the map argument).
Refer to the kafka documentation.
EDIT:
When using the high-level consumer, the kafka inbound channel adapter is polled, so the threads on which the downstream integration flow runs are not related to the kafka client threads; they are managed in the poller configuration.
You could consider using the message-driven channel adapter instead.

Spring Integration Threads Parked at Service Activator

I am having issues with threads getting parked at my Service Activators that leads to files hanging in the SftpGatewayChannel when the pool is depleted. I think it is related to the SA's having a void return, which is correct because they are only incrementing metrics.
I was able to work around the issue by adding a default-reply-timeout to the SftpGateway, but this is not ideal since there is retry advice and I don't want the threads to timeout if there is a connection issue. I would like a solution that returns the threads to the pool after a successful upload and call to the "Success" Service Activator.
<task:executor id="Tasker" rejection-policy="CALLER_RUNS" pool-size="${MaxThreads}" />
<int:channel id="SftpGatewayChannel">
<int:dispatcher task-executor="Tasker" />
</int:channel>
<int:service-activator id="SegmentStart" input-channel="SftpGatewayChannel" ref="SftpGateway" />
<int:gateway id="SftpGateway" default-request-channel="SftpOutboundChannel" error-channel="ErrorChannel" />
<int:channel id="SftpOutboundChannel" datatype="java.lang.String,java.io.File,byte[]" />
<int-sftp:outbound-channel-adapter id="SftpOutboundAdapter"
session-factory="SftpCachingSessionFactory" channel="SftpOutboundChannel" charset="UTF-8" >
<int-sftp:request-handler-advice-chain>
<ref bean="exponentialRetryAdvice" />
<bean id="SuccessAdvice" class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice" >
<property name="successChannel" ref="SuccessChannel"/>
<property name="onSuccessExpression" value="true"/>
</bean>
</int-sftp:request-handler-advice-chain>
</int-sftp:outbound-channel-adapter>
<int:channel id="ErrorChannel">
<int:interceptors>
<int:wire-tap channel="FailureChannel" />
</int:interceptors>
</int:channel>
<int:channel id="AttemptChannel" />
<int:channel id="SuccessChannel" />
<int:channel id="FailureChannel" />
<int:service-activator id="AttemptMetrics" input-channel="AttemptChannel"
expression="T(MetricsCounter).addAttempt()" />
<int:service-activator id="SuccessMetrics" input-channel="SuccessChannel"
expression="T(MetricsCounter).addSuccesses(inputMessage.Headers.messages.size())" />
<int:service-activator id="FailureMetrics" input-channel="FailureChannel"
expression="T(MetricsCounter).addFailures(payload.getFailedMessage().Headers.messages.size())" />
Yes, gateways expect a reply by default. Instead of using the default RequestReplyExchanger you could use a service-interface method with a void return void process(Message<?> m).
Alternatively, as you have done, simply add default-reply-timeout="0" on your gateway and the thread will return immediately without waiting for a reply (that will never come).
... but this is not ideal ...
The reply timeout clock only starts when the thread returns to the gateway, so it will have no impact on the downstream flow.

Resources