I am using spring integration to read file from s3 however this works but my local directory is getting full I want to delete files from local directory after files are processed from s3?
<bean id="credentials" class="org.springframework.integration.aws.core.BasicAWSCredentials">
<property name="accessKey" value="${accessKey}"/>
<property name="secretKey" value="${secretKey}"/>
</bean>
<bean id="clientConfiguration" class="com.amazonaws.ClientConfiguration">
<property name="proxyHost" value="${proxyHost}"/>
<property name="proxyPort" value="${proxyPort}"/>
<property name="preemptiveBasicProxyAuth" value="false"/>
</bean>
<bean id="s3Operations" class="org.springframework.integration.aws.s3.core.CustomC1AmazonS3Operations">
<constructor-arg index="0" ref="credentials"/>
<constructor-arg index="1" ref="clientConfiguration"/>
<property name="awsEndpoint" value="s3.amazonaws.com"/>
<property name="temporaryDirectory" value="${temporaryDirectory}"/>
<property name="awsSecurityKey" value="${awsSecurityKey}"/>
</bean>
<!-- aws-endpoint="https://s3.amazonaws.com" -->
<int-aws:s3-inbound-channel-adapter aws-endpoint="s3.amazonaws.com"
bucket="${bucket}"
s3-operations="s3Operations"
credentials-ref="credentials"
file-name-wildcard="${fileNameWildcard}"
remote-directory="${remoteDirectory}"
channel="splitChannel"
local-directory="${localDirectory}"
accept-sub-folders="false"
delete-source-files="true"
archive-bucket="${archiveBucket}"
archive-directory="${archiveDirectory}">
</int-aws:s3-inbound-channel-adapter>
Looks like you are looking for ExpressionEvaluatingRequestHandlerAdvice.
Please, find Retry and More. There is something like <property name="onSuccessExpression" value="payload.delete()" /> in the expression-advice-context.xml config to take care about the local file after the proper finish of the process.
Related
I am using Spring Session with Redis using Azure Redis.
Things are working fine with the non-SSL port 6379. However with the SSL port 6380, I get this error:
ERROR (org.springframework.data.redis.listener.RedisMessageListenerContainer:651) || - Connection failure occurred. Restarting subscription task after 5000 ms
That’s it. No further information.
Here is my Redis configuration:
<bean id="redisPassword" class="org.springframework.data.redis.connection.RedisPassword">
<constructor-arg index="0" value="${spring.redis.password}"/>
</bean>
<bean id="redisStandaloneConfiguration"
class="org.springframework.data.redis.connection.RedisStandaloneConfiguration">
<property name="hostName" value="${spring.redis.host}"/>
<property name="port" value="${spring.redis.port}"/>
<property name="password" ref="redisPassword"/>
</bean>
<util:constant id="configureRedisAction"
static-field="org.springframework.session.data.redis.config.ConfigureRedisAction.NO_OP"/>
<bean id="lettuceClientConfiguration"
class="org.springframework.data.redis.connection.lettuce.DefaultLettuceClientConfiguration"
factory-method="defaultConfiguration">
</bean>
<context:annotation-config/>
<bean class="org.springframework.session.data.redis.config.annotation.web.http.RedisHttpSessionConfiguration"
p:configureRedisAction-ref="configureRedisAction"/>
<bean class="org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory">
<constructor-arg index="0" ref="redisStandaloneConfiguration"/>
<constructor-arg index="1" ref="lettuceClientConfiguration"/>
</bean>
What is causing the connection failure?
<util:constant id="configureRedisAction"
static-field="org.springframework.session.data.redis.config.ConfigureRedisAction.NO_OP"/>
<context:annotation-config/>
<bean class="org.springframework.session.data.redis.config.annotation.web.http.RedisHttpSessionConfiguration"
p:configureRedisAction-ref="configureRedisAction"/>
<bean id="jedisPoolConfig" class="redis.clients.jedis.JedisPoolConfig">
<property name="maxTotal" value="200" />
<property name="maxIdle" value="50" />
<property name="maxWaitMillis" value="30000" />
<property name="minIdle" value="10"/>
</bean>
<bean class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory">
<property name="hostName" value="${spring.redis.host}" />
<property name="port" value="${spring.redis.port}" />
<property name="poolConfig" ref="jedisPoolConfig" />
<property name="usePool" value="true" />
<property name="useSsl" value="${spring.redis.ssl}"/>
<property name="password" value="${spring.redis.password}"/>
</bean>
Polling dynamically
I am using poller component from integration to poll files from s3.The fixed delay is 15 min and max message rate is 1 .The reason i did this was in down stream messages in xd were clogging since I am using http.Now this is file for 100k records file but when file size is small i still wait 15 min though i can process fast.Now is there any way to dynamically set delay depending in size of file.Because we don't know which files will be polled also to know it?Depending on file size i will pick up or number of records can we change dynamically the fixed delay or fixed rate?
<int:poller fixed-delay="${fixedDelay}" default="true" max-messages-per-poll="${maxMessageRate}">
<int:advice-chain>
<ref bean="pollAdvise"/>
</int:advice-chain>
</int:poller>
<bean id="pollAdvise" class="org.springframework.integration.scheduling.PollSkipAdvice">
<constructor-arg ref="healthCheckStrategy"/>
</bean>
<bean id="healthCheckStrategy" class="test.ServiceHealthCheckPollSkipStrategy">
<property name="url" value="${url}"/>
<property name="doHealthCheck" value="${doHealthCheck}"/>
</bean>
<bean id="credentials" class="org.springframework.integration.aws.core.BasicAWSCredentials">
<property name="accessKey" value="${accessKey}"/>
<property name="secretKey" value="${secretKey}"/>
</bean>
<bean id="clientConfiguration" class="com.amazonaws.ClientConfiguration">
<property name="proxyHost" value="${proxyHost}"/>
<property name="proxyPort" value="${proxyPort}"/>
<property name="preemptiveBasicProxyAuth" value="false"/>
</bean>
<bean id="s3Operations" class="org.springframework.integration.aws.s3.core.CustomC1AmazonS3Operations">
<constructor-arg index="0" ref="credentials"/>
<constructor-arg index="1" ref="clientConfiguration"/>
<property name="awsEndpoint" value="s3.amazonaws.com"/>
<property name="temporaryDirectory" value="${temporaryDirectory}"/>
<property name="awsSecurityKey" value="${awsSecurityKey}"/>
</bean>
<!-- aws-endpoint="https://s3.amazonaws.com" -->
<int-aws:s3-inbound-channel-adapter aws-endpoint="s3.amazonaws.com"
bucket="${bucket}"
s3-operations="s3Operations"
credentials-ref="credentials"
file-name-wildcard="${fileNameWildcard}"
remote-directory="${remoteDirectory}"
channel="splitChannel"
local-directory="${localDirectory}"
accept-sub-folders="false"
delete-source-files="true"
archive-bucket="${archiveBucket}"
archive-directory="${archiveDirectory}">
</int-aws:s3-inbound-channel-adapter>
<int-file:splitter id="s3splitter" input-channel="splitChannel" output-channel="bridge" markers="false" charset="UTF-8">
<int-file:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="onSuccessExpression" value="payload.delete()"/>
</bean>
</int-file:request-handler-advice-chain>
</int-file:splitter>
Starting with Spring Integration 4.2 the AbstractMessageSourceAdvice has been introduced:
This method is called after the receive() method; again, you can reconfigure the source, or take any action perhaps depending on the result (which can be null if there was no message created by the source). You can even return a different message!
Starting with version 4.3 we introduce CompoundTriggerAdvice: http://docs.spring.io/spring-integration/docs/4.3.0.BUILD-SNAPSHOT/reference/html/messaging-channels-section.html#_compoundtriggeradvice
Which you can use for your use-case based on the payload size.
I have a spring xd source module which pulls file from s3 and splits line by line.I have my spring config as below.But I have 3 container and 1 admin server.Now I see duplicate message being processed by each container as each of them is downloading there own copy.
I can solve with making source s3 module deployment count as 1 but my processing of message is getting slow.?Any inputs to solve this?
<int:poller fixed-delay="${fixedDelay}" default="true">
<int:advice-chain>
<ref bean="pollAdvise"/>
</int:advice-chain>
</int:poller>
<bean id="pollAdvise"
</bean>
<bean id="credentials" class="org.springframework.integration.aws.core.BasicAWSCredentials">
<property name="accessKey" value="#{encryptedDatum.decryptBase64Encoded('${accessKey}')}"/>
<property name="secretKey" value="${secretKey}"/>
</bean>
<bean id="clientConfiguration" class="com.amazonaws.ClientConfiguration">
<property name="proxyHost" value="${proxyHost}"/>
<property name="proxyPort" value="${proxyPort}"/>
<property name="preemptiveBasicProxyAuth" value="false"/>
</bean>
<bean id="s3Operations" class="org.springframework.integration.aws.s3.core.CustomC1AmazonS3Operations">
<constructor-arg index="0" ref="credentials"/>
<constructor-arg index="1" ref="clientConfiguration"/>
<property name="awsEndpoint" value="s3.amazonaws.com"/>
<property name="temporaryDirectory" value="${temporaryDirectory}"/>
<property name="awsSecurityKey" value="${awsSecurityKey}"/>
</bean>
<bean id="encryptedDatum" class="abc"/>
<!-- aws-endpoint="https://s3.amazonaws.com" -->
<int-aws:s3-inbound-channel-adapter aws-endpoint="s3.amazonaws.com"
bucket="${bucket}"
s3-operations="s3Operations"
credentials-ref="credentials"
file-name-wildcard="${fileNameWildcard}"
remote-directory="${remoteDirectory}"
channel="splitChannel"
local-directory="${localDirectory}"
accept-sub-folders="false"
delete-source-files="true"
archive-bucket="${archiveBucket}"
archive-directory="${archiveDirectory}">
</int-aws:s3-inbound-channel-adapter>
<int-file:splitter input-channel="splitChannel" output-channel="output" markers="false" charset="UTF-8">
<int-file:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="onSuccessExpression" value="payload.delete()"/>
</bean>
</int-file:request-handler-advice-chain>
</int-file:splitter>
<int:channel id="output"/>
[Updated]
I added the idempotency as suggested by you with a metadata store.But since my xd is running in 3 container cluster with rabbit will simple metadatastore work?I think I should use reds/mongo metadata source.If I use mongo/redis metadatastore howcan i evict/remove the messages because messages will pile up over time?
<int:idempotent-receiver id="expressionInterceptor" endpoint="output"
metadata-store="store"
discard-channel="nullChannel"
throw-exception-on-rejection="false"
key-expression="payload"/>
<bean id="store" class="org.springframework.integration.metadata.SimpleMetadataStore"/>
I can suggest you to take a look to the Idempotent Receiver.
With that you can use shared MetadataStore and don't accept duplicate files.
The <idempotent-receiver> should be configured for that your <int-file:splitter>. And yes: with the discard logic to avoid duplicate message.
UPDATE
.But since my xd is running in 3 container cluster with rabbit will simple metadatastore work?
That doesn't matter because you start the stream from the S3 MessageSource, so you should filter files already there. Therefore you need external shared MetadataStore.
.If I use mongo/redis metadatastore howcan i evict/remove the messages because messages will pile up over time?
That's correct. It is a side affect of the Idempotent Receiver logic. Not sure how it is a problem for you if you use a DataBase...
You can clean the collection/keys by some periodic task. Maybe once a week...
I am pulling files from s3 and processing them using spring xd. I have one processor http client component where i do some RESTful request .Now the problem with this approach is if my webservice is down the files get accumulated in rabbit mq transport .Hence before pulling a individual file from s3 I want to do a health check on my rest service.How can I tackle this my configuration file looks something like this.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:int-aws="http://www.springframework.org/schema/integration/aws"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/aws http://www.springframework.org/schema/integration/aws/spring-integration-aws-1.0.xsd">
<int:poller fixed-delay="${fixed-delay}" default="true"/>
<bean id="credentials" class="org.springframework.integration.aws.core.BasicAWSCredentials">
<property name="accessKey" value="${accessKey}"/>
<property name="secretKey" value="${secretKey}"/>
</bean>
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location">
<value>dms-aws-s3-nonprod.properties</value>
</property>
</bean>
<bean id="clientConfiguration" class="com.amazonaws.ClientConfiguration">
<property name="proxyHost" value="${proxyHost}"/>
<property name="proxyPort" value="${proxyPort}"/>
<property name="preemptiveBasicProxyAuth" value="false"/>
</bean>
<bean id="s3Operations" class="org.springframework.integration.aws.s3.core.CustomC1AmazonS3Operations">
<constructor-arg index="0" ref="credentials"/>
<constructor-arg index="1" ref="clientConfiguration"/>
<property name="awsEndpoint" value="s3.amazonaws.com"/>
<property name="temporaryDirectory" value="${temporaryDirectory}"/>
<property name="awsSecurityKey" value="${awsSecurityKey}"/>
</bean>
<!-- aws-endpoint="https://s3.amazonaws.com" -->
<int-aws:s3-inbound-channel-adapter aws-endpoint="s3.amazonaws.com"
bucket="${bucket}"
s3-operations="s3Operations"
credentials-ref="credentials"
file-name-wildcard="${file-name-wildcard}"
remote-directory="${remote-directory}"
channel="splitChannel"
local-directory="${local-directory}"
accept-sub-folders="false"
delete-source-files="true"
archive-bucket="${archive-bucket}"
archive-directory="${archive-directory}">
</int-aws:s3-inbound-channel-adapter>
int-file:splitter input-channel="splitChannel" output-channel="output" markers="true"/>
<int:channel id="output"/>
My stream defination
xd-shell>stream create feedTest16 --definition "aws-s3-source |processor-http-client| log" --deploy
Starting with Spring Integration 4.1, the PollSkipAdvice has been introduced.
Implement your own ServiceHealthCheckPollSkipStrategy and inject it into the <advice-chain> of the <poller> for your <int-aws:s3-inbound-channel-adapter> and you're good with the requirement!
Only one issue is there that your s3-source is tied with the target service for the http-client...
In my app the message comes to the inque and then is sent to the output queue. We do this thru spring integration. My requirement is if there is a problem in connecting to the output queue then it shud try reconnecting 3 times with a delay of 30 secs and finally if it fails then log the exception. Can you please help on how to achieve this ? My config file -
<bean id="mqQcfParent" class="com.ibm.mq.jms.MQQueueConnectionFactory">
<property name="transportType">
<util:constant static-field="com.ibm.mq.jms.JMSC.MQJMS_TP_CLIENT_MQ_TCPIP"/>
</property>
<property name="hostName" value="${mq.out.hostname}"/>
<property name="channel" value="${mq.out.channel}"/>
<property name="port" value="${mq.out.port}"/>
</bean>
<bean id="remoteConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="mqQcfParent"/>
<property name="sessionCacheSize" value="${mq.out.cacheSize}"/>
<property name="cacheProducers" value="true"/>
<property name="cacheConsumers" value="true"/>
</bean>
<bean id="inQueue" class="com.ibm.mq.jms.MQQueue">
<property name="baseQueueName" value="${mq.in.queue}"/>
</bean>
<bean id="aircraftAssignQueue" class="com.ibm.mq.jms.MQQueue">
<property name="baseQueueName" value="${mq.out.aircraftAssignQueue}"/>
</bean>
</bean>
<bean id="failureQueue" class="com.ibm.mq.jms.MQQueue">
<property name="baseQueueName" value="${mq.out.failureQueue}"/>
</bean>
<bean id="messageListenerContainerParent" class="org.springframework.jms.listener.DefaultMessageListenerContainer" abstract="true">
<property name="destination" ref="inQueue"/>
<property name="sessionTransacted" value="true"/>
<property name="maxConcurrentConsumers" value="${mq.in.max.consumer}"/>
<property name="concurrentConsumers" value="${mq.in.min.consumer}"/>
<property name="receiveTimeout" value="5000"/>
<property name="recoveryInterval" value="60000"/>
<property name="autoStartup" value="true"/>
</bean>
<bean id="messageListenerContainerCDC" parent="messageListenerContainerParent">
<property name="connectionFactory">
<bean parent="remoteConnectionFactory">
<property name="targetConnectionFactory">
<bean parent="mqQcfParent">
<property name="hostName" value="${mq.in.cdc.hostname}"/>
<property name="channel" value="${mq.in.cdc.channel}"/>
<property name="port" value="${mq.in.cdc.port}"/>
</bean>
</property>
</bean>
</property>
</bean>
<bean id="messageListenerContainerPDC" parent="messageListenerContainerParent">
<property name="connectionFactory">
<bean parent="remoteConnectionFactory">
<property name="targetConnectionFactory">
<bean parent="mqQcfParent">
<property name="hostName" value="${mq.in.pdc.hostname}"/>
<property name="channel" value="${mq.in.pdc.channel}"/>
<property name="port" value="${mq.in.pdc.port}"/>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
<context:property-placeholder location="config/application.properties"/>
<import resource="jms-listener-container-config.xml"/>
<!-- Get Input Messages -->
<int-jms:message-driven-channel-adapter id="msgInCDC" channel="toRoute" container="messageListenerContainerCDC" error-channel="errorChannel" acknowledge="transacted"/>
<int-jms:message-driven-channel-adapter id="msgInPDC" channel="toRoute" container="messageListenerContainerPDC" error-channel="errorChannel" acknowledge="transacted"/>
<!-- Route Messages Depending on Root Element -->
<int:channel id="toAircraftAssign"/>
<int:channel id="toDiversionChange"/>
<int:channel id="toFlightCreate"/>
<int:channel id="toFlightPlanRelease"/>
<int:channel id="toGateChange"/>
<int:channel id="toPositionReport"/>
<int:channel id="toScheduleChange"/>
<int-xml:xpath-router id="flightUpdateRouter" input-channel="toRoute" default-output-channel="errorChannel" evaluate-as-string="true">
<int-xml:xpath-expression expression="name(/*)"/>
<int-xml:mapping value="AircraftAssignment" channel="toAircraftAssign"/>
<int-xml:mapping value="DiversionChangesUpdate" channel="toDiversionChange"/>
<int-xml:mapping value="CreateFlight" channel="toFlightCreate"/>
<int-xml:mapping value="FlightPlanRelease" channel="toFlightPlanRelease"/>
<int-xml:mapping value="GateChange" channel="toGateChange"/>
<int-xml:mapping value="PositionReportUpdate" channel="toPositionReport"/>
<int-xml:mapping value="ScheduleChangesUpdate" channel="toScheduleChange"/>
</int-xml:xpath-router>
<int-jms:outbound-channel-adapter id="aircraftAssignMsgOut" channel="toAircraftAssign" connection-factory="remoteConnectionFactory" destination="aircraftAssignQueue"/>
<int-jms:outbound-channel-adapter id="diversionChangeMsgOut" channel="toDiversionChange" connection-factory="remoteConnectionFactory" destination="diversionChangeQueue"/>
<int-jms:outbound-channel-adapter id="flightCreateMsgOut" channel="toFlightCreate" connection-factory="remoteConnectionFactory" destination="flightCreateQueue"/>
<int-jms:outbound-channel-adapter id="flightPlanReleaseMsgOut" channel="toFlightPlanRelease" connection-factory="remoteConnectionFactory" destination="flightPlanReleaseQueue"/>
<int-jms:outbound-channel-adapter id="gateChangeMsgOut" channel="toGateChange" connection-factory="remoteConnectionFactory" destination="gateChangeQueue"/>
<int-jms:outbound-channel-adapter id="positionReportMsgOut" channel="toPositionReport" connection-factory="remoteConnectionFactory" destination="positionReportQueue"/>
<int-jms:outbound-channel-adapter id="scheduleChangeMsgOut" channel="toScheduleChange" connection-factory="remoteConnectionFactory" destination="scheduleChangeQueue"/>
<!-- Error Handling -->
<int-jms:outbound-channel-adapter id="errMsgOut" channel="errorChannel" connection-factory="remoteConnectionFactory" destination="failureQueue"/>
<!-- Logger -->
<int:wire-tap pattern="to*" order="7" channel="wireTapChannel"/>
<int:logging-channel-adapter id="wireTapChannel" level="debug" logger-name="WIRETAP"/>
<!-- Logger -->
<!--
<int:wire-tap pattern="to*" order="0" channel="loggerChannel"/>
-->
<!--
<int:logging-channel-adapter id="loggerChannel" level="DEBUG" expression="'YYYY'"/>
-->
</beans>
You don't show any Spring Integration configuration but, presuming you are using a JMS outbound channel adapter, you an add a retry advice with an appropriately configured SimplyRetryPolicy.
However, if the same broker is being used for the inbound queue too, that session will be broken and the message redelivered anyway; so you might be better off setting the retry policy in the broker.