So I think I need to extend the current redis-sink provided in spring-xd to write into a redis Capped list, rather than creating a new one but unfortunately it seems it gets worse as I will have to go deeper into spring-integration and further back into spring-data (spring-data-redis) because the whole redis-sink seems to be based on the generic pub/sub abstraction on redis - or is there some type of handler that can be defined once the message arrives to the channel handler?
In order to have the "effect of a capped list" when I push data redis, I need to execute both a redis "push" and then an "rtrim" as outlined here - http://redis.io/topics/data-types-intro. If I am to build a custom spring-integration / spring-data module. I believe I see support for the "ltrim" but not the"rtrim" operation here http://docs.spring.io/spring-data/redis/docs/1.7.0.RC1/api/
Any Advice on how/where to start or an easier approach would be appreciated.
Actually even Redis doesn't have such a RTRIM command. We don't need it because we reach the same behavior with the negative indexes for LTRIM:
start and end can also be negative numbers indicating offsets from the end of the list, where -1 is the last element of the list, -2 the penultimate element and so on.
I think you should use <redis:store-outbound-channel-adapter> and add something like this into its configuration:
<int-redis:request-handler-advice-chain>
<beans:bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<beans:property name="onSuccessExpression" value="#redisTemplate.boundListOps(${keyExpression}).trim(1, -1)"/>
</beans:bean>
</int-redis:request-handler-advice-chain>
To remove the oldest element in the Redis List.
Related
I have a spring-integration flow that starts with a file inbound-channel-adapter activated by a transactional poller (tx is handled by atomikos).
The text in the file is processed and the message goes down through the flow until it gets sent to one of the JMS queues (JMS outbound-channel-adapter).
In the middle, there are some database writes within a nested transaction.
The system is meant to run 24/7.
It happens that the single message flow, progressively slows down and when I investigated, I found that the stage that is responsable for the increasing delay is the read from filesystem.
Below, the first portion fo the integration flow:
<logging-channel-adapter id="logger" level="INFO"/>
<transaction-synchronization-factory id="sync-factory">
<after-commit expression="payload.delete()" channel="after-commit"/>
</transaction-synchronization-factory>
<service-activator input-channel="after-commit" output-channel="nullChannel" ref="tx-after-commit-service"/>
<!-- typeb inbound from filesystem -->
<file:inbound-channel-adapter id="typeb-file-inbound-adapter"
auto-startup="${fs.typeb.enabled:true}"
channel="typeb-inbound"
directory="${fs.typeb.directory.in}"
filename-pattern="${fs.typeb.filter.filenamePattern:*}"
prevent-duplicates="${fs.typeb.preventDuplicates:false}" >
<poller id="poller"
fixed-delay="${fs.typeb.polling.millis:1000}"
max-messages-per-poll="${fs.typeb.polling.numMessages:-1}">
<transactional synchronization-factory="sync-factory"/>
</poller>
</file:inbound-channel-adapter>
<channel id="typeb-inbound">
<interceptors>
<wire-tap channel="logger"/>
</interceptors>
</channel>
I read something about issues related to the prevent-duplicates option that stores a list of seen files, but that is not the case because I turned it off.
I don't think that it may be related to the filter (filename-pattern) because the expression I use in my config (*.RCV) is easy to apply and the input folder does not contain a lot of files (less than 100) at the same time.
Still, there is something that gradually makes the read from filesystem slower and slower over time, from a few millis to over 3 seconds within a few days of up-time.
Any hints?
You should remove, or move files after they have been processed; otherwise the whole directory has to be rescanned.
In newer versions, you can use a WatchServiceDirectoryScanner which is more efficient.
But it's still best practice to clean up old files.
Finally I got the solution.
The issue was related to the specific version of Spring I was using (4.3.4) that is affected by a bug I had not discovered yet.
The problem is something about DefaultConversionService and the use of converterCache (look at this for more details https://jira.spring.io/browse/SPR-14929).
Upgrading to a more recent version has resolved.
I have a transacted Camel route with a number of processors
from(Constant.RouteA)
.transacted()
.process(processor1)
.process(processor2)
.process(processor3)
.wireTap(Constant.RouteB)
.wireTap(Constant.RouteC)
.end()
My problem is that I don't want the final part of the route (the wiretaps) to be part of the transaction i.e. I want them to be executed once processor3 has finished and the transaction committed.
Initially I looked at using onCompletion() but it doesn't seem to work together with transacted().
So I found another way which requires using policy() to limit the transaction scope i.e.
from(Constant.RouteA)
.policy("PROPAGATION_REQUIRED")
.process(processor1)
.process(processor2)
.process(processor3)
.end()
.wireTap(Constant.RouteB)
.wireTap(Constant.RouteC)
.end()
The problem is that this solution requires to define the SpringTransactionPolicy in the Spring configuration, but the software I'm working on doesn't use Spring. Transactions are managed by Bitronix and everything works just by using the transacted() method, which as far as I can tell doesn't allow you to limit the scope of a transaction.
Is there a simple way to achieve my goal? Hopefully without bringing Spring into the picture. Thank you!
Try to create two routes. For example:
from(direct:startRoute)
.to(Constant.RouteA)
.wireTap(Constant.RouteB)
.wireTap(Constant.RouteC);
from(Constant.RouteA)
.transacted()
.process(processor1)
.process(processor2)
.process(processor3);
Once route "Constant.RouteA" is finished all changes will committed.
I have an integration that starts with a standard database query and it update the state in the database to indicate that the integration has worked fine. It works.
But if the data cannot be processed and an exception is raised, the state is not updated as intended, but I would like to update my database row with a 'KO' state so the same row won't fail over and over.
Is there a way to provide a second query to execute when integration fails?
It seems to me that it is very standard way of doing things but I couldn't find a simple way to do it. I could catch exception in every step of the integration and update the database, but it creates coupling, so there should be another solution.
I tried a lot of Google search but I could not find anything, but I'm pretty sure the answer is out there.
Just in case, there is my xml configuration to do the database query (nothing fancy) :
<int-jdbc:inbound-channel-adapter auto-startup="true" data-source="datasource"
query="select * FROM MyTable where STATE='ToProcess')"
channel="stuffTransformerChannel"
update="UPDATE MyTable SET STATE='OK' where id in (:id)"
row-mapper="myRowMapper" max-rows-per-poll="1">
<int:poller fixed-rate="1000">
<int:transactional />
</int:poller>
</int-jdbc:inbound-channel-adapter>
I'm using spring-integration version 4.0.0.RELEASE
Since you are within Transaction, it is normal behaviuor, that rallback is caused and your DB returns to the clear state.
And it is classical pattern to get the deal with data on application purpose in that case, not from some built-in tool. That's why we don't provide any on-error-update, because it can't be a use-case for evrything.
As soon as you are going to update the row anyway you should do something on onRallback event and do it within new transaction, though. However it should be in the same Thread, to prevent fetching the same row from the second polling task.
For this purpose we provide a transaction-synchronization-factory feature:
<int-jdbc:inbound-channel-adapter max-rows-per-poll="1">
<int:poller fixed-rate="1000" max-messages-per-poll="1">
<int:transactional synchronization-factory="syncFactory"/>
</int:poller>
</int-jdbc:inbound-channel-adapter>
<int:transaction-synchronization-factory id="syncFactory">
<int:after-rollback channel="stuffErrorChannel"/>
</int:transaction-synchronization-factory>
<int-jdbc:outbound-channel-adapter
query="UPDATE MyTable SET STATE='KO' where id in (:payload[id])"
channel="stuffErrorChannel">
<int-jdbc:request-handler-advice-chain>
<tx:advice id="requiresNewTx">
<tx:attributes>
<tx:method name="handle*Message" propagation="REQUIRES_NEW"/>
</tx:attributes>
</tx:advice>
</int-jdbc:request-handler-advice-chain>
</int-jdbc:outbound-channel-adapter>
Hope I am clear
I understand how aggregating based on size works but I also want to make the release strategy depend on another step in the pipeline to be still running. The idea is that i move files to a certain dir "source", aggregate enough files and then move from "source" to "stage" and then process the staged files. While this process is running I dont want to put more files in stage but I do want to continue to add more files to source folder (that part is handled by using a dispatcher channel connected with file inbound adapter before the aggregator)
<int:aggregator id="filesBuffered"
input-channel="sourceFilesProcessed"
output-channel="stagedFiles"
release-strategy-expression="size() == 10"
correlation-strategy-expression="'mes-group'"
expire-groups-upon-completion="true"
/>
<int:channel id="stagedFiles" />
<int:service-activator input-channel="stagedFiles"
output-channel="readyForMes"
ref="moveToStage"
method="move" />
so as you can see I dont want to release the aggregated messages if an existing instance of moveToStage service activator is running.
I thought about making the stagedFiles channel a queue channel but that doesnt seems right because I do want the files to be passed to moveToStage as a Collection not a single file which I am assuming by making stagedFiles a queue channel it will send a single file. Instead I want to get to a threshold e.g. 10 files, pass those to stagedFiles which allows the moveToStage to process those files but then until this step is done I want the aggregator to continue to aggregate files and then release all aggregated files.
Thanks
I suggest you to have some flag as a AtomicBoolean bean and use it from your moveToStage#move and check it's state from:
release-strategy-expression="size() >= 10 and #stagingFlag.get()"
Our current service has 7 operations. when writing an outbound xquery "local entry" in wso2, we're trying to retrieve the name of the current operation being executed (how can this be so difficult?).
After reading what i could find in wso2's documentation. it appears as if we need to set up both a Property and an Xquery mediator. supposedly the property mediator would pull the value doing something like get-property('OperationName') and then this would be referenced and passed thru the Xquery mediator.
The other idea was that we needed to define it as a variable in the "Local Registry entry definitions" and than it would be around at all parts of the sequence.
I've tried for 2 days but haven't quite got it.
Please tell me what I'm missing...
Did you try the following xquery sample[1]? I modified the query mediator to get the operation name as follows.
<variable xmlns:ax21="http://services.samples/xsd" xmlns:m0="http://services.samples" name="code" expression="get-property('OperationName')" type="STRING" />
this worked fine. I could see the getQuote in the response message.
[1] http://wso2.org/project/esb/java/4.0.2/docs/samples/advanced_mediation_samples.html#Sample390