We have clustered environment with 2 nodes in Oracle Weblogic 10.3.6 server and it is Round-Robin.
I have service which goes and gets the message from a external system and puts them in the Database (Oracle DB).
I am using a jdbc-inbound-adapter to convert these messages and pass it to the channels.
And to have a message processed only once. I am planning to have a column(NODE_NAME) in the DB-table. When the first service which gets the message from the external system also updates the column with the NODE_NAME (weblogic.Name). In the SELECT query of jdbc-inbound-adapter if I specify the NODE_NAME then the messages would be processed only once.
i.e. If the Service1(of Node1) saves the message in DB then inbound-adapter1 (of node1) passes the message to channel.
Example:
<si-jdbc:inbound-channel-adapter id="jdbcInboundAdapter"
channel="queueChannel" data-source="myDataSource"
auto-startup="true"
query="SELECT * FROM STAGE_TABLE WHERE STATUS='WAITING' and NODE_NAME = '${weblogic.Name}'"
update="UPDATE STAGE_TABLE SET STATUS='IN_PROGRESS' WHERE ID IN (:Id)"
max-rows-per-poll="100" row-mapper="rowMapper"
update-per-row="true">
<si:poller fixed-rate="5000">
<si:advice-chain>
<ref bean="txAdvice"/>
<ref bean="inboundAdapterConfiguration"/>
</si:advice-chain>
</si:poller>
</si-jdbc:inbound-channel-adapter>
Is this a good design?
By second approach: using the below Select SQL in the jdbc-inbound-adapter but I am guessing this would fail as I am using Oracle Database.
SELECT * FROM TABLE WHERE STATUS='WAITING' FOR UPDATE SKIP LOCKED
It would be great if some one could point me in the right direction.
Actually, FOR UPDATE SKIP LOCKED is exactly Oracle's feature - https://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:2060739900346201280
If you are in doubt as, here is a code from Spring Integration: https://github.com/spring-projects/spring-integration/blob/master/spring-integration-jdbc/src/main/java/org/springframework/integration/jdbc/store/channel/OracleChannelMessageStoreQueryProvider.java#L39
Related
How can I inject a sql to query property in int-jdbc:outbound-gateway?
Background:
I receive an amqp message with a where clause to query a table and do some logic. Where clause could be anything like state in ('ca','ma) or zipcode = '01760'
Is there an example on using int-jdbc:outbound-gateway passing a query that can change based on the received message?
For example:
we receive amqp message:
1: {"whereClause":"State in ('ca','ma')"}
2: {"whereClause":"id = 1"}
How can I inject to query prop in int-jdbc:outbound-gateway as below?
query="SELECT id FROM account where State in ('ca','ma')"
query="SELECT id FROM account where id = 1"
No, you can't do that with out-of-the-box Spring Integration JDBC components.
The query property is final and can't be changed at runtime.
Consider to use <service-activator> with JdbcTemplate.query() direct usage.
I want to write a batch which reads website access log files(csv file)from a path every day and do some analysis using spring integration.
this is the simplified version of the input csv file.
srcIp1,t1,path1
srcIp2,t2,path2
srcIp1,t3,path2
srcIp1,t4,path1
The access number per source ip and path is to be calculated after some filtering logic.
I made a input channel whose payload is the parsed log line,and a filter is applied,and finally an aggregator to calculate the final result.
The problem is what should be the right group release stragety,the default release stragety(SequenceSizeReleaseStrategy) does not work.
Also any of other spring integraion out of box release
strategies(ExpressionEvaluatingReleaseStrategy,
MessageCountReleaseStrategy, MethodInvokingReleaseStrategy,
SequenceSizeReleaseStrategy, TimeoutCountSequenceSizeReleaseStrategy)
does not seem to fit my needs.
Or Spring integration assumed that a channel carries a message stream where there is no concept of "ending of message" and is not suitalbe for my problem here ?
You can write a custom ReleaseStrategy if you have some way to tell when the group is complete. It is consulted each time a message is added to the group.
Or, you can use a group-timeout to release a partial group after some time when no messages arrived.
I have an integration that starts with a standard database query and it update the state in the database to indicate that the integration has worked fine. It works.
But if the data cannot be processed and an exception is raised, the state is not updated as intended, but I would like to update my database row with a 'KO' state so the same row won't fail over and over.
Is there a way to provide a second query to execute when integration fails?
It seems to me that it is very standard way of doing things but I couldn't find a simple way to do it. I could catch exception in every step of the integration and update the database, but it creates coupling, so there should be another solution.
I tried a lot of Google search but I could not find anything, but I'm pretty sure the answer is out there.
Just in case, there is my xml configuration to do the database query (nothing fancy) :
<int-jdbc:inbound-channel-adapter auto-startup="true" data-source="datasource"
query="select * FROM MyTable where STATE='ToProcess')"
channel="stuffTransformerChannel"
update="UPDATE MyTable SET STATE='OK' where id in (:id)"
row-mapper="myRowMapper" max-rows-per-poll="1">
<int:poller fixed-rate="1000">
<int:transactional />
</int:poller>
</int-jdbc:inbound-channel-adapter>
I'm using spring-integration version 4.0.0.RELEASE
Since you are within Transaction, it is normal behaviuor, that rallback is caused and your DB returns to the clear state.
And it is classical pattern to get the deal with data on application purpose in that case, not from some built-in tool. That's why we don't provide any on-error-update, because it can't be a use-case for evrything.
As soon as you are going to update the row anyway you should do something on onRallback event and do it within new transaction, though. However it should be in the same Thread, to prevent fetching the same row from the second polling task.
For this purpose we provide a transaction-synchronization-factory feature:
<int-jdbc:inbound-channel-adapter max-rows-per-poll="1">
<int:poller fixed-rate="1000" max-messages-per-poll="1">
<int:transactional synchronization-factory="syncFactory"/>
</int:poller>
</int-jdbc:inbound-channel-adapter>
<int:transaction-synchronization-factory id="syncFactory">
<int:after-rollback channel="stuffErrorChannel"/>
</int:transaction-synchronization-factory>
<int-jdbc:outbound-channel-adapter
query="UPDATE MyTable SET STATE='KO' where id in (:payload[id])"
channel="stuffErrorChannel">
<int-jdbc:request-handler-advice-chain>
<tx:advice id="requiresNewTx">
<tx:attributes>
<tx:method name="handle*Message" propagation="REQUIRES_NEW"/>
</tx:attributes>
</tx:advice>
</int-jdbc:request-handler-advice-chain>
</int-jdbc:outbound-channel-adapter>
Hope I am clear
I have the below configuration for my jdbc-inbound-adapter:
<si-jdbc:inbound-channel-adapter id="jdbcInboundAdapter"
channel="queueChannel" data-source="myDataSource"
auto-startup="true"
query="SELECT * FROM STAGE_TABLE WHERE STATUS='WAITING' FOR UPDATE SKIP LOCKED"
update="UPDATE STAGE_TABLE SET STATUS='IN_PROGRESS' WHERE ID IN (:Id)"
max-rows-per-poll="100" row-mapper="rowMapper"
update-per-row="true">
<si:poller fixed-rate="5000">
<si:advice-chain>
<ref bean="txAdvice"/>
<ref bean="inboundAdapterConfiguration"/>
</si:advice-chain>
</si:poller>
</si-jdbc:inbound-channel-adapter>
<tx:advice id="txAdvice">
<tx:attributes>
<tx:method name="get*" read-only="false"/>
<tx:method name="*"/>
</tx:attributes>
</tx:advice>
My question is does the both select and update statements would be executed in the same transaction.
In the spring-integration documentation it does not specify clearly about the transaction when advice-chain is used. (I am using spring-integration-jdbc-2.2.0.RC2.jar)
Please see section 18.1.1 Poller Transaction Support:
http://docs.spring.io/spring-integration/docs/2.0.0.RC1/reference/html/transactions.html
You are using a very old version (and only a release candidate at that, not a GA release. The current version is 3.0.2.RELEASE the latest 2.2.x is 2.2.6.RELEASE. Please upgrade to one of those.
http://projects.spring.io/spring-integration/
Yes, it's all done in the same transaction.
Are you referring to the old documentation too? The current documentation says
"A very important feature of the poller for JDBC usage is the option to wrap the poll operation in a transaction,..".
The transaction is started, the message source is invoked (to get the results) and the downstream flow all occurs in the transaction (unless you hand off to another thread, in which case the transaction will commit at that time).
In fact, the update is done right after the select (and before the message is sent), but the commit doesn't occur until the downstream flow is complete.
Your channel is called queueChannel; if it really is a queue channel, that means that the transaction will commit as soon as the message is stored in the queue.
If you feel documentation improvements are required, please open a JIRA Issue.
I understand how aggregating based on size works but I also want to make the release strategy depend on another step in the pipeline to be still running. The idea is that i move files to a certain dir "source", aggregate enough files and then move from "source" to "stage" and then process the staged files. While this process is running I dont want to put more files in stage but I do want to continue to add more files to source folder (that part is handled by using a dispatcher channel connected with file inbound adapter before the aggregator)
<int:aggregator id="filesBuffered"
input-channel="sourceFilesProcessed"
output-channel="stagedFiles"
release-strategy-expression="size() == 10"
correlation-strategy-expression="'mes-group'"
expire-groups-upon-completion="true"
/>
<int:channel id="stagedFiles" />
<int:service-activator input-channel="stagedFiles"
output-channel="readyForMes"
ref="moveToStage"
method="move" />
so as you can see I dont want to release the aggregated messages if an existing instance of moveToStage service activator is running.
I thought about making the stagedFiles channel a queue channel but that doesnt seems right because I do want the files to be passed to moveToStage as a Collection not a single file which I am assuming by making stagedFiles a queue channel it will send a single file. Instead I want to get to a threshold e.g. 10 files, pass those to stagedFiles which allows the moveToStage to process those files but then until this step is done I want the aggregator to continue to aggregate files and then release all aggregated files.
Thanks
I suggest you to have some flag as a AtomicBoolean bean and use it from your moveToStage#move and check it's state from:
release-strategy-expression="size() >= 10 and #stagingFlag.get()"