Duplicate call in select-sql-parameter-source - spring-integration

I'm working with a dynamic query, using select-sql-parameter-source to search the information that I need.
This is my configuration:
<int-jdbc:inbound-channel-adapter query="SELECT * FROM CUSTOMER WHERE CUSTOMER.LASTUPDATE_ACTIVE < TO_DATE(:last_process_date,'YYYY-MM-DD HH24:Mi:SS') "
channel="headerEnricher.customerBR01"
update=""
row-mapper="customerRowMapper"
data-source="jdbcTemplate"
max-rows-per-poll="0"
select-sql-parameter-source="parameterSource.customerBR01">
<!-- Cron Time -->
<int:poller fixed-rate="50" time-unit="SECONDS">
</int:poller>
</int-jdbc:inbound-channel-adapter>
<!-- This is to get last process date -->
<bean id="parameterSource.customerBR01" factory-bean="parameterSourceFactory.customerBR01" factory-method="createParameterSourceNoCache">
<constructor-arg value="" />
</bean>
<bean id="parameterSourceFactory.customerBR01" class="org.springframework.integration.jdbc.ExpressionEvaluatingSqlParameterSourceFactory">
<property name="parameterExpressions">
<map>
<!-- Here we get the last process date -->
<entry key="last_process_date" value="#hsqlHistoricProcessServiceDateDAO.getLastProcessDate(3,1,'CUSTOMER')" />
</map>
</property>
</bean>
I was looking that loggin appeared twice, so I changed my code in this function :
hsqlHistoricProcessServiceDateDAO.getLastProcessDate
To return only an account variable.
Code of function hsqlHistoricProcessServiceDateDAO.getLastProcessDate is the following:
private int contador = 0;
public String getLastProcessDate(Integer country, Integer business, String tableName) {
contador++;
System.out.println("Contador "+ contador);
return Integer.toString(contador);
}
And its result is :
Contador 1
Contador 2
So, this method is called twice, and I need only one call, because in the "real code" I have all logging twice for that.

For your use case, you don't need to disable the cache; use this instead...
<bean id="parameterSource.customerBR01"
factory-bean="parameterSourceFactory.customerBR01"
factory-method="createParameterSource">
<constructor-arg value="" />
</bean>
The ...NoCache version is needed when the same key is used in multiple parameters and you want each one to be re-evaluated.
Disabling the cache has this additional side-effect because the getValue() method is called twice for each use of the key. One call is from NamedParameterUtils.substituteNamedParameters(); the second is from NamedParameterUtils.buildValueArray().

Related

handling exceptions for Http rest endpoints in spring integration

I need to throw an exception to the end user, can you please assist with this?
<int-http:request-handler-advice-chain>
<!--<bean class="org.springframework.integration.handler.advice.RequestHandlerRetryAdvice">
<property name="recoveryCallback">
<bean class="org.springframework.integration.handler.advice.ErrorMessageSendingRecoverer">
<constructor-arg ref="retryErrorChannel" />
</bean>
</property>
<property name="retryTemplate" ref="retryLoadTemplate" />
</bean>-->
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="successChannel" ref="afterSuccessFetchChannel" />
<property name="failureChannel" ref="afterFailFetchChannel" />
</bean>
</int-http:request-handler-advice-chain>
</int-http:outbound-gateway>
<int:transformer ref="testTransformer" method="processDetails" />
<int:transformer input-channel="afterSuccessFetchChannel" output-channel="goodResultChannel"
expression="'Fetching load id: ' + payload + ' details was successful'" />
<int:transformer input-channel="afterFailFetchChannel" output-channel="badResultChannel"
expression="payload + ' was bad, with reason: ' + payload.cause.message" />
<int:logging-channel-adapter id="badResultChannel" level="ERROR"/>
<int:logging-channel-adapter id="goodResultChannel" level="INFO" />
How and where to handle custom exceptions and how to re throw them?
My api is throwing 404 Not found exception, need to catch and re throw it to the user?
for now, I am just using logging channel, I don't really know the way to deal with exceptions in spring integration, can you please help?
So, instead of that <int:logging-channel-adapter id="badResultChannel" level="ERROR"/> you just need to write some POJO method which would throw some other exception after its handling. The ExpressionEvaluatingRequestHandlerAdvice has this property returnFailureExpressionResult to be set to true.
The logic is like this:
try {
evalResult = this.onFailureExpression.getValue(prepareEvaluationContextToUse(exception), message);
}
catch (Exception e) {
evalResult = e;
logger.error("Failure expression evaluation failed for " + message + ": " + e.getMessage());
}
and then:
if (this.onFailureExpression != null) {
Object evalResult = evaluateFailureExpression(message, actualException);
if (this.returnFailureExpressionResult) {
return evalResult;
}
}
So, your custom thrown exception is going to be sent as a reply message payload back to the caller. And there probably an MVC layer can simply convert it to respective HTTP response for the client.

WARN JdbcChannelMessageStore Message with id was not deleted

I have a queue channel backed by a JdbcChannelMessageStore. I have two instances of this application and with high concurrency I have this warning:
2020-03-13 19:25:38,209 task-scheduler-5 WARN JdbcChannelMessageStore:652 - Message with id '06b73eab-727a-780f-d0fa-1b0e0dd1ea20' was not deleted.
Is there a way to remove them?
As far as my understanding, messages are being read twice, am I correct?
I am using SI 4.3.19.RELEASE. Here is my spring flow
<int:channel id="channel">
<int:queue message-store="messageStoreBean"/>
</int:channel>
<int:header-value-router input-channel="channel
header-name="name" >
<int:poller max-messages-per-poll="2" fixed-rate="500" >
<int:transactional />
</int:poller>
...
</int:header-value-router>
<bean id="storeQueryProviderBean" class="org.springframework.integration.jdbc.store.channel.PostgresChannelMessageStoreQueryProvider" />
<bean id="messageStoreBean" class="org.springframework.integration.jdbc.store.JdbcChannelMessageStore">
<property name="dataSource" ref="messageStoreDataSource" />
<property name="channelMessageStoreQueryProvider" ref="storeQueryProviderBean" />
<property name="region" value="region" />
</bean>
It looks like PostgreSQL doesn't guarantee exclusive reading with transactions and LIMIT 1 FOR UPDATE.
Anyway that WARN is just a note that some other process has removed the message. Nothing is duplicated if other process is similar to that poller:
public Message<?> pollMessageFromGroup(Object groupId) {
final String key = getKey(groupId);
final Message<?> polledMessage = this.doPollForMessage(key);
if (polledMessage != null) {
if (!this.doRemoveMessageFromGroup(groupId, polledMessage)) {
return null;
}
}
return polledMessage;
}
You see if message was not removed, we return null therefore nothing to poll at the moment.
You can turn off the warning level for the org.springframework.integration.jdbc.store.JdbcChannelMessageStore to avoid that message specifying a category level as ERROR in your logging config.

Spring integration no files picked

The following is my configuration. I was able to poll files until recently. Now the filter always gets an empty list of files. The only change i have made is install kaspersky antivirus. Hopefully that is not a problem. I can successfully access the ftp from command prompt as well as from the browser.
The conf:
<int:channel id="ftpChannel"/>
<int-ftp:inbound-channel-adapter id="ftpInbound1"
channel="ftpChannel"
session-factory="ftpClientFactory"
charset="UTF-8"
local-directory="file:${paths.root}"
delete-remote-files="false"
temporary-file-suffix=".writing"
remote-directory="${file.ftpfolder}"
preserve-timestamp="true"
auto-startup="true"
filter="compositeFilterLocal"
>
<int:poller max-messages-per-poll="10000" fixed-rate="1000" error-channel="errorChannel"/>
</int-ftp:inbound-channel-adapter>
<int-ftp:outbound-channel-adapter id="ftpOutbound"
channel="ftpChannel"
session-factory="ftpClientFactory"
charset="UTF-8"
remote-file-separator="/"
auto-create-directory="true"
remote-directory="DMS"
use-temporary-file-name="true"
temporary-file-suffix=".writing">
</int-ftp:outbound-channel-adapter>
<!-- <bean id="acceptAllFilter" class="org.springframework.integration.file.filters.AcceptAllFileListFilter" /> -->
<bean id="compositeFilterLocal" class="org.springframework.integration.file.filters.CompositeFileListFilter">
<constructor-arg>
<list>
<!-- Ensures that the file is whole before processing it -->
<bean class="org.springframework.integration.file.filters.AcceptAllFileListFilter" />
<bean class="com.polling.util.CustomFileFilterLocal"/>
<!-- Ensures files are picked up only once from the directory -->
</list>
</constructor-arg>
</bean>
Please tell me if anything should be changed in it...Thanks
Please let me know if anymore information is needed!
EDIT:: Update
If I use the Apache commons-net-3.3 to retrieve the same file from the same folder, it is working fine and allowing me to take the file and download it. So this has nothing to do with jvm access to the ftp site.
EDIT::Code for the filter is simple. Currently I am only using it for pattern matching.
#Override
public List<File> filterFiles(File[] files)
{
List<File> ret = new ArrayList<File>();
Pattern pattern = Pattern.compile(".*?~.*?");//(".*?#.*?#.*?");
DocumentFile documentFile;
Matcher matcher;
for (File file : files)
{
matcher = pattern.matcher(file.getName());
if(matcher.find())// matching the input file name pattern
{
//get key and documentfile
//create sha key to check file existance
String key = EncodeUtil.generateKey(file);
documentFile = documentDaoImpl.getDocumentFile(key,Constants.INPROGRESS);
if (documentFile != null)
{
ret.add(file);
}
}/*else
{
file.delete();
}*/
}
return ret;
}
I have been successfully working with this for atleast a couple of months and now suddenly I am getting no files!!
Currently I am in process of using a timer cron expression and will do the ftp using apache commons-net within the triggered class. Seems such a waste having to do the ftp inspite of having the spring ftp tag.
I have built a project with the configuration you have used and everything seems to work fine.
There are some pieces in your code (not posted here) that might lead to discard the files in the filter and that you will have to check (adding log messages will help):
if (matcher.find())// matching the input file name pattern
{
// get key and documentfile
// create sha key to check file existance
// TODO: does this call throw any exception? return null?
String key = EncodeUtil.generateKey(file);
documentFile = documentDaoImpl.getDocumentFile(key, Constants.INPROGRESS);
if (documentFile != null) {
ret.add(file);
}
else {
// TODO: Log here that your DAO implementation did not return anything for this specific file
}
}
else {
// TODO: Log here that the file does not meet the naming convention
}

Spring Integration - how to keep the orginal payload and use it later?

I would like to keep the original payload of the original requests and ise it in a xslt-transformer or in other operation. I lose it because I use an xslt-transformer and I need just some of the elements in the transformation. So my scenario is:
1.inbound-gateway (incoming WS req) -> 2.xslt-transformer (mapping for calling an external WS) -> 3.outbound-gateway (calling the external WS) -> 4.xslt-transformer (creating response from the resp. of the external WS and the original req)
At the 4th step I don't have the original req but I'd need it as I have to put values from it to the response. How could I implement it?
Thanks,
V.
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:int="http://www.springframework.org/schema/integration" xmlns:int-ws="http://www.springframework.org/schema/integration/ws" xmlns:int-xml="http://www.springframework.org/schema/integration/xml" xmlns:util="http://www.springframework.org/schema/util" xsi:schemaLocation="http://www.springframework.org/schema/integration/ws http://www.springframework.org/schema/integration/ws/spring-integration-ws.xsd http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/integration/xml http://www.springframework.org/schema/integration/xml/spring-integration-xml.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd">
<bean id="authenticator" class="uk.co.virginmedia.test.Authenticator"/>
<bean id="webserviceDestinationProvider" class="uk.co.virginmedia.test.WebserviceDestinationProvider"/>
<bean id="resultToDocumentTransformer" class="org.springframework.integration.xml.transformer.ResultToDocumentTransformer"/>
<util:map id="orderNamespaceMap">
<entry key="res" value="http://schema/ReserveAppointment/2/0" />
</util:map>
<int-ws:inbound-gateway id="ws-gateway-for-rbta" request-channel="incoming-req-channel" reply-channel=""/>
<int:channel id="incoming-req-channel"/>
<int:service-activator id="authentication" input-channel="incoming-req-channel" ref="authenticator" method="authenticate" output-channel="authenticated-channel" />
<int:channel id="authenticated-channel"/>
<int-xml:xpath-router id="servicetype-router" input-channel="authenticated-channel" evaluate-as-string="true">
<int-xml:xpath-expression expression="//res:ReserveAppointmentRequest/res:serviceType/text()" ns-prefix="res" ns-uri="http://schema/ReserveAppointment/2/0"/>
<int-xml:mapping value="Broadband" channel="broadband-channel"/>
<int-xml:mapping value="FTTC+WholesaleLineRental" channel="fttc-wlr-channel"/>
</int-xml:xpath-router>
<int:channel id="broadband-channel"/>
<int-xml:xslt-transformer id="req_for_bt_xslt_transformer" input-channel="broadband-channel" output-channel="domresult_for_bt_channel" xsl-resource="classpath:/xsl/ToBTReq.xsl" result-type="StringResult"/>
<int:channel id="domresult_for_bt_channel"/>
<int:transformer input-channel="domresult_for_bt_channel" output-channel="document_for_bt_channel" expression="payload.toString()"/>
<int:channel id="document_for_bt_channel"/>
<int-ws:outbound-gateway request-channel="document_for_bt_channel" reply-channel="resp_from_bt_channel" destination-provider="webserviceDestinationProvider" id="call_bt-outbound_gateway" />
<int:channel id="resp_from_bt_channel"/>
<int-xml:xslt-transformer id="resp_for_rbta_xslt_transformer" input-channel="resp_from_bt_channel" output-channel="resp_for_rbta_channel" xsl-resource="classpath:/xsl/ToBTReq.xsl" result-type="StringResult"/>
Since your original message is just text you could copy it to a header field. This should work as long as you don't do anything special in between when you store and afterwards retrieve it.
So what I would try is:
<int:header-enricher input-channel="authenticated-channel" output-channel="pre-routing-channel">
<int:header name="original-payload" expression="payload.toString()" />
</int:header-enricher>
<!-- changed input channel of router -->
<int-xml:xpath-router id="servicetype-router" input-channel="pre-routing-channel" evaluate-as-string="true">
If this is not working for you (maybe because you have to do something more special in between or the payload is too big), you still have the option to use a ClaimCheck. Which is actually exactly what you are asking for. For this you'll need a MessageStore and then just store the message payload before modifying it. So instead of the header-enricher you will call
<int:claim-check-in input-channel="authenticated-channel" output-channel="pre-routing-channel" message-store="payloadstore" />
<!-- MessageStore definition storing payload using in memory map -->
<bean id="simpleMessageStore"
class="org.springframework.integration.store.SimpleMessageStore"/>

Threading issues with Hibernate

I'm experiencing a strange behaviour with Hibernate. I've been banging my head against the wall for a while now, and will award any answer which leads to a solution with a +100 bounty.
I have a JAX-RS (Jersey) REST server, with a filter that associates one Hibernate-session per request.
In one request a client POSTs some data which is stored in the database using one session (and one transaction). In a subsequent call, the client tries to GET this entity, but Hibernate can't find it.
Some observations:
I can only reproduce this if I run multiple simultaneous clients. I've never managed to reproduce it by running one client at a time.)
I can see the entity ID in the database, and if I restart the server, the entity is found by Hibernate as it should.
The error does not occur if I use a thread pool of size 1 (regardless of how many clients I run simultaneously).
Here's the code, with some logging:
chargeables.setId(new SecureRandom().nextLong());
System.out.printf("%s, session: %s [%s]%n",
Thread.currentThread(),
System.identityHashCode(session),
"session.beginTransaction()");
session.beginTransaction();
System.out.printf("%s, session: %s [%s]%n",
Thread.currentThread(),
System.identityHashCode(session),
"session.save(id = "+chargeables.getId()+")");
session.save(chargeables);
System.out.printf("%s, session: %s [%s]%n",
Thread.currentThread(),
System.identityHashCode(session),
"session.getTransaction().commit()");
session.getTransaction().commit();
The code for getting the entity:
System.out.printf("%s, session: %s [%s]%n",
Thread.currentThread(),
System.identityHashCode(session),
"session.get("+id+")");
Chargeables entity = (Chargeables) session.get(Chargeables.class, id);
if (entity == null)
System.out.printf("%s, session: %s [%s]%n",
Thread.currentThread(),
System.identityHashCode(session),
"ENTITY NOT FOUND!");
Now here is an excerpt of the resulting log (with some additional open/close session output):
Thread[Grizzly(5),5,main], session: 2041842357 [factory.openSession()]
Thread[Grizzly(5),5,main], session: 2041842357 [session.beginTransaction()]
Thread[Grizzly(5),5,main], session: 2041842357 [session.save(id = 7939229356942262438)]
Thread[Grizzly(5),5,main], session: 2041842357 [session.getTransaction().commit()]
Thread[Grizzly(5),5,main], session: 2041842357 [session.close()]
[...]
Thread[Grizzly(7),5,main], session: 1717445911 [factory.openSession()]
Thread[Grizzly(7),5,main], session: 1717445911 [session.get(7939229356942262438)]
Thread[Grizzly(7),5,main], session: 1717445911 [ENTITY NOT FOUND!]
Thread[Grizzly(7),5,main], session: 1717445911 [session.close()]
Why on earth do I reach ENTITY NOT FOUND!?
Hibernate version: 4.1.9.Final
MySQL verison: 14.14 Distrib 5.5.29
Mapping file for Chargeables:
<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC
"-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://www.hibernate.org/dtd/hibernate-mapping-3.0.dtd">
<hibernate-mapping
default-cascade="all"
package="se.package.common.chargeables"
default-lazy="false">
<class name="Chargeables">
<id name="id" type="long">
<generator class="assigned"/>
</id>
<property name="startTimeStamp" />
<property name="endTimeStamp" />
<list name="chargeables">
<key column="chargeableId" />
<list-index column="pos" />
<many-to-many class="Chargeable"/>
</list>
</class>
<class name="Chargeable">
<id column="id" type="long">
<generator class="native"/>
</id>
<discriminator />
<property name="timestamp" />
</class>
<subclass name="DataTransfer" extends="Chargeable">
<property name="bytesSent" />
<property name="bytesReceived" />
</subclass>
<subclass name="TelephonyChargeable" extends="Chargeable">
<many-to-one name="num" />
</subclass>
<subclass name="Call" extends="TelephonyChargeable">
<property name="duration" />
</subclass>
<subclass name="OutgoingCall" extends="Call" />
<subclass name="IncomingCall" extends="Call" />
<subclass name="Message" extends="TelephonyChargeable" />
<subclass name="Sms" extends="Message" />
<subclass name="IncomingSms" extends="Sms" />
<subclass name="OutgoingSms" extends="Sms" />
<subclass name="Mms" extends="Message" />
<subclass name="IncomingMms" extends="Mms" />
<subclass name="OutgoingMms" extends="Mms" />
</hibernate-mapping>
IMHO it's an isolation problem. Your second session start before the transaction of the 1st one is commited. As default hibernate isolation level is read_commited, the 2nd session can't so retrieve the entity. How are you passing id between the 2 threads ?
If you can see the commit in the database, it cannot be an isolation issue at the MySQL level: console operations are also client operations, you don't get special powers when running them, so they conform to the isolation policy you selected.
Looking out for solution, I discovered that Hibernate provide some facilities to cache DB results: there are actually 2 levels of cache implemented there. Perhaps your hibernate installation comes bundled with a cache mechanism?
The first level works on a per session basis, which is consistent with the behaviour you observe when running the code on one thread, and seemingly happens to be activated by default. I am not Hibernate proficient, I can't say that for sure, but my opinion is that your setup has this first level of cache set in read-only mode, while you would like it to be in read-write mode.

Resources