Design: Spring Integration jdbc best practice - spring-integration

After using Spring Integration in a project, my observation is to use jdbc adapter or gateway only at the start or end of the flow. If we use them in the middle of flow then it will become too verbose and complex.
For example:
<jdbc:outbound-gateway
query="select * from foo where
c1=:headers[c1] AND
c2=:headers[c2] AND
c3=:headers[c3] AND
c4=:headers[c4]"
row-mapper="fooMapper" data-source="myDataSource" max-rows-per-poll="100000" />
<int:service-activator ref="serviceActivator" method="processFoo" />
In the above <jdbc:outbound-gateway>, we need to pass all the placeholders (c1, c2, c3, c4) in the header of Message. We need to look back and forth in java code and xml file for any change in where condition or when there are too many where clauses.
It is also error prone. For example, if we misspelled :headers[c1] to :headers[d1] then it will not throw any exception and replace :headers[d1] with null.
If query does not return any row then it will throw exception by default. So, we have to use requires-reply="false" to change default behaviour.
If we want to proceed when query does not return any value then we have to add advice to gateway, as shown below:
<jdbc:outbound-gateway ... >
<jdbc:request-handler-advice-chain>
<bean class="com.service.NullReplyAdvice" />
</jdbc:request-handler-advice-chain>
</jdbc:outbound-gateway>
Please correct me if there are flaws in understanding of the concept.

We need to look back and forth in java code and xml file for any change in where condition or when there are too many where clauses.
It's true even for raw Java code around the JDBC: if you change the model you, of course, should change the SELECT, because it is just a String. And that's why there is a lot of work to make it type-safe - ORM, QueryDSL, Spring-Data etc.
if we misspelled :headers[c1] to :headers[d1] then it will not throw any exception and replace :headers[d1] with null.
That's because the headers is just a Map and it's truth that you get null, if there is no such a key in the map. To overcome that typo issue you can use POJO payload with getters, or some custom header, and again - the POJO with getters. In this case you end up with exception that there is no such a property against object. Although you'll see that issue only at runtime, not on compile. And again the same is with Hashtable - only at runtime.
So, we have to use requires-reply="false" to change default behaviour.
You should understand it at design time: allow or not to return nothing for the component.
The last idea is good. Wouldn't you mind to share your NullReplyAdvice?
Actually I achieve the same with <filter> before the JDBC gateway: to determine if there is something to fetch by count(*) query. From there I can lead my flow to the different logic, rather than the direct flow, when SELECT returns rows.
UPDATE
When you want to use Model object to keep business-specific values within Message, it's just enough to put this object to the header:
public class Foo {
private String foo1;
private String foo2;
public String getFoo1() {
return foo1;
}
public String getFoo2() {
return foo2;
}
}
...
MessageBuilder.withPayload(payload).setHeader("foo", foo).build();
...
<jdbc:outbound-gateway
query="select * from foo where
c1=:headers[foo].foo1 AND
c1=:headers[foo].foo2"/>

Related

Does jooq record use column indexes when fetching data?

I'm investigating an issue where we are seeing strange exceptions related to jooq trying to populate a generated Record class, where it gets data type errors because it uses java.sql.ResultSet::getXXX(int) (column index based) to fetch data.
The part of the stacktrace that I can share looks like:
Caused by: java.sql.SQLDataException: Value 'ABC' is outside of valid range for type java.lang.Byte
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:114)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:73)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:92)
at com.mysql.cj.jdbc.result.ResultSetImpl.getObject(ResultSetImpl.java:1423)
at com.mysql.cj.jdbc.result.ResultSetImpl.getByte(ResultSetImpl.java:710)
at com.zaxxer.hikari.pool.HikariProxyResultSet.getByte(HikariProxyResultSet.java)
at org.jooq.tools.jdbc.DefaultResultSet.getByte(DefaultResultSet.java:124)
at org.jooq.tools.jdbc.DefaultResultSet.getByte(DefaultResultSet.java:124)
at org.jooq.impl.CursorImpl$CursorResultSet.getByte(CursorImpl.java:688)
at org.jooq.impl.DefaultBinding$DefaultByteBinding.get0(DefaultBinding.java:1783)
at org.jooq.impl.DefaultBinding$DefaultByteBinding.get0(DefaultBinding.java:1755)
at org.jooq.impl.DefaultBinding$AbstractBinding.get(DefaultBinding.java:871)
at org.jooq.impl.CursorImpl$CursorIterator$CursorRecordInitialiser.setValue(CursorImpl.java:1725)
Which is definitely a column mismatch caused using wrong column index.
The issue comes up because we are using the record on an evolving schema so the underlying table contains columns not available in the record definition.
Note that the actual code that triggers this is:
jooq.insertInto(TABLE)
.set(TABLE.COL, "ABC")
.returning(TABLE.asterisk())
.fetchOne();
What scares me a bit here is, that if it does indeed use column indexes by design, that will make schema evolution somewhat hard (how would you delete a column from a running application).
Long story (sorry), question is: does jooq use column index in records generated by jooq-generator and is there a way to use column names instead?
One thing I have noticed is that when I compare the documentation at https://www.jooq.org/doc/3.14/manual/code-generation/codegen-records/ the shown generated records does not match what the generator actually generates. The documentation show methods like:
// Every column generates a setter and a getter
#Override
public void setId(Integer value) {
setValue(BOOK.ID, value);
}
But in reality the generated code looks like (taken from jOOQ-examples):
/**
* Setter for <code>PUBLIC.BOOK.ID</code>.
*/
public void setId(Integer value) {
set(0, value);
}
Btw we are using jooq 3.14.15.
Ok, this was a local error. What actually caused the issue was that our code was written as:
jooq.insertInto(TABLE)
.set(TABLE.COL, "ABC")
.returning(TABLE.asterisk())
.fetchOne();
And the TABLE.asterisk() is what messes it up (since on the database that contains extra columns it does not return what jooq expects). Fortunately removing it solves the problem so our code now looks like:
jooq.insertInto(TABLE)
.set(TABLE.COL, "ABC")
.returning()
.fetchOne();

HTTP Outbound Gateway - Passing URI parameters

I need to invoke a REST API via Spring Integration's HTTP outbound gateway. How do I substitute the path variable with a value from payload. Payload has just one String value. The following code snippet is sending the place holder as such. Any help is appreciated.
#Bean
public MessageHandler httpGateway(#Value("http://localhost:8080/api/test-resource/v1/{parameter1}/codes") URI uri) {
HttpRequestExecutingMessageHandler httpHandler = new HttpRequestExecutingMessageHandler(uri);
httpHandler.setExpectedResponseType(Map.class);
httpHandler.setHttpMethod(HttpMethod.GET);
Map<String, Expression> uriVariableExp = new HashMap();
SpelExpressionParser parser = new SpelExpressionParser();
uriVariableExp.put("parameter1", parser.parseExpression("payload.Message"));
httpHandler.setUriVariableExpressions(uriVariableExp);
return httpHandler;
}
Let take a look what #Value is for first of all!
* Annotation at the field or method/constructor parameter level
* that indicates a default value expression for the affected argument.
*
* <p>Typically used for expression-driven dependency injection. Also supported
* for dynamic resolution of handler method parameters, e.g. in Spring MVC.
*
* <p>A common use case is to assign default field values using
* "#{systemProperties.myProp}" style expressions.
Looking to your sample there is nothing to resolve as a dependency injection value.
You can just use:
new HttpRequestExecutingMessageHandler("http://localhost:8080/api/test-resource/v1/{parameter1}/codes");
Although it doesn't matter in this case...
You code looks good, unless we don't know what your payload is.
That payload.Message expression looks odd. From big height it may mean like MyClass.getMessage(), but it can't invoke the getter because you use a property name as capitalized. If you really have there such a getter, so use it like payload.message. Otherwise, please, elaborate more. Some logs, StackTrace, the info about payload etc... It's fully unclear what is the problem.

Performing a distributed search through spark-solr

I'm using spark-solr in order to perform Solr queries. However, my searches don't work as they're supposed to because for some reason the requests being generated by spark prevent the searches from being distributed. I have discovered it by looking at the Solr logs where I saw that a distrib=false parameter is added to the sent requests. When executing the queries manually (not using spark) with distrib=true the results were fine.
I was trying to set the parameters sent by spark by changing the "solr.params" value in the options dictionary (I'm using pyspark):
options = {
"collection": "collection_name",
"zkhost": "server:port",
"solr.params": "distrib=true"
}
spark.read.format("solr").options(**options).load().show()
This change did not have any effect: I still see in the logs that a distrib=false parameter is being sent. Other parameters passed through the "solr.params" key (such as fq=something) do have an effect on the results. But it looks like spark insists on sending distrib=false no matter what I do.
How do I force a distributed search through spark-solr?
The easy solution is to configure the request handler to run distributed queries using an invariant. The invariant forces the distrib parameter to have a true value even if spark-solr is trying to change it in query time. Introducing the invariant can be done by adding the following lines under the definition of your request handler entry in solrconfig.xml:
<lst name="invariants">
<str name="distrib">true</str>
</lst>
While the introduction of the invariant is going to fix the problem, I think it's kind of a radical solution. This is because the solution involves hiding a behavior in which you overload the value of a parameter. By introducing the invariant you cannot decide to set distrib to false: even if your request explicitly does so, the value of distrib would still be true. This is too risky in my opinion and that's why I'm suggesting another solution which might be harder to implement but wouldn't suffer from that flaw.
The solution is to implement a query component which is going to force distrib=true only when receiving a forceDistrib=true flag as a parameter.
public class ForceDistribComponent extends SearchComponent {
private static String FORCE_DISTRIB_PARAM = "forceDistrib";
#Override
public void prepare(ResponseBuilder rb) throws IOException {
ModifiableSolrParams params = new ModifiableSolrParams(rb.req.getParams());
if (!params.getBool(FORCE_DISTRIB_PARAM, false)) return;
params.set(CommonParams.DISTRIB, true);
params.set(FORCE_DISTRIB_PARAM, false);
rb.req.setParams(params);
}
}
After building the component you can configure solr to use it by adding the component to solrconfig.xml and set your request handler to use it.
Adding the component to solrconfig.xml is done by adding the following entry to the solrconfig.xml file:
<searchComponent name="forceDistrib" class="ForceDistribComponent"/>
Configuring the request handler to use the forceDistrib component is done by adding it to the list of components under the request handler entry. It must be the first component in the list:
<arr name="components">
<str>forceDistrib</str>
<str>query</str>
...
</arr>
This solution, while more involved than simply introducing an invariant, is much safer.

JsConfig<MyClass>.ExcludePropertyNames example, not working for me

Trying to exclude properties from a model from being included during serialization.
I am using the following syntax:
JsConfig<MyTestClass>.ExcludePropertyNames = new[] { "ShortDescription" };
Just after that I have the following:
return (from o in __someProvider.GetAll() select (new
{
o.Name,
o.ShortDescription
o.InsertDate
}).TranslateTo<MyTestClass>()).ToList()
However once result is returned from the method, it still contains "ShortDescription" field in the Json. Am I doing something wrong?
JsConfig<T>.ExcludePropertyNames appears to be checked only once for each type, in a static constructor for TypeConfig<T>. Thus, if you are configuring ExcludePropertyNames in your service class, just before returning your response, it might be too late -- the TypeConfig properties may already be set up and cached for MyTestClass. I was able to reproduce this.
A more reliable alternative is to move all of your JsConfig<T> configuration to your AppHost setup code.
If you really do need to do this in your service class, e.g. if you are only conditionally excluding property names, then an alternative approach would be to ensure that JsConfig.IncludeNullValues is false (I believe it is by default) and in your service code set ShortDescription to null when appropriate.

How to determine the type of the camel exchange object

I have two different services running on a web server. Both the services have an operation named 'xyz', with the following arguments.
Service 1:
Public String xyx(Student object) {}
Service 2:
public String xyz(Employee object){}
Now i have a client which will invoke the operation of one of these services based on the message that it receives. The message will be received as a camel exchange. So i need to identify the type of the message and then invoke the appropriate service.
How do i identify the original type of the message that is received as a camel exchange.
Thanks.
Or you can do something like this:
from("foo:incommingroute")
.choice()
.when(simple("${body} is 'java.lang.String'"))
.to("webservice:Student")
.when(simple("${body} is 'foo.bar.Employee'"))
.to("webservice:Employee")
.otherwise()
.to("jms:Deadletter")
.end();
Try exchange.getIn().getBody() instanceof Student
I would set the a value in the header to indicate which service it is and then send this off on the camel route. This approach is just but one way of doing this. Christian Schneider has another excellent solution which I will probably use much more now that I have gotten much more into Camel then ever before. However both will achieve the same thing and depending on who you ask one might be more clear than the other.
For example you can do:
public void foo(Exchange exchange){
exchange.getIn().setHeader("MsgType", "Student");
}
You can then filter on the header in either the Java DSL or even spring DSL.
In Java DSL you would do something like this (pseudo code)
from("foo:incommingroute")
.choice()
.when(header("MsgType").equals("Student"))
.to("webservice:Student")
.when(header("MsgType").equals("Employee"))
.to("webservice:Employee")
.otherwise()
.to("jms:Deadletter")
.end();
In Spring DSL you would do something like this (pseudo code)
<route>
<from uri="foo:incommingroute"/>
<choice>
<when>
<simple>${header.MsgType} equals 'Student'</simple>
<to uri="webservice:Student"/>
</when>
<when>
<simple>${header.MsgType} equals 'Employee'</simple>
<to uri="webservice:Employee"/>
</when>
<otherwise>
<to uri="jms:badOrders"/>
<stop/>
</otherwise>
</choice>
<to uri="jms:Deadletter"/>
</route>
You can also look at the enricher pattern at this link http://camel.apache.org/content-enricher.html. Basically what I am suggesting is following the enricher pattern. If you could tell me how you are sending messages to Camel then I could probably help more.
Hope this give you some ideas and if there is syntax mistakes etc in the code sorry I am at a bus stop and did not have time to check it.
I prefer to write this type of logic directly in the route definition rather than in a Processor. Here is the Camel DSL approach that uses a Predicate to determine the body class type. It assumes that you have already deserialized the Exchange body into a Student or Employee object.
choice()
.when(body().isInstanceOf(Student::class))
.to(...)
.when(body().isInstanceOf(Employee::class))
.to(...)
.end()
If you're going to perform various transformations on the body throughout the route, resulting in a variety of Student or Employee object types at various stages (e.g. a Student then a StudentEntity, etc) then saving the type in a header or property as some String constant at the beginning of the route might be the cleaner approach.
// Note that this labelling could be bundled into a processor
choice()
.when(body().isInstanceOf(Student::class))
.setProperty("TYPE", "STUDENT")
.when(body().isInstanceOf(Employee::class))
.setProperty("TYPE", "EMPLOYEE")
.end()
// later after some body transformations
.choice()
.when(exchangeProperty("TYPE").isEqualTo("STUDENT"))
// process student
Lastly, you might be able to do everything in a processor but I think this sort of branch logic combined with service invocation is a Camel anti-pattern.
class MyProcessor implements Processor {
#Override
public void process(Exchange exchange) {
Object body = exchange.getIn().getBody()
if (body instanceOf Student) {
// invoke StudentService
} else if (body instanceOf Employee) {
// invoke EmployeeService
}
}
}
// Route definition
from(...)
.process(myProcessor)

Resources