End to End Test case for Spring batch job - mockito

I'm having an application where i'm using spring batch. I want to write a test case which can test the batch job end to end. I have been exploring various options for the same. I checked if concordion test cases can be useful but i'm not sure if it's ideal way to test spring-batch jobs. So far i think integration test case should fit the best to my case. I would like to know what should be the ideal approach to test my scenario.
<batch:job id="batch-job">
<batch:step id="cleanupData" next="populateExchRates">
<batch:tasklet ref="dataCleanupTasklet" />
</batch:step>
<batch:step id="populateExchRates" next="populateCache">
<batch:tasklet ref="populateExchRatesDBTasklet" />
</batch:step>
<batch:step id="populateCache" next="ExternalDbQuery">
<batch:tasklet ref="populateFxRatesCacheTasklet" />
</batch:step>
<batch:step id="ExternalDbQuery" next="...">
<batch:tasklet ref="ExternalDBQueryTasklet" />
</batch:step>
...
</batch:job>
We have batch job defined as above, there are more than 20 steps which includes interface with external systems(>5 such steps), there are step readers etc.
While developing Integration test cases i was considering using spring-batch-test API along with Mockito so that I can mock steps involving external systems calls. Using this approach i'll have to create the mock objects in the spring configuration itself(Injecting Mockito mocks into a Spring bean). What i'm not sure about is how i'll mock the method calls for the mocked beans if i configure them using spring as below.
<bean id="dao" class="org.mockito.Mockito" factory-method="mock">
<constructor-arg value="com.package.Dao" />
</bean>
I'm not sure if i could explain my scenario clearly or not. Please suggest if you have any other better opinion for testing E2E flow for spring-batch job and also if you can provide any clarity on the above approach that would be helpful.

Your approach seems valid to me. I've used a similar setup many times. Just setup a Spring context from your test and use the JobLauncherTestUtils to work with your jobs.
One thing to consider is what to mock. If multiple consecutive steps read and write the same data from the same datasource, mocking those out, might lead to a lot of mock configuration in your test. Perhaps it is easier then to setup an in memory database for those steps. That depends on the details of your implementation.
Concerning the setup of your mocks...
You can simply autowire your mocks into the test.
#Autowired
private Dao mockedDao;
Then you can simply configure that mock as you would any mock.
when(dao.findById(any())).thenReturn(something);

Related

spring integration with multiple defaultmessagelistenercontainer (with transactionManager) and multiple <jms:outbound-channel-adapter> and

I have a spring integration application with a requirement of 2 DefaultMessageListenerContainer (with transactionManager) and 2 jms:outbound-channel-adapter.
I have configured each listener container with "property name="connectionFactory" ref="jmsConnectionFactory" and also "property name="transactionManager" ref="platformTransactionManager" .
But for the transaction manager i have used again a different jmsConnectionFactory . I am not using a CachingConnectionFactory in the listener container but using it in platformTransactionManager . For jms:outbound-channel-adapter again i am using the same jmsConnectionFactory which i have used in listener container.
My problem is that with this configuration i am hitting the no. of connections exceeding the limit (150) to a MQ channel in a high load condition.
I tried many different ways but i don't see the no. of connection getting down once it gets up , using below configuration for cachingFactory
CachingConnectionFactory cachingConnectionFactory = new
CachingConnectionFactory(factory);
cachingConnectionFactory.setCacheConsumers(true);
cachingConnectionFactory.setCacheProducers(true);
cachingConnectionFactory.setSessionCacheSize(16);`
i tried
1)sharing the jmsConnectionFactory between the listenerContainer and the platformTransactionManager but the txn doesn't work correctly as i see the messages getting sent in incorrect order
2)using the cachingConenctionFactory in thelistenerContainer` , but still i see the same no. of connections
Should i use 3 different jmsConnectionFactory for the listener, txnmanager and the sender?
What would be best configuration here, please suggest , many thanks!
I think with IBM MQ you can't use CachingConnectionFactory, but instead this one as:
<bean id="connectionFactory" class="org.springframework.jms.connection.DelegatingConnectionFactory">
<property name="targetConnectionFactory" ref="imbMqFactory"/>
<property name="shouldStopConnections" value="true"/>
</bean>
This single connectionFactory must be used from the TX manager and from the containers.

Slow performance spring.io flow from ActiveMQ to OracleAQ with XA

I have a Spring integration flow which bridges from ActiveMQ to OracleAQ.
See example project under GitHub - https://github.com/cknzl2014/springio-ora-xa/tree/atomikos.
When I run it without XA, it is blazingly fast.
With XA, it processes only 1 to 2 messages per second.
When profiling the application, I see that for every message a new physical connection is established, and with this, the metadata query is issued on the oracle db.
But I don't understand why it does this, and how I can prevent this from happening.
Does anyone of you guys have experience with OracleAQ and XA?
Could this be a problem with the XA transaction manager (I use Atomikos)?
Thanks for your help,
Chris
We found a solution to the problem.
It consists of four steps.
Step 1: Use the latest Oracle client libraries
The first step ist to use the lastest Oracle 12c client libraries.
There were significant improvements in the ojdbc8.jar, e.g. they use stored procedures to get the metadata now.
This increased the throughput to about 10 msgs/s.
Step 2: Setup connection pooling correctly
The second step was improving the connection pooling according to article http://thinkfunctional.blogspot.ch/2012/05/atomikos-and-oracle-aq-pooling-problem.html:
<bean id="oraXaDataSource" primary="true"
class="oracle.jdbc.xa.client.OracleXADataSource" destroy-method="close">
<property name="URL" value="${oracle.url}" />
<property name="user" value="${oracle.username}" />
<property name="password" value="${oracle.password}" />
</bean>
<bean id="atomikosOraclaDataSource"
class="org.springframework.boot.jta.atomikos.AtomikosDataSourceBean">
<property name="uniqueResourceName" value="xaOracleAQ" />
<property name="xaDataSource" ref="oraXaDataSource" />
<property name="poolSize" value="5" />
</bean>
<bean id="OracleAQConnectionFactory" class="oracle.jms.AQjmsFactory" factory-method="getConnectionFactory">
<constructor-arg ref="atomikosOraclaDataSource" />
</bean>
This configuration alone resultet in exceptions because of 'auto-commit' of the Oracle connection.
Step 3: Set autoCommit to false
The third step was to set the following java system property (see https://docs.oracle.com/database/121/JAJDB/oracle/jdbc/OracleConnection.html#CONNECTION_PROPERTY_AUTOCOMMIT):
-DautoCommit=false
But then the throughput went down to 1 to 2 msg/s again.
Step 4: Set oracle.jdbc.autoCommitSpecCompliant to false
The last step was to set the following java system property (see https://docs.oracle.com/database/121/JAJDB/oracle/jdbc/OracleConnection.html#CONNECTION_PROPERTY_AUTO_COMMIT_SPEC_COMPLIANT):
-Doracle.jdbc.autoCommitSpecCompliant=false
Now we get a throughput of 80 msgs/s.
Conclusion
The setting of oracle.jdbc.autoCommitSpecCompliant to false is not elegant, but solved the problem.
We have to investigate further to see how we can get around this problem without setting oracle.jdbc.autoCommitSpecCompliant to false.
Many thanks to Dani Steinmann (stonie) for the help!
P.S.: I updated the sample project under GitHub - https://github.com/cknzl2014/springio-ora-xa/tree/atomikos.
First of all you should be sure that you use pool for JDBC connections.
On the other hand you may consider to use ChainedTransactionManager isntead of XA for two target transaction managers - JMS and JDBC.
Also see some information in the JDBC extensions project.
There is also some Oracle AQ API as well in that project.

Spring Integration - Processing Pipeline - Design

I am trying to design a message processing pipeline that needs to process a message using multiple end-points. At each stage, the endpoints returns either the payload (could be transformed) or an error message. Here's a generic example of what I have in my mind:
<int:payload-type-router input-channel="preprocessing-output">
<int:mapping type="com.example.Error" channel="error" />
<int:mapping type="com.example.PreprocessedDomainObject" channel="validation-input"/>
</int:payload-type-router>
<int:service-activator input-channel="validation-input"
ref="validationService" method="validate" output-channel="validation-output"/>
<int:payload-type-router input-channel="validation-output">
<int:mapping type="com.example.Error" channel="error" />
<int:mapping type="com.example.CouldBeAnotherObject" channel="processor-input"/>
</int:payload-type-router>
So on, this processing chain can be quite long... Is there a better way to design this instead of a payload type router after every stage? Kinda feels redundant.
Well, I'd make it based on the exceptions. The validation service should just throw exception and all you need is to catch it on the caller or have an errorChannel if it is Inbound Channel Adapter.
That way you don't need those routers definitions at all and just plain services calling chain.

Hybris HAC Update Scenario

I wrote an interceptor in one of my projects to intercept all the requests. So usually in a spring project i will do normal build and start the server and my changes related to an interceptor will start reflecting. However this doesn't seem to be the case with a hybris project.
Do I need to do update in hybris hac as well? And if I do then out of the available below mentioned options which options do i need to choose and why.
1. Update running system
2. Clear the hMC configuration from the database
3. Create essential data
4. Localize types
Thanks,
Ashish
To answer the second part of your question, I have listed at least one reason for selecting each type of options. Hope this helps.
Update running system - Required when hybris type definition changes. For example, changing the content of file <extnesion-name>-items.xml
Clear the hMC configuration from the database - If you have chosen to persist hmc configuration in the Database and changing it. For example, changing content in the file hmc.xml
Create essential data - If there is a change in the content in the impex files which follows the naming pattern of essentialdata*.impex.
Localize types - If there is a change in the properties files for localizations. for example changing the content in file <extension-name>-locales_en.properties
Assuming you've not made any changes to any underlying data objects (Jalo items) then you won't need to run an update in the hybris hAC.
You should just be able to make your changes, run ant clean all from the platform and then start up the hybris ECP instance and your changes will be visible.
If this is a normal Spring MVC interceptor, then it should work fine.
Have you checked your spring configuration in the storefront extension you are working from?
For example, take a look at the accelerator Spring MVC configuration file:
hybris/bin/ext-template/yacceleratorstorefront/web/webroot/WEB-INF/config/spring-mvc-config.xml
This has some examples where this is used out-of-the-box:
<mvc:interceptors>
<ref bean="beforeControllerHandlerInterceptor" />
<ref bean="beforeViewHandlerInterceptor" />
<ref bean="csrfHandlerInterceptor" />
</mvc:interceptors>
As an example, the default before controller handler interceptor is defined as:
<!-- Interceptor that runs once per request and before the controller handler method is called -->
<alias name="defaultBeforeControllerHandlerInterceptor" alias="beforeControllerHandlerInterceptor" />
<bean id="defaultBeforeControllerHandlerInterceptor" class="de.hybris.platform.yacceleratorstorefront.interceptors.BeforeControllerHandlerInterceptor" >
<property name="beforeControllerHandlers">
<ref bean="beforeControllerHandlersList" />
</property>
</bean>
which references:
<alias name="defaultBeforeControllerHandlersList" alias="beforeControllerHandlersList" />
<util:list id="defaultBeforeControllerHandlersList" >
<!-- List of handlers to run -->
<bean class="de.hybris.platform.yacceleratorstorefront.interceptors.beforecontroller.SecurityUserCheckBeforeControllerHandler" />
<bean class="de.hybris.platform.yacceleratorstorefront.interceptors.beforecontroller.RequireHardLoginBeforeControllerHandler" >
<property name="userService" ref="userService"/>
<property name="redirectStrategy" ref="redirectStrategy"/>
...
</bean>
<bean class="de.hybris.platform.yacceleratorstorefront.interceptors.beforecontroller.DeviceDetectionBeforeControllerHandler" />
...
</util:list>
So you could either override this using the alias with your own implementation, or add additional controller handlers to the list.
As there is no change to the underlying data model - this is just wiring up Spring MVC related classes - no need for an update system or anything like that. Just an 'ant clean all' to recompile to pick up your new interceptor classes, and server restart to pick up the change in the Spring cornfiguraton.

Routing based on REST method with Spring integration

I have several "int-http:inbound-gateway" and I need one of them to point at different services based on http method provided by the request.
<int-http:inbound-gateway path="....." supported-methods="POST,PUT"/>
At this moment I have 2 different endpoints and I was looking for some rest-method-based router but I have found nothing about this topic.
Any help?
you can use a header-value-router for this one as the http method is automatically set in the message header.
something like this
<int-http:inbound-channel-adapter channel="input.channel"
path="/log" supported-methods="PUT,POST" request-payload-type="java.lang.String"/>
<int:channel id="input.channel"/>
<int:header-value-router input-channel="input.channel" header-name="#{T(org.springframework.integration.http.HttpHeaders).REQUEST_METHOD">
<int:mapping value="PUT" channel="put.input.channel"/>
<int:mapping value="POST" channel="post.input.channel"/>
</int:header-value-router>
hope that helps

Resources