Slow performance spring.io flow from ActiveMQ to OracleAQ with XA - spring-integration

I have a Spring integration flow which bridges from ActiveMQ to OracleAQ.
See example project under GitHub - https://github.com/cknzl2014/springio-ora-xa/tree/atomikos.
When I run it without XA, it is blazingly fast.
With XA, it processes only 1 to 2 messages per second.
When profiling the application, I see that for every message a new physical connection is established, and with this, the metadata query is issued on the oracle db.
But I don't understand why it does this, and how I can prevent this from happening.
Does anyone of you guys have experience with OracleAQ and XA?
Could this be a problem with the XA transaction manager (I use Atomikos)?
Thanks for your help,
Chris

We found a solution to the problem.
It consists of four steps.
Step 1: Use the latest Oracle client libraries
The first step ist to use the lastest Oracle 12c client libraries.
There were significant improvements in the ojdbc8.jar, e.g. they use stored procedures to get the metadata now.
This increased the throughput to about 10 msgs/s.
Step 2: Setup connection pooling correctly
The second step was improving the connection pooling according to article http://thinkfunctional.blogspot.ch/2012/05/atomikos-and-oracle-aq-pooling-problem.html:
<bean id="oraXaDataSource" primary="true"
class="oracle.jdbc.xa.client.OracleXADataSource" destroy-method="close">
<property name="URL" value="${oracle.url}" />
<property name="user" value="${oracle.username}" />
<property name="password" value="${oracle.password}" />
</bean>
<bean id="atomikosOraclaDataSource"
class="org.springframework.boot.jta.atomikos.AtomikosDataSourceBean">
<property name="uniqueResourceName" value="xaOracleAQ" />
<property name="xaDataSource" ref="oraXaDataSource" />
<property name="poolSize" value="5" />
</bean>
<bean id="OracleAQConnectionFactory" class="oracle.jms.AQjmsFactory" factory-method="getConnectionFactory">
<constructor-arg ref="atomikosOraclaDataSource" />
</bean>
This configuration alone resultet in exceptions because of 'auto-commit' of the Oracle connection.
Step 3: Set autoCommit to false
The third step was to set the following java system property (see https://docs.oracle.com/database/121/JAJDB/oracle/jdbc/OracleConnection.html#CONNECTION_PROPERTY_AUTOCOMMIT):
-DautoCommit=false
But then the throughput went down to 1 to 2 msg/s again.
Step 4: Set oracle.jdbc.autoCommitSpecCompliant to false
The last step was to set the following java system property (see https://docs.oracle.com/database/121/JAJDB/oracle/jdbc/OracleConnection.html#CONNECTION_PROPERTY_AUTO_COMMIT_SPEC_COMPLIANT):
-Doracle.jdbc.autoCommitSpecCompliant=false
Now we get a throughput of 80 msgs/s.
Conclusion
The setting of oracle.jdbc.autoCommitSpecCompliant to false is not elegant, but solved the problem.
We have to investigate further to see how we can get around this problem without setting oracle.jdbc.autoCommitSpecCompliant to false.
Many thanks to Dani Steinmann (stonie) for the help!
P.S.: I updated the sample project under GitHub - https://github.com/cknzl2014/springio-ora-xa/tree/atomikos.

First of all you should be sure that you use pool for JDBC connections.
On the other hand you may consider to use ChainedTransactionManager isntead of XA for two target transaction managers - JMS and JDBC.
Also see some information in the JDBC extensions project.
There is also some Oracle AQ API as well in that project.

Related

Performance Issue using DefaultMessageListenerContainer with CachingConnectionFactory

In our application we are using SingleConnectionFactory with DefaultMessageListenerContainer consuming from IBM MQ server, performance wise the app is doing pretty good, however on MQ end our application is opening too many new connections randomly. Here is the current config we have.
Try with this instead of caching or single:
<bean id="connectionFactory" class="org.springframework.jms.connection.DelegatingConnectionFactory">
<property name="targetConnectionFactory" ref="primaryRawInputConnectionFactory" />
<property name="shouldStopConnections" value="true"/>
</bean>

Synchronizing JMS Message Queue and JDBC transaction- Is Transaction proxy needed

We need to persist message from a JMS Queue to a database in a transaction ensuring that JMS message is not acknowledged in case any error is thrown during DB persist.
Based on the solution provided here- Transaction handling while using message driven channel adapter & service activator
Below is the approach I arrived at
<int-jms:message-driven-channel-adapter id="jmsIn"
transaction-manager="transactionManager"
connection-factory="sConnectionFactory"
destination-name="emsQueue"
acknowledge="transacted" channel="jmsInChannel"/>
<int:channel id=" jmsInChannel " />
<bean id="dataSource"
class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="org.hsqldb.jdbcDriver" />
<property name="url" value="jdbc:hsqldb:mem:testdb" />
</bean>
<!-- Transaction manager for a datasource -->
<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource" />
</bean>
<bean id="sconnectionFactory" class="org.springframework.jms.connection.TransactionAwareConnectionFactoryProxy">
<property name="targetConnectionFactory">
<bean class="project.TestConnectionFactory">
</bean>
</property>
<property name="synchedLocalTransactionAllowed" value="true" />
</bean>
Please confirm if the understanding is correct. Also, have the below queries
Is the TransactionAwareConnectionFactoryProxy necessary in this scenario
JMS Queue and JDBC are two separate transactional resources. Is injecting jdbc transaction manager into JMS adapter as shown in this example equivalent to using a ChainedTransactionManager(chaining JMS Transaction Mananger and JDBC transaction manager)
For TransactionAwareConnectionFactoryProxy, please, consult its JavaDocs:
* Proxy for a target CCI {#link javax.resource.cci.ConnectionFactory}, adding
* awareness of Spring-managed transactions. Similar to a transactional JNDI
* ConnectionFactory as provided by a Java EE server.
*
* <p>Data access code that should remain unaware of Spring's data access support
* can work with this proxy to seamlessly participate in Spring-managed transactions.
* Note that the transaction manager, for example the {#link CciLocalTransactionManager},
* still needs to work with underlying ConnectionFactory, <i>not</i> with this proxy.
I'm not sure how is that related to your request about JMS+DB transaction, but I think you should worry about transaction manager which supports both those transactional resources.
For this purpose Spring suggests JtaTransactionManager for XA transactions.
Or you can consider to use ChainedTransactionManager based on the Dave Syer article: http://www.javaworld.com/article/2077963/open-source-tools/distributed-transactions-in-spring--with-and-without-xa.html
UPDATE
If local transaction are fine for you, you really can go ahead with the TransactionAwareConnectionFactoryProxy and its synchedLocalTransactionAllowed to true. And, of course, use that DataSourceTransactionManager directly from the <int-jms:message-driven-channel-adapter>.
Otherwise you have to go with the ChainedTransactionManager. This solution really adds some overhead over the first TransactionAwareConnectionFactoryProxy-based one.

Implement a connection between my application (Spring integration) with IBM-MQ - High Availability

I am trying to implement a connection between my application (Spring integration) with IBM-MQ, I did see the question spring-integration-support-for-clustered-high-availability-ibm-mq-manager, but in my case each host has different "queueManager" and "channels", it means I will have must have a configuration like follows but, the queueManager and channels properties support a String not a list values:
<bean id="connectionFactory" class="com.ibm.mq.jms.MQQueueConnectionFactory">
<property name="connectionNameList" value="host1(6464),host1(6464)" />
<property name="clientReconnectTimeout" value="15000" />
<property name="transportType" value="1" />
<property name="queueManager" value="QM1, QM1," />
<property name="channel" value="channel1,channel1"/>
</bean>
The simplest thing would be to define a channel with the same name on both hosts and let the client try host1 first and then host2 using connectionNameList. In this setup it would always prefer host1. You would need to specify a queueManager that is blank so that the client will accept the queue manager it connects to. Example follows:
<property name="queueManager" value="" />
Another option that was pointed out in a comment from Morag on the other post you linked to is to use a CCDT (Client channel definition table).
See Using a client channel definition table with IBM WebSphere MQ classes for JMS. The property name is CCDTURL
The CCDT can have multiple CLNTCONN channel entries with different channel names all having the same QMNAME, this is called a Queue Manager Group, you would then specify the queueManager property as *QMNAME, this will allow the app to connect to which ever queue manager you are directed to with out regard to the actual queue manager name. There are other parameters of the CLNTCONN listed at the bottom of the link I provided that can help you to control if one queue manager is preferred over the other(s) as well as which queue manager to connect to if a reconnect is required.

CAS 4.x, & LDAP Attributes

I have the same problem as several other questions here, none of which have really been answered; that is, with CAS 4.x (4.2.6 actually) I cannot get LDAP attributes to return to the client application.
Question 1 This seems like overkill; custom code to get around what is a "simple" problem.
Question 2 Had already done this, and it didn't work.
So, now it's my turn to ask... is there some magic to making it work? We've used 3.5 for a long time without any issues. I'm trying to convert those settings to the 4.x Maven overlayer and new context configuration of 4.x, and it's not doin' it.
I can see in the logs that CAS is requesting, and getting the properties I'm looking for from LDAP. But they are not getting put in the token back to the application.
What more needs to be done beyond what the Apereo documentation lays out? I'm thinking it's the attribute repository maybe??? If something would help you help me through this, just ask: Config, Logs (redacted of course)... anything.
Thanks.
Update #1. Here is my resolvers list. NOTE: I keep code/settings in place commented out until I get it to work before I clean stuff out.
<util:map id="authenticationHandlersResolvers">
<!--
<entry key-ref="proxyAuthenticationHandler" value-ref="proxyPrincipalResolver" />
<entry key-ref="primaryAuthenticationHandler" value-ref="primaryPrincipalResolver" />
-->
<!--<entry key-ref="ldapAuthenticationHandler" value-ref="primaryPrincipalResolver" /> -->
<entry key-ref="ldapAuthenticationHandler" value="#{null}" />
</util:map>
Update #2
I've done more testing, and still am unsuccessful. I think, it's coming down to the principalAttributeMap of the LdapAuthenticationHandler not working, OR, the attributeReleasePolicy of the serviceRegistryDao... anyone see any issues in this config?
<bean id="ldapAuthenticationHandler" class="org.jasig.cas.authentication.LdapAuthenticationHandler"
p:principalIdAttribute="sAMAccountName"
c:authenticator-ref="authenticator"
>
<property name="principalAttributeMap">
<map>
<entry key="cn" value="cn" />
<entry key="mail" value="Email" />
<entry key="memberOf" value="Groups" />
<entry key="displayName" value="displayName" />
</map>
</property>
</bean>
<bean id="serviceRegistryDao" class="org.jasig.cas.services.InMemoryServiceRegistryDaoImpl">
<property name="registeredServices">
<list>
<bean class="org.jasig.cas.services.RegexRegisteredService"
p:id="5"
p:name="All Servicesxxx"
p:description="Allow connections for all services and protocols"
p:serviceId="^(http|https|imaps)://.*"
p:evaluationOrder="5"
>
<property name="attributeReleasePolicy">
<bean class="org.jasig.cas.services.ReturnAllAttributeReleasePolicy" />
</property>
</bean>
</list>
</property>
</bean>
To quote from CAS Security Guide:
All communication with the CAS server MUST occur over a secure channel
(i.e. TLSv1). There are two primary justifications for this
requirement:
The authentication process requires transmission of security
credentials.
The CAS ticket-granting ticket is a bearer token.
Since the disclosure of either data would allow impersonation attacks, it’s
vitally important to secure the communication channel between CAS
clients and the CAS server.
Practically, it means that all CAS urls must use HTTPS, but it also
means that all connections from the CAS server to the application must
be done using HTTPS:
when the generated service ticket is sent back to the application on the “service” url when a
proxy callback url is called.
HTTPS is a must in CAS. However, there are possibilities to disable it but it is highly recommended not to do so. If you have struggle to handle the SSL configuration, I recommend you read my question where I explain in detail how to deal with the keystores.

how to add a retry advice with jms:message-driven-channel-adapter

I am new to spring integration. My requirement is that if there is a connection problem to the jms q then it should try to connect 3 times then log it and exit the process. I am not able to do it. It throws an error saying it needs the ref attribute for service:activator. But I don't have/know reference of which class to provide here. Is there any other way of doing it?
<int-jms:message-driven-channel-adapter id="msgIn" channel="toRoute" container="messageListenerContainer" />
<int:service-activator id="service" input-channel="toRoute" >
<int:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.RequestHandlerRetryAdvice">
<property name="recoveryCallback">
<bean class="org.springframework.integration.handler.advice.ErrorMessageSendingRecoverer">
<constructor-arg ref=“errorChannel" />
</bean>
</property>
</bean>
</request-handler-advice-chain>
</int:service-activator>
You seem to have completely misunderstood what the framework does.
The service-activator gets a message when one is received from JMS (which implies the connection is good), and needs "something" (a reference to a bean or an expression) to invoke as a result of receiving that message.
The retry advice is to retry calling that service if it fails to process the message for some reason. It is unrelated to whatever is the source of the message (JMS in this case).
It's not clear why you are trying to use Spring Integration for something as simple as testing whether a JMS broker is available.
Perhaps if you can provide some larger context someone might be able to help.

Resources