Jetty Clustering via the Database Overrides Web Application Specified Data Source Pool - web

Question

The goal is to create a clustered environment for a web application that is currently running in a single Jetty instance. When clustering configuration is added, it appears to prevent the applications data source pool from initializing. Attaching a remote debugging session and stepping through the code shows the application hanging on startup while waiting for a connection to free up from the pool. Inspecting the details of the pool shows no connections have been created. c3p0 is used for the pooling implementation. When the server starts without the clustering configuration, c3p0 generates a log message showing it is initializing. When started with clustering configured, the log message is not seen. Leading me to believe the application data source pool is never initialized because the clustering data source pool is usurping it in some way.
To get the clustering behavior I added the following to %JETTY_HOME%/etc/jetty.xml.
<New id="DSClustering" class="org.eclipse.jetty.plus.jndi.Resource">
<Arg></Arg>
<Arg>jdbc/DSClustering</Arg>
<Arg>
<New class="com.mchange.v2.c3p0.ComboPooledDataSource">
<Set name="driverClass">oracle.jdbc.OracleDriver</Set>
<Set name="jdbcUrl">jdbc:oracle:thin:#xxxxx:1521:xe</Set>
<Set name="User">xxxx</Set>
<Set name="Password">xxxx</Set>
</New>
</Arg>
</New>
<Set name="sessionIdManager">
<New id="jdbcidmgr" class="org.eclipse.jetty.server.session.JDBCSessionIdManager">
<Arg><Ref id="Server"/></Arg>
<Set name="workerName">jetty1</Set>
<Set name="DatasourceName">jdbc/DSClustering</Set>
<Set name="scavengeInterval">60</Set>
</New>
</Set>
<Call name="setAttribute">
<Arg>jdbcIdMgr</Arg>
<Arg><Ref id="jdbcidmgr"/></Arg>
</Call>
And added the following to the applications jetty-web.xml
<Set name="sessionHandler">
<New class="org.eclipse.jetty.server.session.SessionHandler">
<Arg>
<New class="org.eclipse.jetty.server.session.JDBCSessionManager">
<Set name="idManager">
<Ref id="jdbcidmgr"/>
</Set>
</New>
</Arg>
</New>
</Set>
The applications data source pool is configured as a Spring bean -
<bean id="sysContextAwareDataSource" class="com.mycompany.datasource.SysContextAwareDataSource">
<property name="targetDataSource" ref="myPoolDataSource"/>
<property name="connectionWaitLoggingThreshold" value="1000"/>
</bean>
<bean id="myPoolDataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">
<property name="driverClass" value="${jdbc.driverClassName}"/>
<property name="jdbcUrl" value="${jdbc.url}"/>
<property name="user" value="${jdbc.username}"/>
<property name="password" value="${jdbc.password}"/>
<property name="acquireIncrement" value="${jdbc.pool.acquireIncrement}"/>
<property name="minPoolSize" value="${jdbc.pool.minPoolSize}"/>
<property name="maxPoolSize" value="${jdbc.pool.maxPoolSize}"/>
<property name="initialPoolSize" value="${jdbc.pool.initialPoolSize}"/>
<property name="acquireRetryAttempts" value="${jdbc.pool.acquireRetryAttempts}"/>
<property name="testConnectionOnCheckin" value="${jdbc.pool.testConnectionOnCheckin}"/>
<property name="idleConnectionTestPeriod" value="${jdbc.pool.idleConnectionTestPeriod}"/>
<property name="preferredTestQuery" value="${jdbc.pool.preferredTestQuery}"/>
<property name="maxIdleTime" value="${jdbc.pool.maxIdleTime}"/>
<property name="acquireRetryDelay" value="${jdbc.pool.acquireRetryDelay}"/>
<property name="maxStatements" value="${jdbc.pool.maxStatements}"/>
<property name="maxStatementsPerConnection" value="${jdbc.pool.maxStatementsPerConnection}"/>
</bean>
<bean id="userDAO" class="com.mycompany.dataaccess.UserDAOImpl">
<property name="sessionFactory" ref="sessionFactory"/>
<property name="dataSource" ref="transactionAwareDataSourceProxy"/>
</bean>
Environment: jetty 8.1.9, Oracle 11g, Windows 7 Enterprise, JDK 1.6.0_38

Answer 1

Figured it out! This was caused because I had 2 ojdbc-10.2.0.4.0.jars that were being accessed. To get the session management DB implementation working I added an ojdbc jar to JETTY_HOME\lib\ext. This provided the Oracle classes to the Jetty server. There’s also a ojdbc jar contained in the applications lib directory, which is used by the application. The problem occurred because the ojdbc jar is a “sealed” jar. Meaning, once a class is loaded from one of the jars, an attempt to load the same class from another jar causes a security exception to be thrown. So the Jetty server would load an Oracle class from the lib/ext version of the jar. Then the application would try to load the same class from its ojdbc jar, causing the security exception. It took a while to figure out because the c3p0 class that received the exception, silently swallowed the exception and then tried to establish the connection again. This was happening on another thread then the one I was looking at. That was the thread that was establishing connections for the pool, while I was looking at the thread that was waiting for connections from the pool. Argh!