Why does Spring batch multithreaded step break additional repository hibernate queries in ItemProcessor? - multithreading

Question

I working on transforming a gigantic DB2 schema with about 30-40 tables into a streamlined JSON format via Spring Batch. My process works just fine with one thread, but as soon as I increase the thread pool size to enable multi-threading my step, my ItemProcessor breaks down with infuriatingly cryptic and incomprehensible errors.
I just don't understand how my Processor could possibly not be thread-safe: I'm not maintaining state anywhere, I'm just making a few extra repository calls to enrich the data, since I couldn't get the reader to pull in everything I need. And one such repository call is throwing an ArrayIndexOutOfBoundsException! I even added a #Transactional annotation to my processor and called it using Java's multithreaded java.util.concurrent.ExecutorService - works just fine there too.
I just can't seem to figure out why my Spring Batch multithreaded step would break my Item Processor's simple repository queries. I even intermittently get LazyLoading exceptions! Isn't an Item Processor supposed to be wrapped in a transaction? And intermittently I see absolutely nonsensical exceptions about class-cast exceptions, where a one to many mapping is returning the wrong Entity type! Again, this all works perfectly for the same data set in one thread. Is my configuration wrong?
java.lang.ArrayIndexOutOfBoundsException: 1765
at org.hibernate.engine.internal.EntityEntryContext.reentrantSafeEntityEntries(EntityEntryContext.java:319) ~[hibernate-core-5.3.14.Final.jar!/:5.3.14.Final]
at org.hibernate.engine.internal.StatefulPersistenceContext.reentrantSafeEntityEntries(StatefulPersistenceContext.java:1156) ~[hibernate-core-5.3.14.Final.jar!/:5.3.14.Final]
at org.hibernate.event.internal.AbstractFlushingEventListener.prepareEntityFlushes(AbstractFlushingEventListener.java:145) ~[hibernate-core-5.3.14.Final.jar!/:5.3.14.Final]
at org.hibernate.event.internal.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:83) ~[hibernate-core-5.3.14.Final.jar!/:5.3.14.Final]
at org.hibernate.event.internal.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:46) ~[hibernate-core-5.3.14.Final.jar!/:5.3.14.Final]
at org.hibernate.internal.SessionImpl.autoFlushIfRequired(SessionImpl.java:1433) ~[hibernate-core-5.3.14.Final.jar!/:5.3.14.Final]
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1519) ~[hibernate-core-5.3.14.Final.jar!/:5.3.14.Final]
at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1538) ~[hibernate-core-5.3.14.Final.jar!/:5.3.14.Final]
at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1506) ~[hibernate-core-5.3.14.Final.jar!/:5.3.14.Final]
at org.hibernate.query.Query.getResultList(Query.java:132) ~[hibernate-core-5.3.14.Final.jar!/:5.3.14.Final]
at org.springframework.data.jpa.repository.query.JpaQueryExecution$CollectionExecution.doExecute(JpaQueryExecution.java:129) ~[spring-data-jpa-2.1.14.RELEASE.jar!/:2.1.14.RELEASE]
at org.springframework.data.jpa.repository.query.JpaQueryExecution.execute(JpaQueryExecution.java:91) ~[spring-data-jpa-2.1.14.RELEASE.jar!/:2.1.14.RELEASE]
at org.springframework.data.jpa.repository.query.AbstractJpaQuery.doExecute(AbstractJpaQuery.java:136) ~[spring-data-jpa-2.1.14.RELEASE.jar!/:2.1.14.RELEASE]
at org.springframework.data.jpa.repository.query.AbstractJpaQuery.execute(AbstractJpaQuery.java:125) ~[spring-data-jpa-2.1.14.RELEASE.jar!/:2.1.14.RELEASE]
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:605) ~[spring-data-commons-2.1.14.RELEASE.jar!/:2.1.14.RELEASE]
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.lambda$invoke$3(RepositoryFactorySupport.java:595) ~[spring-data-commons-2.1.14.RELEASE.jar!/:2.1.14.RELEASE]
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor$$Lambda$541/246846952.get(Unknown Source) ~[na:na]
at org.springframework.data.repository.util.QueryExecutionConverters$$Lambda$540/1619129136.apply(Unknown Source) ~[na:na]
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:595) ~[spring-data-commons-2.1.14.RELEASE.jar!/:2.1.14.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.12.RELEASE.jar!/:5.1.12.RELEASE]
at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:59) ~[spring-data-commons-2.1.14.RELEASE.jar!/:2.1.14.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.12.RELEASE.jar!/:5.1.12.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor$$Lambda$539/1493625851.proceedWithInvocation(Unknown Source) ~[na:na]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295) ~[spring-tx-5.1.12.RELEASE.jar!/:5.1.12.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) ~[spring-tx-5.1.12.RELEASE.jar!/:5.1.12.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.12.RELEASE.jar!/:5.1.12.RELEASE]
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139) ~[spring-tx-5.1.12.RELEASE.jar!/:5.1.12.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.12.RELEASE.jar!/:5.1.12.RELEASE]
at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:144) ~[spring-data-jpa-2.1.14.RELEASE.jar!/:2.1.14.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.12.RELEASE.jar!/:5.1.12.RELEASE]
at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$ExposeRepositoryInvocationInterceptor.invoke(CrudMethodMetadataPostProcessor.java:364) ~[spring-data-jpa-2.1.14.RELEASE.jar!/:2.1.14.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.12.RELEASE.jar!/:5.1.12.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93) ~[spring-aop-5.1.12.RELEASE.jar!/:5.1.12.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.12.RELEASE.jar!/:5.1.12.RELEASE]
at org.springframework.data.repository.core.support.SurroundingTransactionDetectorMethodInterceptor.invoke(SurroundingTransactionDetectorMethodInterceptor.java:61) ~[spring-data-commons-2.1.14.RELEASE.jar!/:2.1.14.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.12.RELEASE.jar!/:5.1.12.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) [spring-aop-5.1.12.RELEASE.jar!/:5.1.12.RELEASE]
...
Here's my properties file:
#I don't want all my jobs run, I want to specify which one via an environment variable
spring.batch.job.enabled=false
spring.main.allow-bean-definition-overriding=true
spring.main.web-application-type=none
#If I set this count to 1, everything works just fine.
process.thread.count=4
process.page.size=25
hibernate.jdbc.fetch_size=100
process.chunk.size=25
process.publish.limit=200
My configuration class:
#Configuration
public class FullProductSyncBatchJobConfig {
#Bean
#StepScope
public ItemStreamReader<LegacyProduct> productReader(
LegacyProductRepository legacyProductRepository,
#Value("#{jobParameters[pageSize]}") Integer pageSize,
#Value("#{jobParameters[limit]}") Integer limit) {
RepositoryItemReader<LegacyProduct> legacyProductRepositoryReader = new RepositoryItemReader<>();
legacyProductRepositoryReader.setRepository(legacyProductRepository);
legacyProductRepositoryReader.setMethodName("findAllRelevantProducts");
legacyProductRepositoryReader.setSort(new HashMap<String, Sort.Direction>() {{
put("id.guid", Sort.Direction.ASC);
put("id.modelNumber", Sort.Direction.ASC);
}});
legacyProductRepositoryReader.setPageSize(pageSize);
if(limit > 0) legacyProductRepositoryReader.setMaxItemCount(limit);
legacyProductRepositoryReader.setSaveState(false);
return legacyProductRepositoryReader;
}
#Bean
#StepScope
public ItemProcessor<LegacyProduct, StreamlinedProduct> productDocumentBuilder(
SupplierRepository supplierRepository) {
return new StreamlinedProductBuilder(supplierRepository);
}
#Bean
#StepScope
public ItemWriter<StreamlinedProduct> productDocumentPublisher(
GcpPubsubPublisherService publisherService) {
return new StreamlinedProductPublisher(publisherService);
}
#Bean
public Step fullProductSync(ItemStreamReader<LegacyProduct> productReader,
ItemProcessor<LegacyProduct, StreamlinedProduct> productDocumentBuilder,
ItemWriter<StreamlinedProduct> productDocumentPublisher,
StepBuilderFactory stepBuilderFactory,
TaskExecutor syncProcessThreadPool,
PlatformTransactionManager jpaTransactionManager,
#Value("${process.chunk.size:100}") Integer chunkSize,
#Value("${process.publish.timeout.retry.limit:2}") int timeoutRetryLimit,
#Value("${process.failure.limit:20}") int maximumProcessingFailures) {
return stepBuilderFactory.get("fullProductSync")
.transactionManager(jpaTransactionManager)
.<GtinVendor, AbstractProduct>chunk(chunkSize)
.reader(productReader)
.processor(productDocumentBuilder)
.writer(productDocumentPublisher)
.faultTolerant()
.retryLimit(timeoutRetryLimit)
.retry(TimeoutException.class)
.skipPolicy(new SyncProcessSkipPolicy(maximumProcessingFailures))
.listener(new SyncProcessSkipListener()) // <== just logs them right now
.taskExecutor(syncProcessThreadPool)
.build();
}
#Bean
public Job fullProductSyncJob(Step fullProductSync,
JobBuilderFactory jobBuilderFactory) {
return jobBuilderFactory.get("fullProductSync")
.start(fullProductSync)
.build();
}
}
And my processor class:
#Slf4j
public class StreamlinedProductBuilder implements ItemProcessor<LegacyProduct, StreamlinedProduct> {
private DimensionRepository dimensionRepository;
public LegacyProductBuilder(DimensionRepository dimensionRepository) {
this.dimensionRepository = dimensionRepository;
}
public StreamlinedProduct process(LegacyProduct legacyProduct) {
StreamlinedProduct streamlinedProduct = new StreamlinedProduct();
streamlinedProduct.setPrimarySupplierNumber(parsePrimarySupplierNumber(product.getSuppliers()));
attachProductDimensions(streamlinedProduct, legacyProduct);
return streamlinedProduct;
}
private int parsePrimarySupplierNumber(List<Supplier> suppliers) {
/* This intermittently throws a ClassCastException when using multiple threads,
* saying that a Description can't be cast a Supplier... WHAT??! HOW???! How does
* getSuppliers() ever return a list of a completely different one-to-many entity????
*/
for(Supplier supplier : suppliers) {
if(supplier.isPrimary()) return supplier.getId();
}
return -1;
}
private void attachProductDimensions(LegacyProduct legacyProduct,
StreamlinedProduct streamlinedProduct) {
// The following line occasionally throws the ArrayOutOfBounds index I mentioned above. WHY?
// Works just fine in one thread...
List<Dimension> dimensions= dimensionRepository.findByProductIdAndModel(
legacyProduct.getId().getGuid(), legacyProduct.getId().getModelNumber());
Map<String, Double> dimensionsAsMap = new HashMap<>();
for (Description description : descriptions ) {
dimensionsAsMap.put(dimension.getName(), dimension.getValue());
}
streamlinedProduct.setDimensions(dimensionsAsMap);
}
}
My repository:
#Repository
public interface DimensionRepository extends PagingAndSortingRepository<ProductDimension, DimensionCompositePK> {
#Transactional(isolation = Isolation.READ_UNCOMMITTED) // <== fails with or without this
#Query(value = "select d.name as name, d.value as value " +
"from {h-schema}product p left join {h-schema}dimension d " +
"on p.guid = d.product_guid and p.model_number = d.product_model_number " +
"where p.guid = :guid and p.model_number = :model", nativeQuery = true)
List<Dimension> findVendorsByGtinGlnCountry(#Param("guid") String guid,
#Param("modelNumber") Integer model);
}

Answer 1

I don't know what was going on or why I was seeing exceptions suggesting Spring was trying to fill certain oneToMany relationships with the wrong Entity class from a different oneToMany relationship.
But regardless, adopting the Spring Batch driver-query pattern seems to have fixed things for me. I was under the impression that the same transaction was shared for a given chunk across the reader, processor, and writer. But I think that assumption was incorrect - each chunk of each stage has its own transaction.
So now my reader simply pulls in all relevant IDs of my entity class, and the processor goes out and fetches the oneToMany relationships for that entity.