In our Spring 4 project we would like to have database transactions that involve Redis and Hibernate. Whenever Hibernate fails, for example due to optimistic locking, the Redis transaction should be aborted as well.
This seems to work for
Single-threaded transaction execution.
Multi-threaded transaction execution, as long as the transaction only includes a single Redis call.
Multi-threaded transaction execution with multiple Redis calls, if Hibernate is excluded from our configuration.
As soon as a transaction includes multiple Redis calls, and Hibernate is configured to take part in the transactions, there seems to be a problem with connection binding and multithreading. Threads are stuck at RedisConnectionUtils.bindConnection(), probably since the JedisPool runs out of connections.
This can be reproduced as follows.
#Service
public class TransactionalService {
#Autowired
#Qualifier("redisTemplate")
private RedisTemplate<String, Object> redisTemplate;
#Transactional
public void processTask(int i){
redisTemplate.convertAndSend("testChannel", new Message());
redisTemplate.convertAndSend("testChannel", new Message());
}
}
We use a ThreadPoolTaskExecutor having a core pool size of 50 to simulate multithreaded transactions.
#Service
public class TaskRunnerService {
#Autowired
private TaskExecutor taskExecutor;
#Autowired
private TransactionalService transactionalService;
public void runTasks() {
for (int i = 0; i < 100; i++) {
final int j = i;
taskExecutor.execute(new Runnable() {
#Override
public void run() {
transactionalService.processTask(j);
}
});
}
}
}
Running this results in all taskExecutor threads hanging in JedisPool.getResource():
"taskExecutor-1" - Thread t#18
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <1b83c92c> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:524)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:438)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
at redis.clients.util.Pool.getResource(Pool.java:40)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:84)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:10)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:90)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:143)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:41)
at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:128)
at org.springframework.data.redis.core.RedisConnectionUtils.bindConnection(RedisConnectionUtils.java:66)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:175)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:152)
at org.springframework.data.redis.core.RedisTemplate.convertAndSend(RedisTemplate.java:675)
at test.TransactionalService.processTask(TransactionalService.java:23)
at test.TransactionalService$$FastClassBySpringCGLIB$$9b3de279.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:262)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)
at test.TransactionalService$$EnhancerBySpringCGLIB$$a1b3ba03.processTask(<generated>)
at test.TaskRunnerService$1.run(TaskRunnerService.java:28)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- locked <7d528cf7> (a java.util.concurrent.ThreadPoolExecutor$Worker)
Redis Config
#Configuration
public class RedisConfig {
#Bean
public JedisConnectionFactory jedisConnectionFactory() {
JedisConnectionFactory jedisConnectionFactory = new JedisConnectionFactory();
jedisConnectionFactory.setPoolConfig(new JedisPoolConfig());
return jedisConnectionFactory;
}
#Bean
public Jackson2JsonRedisSerializer<Object> jackson2JsonRedisSerializer() {
Jackson2JsonRedisSerializer jackson2JsonRedisSerializer = new Jackson2JsonRedisSerializer(Object.class);
jackson2JsonRedisSerializer.setObjectMapper(objectMapper());
return jackson2JsonRedisSerializer;
}
#Bean
public StringRedisSerializer stringRedisSerializer() {
return new StringRedisSerializer();
}
#Bean
public RedisTemplate<String, Object> redisTemplate() {
RedisTemplate<String, Object> redisTemplate = new RedisTemplate();
redisTemplate.setConnectionFactory(jedisConnectionFactory());
redisTemplate.setKeySerializer(stringRedisSerializer());
redisTemplate.setValueSerializer(jackson2JsonRedisSerializer());
redisTemplate.setEnableTransactionSupport(true);
return redisTemplate;
}
#Bean
public ObjectMapper objectMapper() {
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY);
objectMapper.enableDefaultTyping(ObjectMapper.DefaultTyping.NON_FINAL);
return objectMapper;
}
}
Hibernate Config
#EnableTransactionManagement
#Configuration
public class HibernateConfig {
#Bean
public LocalContainerEntityManagerFactoryBean admin() {
LocalContainerEntityManagerFactoryBean entityManagerFactoryBean = new LocalContainerEntityManagerFactoryBean();
entityManagerFactoryBean.setJpaVendorAdapter(new HibernateJpaVendorAdapter());
entityManagerFactoryBean.setPersistenceUnitName("test");
return entityManagerFactoryBean;
}
#Bean
public JpaTransactionManager transactionManager(
#Qualifier("admin") LocalContainerEntityManagerFactoryBean entityManagerFactoryBean) {
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setEntityManagerFactory(entityManagerFactoryBean.getObject());
transactionManager.setDataSource(entityManagerFactoryBean.getDataSource());
return transactionManager;
}
}
Is this a bug in spring-data-redis or is something wrong in our configuration?
I found your question (coincidentally) right before I hit the exact same issue using opsForHAsh and putting many keys. A thread dump confirmed it.
What I found helped to get me going was to increase the thread pool in my JedisPoolConfig. I set it as follows, to 128, and that got me on my way again.
#Bean
JedisPoolConfig jedisPoolConfig() {
JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
jedisPoolConfig.setMaxTotal(128);
return jedisPoolConfig;
}
I assume the pool was too small in my case, and all the threads were in use for my transaction, so were waiting indefinitely. Setting to total to 128 allowed me to continue. Try setting your config to a maxTotal that makes sense for your application.
I had a very similar problem but bumping the maxTotal threads bothered me if the threads really weren't being released. Instead I had some code that rapidly did a get and then a set. I put this in a SessionCallback and it behaved much better. Hope that helps.
Related
I'm having trouble grasping the concepts behind multithreading in netty, EventLoopGroup (MultithreadEventLoopGroup), MultithreadEventExecutorGroup, DefaultEventExecutorGroup
I am trying to understand how the server handles multiple clients simultaneously sending requests that will execute some business logic and CRUD operations that add to RTT. Below is my netty server code which works, but I am trying to understand exactly how it will work with concurrent users and multiple open channels.
I have a simple ServerBootstrap
#Component
#RequiredArgsConstructor
public class SocketServer {
private final ContextAwareLogger logger;
private final ServerInitializer serverInitializer;
private final NioEventLoopGroup bossGroup;
private final NioEventLoopGroup workerGroup;
private Channel mainChannel;
#PostConstruct
public void start() {
try {
ServerBootstrap bootstrap = init();
mainChannel = bootstrap.bind(8484).sync().channel(); // save the main channel so we can cleanly close it when app is shutdown
logger.info("Netty server started...");
} catch (Exception e) {
e.printStackTrace();
}
}
#PreDestroy
public void stop() throws InterruptedException {
logger.info("Shutting down Netty server");
bossGroup.shutdownGracefully().sync();
workerGroup.shutdownGracefully().sync();
mainChannel.closeFuture().sync();
logger.info("Netty Server shutdown complete.");
}
private ServerBootstrap init() {
return new ServerBootstrap()
.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.class)
.option(ChannelOption.SO_BACKLOG, 5000)
.option(ChannelOption.TCP_NODELAY, true)
.option(ChannelOption.SO_KEEPALIVE, true)
.childHandler(serverInitializer);
}
}
ChannelInitializer:
#Component
#RequiredArgsConstructor
public class ServerInitializer extends ChannelInitializer<SocketChannel> {
private final PacketDecoder packetDecoder;
private final ServerHandler serverHandler;
private final PacketEncoder packetEncoder;
#Override
protected void initChannel(SocketChannel socketChannel) throws Exception {
socketChannel.pipeline()
.addLast("decoder", packetDecoder) // ByteArrayDecoder
.addLast("encoder", packetEncoder) // ByteArrayEncoder
.addLast("inbound", serverHandler); // ChannelInboundHandlerAdapter
}
}
ChannelInboundHandlerAdapter:
#Component
#Sharable
public class ServerHandler extends ChannelInboundHandlerAdapter {
#Autowired
private SomeService someService;
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
// contains db access
byte[] accept = someService.validateClient(ctx.channel());
ctx.channel().writeAndFlush(accept);
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
// may contain db access
byte[] response = someService.processPacket(ctx.channel(), msg));
ctx.channel().writeAndFlush(response);
}
}
Now when a client connects, I understand that a new Channel will opened and the handlers will be reused. Requirement is each client request/response needs to process immediately without waiting for some other client's CRUD operations to finish.
Are my channelRead and channelActive, etc, async because I am using NioEventLoopGroup (ie will each client's channel operations be run independent of each other)?
If a single client sends multiple requests in series, are they guaranteed to be handled in the same order?
Do I need to specific DefaultEventExecutorGroup for my inbound handler? (https://stackoverflow.com/a/28305019/1738539)
You either would need to use a DefaultEventExecutorGroup for your ServerHandler or dispatch the validateClient(...) / processPacket(...) to your own ThreadPool. Failing todo so will cause the EventLoop thread to block and so no other IO can be processed for this EventLoop until the blocking operation completes.
In my Spring batch job, I'm trying to share data between steps using JobExecutionContext, which works only if i keep the steps single threaded as follows:
#EnableTask
#EnableBatchProcessing
#Configuration
#PropertySource(value = {"classpath:application.properties"})
public class Config{
private static final HashMap<String,Object> OVERRIDDEN_BY_EXPRESSION = null;
private static final String QUERY = "SELECT * FROM \"Config\"";
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
private MongoTemplate mongoTemplate;
#Autowired
EntityManager em;
#Autowired
DataSource dataSource;
/*Config Step*/
#Bean
public JdbcCursorItemReader<BatchConfig> configReader(DataSource dataSource) {
JdbcCursorItemReader<BatchConfig> config = new JdbcCursorItemReader<>();
config.setDataSource(dataSource);
config.setSql(QUERY);
config.setRowMapper(new BatchRowMapper());
return config;
}
#Bean
public ItemWriter<BatchConfig> itemWriter() {
return new ItemWriter<BatchConfig>() {
private StepExecution stepExecution;
#Override
public void write(List<? extends BatchConfig> items) {
ExecutionContext stepContext = this.stepExecution.getExecutionContext();
for (BatchConfig item : items) {
HashMap<String, Object> table = new HashMap<>();
table.put("date", item.getDate_time());
table.put("size", item.getSize());
System.out.println(table);
stepContext.put(item.getName(), table);
}
}
#BeforeStep
public void saveStepExecution(StepExecution stepExecution) {
this.stepExecution = stepExecution;
}
};
}
#Bean
public Step stepConfig(JdbcCursorItemReader<BatchConfig> configReader) throws Exception {
return stepBuilderFactory.get("stepConfig")
.<BatchConfig, BatchConfig>chunk(10)
.reader(configReader)
.writer(itemWriter())
.listener(promotionListener())
.build();
}
#Bean
public ExecutionContextPromotionListener promotionListener() {
ExecutionContextPromotionListener listener = new ExecutionContextPromotionListener();
listener.setKeys(new String[] {"COUNTRY", "CATEGORY", "USER"});
return listener;
}
/*Country Step*/
#JobScope
#Bean
public MongoItemReader<COUNTRY> CountryItemReader(#Value("#{jobExecutionContext['COUNTRY']}") HashMap<String, Object> table) {
int date = (int) table.get("date");
MongoItemReader<COUNTRY> reader = new MongoItemReader<COUNTRY>();
reader.setTemplate(mongoTemplate);
reader.setTargetType(COUNTRY.class);
reader.setCollection("COUNTRY");
reader.setFields("{\"COUNTRY_NAME\": 1,\"SHORT_NAME\": 1,\"DEPT_CODE\": 1}");
reader.setSort(new HashMap<String, Sort.Direction>() {{
put("_id", Sort.Direction.DESC);
}});
reader.setQuery("{DATE_TIME: {$gt:"+date+"}}");
reader.setPageSize(250);
return reader;
}
#Bean
public CountryItemProcessor CountryProcessor(){
return new CountryItemProcessor();
}
#Bean
public JpaItemWriter<COUNTRY> country_writer(){
JpaItemWriter<COUNTRY> jpa = new JpaItemWriter<COUNTRY>();
jpa.setEntityManagerFactory(em.getEntityManagerFactory());
return jpa;
}
#JobScope
#Bean
public Step step1(#Value("#{jobExecutionContext['COUNTRY']}") HashMap<String, Object> tab) {
int size = (int) tab.get("size");
//System.out.println(size);
return stepBuilderFactory.get("step1")
.<COUNTRY, COUNTRY>chunk(20)
.reader(CountryItemReader(OVERRIDDEN_BY_EXPRESSION))
.writer(country_writer())
.build();
}
#Bean
public Job TestJob(Step stepConfig) throws Exception {
return this.jobBuilderFactory.get("TestJob")
.incrementer(new RunIdIncrementer())// because a spring config bug, this incrementer is not really useful
.start(stepConfig)
.next(step1(OVERRIDDEN_BY_EXPRESSION))
.build();
}
}
However when adding SimpleAsyncTaskExecutor an error occured:
org.springframework.beans.factory.support.ScopeNotActiveException: Error creating bean with name 'scopedTarget.CountryItemReader': Scope 'job' is not active for the current thread; consider defining a scoped proxy for this bean if you intend to refer to it from a singleton; nested exception is java.lang.IllegalStateException: No context holder available for job scope
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:383) ~[spring-beans-5.3.6.jar:5.3.6]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.6.jar:5.3.6]
at org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:35) ~[spring-aop-5.3.6.jar:5.3.6]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:676) ~[spring-aop-5.3.6.jar:5.3.6]
at org.springframework.batch.item.data.MongoItemReader$$EnhancerBySpringCGLIB$$67443e4.read(<generated>) ~[spring-batch-infrastructure-4.3.2.jar:4.3.2]
at org.springframework.batch.core.step.item.SimpleChunkProvider.doRead(SimpleChunkProvider.java:99) ~[spring-batch-core-4.3.2.jar:4.3.2]
at org.springframework.batch.core.step.item.SimpleChunkProvider.read(SimpleChunkProvider.java:180) ~[spring-batch-core-4.3.2.jar:4.3.2]
at org.springframework.batch.core.step.item.SimpleChunkProvider$1.doInIteration(SimpleChunkProvider.java:126) ~[spring-batch-core-4.3.2.jar:4.3.2]
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375) ~[spring-batch-infrastructure-4.3.2.jar:4.3.2]
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) ~[spring-batch-infrastructure-4.3.2.jar:4.3.2]
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145) ~[spring-batch-infrastructure-4.3.2.jar:4.3.2]
at org.springframework.batch.core.step.item.SimpleChunkProvider.provide(SimpleChunkProvider.java:118) ~[spring-batch-core-4.3.2.jar:4.3.2]
at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:71) ~[spring-batch-core-4.3.2.jar:4.3.2]
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:407) ~[spring-batch-core-4.3.2.jar:4.3.2]
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331) ~[spring-batch-core-4.3.2.jar:4.3.2]
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) ~[spring-tx-5.3.6.jar:5.3.6]
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273) ~[spring-batch-core-4.3.2.jar:4.3.2]
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82) ~[spring-batch-core-4.3.2.jar:4.3.2]
at org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable.run(TaskExecutorRepeatTemplate.java:262) ~[spring-batch-infrastructure-4.3.2.jar:4.3.2]
at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
Caused by: java.lang.IllegalStateException: No context holder available for job scope
at org.springframework.batch.core.scope.JobScope.getContext(JobScope.java:159) ~[spring-batch-core-4.3.2.jar:4.3.2]
at org.springframework.batch.core.scope.JobScope.get(JobScope.java:92) ~[spring-batch-core-4.3.2.jar:4.3.2]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:371) ~[spring-beans-5.3.6.jar:5.3.6]
I tried solving this issue like in:
https://github.com/spring-projects/spring-batch/issues/1335, but it seems like it is using just one thread in addition to the main.
Is there any way to resolve this issue without adding tweaked code ?
I'm planning to scale the job using remote partitionning on Kubernetes, would this issue persist because of job scope?
Any thoughts or advice are more than welcome.
I'm trying to share data between steps using JobExecutionContext, which works only if i keep the steps single threaded
Relying on the execution context to share data between multi-threaded steps is incorrect, because the keys will be overridden by concurrent threads. The reference documentation explicitly mentions to turn off state management in multi-threaded environment:
Javadoc: remember to use saveState=false if used in a multi-threaded client
Reference doc: it is not recommended to use job-scoped beans in multi-threaded or partitioned steps
That said, I don't see what key could be shared from a multi-threaded step to the next step (as threads are executed in parallel), but if you really need to do that, you should use another method like defining a shared bean that is thread safe.
The task is to call a database, retrieve certain records update and save them.
As the amount of records if fairly large we want to do this Async, however, this doesn't seem to be implemented correctly.
The main class:
#SpringBootApplication
#EnableAsync
MainApplication() {
#Bean("threadPoolExecutor")
public TaskExecutor getAsyncExecutor(){
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(DataSourceConfig.getTHREAD_POOL_SIZE());
executor.setMaxPoolSize(DataSourceConfig.getTHREAD_POOL_SIZE());
executor.setWaitForTasksToCompleteOnShutdown(true);
executor.setThreadNamePrefix("RetryEnhancement-");
return executor;
}
}
Method in the first service:
#Service
public class FirstService() {
#Transactional
public void fullProcess() {
for(int counter = 0; counter < ConfigFile.getTHREADS(); counter++){
secondaryService.threads();
}
}
}
Method in the second service:
#Service
public class SecondService () {
#Async("threadPoolExecutor")
public void threads() {
while(thirdService.threadMethod()) {
//doNothing
}
}
}
Method in the third service:
#Service
public class ThirdService() {
#Transactional
public boolean threads() {
Record record = repository.fetchRecord();
if(record!=null) {
updateRecord(record);
saveRecord(record);
return true;
} else {
return false;
}
}
}
Repository:
public interface repository extends CrudRepository<Record, long> {
#Lock(LockModeType.PESSIMISTIC_WRITE)
Record fetchRecord();
}
The issue I'm finding is that, while the code executes perfectly fine, it seems to have a Synchronous execution (found by adding a .sleep and watching the execution in the logger).
The seperate threads seem to be waiting until the other is executed.
I'm probably doing something wrong and if another thread already explains the issue, than please refer it, though I have not been able to find this issue in a different thread.
Your solution is way to complex. Ditch all of that and just inject the TaskExecutor and do the updateRecord in a separate thread (you might need to retrieve it again as you are now using a different thread and thus connection.
Something like this should do the trick
private final TaskExecutor executor; // injected through constructor
public void process() {
Stream<Record> records = repository.fetchRecords(); // Using a stream gives you a lazy cursor!
records.forEach(this::processRecord);
}
private void processRecord(Record record) {
executor.submit({
updateRecord(record);
saveRecord(record);
});
}
You might want to put the processRecord into another object and make it #Transactional or wrap it in a TransactionTemplate to get that behavior.
We are launching a website that will have a very heavy volume for a short period of time. It is basically giving tickets. The code is written in Java, Spring & Hibernate. I want to mimic the high volume by spawning multiple threads and trying to get the ticket using JUnit test case. The problem is that in my DAO class the code just simply dies after I begin transaction. I mean there is no error trace in the log file or anything like that. Let me give some idea about the way my code is.
DAO code:
#Repository("customerTicketDAO")
public class CustomerTicketDAO extends BaseDAOImpl {// BaseDAOImpl extends HibernateDaoSupport
public void saveCustomerTicketUsingJDBC(String customerId) {
try{
getSession().getTransaction().begin(); //NOTHING HAPPENS AFTER THIS LINE OF CODE
// A select query
Query query1 = getSession().createSQLQuery("my query omitted on purpose");
.
.
// An update query
Query query2 = getSession().createSQLQuery("my query omitted on purpose");
getSession().getTransaction().commite();
} catch (Exception e) {
}
}
Runnable code:
public class InsertCustomerTicketRunnable implements Runnable {
#Autowired
private CustomerTicketDAO customerTicketDAO;
public InsertCustomerTicketRunnable(String customerId) {
this.customerId = customerId;
}
#Override
public void run() {
if (customerTicketDAO != null) {
customerTicketDAO.saveCustomerTicketUsingJDBC(customerId);
}
}
}
JUnit method:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(locations={"file:src/test/resources/applicationContext-test.xml"})
public class DatabaseTest {
#Before
public void init() {
sessionFactory = (SessionFactory)applicationContext.getBean("sessionFactory");
Session session = SessionFactoryUtils.getSession(sessionFactory, true);
TransactionSynchronizationManager.bindResource(sessionFactory, new SessionHolder(session));
customerTicketDAO = (CustomerTicketDAO)applicationContext.getBean("customerTicketDAO");
}
#After
public void end() throws Exception {
SessionHolder sessionHolder = (SessionHolder) TransactionSynchronizationManager.unbindResource(sessionFactory);
SessionFactoryUtils.closeSession(session);
}
#Test
public void saveCustomerTicketInMultipleThreads () throws Exception {
ExecutorService executor = Executors.newFixedThreadPool(NTHREDS);
for (int i=0; i<1000; i++) {
executor.submit(new InsertCustomerTicketRunnable(i));
}
// This will make the executor accept no new threads
// and finish all existing threads in the queue
executor.shutdown();
// Wait until all threads are finish
executor.awaitTermination(1, TimeUnit.SECONDS);
}
I see no data being inserted into the database. Can someone please point me as to where I am going wrong?
Thanks
Raj
SessionFactory is thread safe but Session is not. So my guess is that you need to call SessionFactoryUtils.getSession() from within each thread, so that each thread gets its own instance. You are currently calling it from the main thread, so all children threads try to share the same instance.
Naughty, naughty!
public void saveCustomerTicketUsingJDBC(String customerId) {
try {
getSession().getTransaction().begin(); //NOTHING HAPPENS AFTER THIS LINE OF CODE
.
.
} catch (Exception e) {
}
}
You should never (well, hardly ever) have an empty catch block, if there is a problem you will find that your code 'just simply dies' with no log messages. Oh look, that's what's happening ;)
At the very minimum you should log the exception, that will go a long way towards you helping you find what the problem is (and from there, the solution).
I am trying to write a batch processing application using spring batch with multi-thread step.this is simple application reading data from a table and writing to another table but data is large around 2 million record .
I am using RepositoryItemReader & RepositoryItemWriter for reading and writing data. But after processing some data it failing due to Unable to acquire JDBC Connection.
//Config.Java
#Bean
public TaskExecutor taskExecutor() {
SimpleAsyncTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor();
taskExecutor.setConcurrencyLimit(10);
return taskExecutor;
}
#Bean(name = "personJob")
public Job personKeeperJob() {
Step step = stepBuilderFactory.get("step-1")
.<User, Person> chunk(1000)
.reader(userReader)
.processor(jpaProcessor)
.writer(personWriter)
.taskExecutor(taskExecutor())
.throttleLimit(10)
.build();
Job job = jobBuilderFactory.get("person-job")
.incrementer(new RunIdIncrementer())
.listener(this)
.start(step)
.build();
return job;
}
//Processor.Java
#Override
public Person process(User user) throws Exception {
Optional<User> userFromDb = userRepo.findById(user.getUserId());
Person person = new Person();
if(userFromDb.isPresent()) {
person.setName(userFromDb.get().getName());
person.setUserId(userFromDb.get().getUserId());
person.setDept(userFromDb.get().getDept());
}
return person;
}
//Reader.Java
#Autowired
public UserItemReader(final UserRepository repository) {
super();
this.repository = repository;
}
#PostConstruct
protected void init() {
final Map<String, Sort.Direction> sorts = new HashMap<>();
sorts.put("userId", Direction.ASC);
this.setRepository(this.repository);
this.setSort(sorts);
this.setMethodName("findAll");
}
//Writer.Java
#PostConstruct
protected void init() {
this.setRepository(repository);
}
#Transactional
public void write(List<? extends Person> persons) throws Exception {
repository.saveAll(persons);
}
application.properties
# Datasource
spring.datasource.platform=h2
spring.datasource.url=jdbc:h2:mem:batchdb
spring.main.allow-bean-definition-overriding=true
spring.datasource.hikari.maximum-pool-size=500
Error :
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:447)
......................
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
............................
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30927ms.
You run out of connections.
Try to set the Hikari Connection Pool to a bigger number:
spring.datasource.hikari.maximum-pool-size=20