In my Spring Boot project, I have two JMS listeners listening to one queue. All messages received from the queue have to be processed in the same way and persisted / updated in the database (Oracle). Currently, I have a synchronized method in a class that is doing the parsing of the messages. As expected, all thread read messages simultaneously, but parsing is done one by one as the method (parseMessage()) is synchronized. What I want is to parse the messages simultaneously and do database operations as well.
How can I solve this?
I don't want to create two different classes with the same code and use #Qualifier to call different classes in each listener, as the code for parsing the message is the same.
The ideal solution, I think, is to do database operations using a new synchronized method in a new class, but parsing the message in a multi-threaded way. So, at a time only one thread can say persist / update. When a thread is not waiting to persist / update, it continues the parsing on its own thread.
Please correct me if I am wrong or if you find the optimal solution. Let me know if any other info is needed.
JMS Controller Class
#RestController
#EnableJms
public class JMSController {
#Autowired
private IParseMapXml iParseMapXml;
#JmsListener(destination = "${app.jms_destinaltion}")
public void receiveMessage1(String recvMsg) {
try {
InputSource is = new InputSource(new StringReader(recvMsg.replaceAll("&", "&")));
Document doc = new SAXReader().read(is);
iParseMapXml.parseMessage(doc);
} catch (Exception e) {
}
}
#JmsListener(destination = "${app.jms_destinaltion}")
public void receiveMessage2(String recvMsg) {
try {
InputSource is = new InputSource(new StringReader(recvMsg.replaceAll("&", "&")));
Document doc = new SAXReader().read(is);
iParseMapXml.parseMessage(doc);
} catch (Exception e) {
}
}
}
Parse XML Interface
public interface IParseMapXml {
public void parseMessage(Document doc);
}
Parsing Implementation
public class ParsingMessageClass implements IParseMapXml{
#Override
#Transactional
synchronized public void parseMessage(Document doc) {
// TODO Auto-generated method stub
....
PROCESS DATA/MESSAGE
....
DO DB OPERATIONS
}
}
Related
I am trying to override the transactional behaviour for a service method(someService.updateSomething() in the example) annotated with #Transactional annotation in Spring. To do so, from other class, I am using programmatic transactional code like the next:
#Service
public class MyServiceClass {
private TransactionTemplate transactionTemplate;
public MyClass (PlatformTransactionManager transactionManager) {
transactionTemplate = new TransactionTemplate(transactionManager);
}
#Transactional
public void someMethod(){
transactionTemplate.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRES_NEW);
transactionTemplate.execute(new TransactionCallbackWithoutResult() {
protected void doInTransactionWithoutResult(TransactionStatus status){
try{
someService.updateSomething();
}catch(Exception e){
LOGGER.error("Error has ocurred");
}
}
});
}
}
My problem is that someService.updateSomething() does not run in a new Transaction and I dont understand why. So:
If I call a proxied service method with transactional behaviour like someService.updateSomething() but in the call I create a new transaction like in the example, when the code hits to the proxied method, it will take the new transaction created and not the transaction already running for the someMethod() method, right?
Thanks!
The doc of kafka give an approach about with following describes:
One Consumer Per Thread:A simple option is to give each thread its own consumer > instance.
My code:
public class KafkaConsumerRunner implements Runnable {
private final AtomicBoolean closed = new AtomicBoolean(false);
private final CloudKafkaConsumer consumer;
private final String topicName;
public KafkaConsumerRunner(CloudKafkaConsumer consumer, String topicName) {
this.consumer = consumer;
this.topicName = topicName;
}
#Override
public void run() {
try {
this.consumer.subscribe(topicName);
ConsumerRecords<String, String> records;
while (!closed.get()) {
synchronized (consumer) {
records = consumer.poll(100);
}
for (ConsumerRecord<String, String> tmp : records) {
System.out.println(tmp.value());
}
}
} catch (WakeupException e) {
// Ignore exception if closing
System.out.println(e);
//if (!closed.get()) throw e;
}
}
// Shutdown hook which can be called from a separate thread
public void shutdown() {
closed.set(true);
consumer.wakeup();
}
public static void main(String[] args) {
CloudKafkaConsumer kafkaConsumer = KafkaConsumerBuilder.builder()
.withBootstrapServers("172.31.1.159:9092")
.withGroupId("test")
.build();
ExecutorService executorService = Executors.newFixedThreadPool(5);
executorService.execute(new KafkaConsumerRunner(kafkaConsumer, "log"));
executorService.execute(new KafkaConsumerRunner(kafkaConsumer, "log.info"));
executorService.shutdown();
}
}
but it doesn't work and throws an exception:
java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access
Furthermore, I read the source of Flink (an open source platform for distributed stream and batch data processing). Flink using multi-thread consumer is similar to mine.
long pollTimeout = Long.parseLong(flinkKafkaConsumer.properties.getProperty(KEY_POLL_TIMEOUT, Long.toString(DEFAULT_POLL_TIMEOUT)));
pollLoop: while (running) {
ConsumerRecords<byte[], byte[]> records;
//noinspection SynchronizeOnNonFinalField
synchronized (flinkKafkaConsumer.consumer) {
try {
records = flinkKafkaConsumer.consumer.poll(pollTimeout);
} catch (WakeupException we) {
if (running) {
throw we;
}
// leave loop
continue;
}
}
flink code of mutli-thread
What's wrong?
Kafka consumer is not thread safe. As you pointed out in your question, the document stated that
A simple option is to give each thread its own consumer instance
But in your code, you have the same consumer instance wrapped by different KafkaConsumerRunner instances. Thus multiple threads are accessing the same consumer instance. The kafka documentation clearly stated
The Kafka consumer is NOT thread-safe. All network I/O happens in the
thread of the application making the call. It is the responsibility of
the user to ensure that multi-threaded access is properly
synchronized. Un-synchronized access will result in
ConcurrentModificationException.
That's exactly the exception you received.
It is throwing the exception on your call to subscribe. this.consumer.subscribe(topicName);
Move that block into a synchronized block like this:
#Override
public void run() {
try {
synchronized (consumer) {
this.consumer.subscribe(topicName);
}
ConsumerRecords<String, String> records;
while (!closed.get()) {
synchronized (consumer) {
records = consumer.poll(100);
}
for (ConsumerRecord<String, String> tmp : records) {
System.out.println(tmp.value());
}
}
} catch (WakeupException e) {
// Ignore exception if closing
System.out.println(e);
//if (!closed.get()) throw e;
}
}
Maybe is not your case, but if you are mergin processing of data of serveral topics, then you can read data from multiple topics with the same consumer. If not, then is preferable to create separate jobs consuming each topic.
Having an huge customers profile page if two or more users start using same page and start editing big change will happen in my database so planing to implement Threads concept where only one user can use that customer page
i'm aware about threads concept but confused how to implement it
hope i need to use Singleton class as well
Any suggestion or Logic's will be helpful
I'm using Struts,Hibernate frame work
You may use application context to store a flag variable. Action will use its value to allow only one simultaneous execution.
public class TestAction extends ActionSupport implements ApplicationAware {
private static final String APP_BUSY_KEY = "APP_BUSY";
Map<String, Object> map;
#Override
public void setApplication(Map<String, Object> map) {
this.map = map;
}
#Override
public String execute() throws Exception {
if (map.containsKey(APP_BUSY_KEY)) {
return ERROR;
} else {
map.put(APP_BUSY_KEY, "1");
try {
// action logic here
} finally {
map.remove(APP_BUSY_KEY);
}
return SUCCESS;
}
}
}
If you plan to implement similar logic for two requests (lock after displaying values and release lock after submitting new values) then logic will be more complex and you will also need to handle lock release after timeout.
I am confused with Async feature introduced in Servlet 3.0 spec
From Oracle site (http://docs.oracle.com/javaee/7/tutorial/doc/servlets012.htm):
To create scalable web applications, you must ensure that no threads
associated with a request are sitting idle, so the container can use
them to process new requests.
There are two common scenarios in which a thread associated with a
request can be sitting idle.
1- The thread needs to wait for a resource to become available or process data before building the response. For example, an application
may need to query a database or access data from a remote web service
before generating the response.
2- The thread needs to wait for an event before generating the response. For example, an application may have to wait for a JMS
message, new information from another client, or new data available in
a queue before generating the response.
The first item happens a lot (nearly always, we always query db or call a remote webservice to get some data). And calling an external resource will always consume some time.
Does it mean that we should ALWAYS use servelt async feature for ALL our servelts and filter ?!
I can ask this way too, if I write all my servelts and filters async, will I lose anything (performance)?!
If above is correct the skeleton of ALL our servlets will be:
public class Work implements ServletContextListener {
private static final BlockingQueue queue = new LinkedBlockingQueue();
private volatile Thread thread;
#Override
public void contextInitialized(ServletContextEvent servletContextEvent) {
thread = new Thread(new Runnable() {
#Override
public void run() {
while (true) {
try {
ServiceFecade.doBusiness();
AsyncContext context;
while ((context = queue.poll()) != null) {
try {
ServletResponse response = context.getResponse();
PrintWriter out = response.getWriter();
out.printf("Bussiness done");
out.flush();
} catch (Exception e) {
throw new RuntimeException(e.getMessage(), e);
} finally {
context.complete();
}
}
} catch (InterruptedException e) {
return;
}
}
}
});
thread.start();
}
public static void add(AsyncContext c) {
queue.add(c);
}
#Override
public void contextDestroyed(ServletContextEvent servletContextEvent) {
thread.interrupt();
}
}
We are launching a website that will have a very heavy volume for a short period of time. It is basically giving tickets. The code is written in Java, Spring & Hibernate. I want to mimic the high volume by spawning multiple threads and trying to get the ticket using JUnit test case. The problem is that in my DAO class the code just simply dies after I begin transaction. I mean there is no error trace in the log file or anything like that. Let me give some idea about the way my code is.
DAO code:
#Repository("customerTicketDAO")
public class CustomerTicketDAO extends BaseDAOImpl {// BaseDAOImpl extends HibernateDaoSupport
public void saveCustomerTicketUsingJDBC(String customerId) {
try{
getSession().getTransaction().begin(); //NOTHING HAPPENS AFTER THIS LINE OF CODE
// A select query
Query query1 = getSession().createSQLQuery("my query omitted on purpose");
.
.
// An update query
Query query2 = getSession().createSQLQuery("my query omitted on purpose");
getSession().getTransaction().commite();
} catch (Exception e) {
}
}
Runnable code:
public class InsertCustomerTicketRunnable implements Runnable {
#Autowired
private CustomerTicketDAO customerTicketDAO;
public InsertCustomerTicketRunnable(String customerId) {
this.customerId = customerId;
}
#Override
public void run() {
if (customerTicketDAO != null) {
customerTicketDAO.saveCustomerTicketUsingJDBC(customerId);
}
}
}
JUnit method:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(locations={"file:src/test/resources/applicationContext-test.xml"})
public class DatabaseTest {
#Before
public void init() {
sessionFactory = (SessionFactory)applicationContext.getBean("sessionFactory");
Session session = SessionFactoryUtils.getSession(sessionFactory, true);
TransactionSynchronizationManager.bindResource(sessionFactory, new SessionHolder(session));
customerTicketDAO = (CustomerTicketDAO)applicationContext.getBean("customerTicketDAO");
}
#After
public void end() throws Exception {
SessionHolder sessionHolder = (SessionHolder) TransactionSynchronizationManager.unbindResource(sessionFactory);
SessionFactoryUtils.closeSession(session);
}
#Test
public void saveCustomerTicketInMultipleThreads () throws Exception {
ExecutorService executor = Executors.newFixedThreadPool(NTHREDS);
for (int i=0; i<1000; i++) {
executor.submit(new InsertCustomerTicketRunnable(i));
}
// This will make the executor accept no new threads
// and finish all existing threads in the queue
executor.shutdown();
// Wait until all threads are finish
executor.awaitTermination(1, TimeUnit.SECONDS);
}
I see no data being inserted into the database. Can someone please point me as to where I am going wrong?
Thanks
Raj
SessionFactory is thread safe but Session is not. So my guess is that you need to call SessionFactoryUtils.getSession() from within each thread, so that each thread gets its own instance. You are currently calling it from the main thread, so all children threads try to share the same instance.
Naughty, naughty!
public void saveCustomerTicketUsingJDBC(String customerId) {
try {
getSession().getTransaction().begin(); //NOTHING HAPPENS AFTER THIS LINE OF CODE
.
.
} catch (Exception e) {
}
}
You should never (well, hardly ever) have an empty catch block, if there is a problem you will find that your code 'just simply dies' with no log messages. Oh look, that's what's happening ;)
At the very minimum you should log the exception, that will go a long way towards you helping you find what the problem is (and from there, the solution).