How to create separate Log4j2 rolling file appenders and loggers programatically for multiple threads - multithreading

Question

I am running my TestNG tests on multiple threads (Appium tests on multiple devices simultaneously) and want to write the test logs on different threads in different files. Here the threads are created automatically before the start of the test flow.
So I want to create separate appender and separate logger programatically so that each appender would be attached to its own thread only and then the loggers created in one thread would have the appender created in that thread only.
Please let me know how to achieve it step by step.

Answer 1

First, this feels like an XY Problem especially because you have not provided any reasoning as to why you want to go with a programmatic solution.
It is possible to achieve separate log files on a thread by thread basis without programmatically creating loggers and appenders. Since I believe this to be a more optimal solution I will provide a demo of how it can be done.
Explanation
This solution will use the RoutingAppender and ThreadContext to create appenders for each Thread. The example that follows will use a simple FileAppender but you can swap in a RollingFileAppender very easily. Since you did not state that you have any requirement to use different log levels for each thread the following example will not implement this feature. Finally, the example will not use TestNG as it has some overhead to set up; instead a simple class with a main that creates two threads will be used.
Example Implementation
log4j2.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<Routing name="MyRoutingAppender">
<Routes pattern="$${ctx:threadName}">
<Route>
<File
fileName="logs/${ctx:threadName}/log.txt"
name="appender-${ctx:threadName}">
<PatternLayout>
<Pattern>[%date{ISO8601}][%-5level][%t] %m%n</Pattern>
</PatternLayout>
</File>
</Route>
</Routes>
</Routing>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="[%date{ISO8601}][%-5level][%t] %m%n" />
</Console>
</Appenders>
<Loggers>
<Logger name="Thread" level="TRACE" additivity="false">
<AppenderRef ref="STDOUT" />
<AppenderRef ref="MyRoutingAppender" />
</Logger>
<Root level="WARN">
<AppenderRef ref="STDOUT" />
</Root>
</Loggers>
</Configuration>
Java class that generates logs using 2 concurrent threads:
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.ThreadContext;
public class MultiThreadLog4j2SepFilesMain {
//Create a lock to use for synchronizing the getting of the logger
private static final Object lock = new Object();
public static void main(String[] args) {
Thread t1 = new Thread(new Runnable(){
public void run() {
//Set up the context before getting logger
ThreadContext.put("threadName", Thread.currentThread().getName());
//Get the logger for this thread
Logger log = null;
synchronized(lock){
log = LogManager.getLogger(Thread.currentThread().getName());
}
//Generate some logs
log.info("here's the first thread");
//Wait a while so that threads interleave
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
//Generate more logs
log.debug("some debug in first thread");
log.info("finishing first thread");
}}, "Thread.1"); //Use a name that will allow us to use Thread.getName when getting the logger inside the thread
Thread t2 = new Thread(new Runnable(){
public void run() {
//Set up the context before getting logger
ThreadContext.put("threadName", Thread.currentThread().getName());
//Get logger for this thread
Logger log = null;
synchronized(lock){
log = LogManager.getLogger(Thread.currentThread().getName());
}
//Generate some logs
log.info("here's the second thread");
log.debug("some debug in second thread");
}}, "Thread.2"); //Use a name that will allow us to use Thread.getName when getting the logger inside the thread
//Start both threads
t1.start();
t2.start();
}
}