How to tail a file with rx-java? - multithreading

Using the rx-java package, how can I tail a file?
code:
package tail;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.logging.Logger;
public class FileUtils {
public FileUtils() {
}
public void readFromInputStream(InputStream inputStream)
throws IOException {
StringBuilder resultStringBuilder = new StringBuilder();
try (BufferedReader br
= new BufferedReader(new InputStreamReader(inputStream))) {
String line;
while ((line = br.readLine()) != null) {
resultStringBuilder.append(line).append("\n");
}
}
System.out.println(resultStringBuilder);
}
}
I'd want to parse each line as its written to the file. First, need an equivalent to tail -f foo.log from Java.

I'm not sure about using RxJava specifically. But you can take a general Java solution and combine with RxJava. This of course depends on how you want to be informed.
One basic option is to tail the file using an open InputStream (or some stream wrapper classes or reader class).
Using a running thread, you can wait on the file with InputStream.read or sample the amount of new bytes with InputStream.available. In RxJava you can use the IO scheduler. After finishing to read an entire line, just invoke some observers from the thread.
In core Java, it could look something like:
class ObserveTask implements Runnable {
private File file;
private Listener listener;
#Override
public void run() {
try (InputStream stream = new FileInputStream(file);
BufferedReader reader = new BufferedReader(stream)) {
while (!Thread.interrupted()) {
String line = reader.readLine();
listener.onNewLine(line);
}
}
}
}
Now just run the task:
Scheduler scheduler = Schedulers.newThread();
Scheduler.Worker worker = scheduler.createWorker();
worker.schedule(new ObserveTask());
Of course this is just a basic example.
You can split the task to parts, where each run, a check is made for the bytes, while the stream remains open. Of course this is dangerous, since leaving a stream open like that might end in leaking.
A more advanced solution can include using WatchService.

Related

How to create a multi-threaded custom move factory in OptaPlanner?

I am solving a tough problem in OptaPlanner. The best algorithm I found so far is to use a custom move factory, a computationally intensive one. After noticing that I was utilising a single CPU core, I discovered that OptaPlanner only spreads on multiple threads the score calculation, while it performs the move generation in a single thread.
To mitigate the problem, I implemented the multi-threading in my move factory via the following abstract class, which I then extend with the actual logic (I did this because I actually have three computationally expensive custom move factories):
package my.solver.move;
import lombok.AllArgsConstructor;
import lombok.NonNull;
import org.optaplanner.core.impl.domain.solution.descriptor.SolutionDescriptor;
import org.optaplanner.core.impl.heuristic.move.CompositeMove;
import org.optaplanner.core.impl.heuristic.move.Move;
import org.optaplanner.core.impl.heuristic.selector.move.factory.MoveIteratorFactory;
import org.optaplanner.core.impl.score.director.ScoreDirector;
import java.util.Iterator;
import java.util.Objects;
import java.util.Random;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.Executors;
import java.util.concurrent.ThreadFactory;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Function;
public abstract class MultiThreadedMoveFactory<T> implements MoveIteratorFactory<T> {
private final ThreadPoolExecutor threadPoolExecutor;
public MultiThreadedMoveFactory(
#NonNull String threadPrefix
) {
int availableProcessorCount = Runtime.getRuntime().availableProcessors();
int resolvedThreadCount = Math.max(1, availableProcessorCount);
ThreadFactory threadFactory = new SolverThreadFactory(threadPrefix);
threadPoolExecutor = (ThreadPoolExecutor) Executors.newFixedThreadPool(resolvedThreadCount, threadFactory);
}
#AllArgsConstructor
public class MoveGeneratorData {
T solution;
SolutionDescriptor<T> solutionDescriptor;
Random random;
BlockingQueue<Move<T>> generatedMoves;
}
protected abstract int getNumMoves();
#Override
public long getSize(ScoreDirector<T> scoreDirector) {
return getNumMoves();
}
protected class MovesIterator implements Iterator<Move<T>> {
private final BlockingQueue<Move<T>> generatedMoves = new ArrayBlockingQueue<>(getNumMoves());
public MovesIterator(
#NonNull T solution,
#NonNull SolutionDescriptor<T> solutionDescriptor,
#NonNull Random random,
#NonNull Function<MoveGeneratorData, Runnable> moveGeneratorFactory
) {
MoveGeneratorData moveGeneratorData = new MoveGeneratorData(solution, solutionDescriptor, random, generatedMoves);
for (int i = 0; i < getNumMoves(); i++) {
threadPoolExecutor.submit(moveGeneratorFactory.apply(moveGeneratorData));
}
}
#Override
public boolean hasNext() {
if (!generatedMoves.isEmpty()) {
return true;
}
while (threadPoolExecutor.getActiveCount() > 0) {
try {
//noinspection BusyWait
Thread.sleep(50);
} catch (InterruptedException e) {
return false;
}
}
return !generatedMoves.isEmpty();
}
#Override
public Move<T> next() {
//noinspection unchecked
return Objects.requireNonNullElseGet(generatedMoves.poll(), CompositeMove::new);
}
}
private static final AtomicInteger poolNumber = new AtomicInteger(1);
private static class SolverThreadFactory implements ThreadFactory {
private final ThreadGroup group;
private final AtomicInteger threadNumber = new AtomicInteger(1);
private final String namePrefix;
public SolverThreadFactory(String threadPrefix) {
SecurityManager s = System.getSecurityManager();
group = (s != null) ? s.getThreadGroup() : Thread.currentThread().getThreadGroup();
namePrefix = "MyPool-" + poolNumber.getAndIncrement() + "-" + threadPrefix + "-";
}
#Override
public Thread newThread(#NonNull Runnable r) {
Thread t = new Thread(group, r, namePrefix + threadNumber.getAndIncrement(), 0);
if (t.isDaemon()) {
t.setDaemon(false);
}
if (t.getPriority() != Thread.NORM_PRIORITY) {
t.setPriority(Thread.NORM_PRIORITY);
}
return t;
}
}
#Override
public Iterator<? extends Move<T>> createOriginalMoveIterator(ScoreDirector<T> scoreDirector) {
return createMoveIterator(scoreDirector, new Random());
}
#Override
public Iterator<? extends Move<T>> createRandomMoveIterator(ScoreDirector<T> scoreDirector, Random workingRandom) {
return createMoveIterator(scoreDirector, workingRandom);
}
public abstract Iterator<? extends Move<T>> createMoveIterator(ScoreDirector<T> scoreDirector, Random random);
}
However, the solver seems to hang after a while. The debugger tells me that it's waiting on an innerQueue.take() in OrderByMoveIndexBlockingQueue. This is caused by my move factory: if I revert the above and only use the previous implementation, which was single-threaded, the problem goes away.
I do not quite understand where the problem is, so the question is: how can I fix it?
No, no, no. This approach is doomed. I think. (Prove me wrong.)
JIT selection
First learn about Just In Time selection (see docs) of moves.
Instead of generating all moves (which can be billions) at the beginning of each step, only generate those that will actually be evaluated. Most LS algorithms will only evaluate a few moves per step.
Watch the TRACE log to see how many milliseconds it takes to start a step. Typically you want to do evaluate 10000 moves per second, so it should take 0 or 1 milliseconds to start a step (the log only shows in milliseconds).
Multithreaded solving
Then learn about moveThreadCount to enable multithreaded solving. See this blog post. Know that this still does the move selection on 1 thread, for reproducibility reasons. But the move evaluation is spread across threads.
Caching for move selection
But your custom moves are smart, so the move selection must be smart?
First determine what "solution state" query information you need to generate the moves - for example a Map<Employee, List<Shift>> - then cache that:
either calculate that map at the beginning of each step, if it doesn't take too long (but this won't scale because it doesn't do deltas)
or use a shadow variable (#InverseRelationShadowVariable works fine in this case), because these are updated through deltas. But it does do the delta's for every move and undo move too...
Or hack in an actual new MoveSelector, which can listen to stepEnded() events and actually apply the delta of the last step on that Map, without doing any of the deltas of every move and undo move. We should probably standardize this approach and make it part of our public API some day.
I was able to make the factory work by removing any trace of JIT-ing from hasNext: block the method until all moves have been generated, and only then return true, and keep returning true until all moves have been consumed.
#Override
public boolean hasNext() {
while (!generationComplete && generatedMoves.size() < getNumMoves()) {
try {
// We get a warning because the event we are waiting for could happen earlier than the end of sleep
// and that means we would be wasting time, but that is negligible so we silence it
//noinspection BusyWait
Thread.sleep(50);
} catch (InterruptedException e) {
return false;
}
}
generationComplete = true;
return !generatedMoves.isEmpty();
}
To the best of my understanding, the solution I am using not only works, but it is the best I found in a few months of iterations.

TestNG Close Browsers after Parallel Test Execution

I want to close browsers after completion of all test. Problem is I am not able to close the browser since the object created ThreadLocal driver does not recognize the driver after completion of test value returning is null.
Below is my working code
package demo;
import java.lang.reflect.Method;
import org.openqa.selenium.By;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;
public class ParallelMethodTest {
private static ThreadLocal<dummy> driver;
private int input;
private int length;
#BeforeMethod
public void beforeMethod() {
System.err.println("Before ID" + Thread.currentThread().getId());
System.setProperty("webdriver.chrome.driver", "chromedriver.exe");
if (driver == null) {
driver = new ThreadLocal<dummy>();
}
if (driver.get()== null) {
driver.set(new dummy());
}
}
#DataProvider(name = "sessionDataProvider", parallel = true)
public static Object[][] sessionDataProvider(Method method) {
int len = 12;
Object[][] parameters = new Object[len][2];
for (int i = 0; i < len; i++) {
parameters[i][0] = i;
parameters[i][1]=len;
}
return parameters;
}
#Test(dataProvider = "sessionDataProvider")
public void executSessionOne(int input,int length) {
System.err.println("Test ID---" + Thread.currentThread().getId());
this.input=input;
this.length=length;
// First session of WebDriver
// find user name text box and fill it
System.out.println("Parameter size is:"+length);
driver.get().getDriver().findElement(By.name("q")).sendKeys(input + "");
System.out.println("Input is:"+input);
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
#AfterMethod
public void afterMethod() {
System.err.println("After ID" + Thread.currentThread().getId());
driver.get().close();
}
}
package demo;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.testng.annotations.AfterClass;
public class dummy {
public WebDriver getDriver() {
return newDriver;
}
public void setNewDriver(WebDriver newDriver) {
this.newDriver = newDriver;
}
private WebDriver newDriver;
public dummy() {
newDriver = new ChromeDriver();
newDriver.get("https://www.google.co.in/");
}
#AfterClass
public void close(){
if(newDriver!=null){
System.out.println("In After Class");
newDriver.quit();
}
}
}
Thanks in Advance.
private static ThreadLocal<dummy> driver is added at the class level. What is happening is that you have already declared the variable at class level. i.e. memory is already allocated to it. Multiple threads are just setting and resetting the values of the same variable.
What you need to do is create a factory that will return an instance of Driver based on a parameter you pass to it.Logic can be anything but taking a general use case example the factory will create a new object and return only if an existing object doesn't exist. Declare and initialise the driver (from factory) in your #Test Methods
Sample code for the factory would be something like
static RemoteWebDriver firefoxDriver;
static RemoteWebDriver someOtherDriver;
static synchronized RemoteWebDriver getDriver(String browser, String browserVersion, String platform, String platformVersion)
{
if (browser == 'firefox')
{
if (firefoxDriver == null)
{
DesiredCapabilities cloudCaps = new DesiredCapabilities();
cloudCaps.setCapability("browser", browser);
cloudCaps.setCapability("browser_version", browserVersion);
cloudCaps.setCapability("os", platform);
cloudCaps.setCapability("os_version", platformVersion);
cloudCaps.setCapability("browserstack.debug", "true");
cloudCaps.setCapability("browserstack.local", "true");
firefoxDriver = new RemoteWebDriver(new URL(URL),cloudCaps);
}
}
else
{
if (someOtherDriver == null)
{
DesiredCapabilities cloudCaps = new DesiredCapabilities();
cloudCaps.setCapability("browser", browser);
cloudCaps.setCapability("browser_version", browserVersion);
cloudCaps.setCapability("os", platform);
cloudCaps.setCapability("os_version", platformVersion);
cloudCaps.setCapability("browserstack.debug", "true");
cloudCaps.setCapability("browserstack.local", "true");
someOtherDriver = new RemoteWebDriver(new URL(URL),cloudCaps);
}
return someOtherDriver;
}
You have a concurrency issue: multiple threads can create a ThreadLocal instance because dummy == null can evaluate to true on more than one thread when run in parallel. As such, some threads can execute driver.set(new dummy()); but then another thread replaces driver with a new ThreadLocal instance.
In my experience it is simpler and less error prone to always use ThreadLocal as a static final to ensure that multiple objects can access it (static) and that it is only defined once (final).
You can see my answers to the following Stack Overflow questions for related details and code samples:
How to avoid empty extra browser opens when running parallel tests with TestNG
Session not found exception with Selenium Web driver parallel execution of Data Provider test case
This is happening because you are creating the driver instance in beforeMethod function so it's scope ends after the function ends.
So when your afterMethod start it's getting null because webdriver instance already destroy as beforeMethod function is already completed.
Refer below links:-
http://www.java-made-easy.com/variable-scope.html
What is the default scope of a method in Java?

Spark FileStreaming issue

I am trying simple file streaming example using Sparkstreaming(spark-streaming_2.10,version:1.5.1)
public class DStreamExample {
public static void main(final String[] args) {
final SparkConf sparkConf = new SparkConf();
sparkConf.setAppName("SparkJob");
sparkConf.setMaster("local[4]"); // for local
final JavaSparkContext sc = new JavaSparkContext(sparkConf);
final JavaStreamingContext ssc = new JavaStreamingContext(sc,
new Duration(2000));
final JavaDStream<String> lines = ssc.textFileStream("/opt/test/");
lines.print();
ssc.start();
ssc.awaitTermination();
}
}
When I run this code on single file or director it does not print anything from file, I see in logs its constantly polling but nothing is printed. I tried moving file to directory when this program was running.
Is there something I am missing? I tried applying map function on lines RDD that also does not work.
The API textFileStream is not supposed to read existing directory content, instead, it's purpose is to monitor the given Hadoop-compatible filesystem path for changes, files must be written into monitored location by "moving" them from another location within same file system.
In short, you are subscribing for directory changes and will receive the content of newly appeared files within the monitored location - in that state in which the file(s) appear at the moment of monitoring snapshot (which is 2000 ms duration in your case), and any further file updates will not reach the stream, only directory updates (new files) will do.
The way you can emulate updates is to create new file during your monitoring session:
import org.apache.commons.io.FileUtils;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.streaming.Duration;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
import java.io.File;
import java.io.IOException;
import java.util.List;
public class DStreamExample {
public static void main(final String[] args) throws IOException {
final SparkConf sparkConf = new SparkConf();
sparkConf.setAppName("SparkJob");
sparkConf.setMaster("local[4]"); // for local
final JavaSparkContext sc = new JavaSparkContext(sparkConf);
final JavaStreamingContext ssc = new JavaStreamingContext(sc,
new Duration(2000));
final JavaDStream<String> lines = ssc.textFileStream("/opt/test/");
// spawn the thread which will create new file within the monitored directory soon
Runnable r = () -> {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
try {
FileUtils.write(new File("/opt/test/newfile1"), "whatever");
} catch (IOException e) {
e.printStackTrace();
}
};
new Thread(r).start();
lines.foreachRDD((Function<JavaRDD<String>, Void>) rdd -> {
List<String> lines1 = rdd.collect();
lines1.stream().forEach(l -> System.out.println(l));
return null;
});
ssc.start();
ssc.awaitTermination();
}
}

JavaFX Image Loading in Background and Threads

I thought this would be a simple question but I am having trouble finding an answer. I have a single ImageView object associated with a JavaFX Scene object and I want to load large images in from disk and display them in sequence one after another using the ImageView. I have been trying to find a good way to repeatedly check the Image object and when it is done loading in the background set it to the ImageView and then start loading a new Image object. The code I have come up with (below) works sometimes and sometimes it doesn't. I am pretty sure I am running into issues with JavaFX and threads. It loads the first image sometimes and stops. The variable "processing" is a boolean instance variable in the class.
What is the proper way to load an image in JavaFX in the background and set it to the ImageView after it is done loading?
public void start(Stage primaryStage) {
...
ImageView view = new ImageView();
((Group)scene.getRoot()).getChildren().add(view);
...
Thread th = new Thread(new Thread() {
public void run() {
while(true) {
if (!processing) {
processing = true;
String filename = files[count].toURI().toString();
Image image = new Image(filename,true);
image.progressProperty().addListener(new ChangeListener<Number>() {
#Override public void changed(ObservableValue<? extends Number> observable, Number oldValue, Number progress) {
if ((Double) progress == 1.0) {
if (! image.isError()) {
view.setImage(image);
}
count++;
if (count == files.length) {
count = 0;
}
processing = false;
}
}
});
}
}
}
});
}
I actually think there's probably a better general approach to satisfying whatever your application's requirements are than the approach you are trying to use, but here is my best answer at implementing the approach you describe.
Create a bounded BlockingQueue to hold the images as you load them. The size of the queue may need some tuning: too small and you won't have any "buffer" (so you won't be able to take advantage of any that are faster to load than the average), too large and you might consume too much memory. The BlockingQueue allows you to access it safely from multiple threads.
Create a thread that simply loops and loads each image synchronously, i.e. that thread blocks while each image loads, and deposits them in the BlockingQueue.
Since you want to try to display images up to once per FX frame (i.e. 60fps), use an AnimationTimer. This has a handle method that is invoked on each frame render, on the FX Application Thread, so you can implement it just to poll() the BlockingQueue, and if an image was available, set it in the ImageView.
Here's an SSCCE. I also indicated how to do this where you display each image for a fixed amount of time, as I think that's a more common use case and might help others looking for similar functionality.
import java.io.File;
import java.net.MalformedURLException;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.Executor;
import java.util.concurrent.Executors;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import javafx.animation.AnimationTimer;
import javafx.animation.PauseTransition;
import javafx.application.Application;
import javafx.concurrent.Task;
import javafx.scene.Parent;
import javafx.scene.Scene;
import javafx.scene.control.Button;
import javafx.scene.image.Image;
import javafx.scene.image.ImageView;
import javafx.scene.layout.BorderPane;
import javafx.scene.layout.Pane;
import javafx.scene.layout.StackPane;
import javafx.stage.DirectoryChooser;
import javafx.stage.Stage;
public class ScreenSaver extends Application {
#Override
public void start(Stage primaryStage) {
BorderPane root = new BorderPane();
Button startButton = new Button("Choose image directory...");
startButton.setOnAction(e -> {
DirectoryChooser chooser= new DirectoryChooser();
File dir = chooser.showDialog(primaryStage);
if (dir != null) {
File[] files = Stream.of(dir.listFiles()).filter(file -> {
String fName = file.getAbsolutePath().toLowerCase();
return fName.endsWith(".jpeg") | fName.endsWith(".jpg") | fName.endsWith(".png");
}).collect(Collectors.toList()).toArray(new File[0]);
root.setCenter(createScreenSaver(files));
}
});
root.setCenter(new StackPane(startButton));
primaryStage.setScene(new Scene(root, 800, 800));
primaryStage.show();
}
private Parent createScreenSaver(File[] files) {
ImageView imageView = new ImageView();
Pane pane = new Pane(imageView);
imageView.fitWidthProperty().bind(pane.widthProperty());
imageView.fitHeightProperty().bind(pane.heightProperty());
imageView.setPreserveRatio(true);
Executor exec = Executors.newCachedThreadPool(runnable -> {
Thread t = new Thread(runnable);
t.setDaemon(true);
return t ;
});
final int imageBufferSize = 5 ;
BlockingQueue<Image> imageQueue = new ArrayBlockingQueue<Image>(imageBufferSize);
exec.execute(() -> {
int index = 0 ;
try {
while (true) {
Image image = new Image(files[index].toURI().toURL().toExternalForm(), false);
imageQueue.put(image);
index = (index + 1) % files.length ;
}
} catch (MalformedURLException e) {
throw new RuntimeException(e);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
});
// This will show a new image every single rendering frame, if one is available:
AnimationTimer timer = new AnimationTimer() {
#Override
public void handle(long now) {
Image image = imageQueue.poll();
if (image != null) {
imageView.setImage(image);
}
}
};
timer.start();
// This wait for an image to become available, then show it for a fixed amount of time,
// before attempting to load the next one:
// Duration displayTime = Duration.seconds(1);
// PauseTransition pause = new PauseTransition(displayTime);
// pause.setOnFinished(e -> exec.execute(createImageDisplayTask(pause, imageQueue, imageView)));
// exec.execute(createImageDisplayTask(pause, imageQueue, imageView));
return pane ;
}
private Task<Image> createImageDisplayTask(PauseTransition pause, BlockingQueue<Image> imageQueue, ImageView imageView) {
Task<Image> imageDisplayTask = new Task<Image>() {
#Override
public Image call() throws InterruptedException {
return imageQueue.take();
}
};
imageDisplayTask.setOnSucceeded(e -> {
imageView.setImage(imageDisplayTask.getValue());
pause.playFromStart();
});
return imageDisplayTask ;
}
public static void main(String[] args) {
launch(args);
}
}

Read a String-object out of a .txt file from the res folder of a Blackberry app

I just started to develop a simple Blackberry app which shows a text sequence in a RichTextField on a MainScreen. When I define the String directly in the sourcecode, then I have no problem to display it. But if I try to read it in from a .txt file which is located in the res folder, then I get a NullPointerException.
The code below is what I did so far.
package mypackage;
import java.io.IOException;
import java.io.InputStream;
import net.rim.device.api.io.IOUtilities;
import net.rim.device.api.ui.component.RichTextField;
import net.rim.device.api.ui.container.MainScreen;
public final class MyScreen extends MainScreen{
String str = readFile("Testfile.txt");
public MyScreen(){
setTitle("Read Files");
add(new RichTextField(str));
}
public String readFile(String filename){
InputStream is = this.getClass().getResourceAsStream("/"+filename);
try {
byte[] filebytes = IOUtilities.streamToBytes(is);
is.close();
return new String(filebytes);
}
catch (IOException e){
System.out.println(e.getMessage());
}
return "";
}
}
Parts of this code I found in this forum but my problem is that I don't understand when I have to open a connection and when to close it.
And when do I need a Buffer?
And why do I have to convert a InputStream to a byte[] and then the byte[] to a String?
All I need is one method, where I can type in the Filename and get back a String-Object with the text which is in my .txt file.
And of course the method should save resources...
package mypackage;
import java.io.IOException;
import java.io.InputStream;
import net.rim.device.api.io.IOUtilities;
import net.rim.device.api.ui.component.RichTextField;
import net.rim.device.api.ui.container.MainScreen;
public final class MyScreen extends MainScreen {
public MyScreen() throws IOException {
setTitle("Read Files");
add(new RichTextField(readFileToString("Testfile.txt")));
}
public String readFileToString(String path) throws IOException {
InputStream is = getClass().getResourceAsStream("/"+path);
byte[] content = IOUtilities.streamToBytes(is);
is.close();
return new String(content);
}
}
Yes!!! I found a way to solve my problem.
I don't know why my previous code didn't work but this one works...
The only thing I've changed is that I've added the throws IOException instead of surrounding it with a try - catch block...

Resources