Fetch exception from spark launcher thread - multithreading

Is there a way to fetch the launcher exception from the launcher thread?
SparkLauncher launcher = new SparkLauncher()
.setSparkHome("sparkhome")
.setMaster("yarn-cluster")
.setAppResource("spark job jar file")
.setMainClass("spark job driver class")
.setAppName("appname")
.addAppArgs(argsArray)
.setVerbose(true)
.addSparkArg("--verbose");
SparkAppHandle handle = launcher.startApplication(new LauncherListener());
Need to get exception thrown by launcher.startApplication() method.
Note- Try catch block doesn't work as launcher.startApplication() method creates separate thread.

Related

Not on FX application thread; currentThread = Thread-5 [duplicate]

This question already has answers here:
Why am I getting java.lang.IllegalStateException "Not on FX application thread" on JavaFX?
(3 answers)
Closed 3 years ago.
Exception in thread "Thread-5" java.lang.IllegalStateException: Not on FX application thread; currentThread = Thread-5
at com.sun.javafx.tk.Toolkit.checkFxUserThread(Toolkit.java:279)
at com.sun.javafx.tk.quantum.QuantumToolkit.checkFxUserThread(QuantumToolkit.java:423)
at javafx.scene.Parent$2.onProposedChange(Parent.java:367)
at com.sun.javafx.collections.VetoableListDecorator.setAll(VetoableListDecorator.java:113)
at com.sun.javafx.collections.VetoableListDecorator.setAll(VetoableListDecorator.java:108)
at com.sun.javafx.scene.control.skin.LabeledSkinBase.updateChildren(LabeledSkinBase.java:575)
at com.sun.javafx.scene.control.skin.LabeledSkinBase.handleControlPropertyChanged(LabeledSkinBase.java:204)
at com.sun.javafx.scene.control.skin.LabelSkin.handleControlPropertyChanged(LabelSkin.java:49)
at com.sun.javafx.scene.control.skin.BehaviorSkinBase.lambda$registerChangeListener$70(BehaviorSkinBase.java:197)
at com.sun.javafx.scene.control.MultiplePropertyChangeListenerHandler$1.changed(MultiplePropertyChangeListenerHandler.java:55)
at javafx.beans.value.WeakChangeListener.changed(WeakChangeListener.java:89)
at com.sun.javafx.binding.ExpressionHelper$SingleChange.fireValueChangedEvent(ExpressionHelper.java:182)
at com.sun.javafx.binding.ExpressionHelper.fireValueChangedEvent(ExpressionHelper.java:81)
at javafx.beans.property.StringPropertyBase.fireValueChangedEvent(StringPropertyBase.java:103)
at javafx.beans.property.StringPropertyBase.markInvalid(StringPropertyBase.java:110)
at javafx.beans.property.StringPropertyBase.set(StringPropertyBase.java:144)
at javafx.beans.property.StringPropertyBase.set(StringPropertyBase.java:49)
at javafx.beans.property.StringProperty.setValue(StringProperty.java:65)
at javafx.scene.control.Labeled.setText(Labeled.java:145)
at edu.controller.GUIController.statusShow(GUIController.java:443)
at edu.controller.GUIController.lambda$7(GUIController.java:214)
at java.lang.Thread.run(Unknown Source)
I'm making a text editor and I want to add a status-bar in the footer that tells user different tips after a few seconds and I'm facing this error when I try to set text on the label but when I try to set that text on console that works fine.
new Thread(()->{
statusBarShow();
}).start();
private void statusBarShow(){
try {
statusLable.setText("Tip 1");
Thread.sleep(2000);
statusLable.setText("Tip 2");
Thread.sleep(2000);
statusLable.setText("Tip 3");
Thread.sleep(2000);
statusLable.setText("Tip 4");
Thread.sleep(2000);
statusLable.setText("Tip 5");
Thread.sleep(2000);
} catch (Exception e) {
e.printStackTrace();
}
}
The only thread that is allowed to modify the JavaFX GUI is the JavaFX thread. If any other thread modifies the UI, you will get an exception.
The most simple answer to this common problem is to wrap the code that changes the GUI in Platform.runLater(). Be careful not to put any sleeps in the runlater(), as they will cause the JavaFX GUI thread to sleep, which will cause your program to freeze for the duration of the sleep.
Platform.runLater() has the following javadoc:
Run the specified Runnable on the JavaFX Application Thread at some
unspecified time in the future. This method, which may be called from
any thread, will post the Runnable to an event queue and then return
immediately to the caller. The Runnables are executed in the order
they are posted. A runnable passed into the runLater method will be
executed before any Runnable passed into a subsequent call to
runLater. If this method is called after the JavaFX runtime has been
shutdown, the call will be ignored: the Runnable will not be executed
and no exception will be thrown.
NOTE: applications should avoid flooding JavaFX with too many pending
Runnables. Otherwise, the application may become unresponsive.
Applications are encouraged to batch up multiple operations into fewer
runLater calls. Additionally, long-running operations should be done
on a background thread where possible, freeing up the JavaFX
Application Thread for GUI operations.
This method must not be called before the FX runtime has been
initialized. For standard JavaFX applications that extend Application,
and use either the Java launcher or one of the launch methods in the
Application class to launch the application, the FX runtime is
initialized by the launcher before the Application class is loaded.
For Swing applications that use JFXPanel to display FX content, the FX
runtime is initialized when the first JFXPanel instance is
constructed. For SWT application that use FXCanvas to display FX
content, the FX runtime is initialized when the first FXCanvas
instance is constructed.
I don't think your code is structured in the best way to accomplish this task, but a very simple solution is the following:
new Thread(()->{
statusBarShow();
}).start();
private void statusBarShow(){
try {
Platform.runLater(()->statusLable.setText("Tip 1"));
Thread.sleep(2000);
Platform.runLater(statusLable.setText("Tip 2"));
Thread.sleep(2000);
Platform.runLater(statusLable.setText("Tip 3"));
Thread.sleep(2000);
Platform.runLater(statusLable.setText("Tip 4"));
Thread.sleep(2000);
Platform.runLater(statusLable.setText("Tip 5"));
} catch (Exception e) {
e.printStackTrace();
}
}
A better solution to your problem may be to use an AnimationTimer.
Here is a useful thread on how to accomplish that: JavaFX periodic background task

How To get the values returned from Spark Job class into Spark Launcher class

I am using the below piece of code to launch the Spark Job.
I want to get the result from com.test.Foo Class into the Launcher class which starts the spark application.
I don't want to write the values to a temporary file and read it from the Launcher Class. Is there a way to get the required values returned from the spark main class com.test.Foo .
Is there a way to set the values to current session and get it wherever required?
SparkAppHandle handler = new SparkLauncher()
.setAppResource(<path to jar>)
.setMaster("yarn-cluster")
.setDeployMode("cluster")
.setVerbose(true)
.setMainClass("com.test.Foo")
.addAppArgs(args[0],args[1])
.startApplication();

Got UnsatisfiedLinkError when start Spark program from Java code

I am using SparkLauncher to launch my spark app from Java.The code looks like
Map<String, String> envMap = new HashMap<>();
envMap.put("HADOOP_CONF_DIR","/etc/hadoop/conf");
envMap.put("JAVA_LIBRARY_PATH", "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/lib/native");
envMap.put("LD_LIBRARY_PATH", "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/lib/native");
envMap.put("SPARK_HOME","/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/spark");
envMap.put("DEFAULT_HADOOP_HOME","/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop");
envMap.put("SPARK_DIST_CLASSPATH","all jars under /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars");
envMap.put("HADOOP_HOME","/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop");
SparkLauncher sparklauncher = new SparkLauncher(envMap)
.setAppResource("myapp.jar")
.setSparkHome("/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/spark/")
.setMainClass("spark.HelloSpark")
.setMaster("yarn-cluster")
.setConf(SparkLauncher.DRIVER_MEMORY, "2g")
.setConf("spark.driver.userClassPathFirst", "true")
.setConf("spark.executor.userClassPathFirst", "true").launch();
Every time, I got
User class threw exception: java.lang.UnsatisfiedLinkError:
org.xerial.snappy.SnappyNative.maxCompressedLength(I)I
It looks like your jar is including Spark/Hadoop libraries that have conflict with other libraries in the cluster. Check that your Spark and Hadoop dependencies are marked as provided.

Spark uses only one core when using forEachRemaining

I'm still new to Spark. I wrote this code to parse large string list.
I had to use forEachRemaining because I had to initialize some non-serializable objects in each partition.
JavaRDD<String> lines=initRetriever();
lines.foreachPartition(iter->{
Obj1 obj1=initObj1()
MyStringParser parser=new MyStringParser(obj1);
iter.forEachRemaining(str->{
try {
parser.parse(str);
} catch (ParsingException e) {
e.printStackTrace();
}
});
System.out.print("all parsed");
obj1.close();
});
I believe Spark is all about parallelism. But this program uses only a single thread on my local machine. Did I do something wrong? Missing configuration? Or maybe the iter doesn't allow it to execute all in parallel.
EDIT
I have no configuration files for Spark.
That's how I initialize Spark
SparkConf conf = new SparkConf()
.setAppName(AbstractSparkImporter.class.getCanonicalName())
.setMaster("local");
I run it on IDE and using mvn:exec command.
As #Alberto-Bonsanto indicated, using local[*] triggers Spark to use all available threads. More info here.
SparkConf conf = new SparkConf()
.setAppName(AbstractSparkImporter.class.getCanonicalName())
.setMaster("local[*]");

Cause of assertion failure

Any idea what would cause this assertion failure in a Groovy script?
Exception in thread "Thread-2" groovy.lang.GroovyRuntimeException: exception while dumping process stream
at org.codehaus.groovy.runtime.ProcessGroovyMethods$ByteDumper.run(ProcessGroovyMethods.java:488)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Bad file descriptor
at java.io.FileInputStream.available(Native Method)
at java.io.BufferedInputStream.read(BufferedInputStream.java:325)
at java.io.FilterInputStream.read(FilterInputStream.java:90)
at org.codehaus.groovy.runtime.ProcessGroovyMethods$ByteDumper.run(ProcessGroovyMethods.java:484)
... 1 more
I'm running Groovy 1.8.3 with JVM 1.6.0_14. I noticed an old bug report that was similar but I perform a waitFor() call after every process is started and it's reported to be fixed in 1.6.x.
I can't trap the exception apparently so I don't know exactly where it is happening. However, I'm guessing it occurs in this code segment:
def pipedOutputStream = new PipedOutputStream()
process.consumeProcessOutput(
new TeeOutputStream(System.out, pipedOutputStream),
new TeeOutputStream(System.err, pipedOutputStream))
new PipedInputStream(pipedOutputStream).eachLine {
(it =~ /condition/).find { process.destroy() }
}
System.exit(0)

Resources