How to run a Scala program via cron? - linux

I wrote a small Scala application. I have 2 classes in one source file including the App trait runner to start the program. It works just fine when I run it in the terminal:
scalac update.scala // compiling
scala update // run it
Now I want to run it with a cron job. For this I edited sudo crontab -e and added this:
*/2 * * * * scala /usr/bin/local/update
and made the script executable but nothing happend so far. I'm not sure how to do it:
Do I have to make a jar file for this?
Do I have to add this before my classes or not?
#!/bin/sh
exec scala -savecompiled $0 $#
!#
Does anyone have some experience with this?
Thanks in advance.

I suspect scala isn't in $PATH where cron can see it.
Try the following in a shell session:
$ which scala
Which should output something like "/opt/scala/2.9.1/bin/scala" or something. Could be in /usr/local, any number of places - java and the unix filesystem don't really play together nicely.
So now you have two options:
Put the folder where scala lives in the system path (This will usually involve editing /etc/profile, but you don't specify the OS so I can't say for sure)
(Easier) Just change the the cron entry to call /full/path/to/scala rather than just "scala"

The scala command expects the name of a compiled runnable object or a file containing a scala script source (or a runnable jar file) as the thing to run.
If you have in update.scala object update extends App (and no package declaration) then after scalac update.scala (which should have produced a bunch of *.class files) scala update is the right thing to run.
If the produced class files are not in the current directory then the -classpath option should be used to tell scala where to find them, as in eg. scala -classpath /usr/bin/local update, if the class files are indeed in /usr/bin/local.
Saying scala /usr/bin/local/update would make sense if the file /usr/bin/local/update (this exact name) contained scala script source (that is more or less a sequence of scala expressions not wrapped in a class or object).

Related

Getting Started with Mobius SparkClr (on Linux)

I am looking to try the C# driver with an existing (stand alone) spark cluster (on Ubuntu Linux) which I interact happily with via python or scala.
I am unclear as to how to run a simple c# example having downloaded the latest Mobius release to the linux box. What I am unclear about are those two extra parameters required for the clr spark submit (over and above the ones that are normally required). I am encountering various errors when i try to follow the submit args as documented (or I have misunderstood the instructions)
Firstly, for the --exe, does one simply point to the .exe file or is it required to pass; --exe [mono] [my_app.exe] [params]
Secondly, remote-spark-clr seems to insist on a HDFS path but I am running spark without HDFS. Is HDFS actually necessary?
Thirdly, and related to question (two), if distributing exe/packages for workers, must these also be in a hdfs path or can I put them somewhere sensible on the "regular" file system.
In short, I am looking for confirmation that HDFS is not required and a simple one-liner submit example that can run an exe in some location. The combinations I have tried are not working for me sadly.
Running Mobius on Linux requires a small trick:
Create shell scripts that are launching your executables using mono
Add the extension .exe to your shell scripts so that they are accepted by sparkclr-submit.
Make sure your shell scripts are linux encoded - we had issues when they had CRLF line endings.
If your application is called Driver.exe, I recommend to create a file driver.sh.exe with the following content:
#!/bin/sh
exec mono ./Driver.exe "$#"
Similarly, create a file CSharpDriver.sh.exe with the following content:
#!/bin/sh
exec mono ./CSharpWorker.exe "$#"
In your App.config set the following value in appSettings:
<add key="CSharpWorkerPath" value="CSharpWorker.sh.exe"/>
Finally, when submitting your application, use the following arguments:
$SPARKCLR_HOME/scripts/sparkclr-submit.sh \
--master yarn \
--deploy-mode client \
--exe driver.sh.exe \
/path/to/driver
Note that the --exe argument only takes the name of the file, the path is the next argument.
You can place your applications on the regular file system (don't need to use HDFS), but according to my experience, Mobius will internally use HDFS to distribute the application to the workers. I don't know if you can avoid it.

Installing Apache Spark on linux

I am installing Apache Spark on linux. I already have Java, Scala and Spark downloaded and they are all in the Downloads folder inside the Home folder with the path /home/alex/Downloads/X where X=scala, java, spark, literally that's what the folders are called.
I got scala to work but when I try to run spark by typing ./bin/spark-shell it says:
/home/alex/Downloads/spark/bin/saprk-class: line 100: /usr/bin/java/bin/java: Not a directory
I have already included the file path by editing the bashrc with sudo gedit ~/.bashrc:
# JAVA
export JAVA_HOME=/home/alex/Downloads/java
export PATH=$PATH:$JAVA_HOME/bin
# scala
export SCALA_HOME=/home/alex/Downloads/scala
export PATH=$PATH:$SCALA_HOME/bin
# spark
export SPARK_HOME=/home/alex/Downloads/spark
export PATH=$PATH:$SPARK_HOME/bin
When I try to type sbt/sbt package in the spark folder it say no such file or directory is found also. What should I do from here?
It seems you have a few issues, namely your JAVA_HOME is not pointed to a directory with java, when you are running sbt in spark you should run ./sbt/sbt (or in new versions ./build/sbt). While you can download Java & Scala by hand, you may find that your system packages are sufficient (make sure to get jdk 7 or later).
Furthermore, after using system packages as Holden points out, in Linux you may use the command whereis to make sure of the right paths.
Finally, the following link may prove useful:
http://www.tutorialspoint.com/apache_spark/apache_spark_installation.htm
Hope this helps.
Note: It looks like there may be a configuration issue, misspelling, of the directory name
/home/alex/Downloads/spark/bin/saprk-class: line 100: /usr/bin/java/bin/java: Not a directory
saprk-class
That could be a configuration issue only, but it's worth a look if it is called /spark-class elsewhere to see if it's causing related issues.

run Groovy CLI from groovy jar placed in war

I know about Groovysh, but I need to know if it is possible to run groovy CLI directly from groovy.jar placed in deployed war servlet (using one command). I can simplify question. Is there possibility to run Groovy CLI like it works, for example, in Clojure?
java -cp clojure-1.4.0.jar clojure.main
And CLI appears in terminal. This is how things look in clojure. I am looking for one line command which will run groovy CLI in terminal (using only groovy library to run it). I was looking for help in javadoc, found classes which should help, but don't know how to run it. :f
http://groovy.codehaus.org/api/groovy/lang/GroovyShell.html
(If someone knows solution which doesn't meet all criterias, also please answer.)
//EDIT
It seems that it needs groovy.jar, commons-cli.jar, antlr.jar, asm-util.jar and jline.jar. So I've added those files in my war file in WEB-INF/lib directory. Maybe it's good solution to make my own jar which role will be to call Groovy CLI from other jars, but now the question is, how ro run jar placed in WEB-INF/lib directory inside deployed war application via command line?
Greets
Seems like groovysh depends on more libs than the embeddable version contains. I managed to do what you want by using the following command:
$ java -classpath '*' org.codehaus.groovy.tools.shell.Main
But the current directory had to be the dir where all the groovy libs are; i.e. $GROOVY_HOME/lib:
$ cd $GROOVY_HOME/lib
$ java -classpath '*' org.codehaus.groovy.tools.shell.Main
Groovy Shell (2.0.6, JVM: 1.6.0_24)
Type 'help' or '\h' for help.
------------------------------------------------------------------------------
groovy:000>

javac error message does not display entire filepath

When using javac (or the ant task ), the error message does not include the entire filepath, it only includes the file name. For example,
$ javac src/path/to/Filename.java
Filename.java:1: package foo.bar does not exist
import foo.bar.Baz;
^
What I would like is,
$ javac src/path/to/Filename.java
src/path/to/Filename.java:1: package foo.bar does not exist
import foo.bar.Baz;
^
My problem is that vim quickfix does not work if it's not given the entire filepath, not just the filename. With just the filename, a new empty file is opened up after running :make.
I'm using:
Debian wheezy
openjdk-6-jdk v6b23~pre7-1
javac v1.6.0_23
I don't think there is a simple way to do this.
However, I believe that there is a complicated way. It basically involves writing your own compiler runner that makes use of a JDK installation's ability to load and run the Java compiler inside of a running program. You need to implement a lot of stuff, but the key thing is a diagnostic processor that formats the compiler error messages in the way that you need them to be formatted.
Here are some relevant links:
Package javax.tools - provides interfaces for tools which can be invoked from a program, for example, compilers.
JavaCompiler - the interface implemented by the compiler
FileObject - the interface that the compiler uses to represent source files; e.g. in diagnostics. Note the toUri() method!

Ant build in linux

I'm trying to run an ant build in linux and I'm getting odd results. I can't seem to get anything to run without specifying each and every library on the classpath, so I systematically added every jar in the "lib" and "ant/dep/lib" locations to the classpath explicitly.
It runs a bit without any issues (it's a setup batch, so in the first phase of the script, it uses a lot of "input" tasks), then I receive the message that org.apache.tools.ant.taskdefs.optional.PropertyFile was not found. Looking around, I discovered it was located in ant-nodeps.jar and I had added it to the class path, but no change.
Complete failure due to missing libraries I can understand, but why would the build partially function if I was sure I was including every library (including ant-nodeps.jar)?
The script file to launch it is as follows:
#!/QOpenSys/usr/bin/bsh
ANT_HOME=dep/ant
PARAMS="-Dant.home=$ANT_HOME -Dsystem.type=as400 -Dis-as400=y"
LIBS="lib/ant-contrib-1.0b3.jar:lib/ant-launcher.jar:lib/ant.jar:lib/catalina-ant.jar:lib/catalina-deployer.jar:lib/commons-net-2.2.jar:lib/el-api.jar:lib/ganymed.jar:lib/jakarta-oro-2.0.8.jar:lib/jasper-el.jar:lib/jasper.jar:lib/jsp-api.jar:lib/log4j-1.2.16.jar:lib/mail.jar"
ANTLIBS="$ANT_HOME/lib/ant-antlr.jar:$ANT_HOME/lib/ant-jai.jar:$ANT_HOME/lib/ant-starteam.jar:$ANT_HOME/lib/ant-apache-bcel.jar:$ANT_HOME/lib/ant-javamail.jar:$ANT_HOME/lib/ant-stylebook.jar:$ANT_HOME/lib/ant-apache-bsf.jar:$ANT_HOME/lib/ant-jdepend.jar:$ANT_HOME/lib/ant-swing.jar:$ANT_HOME/lib/ant-apache-log4j.jar:$ANT_HOME/lib/ant-jmf.jar:$ANT_HOME/lib/ant-testutil.jar:$ANT_HOME/lib/ant-apache-oro.jar:$ANT_HOME/lib/ant-jsch.jar:$ANT_HOME/lib/ant-trax.jar:$ANT_HOME/lib/ant-apache-regexp.jar:$ANT_HOME/lib/ant-junit.jar:$ANT_HOME/lib/ant-weblogic.jar:$ANT_HOME/lib/ant-apache-resolver.jar:$ANT_HOME/lib/ant-launcher.jar:$ANT_HOME/lib/ant.jar:$ANT_HOME/lib/ant-commons-logging.jar:$ANT_HOME/lib/ant-netrexx.jar:$ANT_HOME/lib/xercesImpl.jar:$ANT_HOME/lib/ant-commons-net.jar:$ANT_HOME/lib/ant-nodeps.jar:$ANT_HOME/lib/xml-apis.jar"
java -classpath $LIBS:$ANTLIBS $PARAMS org.apache.tools.ant.launch.Launcher -buildfile install/install.xml
Any help would be appreciated, even if only suggestions for what to try.
EDIT: In light of oers observation, I realized that there are shell scripts for ant afterall (have pity, I'm no expert in linux), so I'm going to attempt to use it to launch ant rather than try to do the same thing myself. I'll let everyone know of my progress and/or if I was able to fix it doing it that way.
I looked into the shellscript on my machine.
There is an additional parameter set:
-Dant.library.dir=\"$ANT_LIB\"
Perhaps this is needed?
ant_exec_command="exec \"$JAVACMD\" $ANT_OPTS
-classpath \"$LOCALCLASSPATH\"
-Dant.home=\"$ANT_HOME\"
-Dant.library.dir=\"$ANT_LIB\" $ant_sys_opts
org.apache.tools.ant.launch.Launcher
$ANT_ARGS
-cp \"$CLASSPATH\"
$ant_exec_args"
Using the ant scripts to launch the setup seemed to do the trick, specifying parameter -lib as lib and -buildfile as install/install.xml.

Resources