SparkCLR: process Text file fails - apache-spark

Question

I am trying to learn SparkCLR to process a text file and run a Spark SQL query on it using a Sample like below:
[Sample]
internal static void MyDataFrameSample()
{
var schemaTagValues = new StructType(new List<StructField>
{
new StructField("tagname", new StringType()),
new StructField("time", new LongType()),
new StructField("value", new DoubleType()),
new StructField("confidence", new IntegerType()),
new StructField("mode", new IntegerType())
});
var rddTagValues1 = SparkCLRSamples.SparkContext.TextFile(SparkCLRSamples.Configuration.GetInputDataPath(myDataFile))
.Map(r => r.Split('\t')
.Select(s => (object)s).ToArray());
var dataFrameTagValues = GetSqlContext().CreateDataFrame(rddTagValues1, schemaTagValues);
dataFrameTagValues.RegisterTempTable("tagvalues");
//var qualityFilteredDataFrame = GetSqlContext().Sql("SELECT tagname, value, time FROM tagvalues where confidence > 85");
var qualityFilteredDataFrame = GetSqlContext().Sql("SELECT * FROM tagvalues");
var data = qualityFilteredDataFrame.Collect();
var filteredCount = qualityFilteredDataFrame.Count();
Console.WriteLine("Filter By = 'confidence', RowsCount={0}", filteredCount);
}
but this keeps giving me the error that says:
[2016-01-13 08:56:28,593] [8] [ERROR] [Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge] - JVM method execution failed: Static method collectAndServe failed for class org.apache.spark.api.python.PythonRDD when called with 1 parameters ([Index=1, Type=JvmObjectReference, Value=19], )
[2016-01-13 08:56:28,593] [8] [ERROR] [Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge] -
*******************************************************************************************************************************
at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] parameters) in d:\SparkCLR\csharp\Adapter\Microsoft.Spark.CSharp\Interop\Ipc\JvmBridge.cs:line 91
*******************************************************************************************************************************
my text file looks like below:
10PC1008.AA 130908762000000000 7.059829 100 0
10PC1008.AA 130908762050000000 7.060376 100 0
10PC1008.AA 130908762100000000 7.059613 100 0
10PC1008.BB 130908762150000000 7.059134 100 0
10PC1008.BB 130908762200000000 7.060124 100 0
Is there something wrong in the way I am using this?
Edit 1
I have set the following as my Samples project properties:
my User Environmentalvariable is as below:(not sure if that matters)
Also what i see in SparkCLRWorker log that its unable to load the assembly as per the logs:
[2016-01-14 08:37:01,865] [1] [ERROR] [Microsoft.Spark.CSharp.Worker] - System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
---> System.IO.FileNotFoundException: Could not load file or assembly 'SparkCLRSamples, Version=1.5.2.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.
at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean forIntrospection)
at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
at System.Reflection.Assembly.Load(String assemblyString)
at System.Runtime.Serialization.FormatterServices.LoadAssemblyFromString(String assemblyName)
at System.Reflection.MemberInfoSerializationHolder..ctor(SerializationInfo info, StreamingContext context)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.SerializationInvoke(IRuntimeMethodInfo method, Object target, SerializationInfo info, StreamingContext& context)
at System.Runtime.Serialization.ObjectManager.CompleteISerializableObject(Object obj, SerializationInfo info, StreamingContext context)
at System.Runtime.Serialization.ObjectManager.FixupSpecialObject(ObjectHolder holder)
at System.Runtime.Serialization.ObjectManager.DoFixups()
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream)
at Microsoft.Spark.CSharp.Worker.Main(String[] args) in d:\SparkCLR\csharp\Worker\Microsoft.Spark.CSharp\Worker.cs:line 149

Answer 1

Did you specify the sample data location and copy your source text file to that location? If not you can refer to
https://github.com/Microsoft/SparkCLR/blob/master/csharp/Samples/Microsoft.Spark.CSharp/samplesusage.md
to set your sample data location with argument [--data | sparkclr.sampledata.loc].

Answer 2

Try explicitly setting [--temp | spark.local.dir] option (see sampleusage.md for more info on supported arguments) when you run your code. SparkCLR worker executable gets downloaded into this directory at the time of execution. If you use default temp directory, there is a possibility that the worker executable will get quarantined by your anti-virus software mistaking it for some malicious program downloaded by your browser. Overriding the default value to something like c:\temp\SparkCLRTemp will help avoiding that problem.
If setting temp directory does not help, share the entire commandline argument list you are using when launching SparkCLR driver code.

Answer 3

this is how you change the port number, I hope this helps
in App.config add the following
for the sake of completeness, you must also add the tag that specifies the path to the csharpworker
<appSettings>
<add key="CSharpBackendPortNumber" value="num"/>
<add key="CSharpWorkerPath" value="C:\MobiusRelease\samples\CSharpWorker.exe"/>
</appSettings>
notice, to make this work in debug mode you should first run this command using command line,
from the (mobius home) directory
%SPARKCLR_HOME%\scripts
run
sparkclr-submit.cmd debug
this will give you a message like this that contains a port number
[CSharpRunner.main] Port number used by CSharpBackend is 5567
* [CSharpRunner.main] Backend running debug mode. Press enter to exit *