How to convert messages from socket streaming source to custom domain object? - apache-spark

Question

I'm very new to spark streaming. I have a Spark Standalone 2.2 running with one worker. I'm using a socket source and trying to read the incoming stream into an object called MicroserviceMessage.
val message = spark.readStream
.format("socket")
.option("host", host)
.option("port", port)
.load()
val df = message.as[MicroserviceMessage].flatMap(microserviceMessage =>
microserviceMessage.DataPoints.map(datapoint => (datapoint, microserviceMessage.ServiceProperties, datapoint.EpochUTC)))
.toDF("datapoint", "properties", "timestamp")
I'm hoping this will a DataFrame with columns of "datapoint", "properties" and "timestamp"
The data i'm pasting into my netcat terminal looks like this (this is what I'm trying to read in as MicroserviceMessage):
{
"SystemType": "mytype",
"SystemGuid": "6c84fb90-12c4-11e1-840d-7b25c5ee775a",
"TagType": "Raw Tags",
"ServiceType": "FILTER",
"DataPoints": [
{
"TagName": "013FIC003.PV",
"EpochUTC": 1505247956001,
"ItemValue": 25.47177,
"ItemValueStr": "NORMAL",
"Quality": "Good",
"TimeOffset": "P0000"
},
{
"TagName": "013FIC003.PV",
"EpochUTC": 1505247956010,
"ItemValue": 26.47177,
"ItemValueStr": "NORMAL",
"Quality": "Good",
"TimeOffset": "P0000"
}
],
"ServiceProperties": [
{
"Key": "OutputTagName",
"Value": "FI12102.PV_CL"
},
{
"Key": "OutputTagType",
"Value": "Cleansing Flow Tags"
}
]
}
Instead what I see is:
Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve '`SystemType`' given input columns: [value];
MicroserviceMessage case class looks like this:
case class DataPoints
(
TagName: String,
EpochUTC: Double,
ItemValue: Double,
ItemValueStr: String,
Quality: String,
TimeOffset: String
)
case class ServiceProperties
(
Key: String,
Value: String
)
case class MicroserviceMessage
(
SystemType: String,
SystemGuid: String,
TagType: String,
ServiceType: String,
DataPoints: List[DataPoints],
ServiceProperties: List[ServiceProperties]
)
EDIT:
After reading this post I was able to start the job by doing
val messageEncoder = Encoders.bean(classOf[MicroserviceMessage])
val df = message.select($"value").as(messageEncoder).map(
msmg => (msmg.ServiceType, msmg.SystemGuid)
).toDF("service", "guid")
But this causes issues when I start sending data.
Caused by: java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize
Full stacktrace

Answer 1

This:
message.as[MicroserviceMessage]
is incorrect as explained by the error message:
cannot resolve 'SystemType' given input columns: [value];
Data that comes from SocketStream is just string (or string and timestamp). To make it usable for strongly typed Dataset you have to parse it, for example with org.apache.spark.sql.functions.from_json.

Answer 2

The reason for the exception
Caused by: java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize
is that you compiled your Spark Structured Streaming application using Scala 2.12.4 (or any other in 2.12 stream) which is unsupported in Spark 2.2.
From the scaladoc of scala.runtime.LambdaDeserializer:
This class is only intended to be called by synthetic $deserializeLambda$ method that the Scala 2.12 compiler will add to classes hosting lambdas.
Spark 2.2 supports up to and including Scala 2.11.12 with 2.11.8 being the most "blessed" version.