Using self-defined data transform function in Spark Structured Stream - apache-spark

Question

I read the following blog and find the API is very useful.
https://databricks.com/blog/2017/02/23/working-complex-data-formats-structured-streaming-apache-spark-2-1.html
In the blog, there are lots of data selection example. Like using input
{
"a": {
"b": 1
}
}
Apply Scala: events.select("a.b"), the output would be
{
"b": 1
}
But data type conversion are not mentioned in the blog. Saying I have the following input:
{
"timestampInSec": "1514917353",
"ip": "123.39.76.112",
"money": "USD256",
"countInString": "6"
}
The expected output is:
{
"timestamp": "2018-01-02 11:22:33",
"ip_long": 2066173040,
"currency": "USD",
"money_amount": 256,
"count": 6
}
There are some transformations that not included in org.apache.spark.sql.functions._:
Timestamp is in second and is a string type
Convert IP to long
Split USD256 to two columns and convert one of the column to number
Convert string to number
Another thing is error handling and default value. If there is an invalid input like:
{
"timestampInSec": "N/A",
"money": "999",
"countInString": "Number-Six"
}
It is expected that the output can be
{
"timestamp": "1970-01-01 00:00:00",
"ip_long": 0,
"currency": "NA",
"money_amount": 999,
"count": -1
}
input timestampInSec is not a number. It is expected to use 0 and create a timestamp string as return value
ip is missing in the input. It is expected to usea default value 0.
money field is not complete. It has money amount but missed currency. It is expected to use NA as default currency and correctly translate the money_amount
countInString is not a number. It is expected to use -1 (not 0) as default value .
These requirments are not common and need some self-defined business logic code.
I do checked some function like to_timestamp. There are some codegen stuff and seems not very easy to add new functions. Is there some guide/document about writing self-defined transformation function? Is there a easy way to meet the requirments?

Answer 1

For all:
import org.apache.spark.sql.functions._
Timestamp is in second and is a string type
val timestamp = coalesce(
$"timestampInSec".cast("long").cast("timestamp"),
lit(0).cast("timestamp")
).alias("timestamp")
Split USD256 to two columns and convert one of the column to number
val currencyPattern = "^([A-Z]+)?([0-9]+)$"
val currency = (trim(regexp_extract($"money", currencyPattern, 1)) match {
case c => when(length(c) === 0, "NA").otherwise(c)
}).alias("currency")
val amount = regexp_extract($"money", currencyPattern, 2)
.cast("decimal(38, 0)").alias("money_amount")
Convert string to number
val count = coalesce($"countInString".cast("long"), lit(-1)).alias("count")
Convert IP to long
val ipPattern = "^([0-9]{1,3})\\.([0-9]{1,3})\\.([0-9]{1,3})\\.([0-9]{1,3})"
val ip_long = coalesce(Seq((1, 24), (2, 16), (3, 8), (4, 0)).map {
case (group, numBits) => shiftLeft(
regexp_extract($"ip", ipPattern, group).cast("long"),
numBits
)
}.reduce(_ + _), lit(0)).alias("ip_long")
Result
val df = Seq(
("1514917353", "123.39.76.112", "USD256", "6"),
("N/A", null, "999", null)
).toDF("timestampInSec", "ip", "money", "countInString")
df.select(timestamp, currency, amount, count, ip_long).show
// +-------------------+--------+------------+-----+----------+
// | timestamp|currency|money_amount|count| ip_long|
// +-------------------+--------+------------+-----+----------+
// |2018-01-02 18:22:33| USD| 256| 6|2066173040|
// |1970-01-01 00:00:00| NA| 999| -1| 0|
// +-------------------+--------+------------+-----+----------+