How do I convert Binary string to scala string in spark scala - string

Question

I am reading an avro file which contains a field as binary string, I need to convert it into a java.lang.string to pass it to another library(spark-xml-util), how do I convert it into java.lang.string efficiently. This is the code I have got so far : -
val df = sqlContext.read.format("com.databricks.spark.avro").load("filePath/fileName.avro")
df.select("myField").collect().mkString
The last line gives me the following exception: -
Exception in thread "main" java.lang.ClassCastException: [B cannot be cast to java.lang.String
at org.apache.spark.sql.Row$class.getString(Row.scala:255)
at org.apache.spark.sql.catalyst.expressions.GenericRow.getString(rows.scala:165)
df schema is: -
root
|-- id: string (nullable = true)
|-- myField: binary (nullable = true)

Answer 1

Considering the state of the API right now (2.2.0), your best call is to create a UDF to do just that and replace the column :
import org.apache.spark.sql.functions.udf
val toString = udf((payload: Array[Byte]) => new String(payload))
df.withColumn("myField", toString(df("myField")))
or if as you seem to imply the data is compressed using GZIP you can :
import org.apache.spark.sql.functions.udf
val toString = udf((payload: Array[Byte]) => {
val inputStream = new GZIPInputStream(new ByteArrayInputStream(payload))
scala.io.Source.fromInputStream(inputStream).mkString
})
df.withColumn("myField", toString(df("myField")))

Answer 2

In Spark 3.0 you can cast between BINARY and STRING data.
scala> val df = sc.parallelize(Seq("ABC", "BCD", "CDE", "DEF")).toDF("value")
df: org.apache.spark.sql.DataFrame = [value: string]
scala> df.select($"value", $"value".cast("BINARY"),
$"value".cast("BINARY").cast("STRING")).show()
+-----+----------+-----+
|value| value|value|
+-----+----------+-----+
| ABC|[41 42 43]| ABC|
| BCD|[42 43 44]| BCD|
| CDE|[43 44 45]| CDE|
| DEF|[44 45 46]| DEF|
+-----+----------+-----+
I don't have your data to test with, but you should be able to do:
df.select($"myField".cast("STRING"))
This obviously depends on the actual data (i.e. don't cast a JPEG to STRING) but assuming it's UTF-8 encoded this should work.

Answer 3

In the previous solution, the code new String(payload) did not work for me on true binary data.
Ultimately the solution was a little more involved, with the length of the binary data required as a 2nd parameter.
def binToString(payload: Array[Byte], payload_length: Int): String = {
val ac: Array[Char] = Range(0,payload_length).map(i => payload(i).toChar).toArray
return ac.mkString
}
val binToStringUDF = udf( binToString(_: Array[Byte], _: Int): String )