form_json return null values - apache-spark

Question

I am trying a parse a string column(containing json string) using from_json, but when i show my result dataframe it shows all the value as null. i am using all type as string, so there should not be any type conversion problem, but still final result is null.
i can show my originaldf and it shows the json string.
sample json :
{"type": "mytype", "version": "0.2", "id": "dc771a5f-336e-4f65-be1c-79de1848d859"}
i am reading the json string from file
originaldf = spark.read.option("header",false).schema("message as string").csv(myfilepath)
originaldf shows. it's not showing full value in console(running in local mode)
root
|-- message: string (nullable = true)
{"fields":[{"metadata":{},"name":"message","nullable":true,"type":"string"}],"type":"struct"}
+-----------------+
| message|
+-----------------+
|{"type": "mytype"|
+-----------------+
schema passed to from_json
{
"fields":[
{
"metadata":{
},
"name":"id",
"nullable":true,
"type":"string"
},
{
"metadata":{
},
"name":"version",
"nullable":true,
"type":"string"
},
{
"metadata":{
},
"name":"type",
"nullable":true,
"type":"string"
}
],
"type":"struct"
}
newdf = originaldf.select(from_json("message",schema).alias("parsedjson")).select("parsedjson.*")
newdf.show(), gives output
+----+--------+---------+
|id | version| type |
+----+--------+----------+
|null| null | null |
+----+--------+----------+

Answer 1

This is strange. I have reproduced it and it worked. I used Spark 2.4.3.
from pyspark.sql import *
row = Row(message='''{"type": "mytype", "version": "0.2", "id": "dc771a5f-336e-4f65-be1c-79de1848d859"}''')
df = spark.createDataFrame([row])
>>> df.show()
+--------------------+
| message|
+--------------------+
|{"type": "mytype"...|
+--------------------+
>>> schema = '''
... {
... "fields":[
... {
... "metadata":{
...
... },
... "name":"id",
... "nullable":true,
... "type":"string"
... },
... {
... "metadata":{
...
... },
... "name":"version",
... "nullable":true,
... "type":"string"
... },
... {
... "metadata":{
...
... },
... "name":"type",
... "nullable":true,
... "type":"string"
... }
... ],
... "type":"struct"
... }
... '''
>>> from pyspark.sql.functions import *
>>> newdf = df.select(from_json("message",schema).alias("parsedjson")).select("parsedjson.*")
>>> newdf.show()
+--------------------+-------+------+
| id|version| type|
+--------------------+-------+------+
|dc771a5f-336e-4f6...| 0.2|mytype|
+--------------------+-------+------+

Answer 2

thanks for your help. i was reading in originaldf as .csv, due to which data was not coming in df as complete json(df.show was showing partial data, so it looked like it has loaded full data but df.col().first.getstring(0) showed it was not full json but string till ',' as i was reading csv).when i used an dummy udf to return json string and it worked. –