Spark fillNa not replacing the null value - apache-spark

Question

I have the following dataset and its contain some null values, need to replace the null value using fillna in spark.
DataFrame:
df = spark.read.format("com.databricks.spark.csv").option("header‌​","true").load("/sam‌​ple.csv")
>>> df.printSchema();
root
|-- Age: string (nullable = true)
|-- Height: string (nullable = true)
|-- Name: string (nullable = true)
>>> df.show()
+---+------+-----+
|Age|Height| Name|
+---+------+-----+
| 10| 80|Alice|
| 5| null| Bob|
| 50| null| Tom|
| 50| null| null|
+---+------+-----+
>>> df.na.fill(10).show()
when i'll give the na values it dosen't changed the same dataframe appeared again.
+---+------+-----+
|Age|Height| Name|
+---+------+-----+
| 10| 80|Alice|
| 5| null| Bob|
| 50| null| Tom|
| 50| null| null|
+---+------+-----+
tried create a new dataframe and store the fill values in dataframe but the result showing like unchanged.
>>> df2 = df.na.fill(10)
how to replace the null values? please give me the possible ways by using fill na.
Thanks in Advance.

Answer 1

It seems that your Height column is not numeric. When you call df.na.fill(10) spark replaces only nulls with column that match type of 10, which are numeric columns.
If Height column need to be string, you can try df.na.fill('10').show(), otherwise casting to IntegerType() is neccessary.

Answer 2

You can also provide a specific default value for each column if you prefer.
df.na.fill({'Height': '10', 'Name': 'Bob'})

Answer 3

To add to #Mariusz answer, here is the exact code to cast and fill NA values:
from pyspark.sql.types import IntegerType
from pyspark.sql.functions import col
df = df.withColumn("Height", col("Height").cast(IntegerType()))
df2 = df.na.fill(value=10, subset=["Height"])
maybe the simpler solution would've been to give a string value, if you don't care about the column type:
df2 = df.na.fill(value="10", subset=["Height"])