How to pass decimal as a value when creating a PySpark dataframe? - apache-spark

Question

I want to create a dummy dataframe with one row which has Decimal values in it. But when do so it automatically converts it to a double. I want the data type to be Decimal(18,2) or etc.
dummy_row = spark.createDataFrame([(0,-1,'missing','missing',0.0],df.columns)
I want the schema to be
unique_id:integer
line_id:long
line_name:string
line_type:string
pct:decimal(18,5)
But I get
unique_id:integer
line_id:long
line_name:string
line_type:string
pct:double
How do I cast the double to decimal type in PySpark?

Answer 1

There are 2 ways.
My preferred way is to cast decimal type after the dataframe is created.
It is safest to provide values as strings.
df = spark.createDataFrame([("92.34567890123456789", )], ["col_1"])
df = df.withColumn("col_1", F.col("col_1").cast("decimal(30,20)"))
df.show(truncate=False)
df.printSchema()
# +-----------------------+
# |col_1 |
# +-----------------------+
# |92.34567890123456789000|
# +-----------------------+
#
# root
# |-- col_1: decimal(30,20) (nullable = true)
If values were provided as numbers, Python may "truncate" your values, because it would first create double precision floating point numbers (16-17 significant digits) out of what was provided in the code. Note how the last digits 789 would disappear:
df = spark.createDataFrame([(92.34567890123456789, )], ["col_1"])
df = df.withColumn("col_2", F.col("col_1").cast("decimal(30,20)"))
df.show(truncate=False)
df.printSchema()
# +-----------------+-----------------------+
# |col_1 |col_2 |
# +-----------------+-----------------------+
# |92.34567890123456|92.34567890123456000000|
# +-----------------+-----------------------+
#
# root
# |-- col_1: double (nullable = true)
# |-- col_2: decimal(30,20) (nullable = true)
The other way is to create decimal numbers beforehand.
from decimal import Context
dec_number = Context(prec=38).create_decimal('92.34567890123456789')
df = spark.createDataFrame([(dec_number, )], 'col_1 decimal(30,20)')
df.show(truncate=False)
df.printSchema()
# +-----------------------+
# |col_1 |
# +-----------------------+
# |92.34567890123456789000|
# +-----------------------+
#
# root
# |-- col_1: decimal(30,20) (nullable = true)
38 is the maximum decimal precision which is supported in Spark.
So, in your case you could:
1.1. - create dataframe from scratch - casting to decimal after df is created
dummy_row = spark.createDataFrame(
[(0, -1, 'missing', 'missing', '0.0')],
"unique_id:int, line_id:long, line_name:string, line_type:string, pct:string")
dummy_row = dummy_row.withColumn("pct", F.col("pct").cast("decimal(18,5)"))
dummy_row.show(truncate=False)
dummy_row.printSchema()
# +---------+-------+---------+---------+-------+
# |unique_id|line_id|line_name|line_type|pct |
# +---------+-------+---------+---------+-------+
# |0 |-1 |missing |missing |0.00000|
# +---------+-------+---------+---------+-------+
#
# root
# |-- unique_id: integer (nullable = true)
# |-- line_id: long (nullable = true)
# |-- line_name: string (nullable = true)
# |-- line_type: string (nullable = true)
# |-- pct: decimal(18,5) (nullable = true)
1.2. - create dataframe from scratch - using Python's decimal module
from decimal import Context
dec_number = Context(prec=38).create_decimal('0.0')
dummy_row = spark.createDataFrame(
[(0, -1, 'missing', 'missing', dec_number)],
"unique_id:int, line_id:long, line_name:string, line_type:string, pct:decimal(18,5)")
dummy_row.show(truncate=False)
dummy_row.printSchema()
# +---------+-------+---------+---------+-------+
# |unique_id|line_id|line_name|line_type|pct |
# +---------+-------+---------+---------+-------+
# |0 |-1 |missing |missing |0.00000|
# +---------+-------+---------+---------+-------+
#
# root
# |-- unique_id: integer (nullable = true)
# |-- line_id: long (nullable = true)
# |-- line_name: string (nullable = true)
# |-- line_type: string (nullable = true)
# |-- pct: decimal(18,5) (nullable = true)
2.1. - borrow schema from existing df (it seems you originally wanted this) - casting to decimal after df is created
df = spark.createDataFrame(
[],
"unique_id:int, line_id:long, line_name:string, line_type:string, pct:decimal(18,5)")
dummy_row = spark.createDataFrame([(0, -1, 'missing', 'missing', '0.0')], df.columns)
dummy_row = dummy_row.withColumn("pct", F.col("pct").cast("decimal(18,5)"))
dummy_row.show(truncate=False)
dummy_row.printSchema()
# +---------+-------+---------+---------+-------+
# |unique_id|line_id|line_name|line_type|pct |
# +---------+-------+---------+---------+-------+
# |0 |-1 |missing |missing |0.00000|
# +---------+-------+---------+---------+-------+
#
# root
# |-- unique_id: long (nullable = true)
# |-- line_id: long (nullable = true)
# |-- line_name: string (nullable = true)
# |-- line_type: string (nullable = true)
# |-- pct: decimal(18,5) (nullable = true)
2.2. - borrow schema from existing df (it seems you originally wanted this) - using Python's decimal module
from decimal import Context
df = spark.createDataFrame(
[],
"unique_id:int, line_id:long, line_name:string, line_type:string, pct:decimal(18,5)")
dec_number = Context(prec=38).create_decimal('0.0')
dummy_row = spark.createDataFrame([(0, -1, 'missing', 'missing', dec_number)], df.schema)
dummy_row.show(truncate=False)
dummy_row.printSchema()
# +---------+-------+---------+---------+-------+
# |unique_id|line_id|line_name|line_type|pct |
# +---------+-------+---------+---------+-------+
# |0 |-1 |missing |missing |0.00000|
# +---------+-------+---------+---------+-------+
#
# root
# |-- unique_id: integer (nullable = true)
# |-- line_id: long (nullable = true)
# |-- line_name: string (nullable = true)
# |-- line_type: string (nullable = true)
# |-- pct: decimal(18,5) (nullable = true)