NORM PPF function throws error on a column - apache-spark

Question

norm.ppf function works on single value.
from scipy import stats
from scipy.stats import norm
pct_5 = norm.ppf(0.008)
print(pct_5)
When I use it on a column; it throws me error.
I have tried two methods
Applying the function directly on a column
df1 = df.withColumn('ppf_col',norm.ppf(col(col1)))
Error : ValueError: Cannot convert column into bool
Using a UDF
def ppf():
return (norm.ppf(col('col1')))
my_udf = udf(ppf,FloatType())
df = df1.withColumn('ppf_col',my_udf(col('col1')))
df.show()
Error:
Py4JJavaError: An error occurred while calling o1780.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 8.0 failed 4 times, most recent failure: Lost task 0.3 in stage 8.0 (TID 76, d2-td-cdh.boigroup.net, executor 14): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/var/opt/teradata/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/python/lib/pyspark.zip/pyspark/worker.py", line 361, in main
func, profiler, deserializer, serializer = read_udfs(pickleSer, infile, eval_type)
File "/var/opt/teradata/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/python/lib/pyspark.zip/pyspark/worker.py", line 236, in read_udfs
arg_offsets, udf = read_single_udf(pickleSer, infile, eval_type, runner_conf)
File "/var/opt/teradata/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/python/lib/pyspark.zip/pyspark/worker.py", line 163, in read_single_udf
f, return_type = read_command(pickleSer, infile)
File "/var/opt/teradata/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/python/lib/pyspark.zip/pyspark/worker.py", line 64, in read_command
command = serializer._read_with_length(file)
File "/var/opt/teradata/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 172, in _read_with_length
return self.loads(obj)
File "/var/opt/teradata/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 577, in loads
return pickle.loads(obj, encoding=encoding)
ModuleNotFoundError: No module named 'scipy'
Note : I have scipy version 1.2.0; it stills throws error ModuleNotFoundError: No module named 'scipy'
I want to understand why it doesnt work on a column
col1 values:
7.999999999999999E-4
0.013793103448275862
0.013612808415190657
1.0
1.0
0.05449976056308704
1.0