How to troubleshoot this PySpark GLM error? - apache-spark

Question

Trying to run a GLM using poisson family and log link function and getting the following errors:
2022-01-11 15:56:55,143 root ERROR An error occurred while calling o266.fit.
: java.lang.NullPointerException
at scala.collection.immutable.StringOps$.length$extension(StringOps.scala:51)
at scala.collection.immutable.StringOps.length(StringOps.scala:51)
at scala.collection.IndexedSeqOptimized.isEmpty(IndexedSeqOptimized.scala:30)
at scala.collection.IndexedSeqOptimized.isEmpty$(IndexedSeqOptimized.scala:30)
at scala.collection.immutable.StringOps.isEmpty(StringOps.scala:33)
at scala.collection.TraversableOnce.nonEmpty(TraversableOnce.scala:114)
at scala.collection.TraversableOnce.nonEmpty$(TraversableOnce.scala:114)
at scala.collection.immutable.StringOps.nonEmpty(StringOps.scala:33)
at org.apache.spark.ml.PredictorParams.validateAndTransformSchema(Predictor.scala:58)
at org.apache.spark.ml.PredictorParams.validateAndTransformSchema$(Predictor.scala:47)
at org.apache.spark.ml.regression.GeneralizedLinearRegression.org$apache$spark$ml$regression$GeneralizedLinearRegressionBase$$super$validateAndTransformSchema(GeneralizedLinearRegression.scala:246)
at org.apache.spark.ml.regression.GeneralizedLinearRegressionBase.validateAndTransformSchema(GeneralizedLinearRegression.scala:213)
at org.apache.spark.ml.regression.GeneralizedLinearRegressionBase.validateAndTransformSchema$(GeneralizedLinearRegression.scala:188)
at org.apache.spark.ml.regression.GeneralizedLinearRegression.validateAndTransformSchema(GeneralizedLinearRegression.scala:246)
at org.apache.spark.ml.Predictor.transformSchema(Predictor.scala:177)
at org.apache.spark.ml.PipelineStage.transformSchema(Pipeline.scala:71)
at org.apache.spark.ml.Predictor.fit(Predictor.scala:133)
at org.apache.spark.ml.Predictor.fit(Predictor.scala:115)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
This is how I'm building the GLM:
data = getattr(self, data_name)
train, test = data.randomSplit(
[train_fraction, 1 - train_fraction], seed=42)
self.store_metadata({"training_samples": train.count()})
self.store_metadata({"test_samples": test.count()})
std_scaler = StandardScaler(inputCol=feature_col,
outputCol="scaled_features")
glm = GeneralizedLinearRegression(labelCol=label_col,
featuresCol="scaled_features",
offsetCol=offset_col,
family=family,
link=link,
maxIter=max_iter,
tol=tol,
weightCol=weight_col
)
pipeline = Pipeline(stages=[std_scaler, glm])
logging.info(pipeline)
grid = (ParamGridBuilder()
.addGrid(glm.regParam, reg_params or [1, 0.1])
.build())
ev = RegressionEvaluator(labelCol=label_col,
metricName=metric_name)
cv = CrossValidator(estimator=pipeline, estimatorParamMaps=grid, evaluator=ev,
numFolds=k_folds, seed=seed)
logging.debug(cv.explainParams())
cv_model = cv.fit(train)
The dataset is insurance claims frequencies with the offsetCol being floating point number of months (exposure) and the label is the number of claims in that period. The features are a 100-dimensional vector of floats. Any suggestions what to look into here?

Answer 1

It turns out I ran into this issue with Spark 5 years ago and forgot about it. In fact I even posted a bug report that was never fixed (because supposedly it is the "correct" behavior). The issue is setting weightCol=None causes an error, whereas simply leaving it out is fine, even though the default according to the api doc is weightCol=None. Go figure. For reference, here is the bug report I filed about this in 2016:
https://issues.apache.org/jira/browse/SPARK-17508