StreamingQueryException: Option 'basePath' must be a directory - apache-spark

Question

I wrote this code and I got this error: StreamingQueryException: Option 'basePath' must be a directory. My goal is to write the streams in file csv sink. The directories output_path/ and checkpoint/ were created but are empty.
pipe = Pipeline(stages=indexers)
pipe_model = pipe.fit(dataset)
dataset= pipe_model.transform(dataset)
pipe_model.save("pipe_model")
df = spark\
.readStream\
.option("header", "true")\
.schema(schema)\
.csv("KDDTrain+.txt")
model = PipelineModel.load("pipe_model")
dataset= model.transform(df)
q=dataset.writeStream\
.format("csv")\
.option("header", "true")\
.option("format", "append") \
.queryName("okk")\
.trigger(processingTime="10 seconds")\
.option("checkpointLocation", "checkpoint/")\
.option("path", "output_path/")\
.outputMode("append") \
.start()
q.awaitTermination()
I got this error:
---------------------------------------------------------------------------
StreamingQueryException Traceback (most recent call last)
Input In [7], in <cell line: 1>()
----> 1 q.awaitTermination()
File /usr/local/spark/python/pyspark/sql/streaming.py:101, in StreamingQuery.awaitTermination(self, timeout)
99 return self._jsq.awaitTermination(int(timeout * 1000))
100 else:
--> 101 return self._jsq.awaitTermination()
File /usr/local/spark/python/lib/py4j-0.10.9.3-src.zip/py4j/java_gateway.py:1321, in
JavaMember.__call__(self, *args)
1315 command = proto.CALL_COMMAND_NAME +\
1316 self.command_header +\
1317 args_command +\
1318 proto.END_COMMAND_PART
1320 answer = self.gateway_client.send_command(command)
-> 1321 return_value = get_return_value(
1322 answer, self.gateway_client, self.target_id, self.name)
1324 for temp_arg in temp_args:
1325 temp_arg._detach()
File /usr/local/spark/python/pyspark/sql/utils.py:117, in capture_sql_exception.<locals>.deco(*a, **kw)
113 converted = convert_exception(e.java_exception)
114 if not isinstance(converted, UnknownException):
115 # Hide where the exception came from that shows a non-Pythonic
116 # JVM exception message.
--> 117 raise converted from None
118 else:
119 raise
StreamingQueryException: Option 'basePath' must be a directory === Streaming Query ===
Identifier: okk [id = a5a4ac3d-a533-409b-be0b-015ead8d2f4a, runId = 3e3b747d-ba4f 4948-bc9f-4ef360e08979]
Current Committed Offsets: {}
Current Available Offsets: {FileStreamSource[file:/home/jovyan/work/KDDTrain+.txt {"logOffset":0}}
Current State: ACTIVE
Thread State: RUNNABLE
where is the problem? and how to fix it please?

Answer 1

I fixed the issue by reading from a directory not a file. i.e.:
df = spark\
.readStream\
.option("header", "true")\
.schema(schema)\
.csv("KDD/")
instead of:
df = spark\
.readStream\
.option("header", "true")\
.schema(schema)\
.csv("KDDTrain+.txt")