pyspark textFileStreaming can not detect txt file while textFile works - apache-spark

Question

To explain my question is different: This question is different from the marked one. First, the input parameter is already a directory(which is correct but the marked question is wrong). Second, I copied the txt file to the directory during streaming running to simulate the new txt file arriving(So new files are generated instead of same files existing in this directory)
My questionis below
I have a directory and txt file /tmp/a.txt, the contents in file is
aaa
bbb
I use pyspark and manually copy this file in same directory, continuously (during streaming running the files are created at the same time)
def count(x):
if x.isEmpty:
print("empty")
return
print(x.count())
sc = SparkContext()
ssc = StreamingContext(sc, 3)
ssc.textFileStream("/tmp/").foreachRDD(count)
The output shows the RDD is empty
However I use
c = sc.textFile("/tmp/").count()
print(c)
it shows c is 2 (consistent with the txt file contents)
Why streaming does not work?

Answer 1

Are you trying to pick up new lines being appended to the /tmp/a.txt file or are you trying to pick up new files being added to the tmp directory?
If its the latter try replacing your last line with this
ssc.textFileStream("/tmp/*").foreachRDD(count)

Answer 2

I have found the solution in scala (still can not pick up new files in python)
First, the sc.textFile and sc.textFileStream takes same paramter, which is a directory name. So above code is right.
However, the difference is, it's Okay for sc.textFile to pick up the files if the directory exists (and it must exist otherwise InvalidInputException would be raised), but in streaming mode sc.textFileStream (local file system), it demands that the directory does not exist and created by streaming program, otherwise the new files could not be picked up (seems to be a bug, only exists in local file system, in HDFS seems to work well according to others experience).
What's more, from some others experience they say that if you delete the directory and re-run the program, the recycle bin must also be emptied.
However, in python this issue still exists, and during no files exist in the directory, scala program would just print 0 but python program would raise warning of
WARN FileInputDStream:87 - Error finding new files
java.lang.NullPointerException
Here is my code in python and scala, the way of writing new files are same so I do not post it here
python code:
if __name__ == "__main__":
sc = SparkContext()
ssc = StreamingContext(sc, 3)
ssc.textFileStream(path).foreachRDD(lambda x: print(x.count()))
ssc.start()
ssc.awaitTermination()
scala code:
def main(args: Array[String]): Unit = {
val sc = new SparkContext()
val ssc = new StreamingContext(sc, Seconds(3))
ssc.textFileStream(params.inputPath).foreachRDD { x =>
print(x.count())
}
ssc.start()
ssc.awaitTermination()
}