Spark DataFrame filtering gives inconsistent output - apache-spark

Question

I have a MongoDB collection with 26,000 records which I am reading into a DataFrame. It has a column, column_1 with String value column_1_value across all the records. I am trying to filter the DataFrame and get the count as follows
val readConfig = ReadConfig(Map("collection" -> collectionName,"spark.mongodb.input.readPreference.name" -> "primaryPreferred", "spark.mongodb.input.database" -> dataBaseName, "spark.mongodb.input.uri" -> hostName))
val dataFrame = MongoSpark.load(spark, readConfig)
df.filter(df.col("column_1") === "column_1_value").count()
Where spark is the instance of SparkSession.
The MongoDB record structure looks something like this.
{
"_id" : ObjectId("SOME_ID"),
"column_1" : "column_1_value",
"column_2" : SOME_VALUE,
"column_3" : SOME_OTHER_VALUE
}
There is no nested structure, and all the records have the same set of fields. I am not accessing the DB during any time while Spark is running
As all the records have the same value of column_1, I expected to get the DataFrame size itself as the output, but instead I am getting a lower value. Not only that, I am getting different outcomes each time I run the above. The outcome is normally varying between 15,000 and 24,000.
But the same approach seems to work when the collection size is less, around 5,000.
I have tried the following approaches with no success
Used equalTo instead of ===
Used $column_1
Used isin
Used df.where instead of df.filter
Used createOrReplaceTempView and ran SQL query
The only thing that seems to work is either df.cache() or df.persist(), none of which I think will be good for the overall performance when working with large data.
What can be the possible reason for this behavior and the ways by which this can be resolved?
My Spark version is 2.2.0, MongoDB version is 3.4. I am running Spark in local mode, with 16 GB of RAM and an 8 core processor.
Edit 1
Tried changing the Mongo Spark connector partitioning policy as follows, but with no success.
val readConfig = ReadConfig(Map("collection" -> collectionName,"spark.mongodb.input.readPreference.name" -> "primaryPreferred", "spark.mongodb.input.database" -> dataBaseName, "spark.mongodb.input.uri" -> hostName, "spark.mongodb.input.partitionerOptions" -> "MongoPaginateBySizePartitioner"))
val dataFrame = MongoSpark.load(spark, readConfig)
df.count()
df.filter(df.col("column_1") === "column_1_value").count()
The first count returns the correct value, even with the default partitioning strategy, which leads me to believe that the Mongo Connector is working fine.