Hashmap load behaving inconsistently in spark streaming code client/cluster mode - apache-spark

Question

I am creating a simple kafka spark streaming program where, I read the data from topic, process it using UDF and print the key,processed value in console.
Processig in uDF is like:
1.We get Ptyid and Date as Input.
2.If we get new Pty ID, we return a value as I+Date (Insert with date) to UDF and store it in hashmap for further comparison.
3.if we get existing ptyid, we compare the date of current record with date of record present in hashmap.
3a) If the date of current record is less than date available in hashmap,we return the value as "D"(Discard) to UDF.
3b) if the date of current record is more than date available in hashmap, we return the value as U+Date (Update) to UDF and store it in hashmap for further comparison.
Issue:
When i run the application in local mode, there is no issue and the I,U,D for each ptyid is behaving as expected.
When i run the same application in client mode, if I post same data in input topic multiple times, it behaves inconsistently and I+Date is generated multiple times for the same PtyID which has been already consumed.Expectation is if we post same data, it should give I(insert) for the first time, U/D as per comparison.But i get I+Date multiple times
below is the code snippet. Any help would be appreciated.
object teststr{
var hashMap = new ConcurrenthashMap[String,String]()
def main(args: Array[Sttring]):Unit = {
val spark = SparkSession.builder.appname("test").getOrCreate()
import spark.impicits._
spark.readstream.format("kafka")
.option ......(all properties)
.load()
.selectExpr("CAST(key AS STRING) as key", "CAST(value AS STRING) as value",offset","partition","timestamp")
.withColumn("value", processRecordsUDF(col("key"),col("value")))
.select("key","value")
.writestream
.format("console")
.outputMode("append")
.start().awaitTermination()
}
def processRecords(partyId: String,value:String):String = {
if (hashMap.containsKey(partyId))
{
if (value > hashMap.get(partyId))
{
hashMap.put(partyId,value)
return ("U"+value)
}
else
{
return ("D")
}}
else
{
hashMap.put(partyId,value)
return ("I"+value)
}}
def processRecordsUDF = udf(processRecords(_:String,_:String):String)
}