Groovy GPars Script - groovy

Question

GParsPool.withPool(numberPool) {
connection.withBatch(10000) { stmt ->
inputFile.eachParallel { data ->
//GParsPool.withPool() {
stmt.addBatch("DELETE FROM user WHERE number = ${data.toLong()} ")
println "IN"
//}
}
println "OUT"
Long startTimee = System.currentTimeMillis()
stmt.executeBatch()
println "deleted Batch"
Long endTime = System.currentTimeMillis()
println "Time taken for each batch: " + ((endTime - startTimee) / 1000)
}
}
The Above code is used to delete data from database. I First get the data from file and then match the each file data with database data and perform the delete query. But I have 5533179 records its take to much time. Even I have used the gpars but i get the same performance issue that is given by without using gpars. I have set the numberPool=5 but same issue. Even I increase the numberPool again same issue

Answer 1

why don't you use the SQL in operator? thus you can process the data much faster.
UPDATE:
from the top of my head:
GParsPool.withPool(numberPool) {
Map buffPerThread = [:].withDefaults{ [] }
inputFile.eachParallel { data ->
def buff = buffPerThread[ Thread.currentThread().id ]
buff << data.toLong()
if( 1000 == buff.size() ){
sql.execute 'DELETE FROM user WHERE number in (?)', [ buff ]
buff.clear()
}
}
}
I wouldn't use conn.withBatch here, as the in statement give the desired batching already