How to safely select across channels where some may get concurrently closed? - multithreading

Question

While answering a question I attempted to implement a setup where the main thread joins the efforts of the CommonPool to execute a number of independent tasks in parallel (this is how java.util.streams operates).
I create as many actors as there are CommonPool threads, plus a channel for the main thread. The actors use rendezvous channels:
val resultChannel = Channel<Double>(UNLIMITED)
val poolComputeChannels = (1..commonPool().parallelism).map {
actor<Task>(CommonPool) {
for (task in channel) {
task.execute().also { resultChannel.send(it) }
}
}
}
val mainComputeChannel = Channel<Task>()
val allComputeChannels = poolComputeChannels + mainComputeChannel
This allows me to distribute the load by using a select expression to find an idle actor for each task:
select {
allComputeChannels.forEach { chan ->
chan.onSend(task) {}
}
}
So I send all the tasks and close the channels:
launch(CommonPool) {
jobs.forEach { task ->
select {
allComputeChannels.forEach { chan ->
chan.onSend(task) {}
}
}
}
allComputeChannels.forEach { it.close() }
}
Now I have to write the code for the main thread. Here I decided to serve both the mainComputeChannel, executing the tasks submitted to the main thread, and the resultChannel, accumulating the individual results into the final sum:
return runBlocking {
var completedCount = 0
var sum = 0.0
while (completedCount < NUM_TASKS) {
select<Unit> {
mainComputeChannel.onReceive { task ->
task.execute().also { resultChannel.send(it) }
}
resultChannel.onReceive { result ->
sum += result
completedCount++
}
}
}
resultChannel.close()
sum
}
This gives rise to the situation where mainComputeChannel may be closed from a CommonPool thread, but the resultChannel still needs serving. If the channel is closed, onReceive will throw an exception and onReceiveOrNull will immediately select with null. Neither option is acceptable. I didn't find a way to avoid registering the mainComputeChannel if it's closed, either. If I use if (!mainComputeChannel.isClosedForReceive), it will not be atomic with the registration call.
This leads me to my question: what would be a good idiom to select over channels where some may get closed by another thread while others are still live?

Answer 1

The kotlinx.coroutines library is currently missing a primitive to make it convenient. The outstanding proposal is to add receiveOrClose function and onReceiveOrClosed clause for select that would make writing code like this possible.
However, you will still have to manually track the fact that your mainComputeChannel was closed and stop selecting on it when it was. So, using a proposed onReceiveOrClosed clause you'll write something like this:
// outside of loop
var mainComputeChannelClosed = false
// inside loop
select<Unit> {
if (!mainComputeChannelClosed) {
mainComputeChannel.onReceiveOrClosed {
if (it.isClosed) mainComputeChannelClosed = true
else { /* do something with it */ }
}
}
// more clauses
}
See https://github.com/Kotlin/kotlinx.coroutines/issues/330 for details.
There are no proposals on the table to further simplify this kind of pattern.