Runspace performance improvement less than expected - multithreading

Question

I am profiling some code for performance, and not getting the results from Runspaces that I would expect.
My source files are 7 Autodesk Revit journal files, ranging from 13MB and 150K lines to 90MB and 900K lines. Each file contains reference to it's own name some number of times, so I am getting that count as a proxy for some real work I want to do later. In the code below, I process all the files with a simple foreach, and then again using runspaces throttled to 8. In both cases I am using a stream reader to parse the files since the files can get rather larger than the ones I am testing with. I wouldn't expect the runspace example to be 25% the time of the loop, but I certainly would expect it to be closer to 25% than even 50%. Instead, I am seeing less than a 50% improvement. The last run was 14.26 seconds for the single thread and 8.74 seconds for 8 runspaces. Am I doing something wrong in my code, or are my expectations incorrect? FWIW I am testing on a VM at the moment. I have tried assigning 4, 6, 8 & 12 cores to the VM with little difference in results. That last test was 12 cores assigned, runspaces throttled to 8. This with a 10 cores hyper threaded Xeon on the host machine.
EDIT: I modified the code to copy the resource files to temp, to remove the network variable, and I added a Jobs based test, again constrained to the same 8 concurrent threads the Runspaces are throttled to. Times are along the lines of 16.8 vs 9.6 vs 7.3. So, Jobs are consistently better, even though my understanding was that runspaces are more efficient and should be faster, and still performance is barely better than a 50% savings, even with 8 threads.
$source = '\\Px\Support\Profiling\Source'
$localSource = "$env:TEMP\Px"
Clear-Host
if (Test-Path $localSource) {
Remove-Item "$localSource\*" -Recurse -force
} else {
New-Item $localSource -ItemType:Directory > $null
}
Copy-Item "$source\*" $localSource
$journals = Get-ChildItem $localSource
Write-Host "Single Thread"
(Measure-Command {
foreach ($journal in $journals) {
$count = 0
#$reader = [IO.StreamReader]::New($journal.fullName, $true)
$reader = New-Object -typeName:System.IO.StreamReader -argumentList $journal.fullName
while (-not ($reader.EndOfStream)) {
$line = ($reader.ReadLine()).Trim()
if ($line -match $journal) {
$count ++
}
}
Write-Host "$journal $count"
$reader.Close()
$reader.Dispose()
}
}).totalSeconds
Write-Host
Write-Host "Runspace 1,8"
(Measure-Command {
$runspacePool = [RunspaceFactory]::CreateRunspacePool(1,8)
$runspacePool.Open()
$runspaceCollection = New-Object system.collections.arraylist
$scriptBlock = {
param (
[string]$journal
)
$journalName = Split-Path $journal -leaf
$count = 0
#$reader = [IO.StreamReader]::New($journal, $true)
$reader = New-Object -typeName:System.IO.StreamReader -argumentList $journal
while (-not ($reader.EndOfStream)) {
$line = ($reader.ReadLine()).Trim()
if ($line -match $journalName) {
$count ++
}
}
$reader.Close()
$reader.Dispose()
"$journalName $count"
}
foreach ($journal in $journals) {
$parameters = #{
journal = $journal.fullName
}
$powershell = [PowerShell]::Create()
$powershell.RunspacePool = $RunspacePool
$powershell.AddScript($scriptBlock) > $null
$powershell.AddParameters($parameters) > $null
$runspace = New-Object -TypeName PSObject -Property #{
runspace = $powershell.BeginInvoke()
powerShell = $powershell
}
$runspaceCollection.Add($runspace) > $null
}
while($runspaceCollection){
foreach($runspace in $runspaceCollection.ToArray()){
if($runspace.RunSpace.IsCompleted -eq $true){
Write-Host "$($runspace.Powershell.EndInvoke($runspace.RunSpace))"
$runspace.Powershell.dispose()
$runspaceCollection.Remove($runspace)
#[System.GC]::Collect()
Start-Sleep -m:100
}
}
}
}).totalSeconds
Write-Host
Write-Host "Jobs 8"
Remove-Job *
(Measure-Command {
$scriptBlock = {
param (
[string]$journal
)
$journalName = Split-Path $journal -leaf
$count = 0
#$reader = [IO.StreamReader]::New($journal, $true)
$reader = New-Object -typeName:System.IO.StreamReader -argumentList:$journal
while (-not ($reader.EndOfStream)) {
$line = ($reader.ReadLine()).Trim()
if ($line -match $journalName) {
$count ++
}
}
$reader.Close()
$reader.Dispose()
Write-Output "$journalName $count"
}
foreach ($journal in $journals) {
Start-Job -ScriptBlock:$scriptBlock -argumentlist:$journal.fullName
While($(Get-Job -State 'Running').Count -ge 8) {
sleep -m:100
}
}
Get-Job | Wait-Job
foreach ($job in Get-Job) {
Write-Host "$(Receive-Job $job)"
Remove-Job $job
}
}).totalSeconds
Write-Host
Remove-Item $localSource -Recurse -force

Answer 1

It's interesting that the Start-Sleep command improves your performance - that suggests that your while($runspaceCollection) loop is part of what's bottlenecking the run speed. After your scripts have all been set running, this loop is constantly re-checking every RunSpace, and only pauses for 100ms whenever one has completed. I think you've built this step the wrong way round - it's probably more important for it to sleep when it hasn't found anything to do:
while($runspaceCollection){
foreach($runspace in $runspaceCollection.ToArray()){
if($runspace.RunSpace.IsCompleted -eq $true){
Write-Host "$($runspace.Powershell.EndInvoke($runspace.RunSpace))"
$runspace.Powershell.dispose()
$runspaceCollection.Remove($runspace)
}else{
Start-Sleep -m:100
}
}
}
Alternatively, you could refactor this part of the code using something like if($RunspacePool.GetAvailableRunspaces() -eq 8) to only begin cleanup of the runspaces once they have all completed. I've left over a thousand runspaces waiting for decomissioning at the end of a script without any noticeable performance drop, so you may be wasting system resources cleaning up earlier than you need to.
Beyond this I'd suggest monitoring CPU and RAM usage on the local machine as you're running the script, to see if there are any obvious performance bottlenecks there that would indicate it slowing down.