syntax for Start-ThreadJob Powershell - multithreading

I need to execute a program (.exe) from a powershell script multiple times and limit the execution time to 20 min for each instance - after that I want to abandon the program(exit execution) and report an error. Currently the code below uses the Invoke-Expression statement to run the program with parameters and pass the exit code to $key.
foreach($acct in $accts)
# run batch
{
$Startdate = get-date
$cmd = $prgPath + $acct.clientID + ' ' + '{0:yyyyMMdd}' -f $rundate + ' ' + '{0:yyyyMMdd}' -f $rundate + ' E'
$key = $cmd | Invoke-Expression
$ac = $acct.clientID
$enddate = Get-Date
$insertSQL = "insert into tbluserkeys values('$key', '$ac','$Startdate','$enddate')"
# insert key
Invoke-Sqlcmd -ServerInstance 'dbsvr' -Database 'websb' -Query $insertSQL
}
For the runtime limit I want to use a thread job to start the program in a separate thread - I tried the code below - but when I check the status I see that PS takes the literal string and not command
3 Job3 ThreadJob Completed True PowerShell $key = $cmd | Invoke-...
how do I pass the command and retrieve the exit code similar to the invoke-expression statement? Also - should I use Get-process to check for the image and track the execution time or or is there another way to restrict the duration of the running process? It is important that the running job is killed to free system resources.
Thanks for any advice!
$Startdate = get-date
$cmd = $prgPath + $acct.clientID + ' ' + '{0:yyyyMMdd}' -f $rundate + ' ' + '{0:yyyyMMdd}' -f $rundate + ' E'
$ac = $acct.clientID
# execute program
Start-ThreadJob -ScriptBlock {$key = $cmd | Invoke-Expression}
# check if program run time exceeds maximum execution time
$process = Get-Process | Where-Object {$_.ProcessName -eq 'AS400SCRIPT'}
while($process -ne $null)

Avoid using Invoke-Expression in general, and in particular for invoking commands (see this answer), even if you need to construct their arguments programmatically - use argument splatting instead.
Use the $using: scope in the thread-job script block to refer to values from the caller's scope.
Use Wait-Job with its -Timeout parameter to wait for a job to finish within the specified period.
Use Receive-Job to collect the output emitted by the job.
Note: Receive-Job reports the content of the job's output streams, but has no concept of an exit code.
If you want to report the process exit code of an external program that ran in the job script block, output $LASTEXITCODE from there, which, however, requires you to separate that value from the program's output itself.
Generally, it would be useful if a thread / background job automatically provided by a way to query the value of $LASTEXITCODE via the job's properties - GitHub issue #5422 requests just that.
A simplified example:
# Sample object
$acct = [pscustomobject] #{ clientID = 'foo' }
# Define the name / path of the program to invoke.
# 'echo' is used as a stand-in for your program.
$prgPath = 'echo'
# Define its arguments as an *array*.
# Note: If you were calling a PowerShell command instead, it's
# better to use a *hashtable*. See about_Splatting.
$prgArgs = #(
$acct.clientID,
('{0:yyyyMMdd}' -f $rundate)
('{0:yyyyMMdd}' -f $rundate),
'E'
)
# Start the thread job.
# Use the $using: scope to refer to the values of variables in the
# caller's scope.
# This sample script block runs the given command then sleeps
# for a day, provoking a timeout below.
$jb = Start-ThreadJob { & $using:prgPath #using:prgArgs; Start-Sleep 86400 }
# Wait for the job to finish with a given timeout (in seconds).
# Note that receiving *no output* implies that waiting timed out.
# Applying -not to no output is equivalent to -not $false and therefore $true.
$timedOut = -not ($jb | Wait-Job -Timeout 2)
if ($timedOut) {
Write-Warning "Terminating job due to timeout."
$jb | Stop-Job
}
# Collect all job output and remove the job.
$outputReceived = $jb | Receive-Job -Wait -AutoRemoveJob
# Show results
[pscustomobject] #{
Job = $jb.Command
TimedOut = $timedOut
JobOutput = $outputReceived
}

Related

$MyInvocation in a runspace Thread

A script (foo.ps1) creates a thread, and that thread creates more threads. Foo's thread is my control thread, and it creates one or more worker threads. The worker threads run a script block. The script block calls functions from a library-script. The library script has a configuration file.
The script block loads the library-script by dot-sourcing it.
$block = {
Param($library_script)
. $library_script
...stuff...
}
When the script loads, the first thing it does is find its configuration file, which is in the script's directory. The code for that looks like...
## Global variables and enumerations
$script:self_location = $script:MyInvocation.MyCommand.Path
$script:configuration_file_location = "{0}.config" -f $script:self_location
My problem is $MyInvocation doesn't appear to exist. As result, the library script can't find it's configuration file.
I'm running Powershell 5.1 on Windows 10. The control thread was made in a runspace. The worker threads are made in a runspace pool.
Does anyone know the rules around the automatic $MyInvocation variable in runspace threads?
Create a file foo.ps1 and add the following to it:
Write-Output '[1] Executed in-scope'
$MyInvocation.MyCommand.Path
Write-Output '[2] Executed in-thread'
$p1 = [PowerShell]::Create()
$p1.AddScript({ $MyInvocation.MyCommand.Path }) | Out-Null
$p1.Invoke()
$p1.Dispose()
Write-Output '[3] Executed in-thread in-thread'
$t = {
$p = [PowerShell]::Create()
$p.AddScript({ $MyInvocation.MyCommand.Path }) | Out-Null
$p.Invoke()
}
$p2 = [PowerShell]::Create()
$p2.AddScript( $t ) | Out-Null
$p2.Invoke()
$p2.Dispose()
Run it. You should see something like the following...
[1] Executed in-scope
C:\Users\deezNuts\development\comcast\sandbox\thing.ps1
[2] Executed in-thread
[3] Executed in-thread in-thread
And, I think I just answered my own question.
I see variable in thread.
$rsp = [runspacefactory]::CreateRunspacePool(1, 2, $iss, $Host)
$rsp.ApartmentState = "STA"
$rsp.ThreadOptions = "ReuseThread"
$rsp.Open()
$p = [PowerShell]::Create()
$p.RunspacePool = $rsp
$p.AddScript({ write-host $MyInvocation.MyCommand.Path })
$h = $p.BeginInvoke()
$p.EndInvoke($h)
$p.Dispose()
$rsp.Dispose()
What are you doing differently?
Im not sure that $MyInvocation variable is supported in a background job
Start-Job -Name Test -ScriptBlock {Get-Variable}
Receive-Job Test
Can you pass the path as a parameter ?

starting a job with dot source

quick story:
for the sake of experimentation with powershell I'm trying to learn how to effectively multithread a script.
now i know how to start jobs and pass variables to my second script, however i have decided to try and figure out how to turn this:
start-job ((Split-Path -parent $PSCommandPath) + "\someScript.ps1") -ArgumentList (,$argList)
into this:
start-job (. ((Split-Path -parent $PSCommandPath) + "\someScript.ps1")) -ArgumentList (,$argList)
reason for this is i have a variable declared in the parent script like this:
New-Variable var -value 0 -Option AllScope
and in the child script: var = "something"
the first start-job passes my argument but the child doesn't set the global 'var' variable
the second doesn't pass my argument but the child script sets the global variable defined in the the parent just fine. $argList variable will be populated right up to this line of code in the second start-job but right after execution of the line, debug reveals the $argList variable to be null and i get "Start-Job : Cannot bind argument to parameter 'ScriptBlock' because it is null."
for the sake of argument assume that right up to the stated lines of code the variables contain the data they should.
can someone help me out with what is wrong with both attempts.
Google has failed to give me any specifics answers to my problem.
thanks in advance for any help i can get.
EDIT:
using Start-Job (. ((Split-Path -parent $PSCommandPath) + "\someScript.ps1") $argList)
accomplishes my goals however i keep getting Start-Job : Cannot bind argument to parameter 'ScriptBlock' because it is null.
even though the arguments are in the script block and the child script is getting and processing the argument.
When you call Start-Job, the script runs in an entirely separate scope (PowerShell Runspace). You can't dot-source a script called directly through Start-Job. You'll have to have the external script process the parameter that's passed in via -ArgumentList, and then return it to the original host Runspace via Receive-Job.
Here's a complete example:
$a = 1;
$Job = Start-Job -FilePath C:\test\script.ps1 -ArgumentList $a;
Write-Host -Object "Before: $a"; # Before
Wait-Job -Job $Job;
$a = Receive-Job -Job $Job -Keep;
Write-Host -Object "After: $a"; # After
c:\test\script.ps1
Here's the contents of the file c:\test\script.ps1:
Write-Output -InputObject (([int]$args[0]) += 5);
Thread and Runspace Exploration
If you want to prove my earlier point about Start-Job creating a new thread and PowerShell Runspace, and a new Thread, then run this script:
# 1. Declare a thread block that retrieves the Runspace ID & ThreadID
$ThreadBlock = {
[runspace]::DefaultRunspace.InstanceId.ToString();
[System.Threading.Thread]::CurrentThread.ManagedThreadId;
};
# 2. Start a job and wait for it to finish
$Job = Start-Job -ScriptBlock $ThreadBlock;
[void](Wait-Job -Job $Job);
Receive-Job -Job $Job -Keep;
# 3. Call the same ScriptBlock locally
& $ThreadBlock;
# 4. Note the differences in the Runspace InstanceIDs and ThreadIDs
Receiving Results Before Job Finish
You can call Receive-Job multiple times before a PowerShell Job has completed, to retrieve results. Here's an example of how that could theoretically work:
$ScriptBlock = {
1..5 | % { Start-Sleep -Seconds 2; Write-Output -InputObject $_; };
};
$Job = Start-Job -ScriptBlock $ScriptBlock;
while ($Job.JobStateInfo.State -notin ([System.Management.Automation.JobState]::Completed,[System.Management.Automation.JobState]::Failed)) {
Start-Sleep -Seconds 3;
$Results = Receive-Job -Job $Job;
Write-Host -Object ('Received {0} items: {1}' -f $Results.Count, ($Results -join ' '));
}

Powershell runspaces and color output

Short issue description:
I need non-interspersed output from additional powershell runspaces that my script spawns. I need to be able to set the color of individual lines in the output.
Long issue description:
I wrote a script to deploy updates to a vendor application on multiple remote servers. Originally the script was meant to only deploy a single update at a time, but I have a new requirement that I now process multiple updates interactively. So I've re-written my script to use runspaces so that I can open sessions on all the servers, initialize them, then for every update the user inputs a runspace is created for each of the initialized sessions and the deployment work is done. I previously used jobs, but had to go to runspaces because you cannot pass a PSSession to a job.
I now have my script working, but the output is less than desirable. The job version had nice output because I could call Receive-Job and get all all my output grouped by thread. Also I could use write-host to log output which allows for colored responses (ie. Red text when updates fail to apply). I have been able to get my runspace version of the job to group output by thread, but only by using write-output. If I use write-host output occurs immediately, causing interspersed output which is unacceptable. Unfortunately write-output does not allow colored output. Setting $host.UI.RawUI.ForegroundColor also does not work because if the output does not happen at the moment in time when the color has been set, the effect does not happen. Since my output won't happen until the end, the $host settings are no longer in play.
Below is a quick demo script to illustrate my woes:
#Runspacing output example. Fix interleaving
cls
#region Setup throttle, pool, iterations
$iterations = 5
$throttleLimit = 2
write-host ('Throttled to ' + $throttleLimit + ' concurrent threads')
$iss = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()
$Pool = [runspacefactory]::CreateRunspacePool(1,$throttleLimit,$iss,$Host)
$Pool.Open()
#endregion
#This works because the console color is set at the time output occurs
$OrigfC = $host.UI.RawUI.ForegroundColor
$host.UI.RawUI.ForegroundColor = 'red'
write-host 'THIS IS RED TEXT!'
start-sleep 2
$host.UI.RawUI.ForegroundColor = $OrigfC
#define code to run off the main thread
$scriptBlock = {
$nl = ([Environment]::NewLine.Chars(0)+[Environment]::NewLine.Chars(1))
#This does not work because output won't occur until after color has been reset
$OrigfC = $host.UI.RawUI.ForegroundColor
$host.UI.RawUI.ForegroundColor = 'yellow'
write-output ($nl + ' TEST: ' + $args[0])
Write-Output (' Some write-output: ' + $args[0])
Start-Sleep 1
write-host (' Some write-host: ' + $args[0]) -ForegroundColor Cyan # notice write-host occurs immediately
Start-Sleep 1
$host.UI.RawUI.ForegroundColor = $OrigfC
}
#Start new runspaces
$threads = #()
$handles = #(for($x = 1; $x -le $iterations; $x++)
{
$powerShell = [PowerShell]::Create().AddScript($scriptBlock).AddParameters(#($x))
$powershell.RunspacePool = $Pool
$powerShell.BeginInvoke()
$threads += $powerShell
})
#Wait for threads to complete
$completedCount = 0
$completed = ($handles | where-object {$_.IsCompleted -eq $true}).count
while($handles.IsCompleted.Contains($false))
{
if($completedCount -ne ($handles | where-object {$_.IsCompleted -eq $true}).count)
{
$completedCount = ($handles | where-object {$_.IsCompleted -eq $true}).count
write-host ('Threads Completed: ' + $completedCount + ' of ' + $iterations)
}
write-host '.' -nonewline
Start-Sleep 1
}
write-host ('Threads Completed: ' + ($handles | where-object {$_.IsCompleted -eq $true}).count + ' of ' + $iterations)
#output from threads
for($z = 0; $z -lt $handles.Count; $z++)
{
$threads[$z].EndInvoke($handles[$z]) #causes output
$threads[$z].Dispose()
$handles[$z] = $null
}
$Pool.Dispose()
The output is below, unless specified the output is in gray:
My goal is to be able to get the lines that say "Some write-output: X" set to one color, and the "TEST: X" set to a different color. Ideas? Please note I'm running from the powershell prompt. You will get different results if you run in the ISE.
Throttled to 2 concurrent threads
THIS IS RED TEXT! #Outputs in Red
. Some write-host: 1 #Outputs in Cyan
Some write-host: 2 #Outputs in Cyan
.Threads Completed: 2 of 5 #Outputs in Yellow
. Some write-host: 3 #Outputs in Cyan
Some write-host: 4 #Outputs in Cyan
.Threads Completed: 4 of 5 #Outputs in Yellow
. Some write-host: 5 #Outputs in Cyan
.Threads Completed: 5 of 5
TEST: 1
Some write-output: 1
TEST: 2
Some write-output: 2
TEST: 3
Some write-output: 3
TEST: 4
Some write-output: 4
TEST: 5
Some write-output: 5
Edit: Adding another example to address Mjolinor's Answer. M, you are correct I can pass sessions to a job; I have oversimplified my example above. Please consider this example below where I am sending a function to the job. If this line ( if(1 -ne 1){MyFunc -ses $args[0]} ) is commented out below, it will run. If the line is not commented out the session (type System.Management.Automation.Runspaces.PSSession) gets converted to a type of Deserialized.System.Management.Automation.Runspaces.PSSession, even though the MyFunc call cannot be hit. I have not been able to figure this out so I started moving towards runspaces. Do you think there is a job oriented solution?
cls
$ses = New-PSSession -ComputerName XXXX
Write-Host ('Outside job: '+$ses.GetType())
$func = {
function MyFunc {
param([parameter(Mandatory=$true)][PSSession]$ses)
Write-Host ('Inside fcn: '+$ses.GetType())
}
}
$scriptBlock = {
Write-Host ('Inside job: '+$args[0].GetType())
if(1 -ne 1){MyFunc -ses $args[0]}
}
Start-Job -InitializationScript $func -ScriptBlock $scriptBlock -Args #($ses) | Out-Null
While (Get-Job -State "Running") { }
Get-Job | Receive-Job
Remove-Job *
Remove-PSSession -Session $ses
I don't quite understand the statement that you can't pass a PSSession to a job. You can run Invoke-Command with both -Session and -AsJob parameters, creating at job targeted to the session.
As far as the coloring conundrum, have you considered using the Verbose or Debug streams for the output you want to be a different color? Powershell should automatically make it a different color depending on the stream it came from.
This gets a bunch easier in Powershell 7, which wasn't yet available when the question was originally asked.
In powershell 7 you can use the foreach-object -parallel {} -asjob command and then fetch the results when all the jobs are completed.
In this example:
There are 5 different jobs with 5 sub-steps
Each step has a 0.5 second delay to demonstrate interleaving
Each job get a color specific to the job
Each step gets a color specific to the step
The output is color coded, and grouped by job
$script = {
foreach ( $Step in 1..5 ) {
write-host "Job $_" -ForegroundColor #{'1'='red';'2'='yellow';'3'='cyan';'4'='magenta';'5'='green'}[[string]$_] -NoNewLine
write-host " Step $Step" -ForegroundColor #{'1'='red';'2'='yellow';'3'='cyan';'4'='magenta';'5'='green'}[[string]$Step]
sleep 0.5
} # next step
} # end script
$Job = 1..5 | ForEach-Object -Parallel $Script -ThrottleLimit 5 -AsJob
$job | Wait-Job | Receive-Job
It should be noted:
The challenge with $job | Wait-Job | Receive-Job is that if any one job hangs, then this command will never return. There are a plethora of posts which describe how to deal with this limitation.
Foreach-object and Foreach-Object -Parallel are two entirely different animals. The former can make calls to predefined functions whereas the later will only have access to what was defined inside the script block.

Powershell to wake up multiple media drives simultaneously

I have a server with lots of media drives ~43TB. An areca 1882ix-16 is set to spin the drives down after 30 minutes of inactivity since most days an individual drive is not even used. This works nicely to prevent unnecessary power and heat. In this case the drives still show up in windows explorer but when you click to access them it takes about 10 seconds for the folder list to show up since it has to wait for the drive to spin up.
For administrative work I have a need to spin up all the drives to be able to search among them. Clicking on each drive in windows explorer and then waiting for it to spin up before clicking the next drive is very tedious. Obviously multiple explorer windows makes it faster but it is still tedious. I thought a powershell script may ease the pain.
So I started with the following:
$mediaDrives = #('E:', 'F:', 'G:', 'H:', 'I:', 'J:', 'K:', 'L:',
'M:','N:', 'O:', 'P:', 'Q:', 'R:', 'S:')
get-childitem $mediaDrives | foreach-object -process { $_.Name }
This is just requesting that each drive in the array have its root folder name listed. That works to wake the drive but it is again a linear function. The script pauses for each drive before printing. Looking for a solution as to how to wake each drive simultaneously. Is there a way to multi-thread or something else?
Here's a script that will do what you want, but it must be run under powershell using the MTA threading mode (which is the default for powershell.exe 2.0, but powershell.exe 3.0 must be launched with the -MTA switch.)
#require -version 2.0
# if running in ISE or in STA console, abort
if (($host.runspace.apartmentstate -eq "STA") -or $psise) {
write-warning "This script must be run under powershell -MTA"
exit
}
$mediaDrives = #('E:', 'F:', 'G:', 'H:', 'I:', 'J:', 'K:', 'L:',
'M:','N:', 'O:', 'P:', 'Q:', 'R:', 'S:')
# create a pool of 8 runspaces
$pool = [runspacefactory]::CreateRunspacePool(1, 8)
$pool.Open()
$jobs = #()
$ps = #()
$wait = #()
$count = $mediaDrives.Length
for ($i = 0; $i -lt $count; $i++) {
# create a "powershell pipeline runner"
$ps += [powershell]::create()
# assign our pool of 8 runspaces to use
$ps[$i].runspacepool = $pool
# add wake drive command
[void]$ps[$i].AddScript(
"dir $($mediaDrives[$i]) > `$null")
# start script asynchronously
$jobs += $ps[$i].BeginInvoke();
# store wait handles for WaitForAll call
$wait += $jobs[$i].AsyncWaitHandle
}
# wait 5 minutes for all jobs to finish (configurable)
$success = [System.Threading.WaitHandle]::WaitAll($wait,
(new-timespan -Minutes 5))
write-host "All completed? $success"
# end async call
for ($i = 0; $i -lt $count; $i++) {
write-host "Completing async pipeline job $i"
try {
# complete async job
$ps[$i].EndInvoke($jobs[$i])
} catch {
# oops-ee!
write-warning "error: $_"
}
# dump info about completed pipelines
$info = $ps[$i].InvocationStateInfo
write-host "State: $($info.state) ; Reason: $($info.reason)"
}
So, for example, save as warmup.ps1 and run like: powershell -mta c:\scripts\warmup.ps1
To read more about runspace pools and the general technique above, take a look at my blog entry about runspacepools:
http://nivot.org/blog/post/2009/01/22/CTP3TheRunspaceFactoryAndPowerShellAccelerators
I chose 8 pretty much arbitrarily for the parallelism factor - experiment yourself with lower or higher numbers.
Spin up a separate powershell instance for each drive or use workflows in PowerShell 3.0.
Anyhow, you can pass drives directly to the Path parameter and skip Foreach-Object all togeteher:
Get-ChildItem $mediaDrives
Have you considered approaching this with the Start-Job cmdlet:
$mediaDrives = #('E:', 'F:', 'G:', 'H:', 'I:', 'J:', 'K:')
$mediaDrives | ForEach-Object {
Start-Job -ArgumentList $_ -ScriptBlock {param($drive)
Get-ChildItem $drive
}
}
The only clever part is that you need to use the -ArgumentList parameter on the Start-Job cmdlet to pass the correct value through for each iteration. This will create a background task that runs in parallel with the execution of the script. If you are curious
If you don't want to wait, well, don't wait: start those wake-up calls in the background.
In bash one would write
foreach drive ($mediadrives) {tickle_and_wake $drive &}
(note the ampersand, which means: start the command in the background, don't wait for it to complete)
In PowerShell that would translate to something like
foreach ($drive in $mediadrives) {
Start-Job {param($d) tickle_and_wake $d} -Arg $drive
}
If you want confirmation that all background jobs have completed, use wait in bash or Wait-Job in Powershell

Can Powershell Run Commands in Parallel?

I have a powershell script to do some batch processing on a bunch of images and I'd like to do some parallel processing. Powershell seems to have some background processing options such as start-job, wait-job, etc, but the only good resource I found for doing parallel work was writing the text of a script out and running those (PowerShell Multithreading)
Ideally, I'd like something akin to parallel foreach in .net 4.
Something pretty seemless like:
foreach-parallel -threads 4 ($file in (Get-ChildItem $dir))
{
.. Do Work
}
Maybe I'd be better off just dropping down to c#...
You can execute parallel jobs in Powershell 2 using Background Jobs. Check out Start-Job and the other job cmdlets.
# Loop through the server list
Get-Content "ServerList.txt" | %{
# Define what each job does
$ScriptBlock = {
param($pipelinePassIn)
Test-Path "\\$pipelinePassIn\c`$\Something"
Start-Sleep 60
}
# Execute the jobs in parallel
Start-Job $ScriptBlock -ArgumentList $_
}
Get-Job
# Wait for it all to complete
While (Get-Job -State "Running")
{
Start-Sleep 10
}
# Getting the information back from the jobs
Get-Job | Receive-Job
The answer from Steve Townsend is correct in theory but not in practice as #likwid pointed out. My revised code takes into account the job-context barrier--nothing crosses that barrier by default! The automatic $_ variable can thus be used in the loop but cannot be used directly within the script block because it is inside a separate context created by the job.
To pass variables from the parent context to the child context, use the -ArgumentList parameter on Start-Job to send it and use param inside the script block to receive it.
cls
# Send in two root directory names, one that exists and one that does not.
# Should then get a "True" and a "False" result out the end.
"temp", "foo" | %{
$ScriptBlock = {
# accept the loop variable across the job-context barrier
param($name)
# Show the loop variable has made it through!
Write-Host "[processing '$name' inside the job]"
# Execute a command
Test-Path "\$name"
# Just wait for a bit...
Start-Sleep 5
}
# Show the loop variable here is correct
Write-Host "processing $_..."
# pass the loop variable across the job-context barrier
Start-Job $ScriptBlock -ArgumentList $_
}
# Wait for all to complete
While (Get-Job -State "Running") { Start-Sleep 2 }
# Display output from all jobs
Get-Job | Receive-Job
# Cleanup
Remove-Job *
(I generally like to provide a reference to the PowerShell documentation as supporting evidence but, alas, my search has been fruitless. If you happen to know where context separation is documented, post a comment here to let me know!)
There's so many answers to this these days:
jobs (or threadjobs in PS 6/7 or the module for PS 5)
start-process
workflows (PS 5 only)
powershell api with another runspace
invoke-command with multiple computers, which can all be localhost (have to be admin)
multiple session (runspace) tabs in the ISE, or remote powershell ISE tabs
Powershell 7 has a foreach-object -parallel as an alternative for #4
Using start-threadjob in powershell 5.1. I wish this worked like I expect, but it doesn't:
# test-netconnection has a miserably long timeout
echo yahoo.com facebook.com |
start-threadjob { test-netconnection $input } | receive-job -wait -auto
WARNING: Name resolution of yahoo.com microsoft.com facebook.com failed
It works this way. Not quite as nice and foreach-object -parallel in powershell 7 but it'll do.
echo yahoo.com facebook.com |
% { $_ | start-threadjob { test-netconnection $input } } |
receive-job -wait -auto | ft -a
ComputerName RemotePort RemoteAddress PingSucceeded PingReplyDetails (RTT) TcpTestS
ucceeded
------------ ---------- ------------- ------------- ---------------------- --------
facebook.com 0 31.13.71.36 True 17 ms False
yahoo.com 0 98.137.11.163 True 97 ms False
Here's workflows with literally a foreach -parallel:
workflow work {
foreach -parallel ($i in 1..3) {
sleep 5
"$i done"
}
}
work
3 done
1 done
2 done
Or a workflow with a parallel block:
function sleepfor($time) { sleep $time; "sleepfor $time done"}
workflow work {
parallel {
sleepfor 3
sleepfor 2
sleepfor 1
}
'hi'
}
work
sleepfor 1 done
sleepfor 2 done
sleepfor 3 done
hi
Here's an api with runspaces example:
$a = [PowerShell]::Create().AddScript{sleep 5;'a done'}
$b = [PowerShell]::Create().AddScript{sleep 5;'b done'}
$c = [PowerShell]::Create().AddScript{sleep 5;'c done'}
$r1,$r2,$r3 = ($a,$b,$c).begininvoke() # run in background
$a.EndInvoke($r1); $b.EndInvoke($r2); $c.EndInvoke($r3) # wait
($a,$b,$c).streams.error # check for errors
($a,$b,$c).dispose() # clean
a done
b done
c done
In Powershell 7 you can use ForEach-Object -Parallel
$Message = "Output:"
Get-ChildItem $dir | ForEach-Object -Parallel {
"$using:Message $_"
} -ThrottleLimit 4
http://gallery.technet.microsoft.com/scriptcenter/Invoke-Async-Allows-you-to-83b0c9f0
i created an invoke-async which allows you do run multiple script blocks/cmdlets/functions at the same time. this is great for small jobs (subnet scan or wmi query against 100's of machines) because the overhead for creating a runspace vs the startup time of start-job is pretty drastic. It can be used like so.
with scriptblock,
$sb = [scriptblock] {param($system) gwmi win32_operatingsystem -ComputerName $system | select csname,caption}
$servers = Get-Content servers.txt
$rtn = Invoke-Async -Set $server -SetParam system -ScriptBlock $sb
just cmdlet/function
$servers = Get-Content servers.txt
$rtn = Invoke-Async -Set $servers -SetParam computername -Params #{count=1} -Cmdlet Test-Connection -ThreadCount 50
Backgrounds jobs are expensive to setup and are not reusable. PowerShell MVP Oisin Grehan
has a good example of PowerShell multi-threading.
(10/25/2010 site is down, but accessible via the Web Archive).
I'e used adapted Oisin script for use in a data loading routine here:
http://rsdd.codeplex.com/SourceControl/changeset/view/a6cd657ea2be#Invoke-RSDDThreaded.ps1
To complete previous answers, you can also use Wait-Job to wait for all jobs to complete:
For ($i=1; $i -le 3; $i++) {
$ScriptBlock = {
Param (
[string] [Parameter(Mandatory=$true)] $increment
)
Write-Host $increment
}
Start-Job $ScriptBlock -ArgumentList $i
}
Get-Job | Wait-Job | Receive-Job
If you're using latest cross platform powershell (which you should btw) https://github.com/powershell/powershell#get-powershell, you can add single & to run parallel scripts. (Use ; to run sequentially)
In my case I needed to run 2 npm scripts in parallel: npm run hotReload & npm run dev
You can also setup npm to use powershell for its scripts (by default it uses cmd on windows).
Run from project root folder: npm config set script-shell pwsh --userconfig ./.npmrc
and then use single npm script command: npm run start
"start":"npm run hotReload & npm run dev"
This has been answered thoroughly. Just want to post this method i have created based on Powershell-Jobs as a reference.
Jobs are passed on as a list of script-blocks. They can be parameterized.
Output of the jobs is color-coded and prefixed with a job-index (just like in a vs-build-process, as this will be used in a build)
Can be used to startup multiple servers at a time or running build steps in parallel or so..
function Start-Parallel {
param(
[ScriptBlock[]]
[Parameter(Position = 0)]
$ScriptBlock,
[Object[]]
[Alias("arguments")]
$parameters
)
$jobs = $ScriptBlock | ForEach-Object { Start-Job -ScriptBlock $_ -ArgumentList $parameters }
$colors = "Blue", "Red", "Cyan", "Green", "Magenta"
$colorCount = $colors.Length
try {
while (($jobs | Where-Object { $_.State -ieq "running" } | Measure-Object).Count -gt 0) {
$jobs | ForEach-Object { $i = 1 } {
$fgColor = $colors[($i - 1) % $colorCount]
$out = $_ | Receive-Job
$out = $out -split [System.Environment]::NewLine
$out | ForEach-Object {
Write-Host "$i> "-NoNewline -ForegroundColor $fgColor
Write-Host $_
}
$i++
}
}
} finally {
Write-Host "Stopping Parallel Jobs ..." -NoNewline
$jobs | Stop-Job
$jobs | Remove-Job -Force
Write-Host " done."
}
}
sample output:
There is a new built-in solution in PowerShell 7.0 Preview 3.
PowerShell ForEach-Object Parallel Feature
So you could do:
Get-ChildItem $dir | ForEach-Object -Parallel {
.. Do Work
$_ # this will be your file
}-ThrottleLimit 4

Resources