A script (foo.ps1) creates a thread, and that thread creates more threads. Foo's thread is my control thread, and it creates one or more worker threads. The worker threads run a script block. The script block calls functions from a library-script. The library script has a configuration file.
The script block loads the library-script by dot-sourcing it.
$block = {
Param($library_script)
. $library_script
...stuff...
}
When the script loads, the first thing it does is find its configuration file, which is in the script's directory. The code for that looks like...
## Global variables and enumerations
$script:self_location = $script:MyInvocation.MyCommand.Path
$script:configuration_file_location = "{0}.config" -f $script:self_location
My problem is $MyInvocation doesn't appear to exist. As result, the library script can't find it's configuration file.
I'm running Powershell 5.1 on Windows 10. The control thread was made in a runspace. The worker threads are made in a runspace pool.
Does anyone know the rules around the automatic $MyInvocation variable in runspace threads?
Create a file foo.ps1 and add the following to it:
Write-Output '[1] Executed in-scope'
$MyInvocation.MyCommand.Path
Write-Output '[2] Executed in-thread'
$p1 = [PowerShell]::Create()
$p1.AddScript({ $MyInvocation.MyCommand.Path }) | Out-Null
$p1.Invoke()
$p1.Dispose()
Write-Output '[3] Executed in-thread in-thread'
$t = {
$p = [PowerShell]::Create()
$p.AddScript({ $MyInvocation.MyCommand.Path }) | Out-Null
$p.Invoke()
}
$p2 = [PowerShell]::Create()
$p2.AddScript( $t ) | Out-Null
$p2.Invoke()
$p2.Dispose()
Run it. You should see something like the following...
[1] Executed in-scope
C:\Users\deezNuts\development\comcast\sandbox\thing.ps1
[2] Executed in-thread
[3] Executed in-thread in-thread
And, I think I just answered my own question.
I see variable in thread.
$rsp = [runspacefactory]::CreateRunspacePool(1, 2, $iss, $Host)
$rsp.ApartmentState = "STA"
$rsp.ThreadOptions = "ReuseThread"
$rsp.Open()
$p = [PowerShell]::Create()
$p.RunspacePool = $rsp
$p.AddScript({ write-host $MyInvocation.MyCommand.Path })
$h = $p.BeginInvoke()
$p.EndInvoke($h)
$p.Dispose()
$rsp.Dispose()
What are you doing differently?
Im not sure that $MyInvocation variable is supported in a background job
Start-Job -Name Test -ScriptBlock {Get-Variable}
Receive-Job Test
Can you pass the path as a parameter ?
Related
All the tuts I have found use a pre defined sleep time to throttle jobs.
I need the throttle to wait until a job is completed before starting a new one.
Only 4 jobs can be running at one time.
So The script will run up 4 and currently pauses for 10 seconds then runs up the rest.
What I want is for the script to only allow 4 jobs to be running at one time and as a job is completed a new one is kicked off.
Jobs are initialised via a list of servers names.
Is it possible to archive this?
$servers = Get-Content "C:\temp\flashfilestore\serverlist.txt"
$scriptBlock = { #DO STUFF }
$MaxThreads = 4
foreach($server in $servers) {
Start-Job -ScriptBlock $scriptBlock -argumentlist $server
While($(Get-Job -State 'Running').Count -ge $MaxThreads) {
sleep 10 #Need this to wait until a job is complete and kick off a new one.
}
}
Get-Job | Wait-Job | Receive-Job
You can test the following :
$servers = Get-Content "C:\temp\flashfilestore\serverlist.txt"
$scriptBlock = { #DO STUFF }
invoke-command -computerName $servers -scriptblock $scriptBlock -jobname 'YourJobSpecificName' -throttlelimit 4 -AsJob
This command uses the Invoke-Command cmdlet and its AsJob parameter to start a background job that runs a scriptblock on numerous computers. Because the command must not be run more than 4 times concurrently, the command uses the ThrottleLimit parameter of Invoke-Command to limit the number of concurrent commands to 4.
Be careful that the file contains the computer names in a domain.
In order to avoid inventing a wheel I would recommend to use one of the
existing tools.
One of them is the script
Invoke-Parallel.ps1.
It is written in PowerShell, you can see how it is implemented directly. It is
easy to get and it does not require any installation for using it.
Another one is the module SplitPipeline.
It may work faster because it is written in C#. It also covers some more use
cases, for example slow or infinite input, use of initialization and cleanup scripts.
In the latter case the code with 4 parallel pipelines will be
$servers | Split-Pipeline -Count 4 {process{ <# DO STUFF on $_ #> }}
I wrote a blog article which covers multithreading any given script via actual threads. You can find the full post here:
http://www.get-blog.com/?p=189
The basic setup is:
$ISS = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()
$RunspacePool = [runspacefactory]::CreateRunspacePool(1, $MaxThreads, $ISS, $Host)
$RunspacePool.Open()
$Code = [ScriptBlock]::Create($(Get-Content $FileName))
$PowershellThread = [powershell]::Create().AddScript($Code)
$PowershellThread.RunspacePool = $RunspacePool
$Handle = $PowershellThread.BeginInvoke()
$Job = "" | Select-Object Handle, Thread, object
$Job.Handle = $Handle
$Job.Thread = $PowershellThread
$Job.Object = $Object.ToString()
$Job.Thread.EndInvoke($Job.Handle)
$Job.Thread.Dispose()
Instead of sleep 10 you could also just wait on a job (-any job):
Get-Job | Wait-Job -Any | Out-Null
When there are no more jobs to kick off, start printing the output. You can also do this within the loop immediately after the above command. The script will receive jobs as they finish instead of waiting until the end.
Get-Job -State Completed | % {
Receive-Job $_ -AutoRemoveJob -Wait
}
So your script would look like this:
$servers = Get-Content "C:\temp\flashfilestore\serverlist.txt"
$scriptBlock = { #DO STUFF }
$MaxThreads = 4
foreach ($server in $servers) {
Start-Job -ScriptBlock $scriptBlock -argumentlist $server
While($(Get-Job -State Running).Count -ge $MaxThreads) {
Get-Job | Wait-Job -Any | Out-Null
}
Get-Job -State Completed | % {
Receive-Job $_ -AutoRemoveJob -Wait
}
}
While ($(Get-Job -State Running).Count -gt 0) {
Get-Job | Wait-Job -Any | Out-Null
}
Get-Job -State Completed | % {
Receive-Job $_ -AutoRemoveJob -Wait
}
Having said all that, I prefer runspaces (similar to Ryans post) or even workflows if you can use them. These are far less resource intensive than starting multiple powershell processes.
Your script looks good, try and add something like
Write-Host ("current count:" + ($(Get-Job -State 'Running').Count) + " on server:" + $server)
after your while loop to work out whether the job count is going down where you wouldn't expect it.
I noticed that every Start-Job command resulted in an additional conhost.exe process in the task manager. Knowing this, I was able to throttle using the following logic, where 5 is my desired number of concurrent threads (so I use 4 in my -gt statement since I am looking for a count greater than):
while((Get-Process conhost -ErrorAction SilentlyContinue).Count -gt 4){Start-Sleep -Seconds 1}
quick story:
for the sake of experimentation with powershell I'm trying to learn how to effectively multithread a script.
now i know how to start jobs and pass variables to my second script, however i have decided to try and figure out how to turn this:
start-job ((Split-Path -parent $PSCommandPath) + "\someScript.ps1") -ArgumentList (,$argList)
into this:
start-job (. ((Split-Path -parent $PSCommandPath) + "\someScript.ps1")) -ArgumentList (,$argList)
reason for this is i have a variable declared in the parent script like this:
New-Variable var -value 0 -Option AllScope
and in the child script: var = "something"
the first start-job passes my argument but the child doesn't set the global 'var' variable
the second doesn't pass my argument but the child script sets the global variable defined in the the parent just fine. $argList variable will be populated right up to this line of code in the second start-job but right after execution of the line, debug reveals the $argList variable to be null and i get "Start-Job : Cannot bind argument to parameter 'ScriptBlock' because it is null."
for the sake of argument assume that right up to the stated lines of code the variables contain the data they should.
can someone help me out with what is wrong with both attempts.
Google has failed to give me any specifics answers to my problem.
thanks in advance for any help i can get.
EDIT:
using Start-Job (. ((Split-Path -parent $PSCommandPath) + "\someScript.ps1") $argList)
accomplishes my goals however i keep getting Start-Job : Cannot bind argument to parameter 'ScriptBlock' because it is null.
even though the arguments are in the script block and the child script is getting and processing the argument.
When you call Start-Job, the script runs in an entirely separate scope (PowerShell Runspace). You can't dot-source a script called directly through Start-Job. You'll have to have the external script process the parameter that's passed in via -ArgumentList, and then return it to the original host Runspace via Receive-Job.
Here's a complete example:
$a = 1;
$Job = Start-Job -FilePath C:\test\script.ps1 -ArgumentList $a;
Write-Host -Object "Before: $a"; # Before
Wait-Job -Job $Job;
$a = Receive-Job -Job $Job -Keep;
Write-Host -Object "After: $a"; # After
c:\test\script.ps1
Here's the contents of the file c:\test\script.ps1:
Write-Output -InputObject (([int]$args[0]) += 5);
Thread and Runspace Exploration
If you want to prove my earlier point about Start-Job creating a new thread and PowerShell Runspace, and a new Thread, then run this script:
# 1. Declare a thread block that retrieves the Runspace ID & ThreadID
$ThreadBlock = {
[runspace]::DefaultRunspace.InstanceId.ToString();
[System.Threading.Thread]::CurrentThread.ManagedThreadId;
};
# 2. Start a job and wait for it to finish
$Job = Start-Job -ScriptBlock $ThreadBlock;
[void](Wait-Job -Job $Job);
Receive-Job -Job $Job -Keep;
# 3. Call the same ScriptBlock locally
& $ThreadBlock;
# 4. Note the differences in the Runspace InstanceIDs and ThreadIDs
Receiving Results Before Job Finish
You can call Receive-Job multiple times before a PowerShell Job has completed, to retrieve results. Here's an example of how that could theoretically work:
$ScriptBlock = {
1..5 | % { Start-Sleep -Seconds 2; Write-Output -InputObject $_; };
};
$Job = Start-Job -ScriptBlock $ScriptBlock;
while ($Job.JobStateInfo.State -notin ([System.Management.Automation.JobState]::Completed,[System.Management.Automation.JobState]::Failed)) {
Start-Sleep -Seconds 3;
$Results = Receive-Job -Job $Job;
Write-Host -Object ('Received {0} items: {1}' -f $Results.Count, ($Results -join ' '));
}
I have a little performance issue in my script, so i would like to implement some sort of worker theads. but so far i have not been able to find a solution..
what im hoping for is something like this:
start a pool of worker threads - these threads takes "commands" from a queue and process them
the main script will write "commands" to the queue as it runs
once complete the main will tell each thread to stop
main will wait for all workers to end before exiting.
does anybody have en idea on how to do this?
You can do this with Powershell workflows.
From Windows PowerShell: What is Windows PowerShell Workflow?
Workflows can also execute things in parallel, if you like. For
example, if you have a set of tasks that can run in any order, with no
interdependencies, then you can have them all run at more or less the
same time
Just do a search on "Powershell workflows" and you will find a good amount of documentation to get you started.
The basic approach to using a job is this:
$task1 = { ls c:\windows\system32 -r *.dll -ea 0 | where LastWriteTime -gt (Get-Date).AddDays(-21) }
$task2 = { ls E:\Symbols -r *.dll | where LastWriteTime -gt (Get-Date).AddDays(-21) }
$task3 = { Invoke-WebRequest -Uri http://blogs.msdn.com/b/mainfeed.aspx?Type=BlogsOnly | % Content }
$job1 = Start-Job $task1; $job2 = Start-Job $task2; $job3 = Start-Job $task3
Wait-Job $job1,$job2,$job3
$job1Data = Receive-Job $job1
$job2Data = Receive-Job $job2
$job3Data = Receive-Job $job3
If you need to have those background jobs waiting in a loop to do work as the main script dictates have a look at this SO answer to see how to use MSMQ to do this.
With some help from the pointers made by Keith hill - i got it working - thanks a bunch...
Here is a snipping of the code that did my prove of concept:
function New-Task([int]$Index,[scriptblock]$ScriptBlock) {
$ps = [Management.Automation.PowerShell]::Create()
$res = New-Object PSObject -Property #{
Index = $Index
Powershell = $ps
StartTime = Get-Date
Busy = $true
Data = $null
async = $null
}
[Void] $ps.AddScript($ScriptBlock)
[Void] $ps.AddParameter("TaskInfo",$Res)
$res.async = $ps.BeginInvoke()
$res
}
$ScriptBlock = {
param([Object]$TaskInfo)
$TaskInfo.Busy = $false
Start-Sleep -Seconds 1
$TaskInfo.Data = "test $($TaskInfo.Data)"
}
$a = New-Task -Index 1 -ScriptBlock $ScriptBlock
$a.Data = "i was here"
Start-Sleep -Seconds 5
$a
And here is the result proving that the data was communicated into the thread and back again:
Data : test i was here
Busy : False
Powershell : System.Management.Automation.PowerShell
Index : 1
StartTime : 11/25/2013 7:37:07 AM
async : System.Management.Automation.PowerShellAsyncResult
as you can see the $a.data now have "test" in front
So thanks a lot...
Below is a small PowerShell script that runs some PowerShell code asynchronously to show a dialog box (for the purpose of demonstrating my issue).
If I close the parent PowerShell process, the child process will also close and the dialog box disappears. Is there any way to launch a PowerShell scriptblock, complete with functions and arguments, asynchronously and without a dependence on the parent PowerShell process?
$testjob = [PowerShell]::Create().AddScript({ $a = new-object -comobject wscript.shell
$b = $a.popup('This is a test',5,'Test Message Box',1) })
$result = $testJob.BeginInvoke()
Update #2
I am trying to execute a script block, rather than an external script. The script block should use functions and variables from the parent script. The problem is, I can't pass those functions or variables in to the new process unless they are contained directly within the script block. Any idea if this is doable?
Function Show-Prompt {
Param ($title,$message)
$a = new-object -comobject wscript.shell
$b = $a.popup($message,5,$title,1)
}
$scriptContent = {Show-Prompt -Message 'This is a test' -Title 'Test Message Box'}
$scriptBlock = [scriptblock]::Create($scriptContent)
Start-process powershell -argumentlist "-noexit -command &{$scriptBlock}"
You can use an intermediate PowerShell process. There has to be a direct parent to process the return value from the async script. For instance, put your script in a file called popup.ps1 and try execute like so:
Start-Process PowerShell -Arg c:\popup.ps1
You might want to bump the timeout up a bit from say 5 to 10 seconds. You can close the original PowerShell and the popup will stay. When you close the popup (or it times out) the secondary PowerShell window will disappear.
You can do this with WMI. If you use Win32_Process to create the process, it will persist after you close PowerShell.
For instance:
invoke-wmimethod -path win32_process -name create -argumentlist calc
function GeneratePortableFunction {
param ([string] $name, [scriptblock] $sb)
$block = [ScriptBlock]::Create("return `${function:$name};");
$script = $block.Invoke();
$block = [ScriptBlock]::Create($script);
return ("function {0} {{ {1} }}" -f $name, $block.ToString());
}
function RemoteScript {
param ([string] $header, [string[]] $functions, [string] $footer)
$sb = New-Object System.Text.StringBuilder;
[void]$sb.Append("$header`n");
$functions | % {
[void]$sb.Append($_);
[void]$sb.Append("`n");
}
[void]$sb.Append($footer);
return [ScriptBlock]::Create($sb.ToString());
}
$fnc = #();
$fnc += GeneratePortableFunction -name "NameOfYourFunction1";
$fnc += GeneratePortableFunction -name "NameOfYourFunction2(CallsNameOfYourFunction1)";
$script = RemoteScript -header "param([int] `$param1)" -functions #($fnc) -footer "NameOfYourFunction2 -YourParameter `$param1;";
$p1 = 0;
$job = Start-Job -ScriptBlock $script -ArgumentList #($p1);
while($job.State -eq "Running")
{
write-host "Running...";
Start-Sleep 5;
}
$result = $job | Receive-Job;
Remove-Job -Id $job.Id;
I have a server with lots of media drives ~43TB. An areca 1882ix-16 is set to spin the drives down after 30 minutes of inactivity since most days an individual drive is not even used. This works nicely to prevent unnecessary power and heat. In this case the drives still show up in windows explorer but when you click to access them it takes about 10 seconds for the folder list to show up since it has to wait for the drive to spin up.
For administrative work I have a need to spin up all the drives to be able to search among them. Clicking on each drive in windows explorer and then waiting for it to spin up before clicking the next drive is very tedious. Obviously multiple explorer windows makes it faster but it is still tedious. I thought a powershell script may ease the pain.
So I started with the following:
$mediaDrives = #('E:', 'F:', 'G:', 'H:', 'I:', 'J:', 'K:', 'L:',
'M:','N:', 'O:', 'P:', 'Q:', 'R:', 'S:')
get-childitem $mediaDrives | foreach-object -process { $_.Name }
This is just requesting that each drive in the array have its root folder name listed. That works to wake the drive but it is again a linear function. The script pauses for each drive before printing. Looking for a solution as to how to wake each drive simultaneously. Is there a way to multi-thread or something else?
Here's a script that will do what you want, but it must be run under powershell using the MTA threading mode (which is the default for powershell.exe 2.0, but powershell.exe 3.0 must be launched with the -MTA switch.)
#require -version 2.0
# if running in ISE or in STA console, abort
if (($host.runspace.apartmentstate -eq "STA") -or $psise) {
write-warning "This script must be run under powershell -MTA"
exit
}
$mediaDrives = #('E:', 'F:', 'G:', 'H:', 'I:', 'J:', 'K:', 'L:',
'M:','N:', 'O:', 'P:', 'Q:', 'R:', 'S:')
# create a pool of 8 runspaces
$pool = [runspacefactory]::CreateRunspacePool(1, 8)
$pool.Open()
$jobs = #()
$ps = #()
$wait = #()
$count = $mediaDrives.Length
for ($i = 0; $i -lt $count; $i++) {
# create a "powershell pipeline runner"
$ps += [powershell]::create()
# assign our pool of 8 runspaces to use
$ps[$i].runspacepool = $pool
# add wake drive command
[void]$ps[$i].AddScript(
"dir $($mediaDrives[$i]) > `$null")
# start script asynchronously
$jobs += $ps[$i].BeginInvoke();
# store wait handles for WaitForAll call
$wait += $jobs[$i].AsyncWaitHandle
}
# wait 5 minutes for all jobs to finish (configurable)
$success = [System.Threading.WaitHandle]::WaitAll($wait,
(new-timespan -Minutes 5))
write-host "All completed? $success"
# end async call
for ($i = 0; $i -lt $count; $i++) {
write-host "Completing async pipeline job $i"
try {
# complete async job
$ps[$i].EndInvoke($jobs[$i])
} catch {
# oops-ee!
write-warning "error: $_"
}
# dump info about completed pipelines
$info = $ps[$i].InvocationStateInfo
write-host "State: $($info.state) ; Reason: $($info.reason)"
}
So, for example, save as warmup.ps1 and run like: powershell -mta c:\scripts\warmup.ps1
To read more about runspace pools and the general technique above, take a look at my blog entry about runspacepools:
http://nivot.org/blog/post/2009/01/22/CTP3TheRunspaceFactoryAndPowerShellAccelerators
I chose 8 pretty much arbitrarily for the parallelism factor - experiment yourself with lower or higher numbers.
Spin up a separate powershell instance for each drive or use workflows in PowerShell 3.0.
Anyhow, you can pass drives directly to the Path parameter and skip Foreach-Object all togeteher:
Get-ChildItem $mediaDrives
Have you considered approaching this with the Start-Job cmdlet:
$mediaDrives = #('E:', 'F:', 'G:', 'H:', 'I:', 'J:', 'K:')
$mediaDrives | ForEach-Object {
Start-Job -ArgumentList $_ -ScriptBlock {param($drive)
Get-ChildItem $drive
}
}
The only clever part is that you need to use the -ArgumentList parameter on the Start-Job cmdlet to pass the correct value through for each iteration. This will create a background task that runs in parallel with the execution of the script. If you are curious
If you don't want to wait, well, don't wait: start those wake-up calls in the background.
In bash one would write
foreach drive ($mediadrives) {tickle_and_wake $drive &}
(note the ampersand, which means: start the command in the background, don't wait for it to complete)
In PowerShell that would translate to something like
foreach ($drive in $mediadrives) {
Start-Job {param($d) tickle_and_wake $d} -Arg $drive
}
If you want confirmation that all background jobs have completed, use wait in bash or Wait-Job in Powershell