Add a 5 minutes progress bar in powershell - multithreading

I would like to add a 5 minutes progress bar in my Powershell script. I don't find the solution to run my progress bar and the other part of my script at the same time... Can someone help please ? Do I need to implement any thread ? Thank you.
Here is the progress bar :
$seconds = 60
$minutes = $seconds * 5
1..$minutes |ForEach-Object {
$percent = $_ * 100 / $minutes
Write-Progress -Activity "Créations des machines virtuelles" -Status "$([math]::ceiling((($minutes - $_) / 60))) minutes remaining..." -PercentComplete $percent
Start-Sleep -Seconds 1
}

You can use a System.Timers.Timer instance and subscribe to its Elapsed events using Register-ObjectEvent with a script block ({ ... }) passed to its -Action parameter, from where you can call Write-Progress periodically.
This allows you to perform foreground operations as usual.
Here's a self-contained example:
# Initialize the progress display
$activity = 'Creating VMs...'
Write-Progress -Activity $activity -PercentComplete 0
$totalMinutes = 1 # How long to show the progress bar for.
$timerIntervalSecs = 3 # The interval for firing timer events
# Create an initially disabled timer that fires every $intervalSecs when enabled.
$timer = [System.Timers.Timer]::new($timerIntervalSecs * 1000)
# Register for the timer's "Elapsed" event.
# The relevant values are passed via a hashtable passed to the -MessageData parameter,
# which can be accesssed with $Event.MessageData inside the -Action script block.
# -Action script block - which runs in a dynamic module - full access to the caller's variables.
$eventJob = Register-ObjectEvent -InputObject $timer -EventName Elapsed -Action {
$endTime, $totalMinutes, $activity = $Event.MessageData.EndTime, $Event.MessageData.TotalMinutes, $Event.MessageData.Activity
$timeLeft = $endTime - (Get-Date)
$percent = (1 - $timeLeft.TotalSeconds / ($totalMinutes * 60)) * 100
if ($timeLeft -lt 0) { $percent = 100; $timeLeft = [timespan] 0 } # fully elapsed already.
Write-Progress -Activity $activity -Status "$([math]::Floor($percent))% complete, $([math]::Ceiling($timeLeft.TotalMinutes)) minute(s) remaining..." -PercentComplete $percent
} -MessageData (#{ EndTime = (Get-Date).AddMinutes($totalMinutes); TotalMinutes = $totalMinutes; Activity = $activity })
$timer.Start() # Start the timer.
Write-Verbose -vb 'Simulating foreground processing. Press Ctrl-C to exit.'
try {
# Simulate foreground processing.
while ($true) {
Write-Host -NoNewline .
# Sleep a little. Note: Event-processing is blocked during sleep.
Start-Sleep -Seconds 1
}
} finally {
# Cleanup.
$timer.Stop()
Remove-Job $eventJob -Force
Write-Progress -Activity $activity -Completed
}

Related

Not able to move to the next loop of vm list

I have a code that starts VM's in batches. I am able to start vm's in batches. I am facing an issue here where ('*adds*','*DB*','*') are vm's containing the following wildcard names and need to be started in the following order.The script only starts the first list of vms with 'adds' and continues to restart it in loop it does not move to the next. Any help guys.
$bubbleName="VmList"
do{
$vname=#('*adds*','*DB*','*')
$i=0
Write-Host "Starting VM with "$vname[$i]
$vmList = Get-AzVM -ResourceGroupName $bubbleName -Name $vname[$i]
$batch = #{
Skip = 0
First = 2
}
do{
do{
foreach($vm in ($vmList | Select-Object #batch)){
$params = #($vm.Name, $vm.ResourceGroupName)
$job = Start-Job -ScriptBlock {
param($ComputerName,$serviceName)
Start-AzVM -Name $ComputerName -ResourceGroupName $serviceName
} -ArgumentList $params
}
Wait-Job -Job $job
Get-Job | Receive-Job
Write-Host $batch
$batch.Skip += 2
}
until($batch.skip -ge $vmList.count)
}while($job.state -ne "Completed")
$i++
}while($vname[$i] -ne $null)
You changed this a bit from an earlier (now deleted) version. However, I continued working on the old one and I'll share my thoughts.
If I understand correctly you want to start VMs in orders
*adds*
*db*
* (everything else)
Honestly, I think you're going about your loops are a little convoluted. If it were me, I'd sort the a list of all VMs according to the above order. Then code a single traditional for loop to start 2 at a time and use Wait-Job in the loop.
I don't have an Azure environment to test with but here are my notes from earlier:
Step 1: get the list in the order you need it. This will prevent the funky stuff you're doing with -First & -Skip:
# Example data, substituting for VM objects
$Objects =
#(
[PSCustomObject]#{ Name = "adds1" }
[PSCustomObject]#{ Name = "DB1" }
[PSCustomObject]#{ Name = "adds2" }
[PSCustomObject]#{ Name = "DB2" }
[PSCustomObject]#{ Name = "something" }
[PSCustomObject]#{ Name = "another" }
[PSCustomObject]#{ Name = "last" }
)
$Objects =
$Objects.Where( { $_.Name -match 'adds'} ) +
$Objects.Where( { $_.Name -match 'db'} ) +
$Objects.Where( { $_.Name -notmatch '(adds|db)'} )
Obviously this is a simulation that you'd have to adjust to your VMs. Maybe Objects will actually be defined by Get-AzVM -ResourceGroupName $bubbleName -Name *. At any rate, you should be able to get the VMs in the desired order.
Step 2: a loop to start 2 at a time, waiting for them to start before moving on to the next 2:
$Step = 1
# Note: 1 means 2 at a time, $i and $i+$Step, in this case 1.
# So the first iteration will work on index 0 & 1
# You can modify $Step to different size chunks.
For($i = 0; $i -lt $Objects.Count; ++$i )
{
$ii = $i + $Step
$Jobs =
$Objects[$i..$ii] |
ForEach-Object{
# Run the Start-Job commands against the appropriate machines
}
# Note: The assignment of $Jobs solves another problem. You were only waiting
# for the 2nd of the 2 jobs that were started. Now Wait-Job -Job $Job
# will be an array of jobs, which is allowed by the cmdlet without piping..
# Wait-Job -Job $Jobs
# ...
# I also didn't understand why one of the loops was conditioned on While
# $job.State -ne 'Completed'. Wait job should be enough and the above code
# assumed that.
# Reassign $i so when the loop increments naturally it's 1 past $ii.
# Effectively this chunks 2 elements per iteration.
$i = $ii
}
Again, I don't have the right environment to test all this. But conceptually this should be enough to get you through. I'm sorry about the variable renaming, but given the disadvantage, I had to hack my way through purely on concept.

Powershell Jobs not returning what I expect

I have this PowerShell script that reads lines of integers from a file and creates a new job with $CHUNK_SIZE amount of lines to get the sum of all the prime numbers until the end of the file. The $MAX_THREADS is the amount of jobs hat can be running at the same time, I'm testing with 1, but will change to 2, 4, and 8 later. I wait for all the jobs to complete and then receive all the subtotals from the jobs to get the actual sum of all primes in the file. My problem is that the accumulated total at the end should be 2844292, but I keep getting 2766271. I have checked my prime function and whats being read from the file and being sent to the job - but they do not seem to be the problem. When I look at my output, I notice that I receive the same value twice in a row for the last two jobs..i'm not sure why that's happening.
These photos show my output:
1.Get-Job Info
2.What I get back from Receive-Job and my total increasing
Any help as to why my total is off would be greatly appreciated! Thanks!
Set-StrictMode -Version latest
$CHUNK_SIZE = 1024
$MAX_THREADS = 1
$scriptBlock = {
param($chunkArr)
function isPrime([int]$data){
if($data -lt 2){return $FALSE}
if($data -eq 2){return $TRUE}
if($data % 2 -eq 0){return $FALSE}
for($i=3;$i*$i-le$data;$i+=2){
if($data % $i -eq 0){return $FALSE}
}
return $TRUE
}
$total = 0
foreach($line in $chunkArr){
$data = [int]$line
if(isPrime $data){
$total += $data
}
}
$total
}
$eof = $FALSE
$reader = New-Object System.IO.StreamReader("$PWD/ass2-20000.txt")
$chunkArr = New-Object System.Collections.ArrayList
while(!$eof){
$chunkArr.Clear()
for($i=0;$i -lt $CHUNK_SIZE;$i++){
$line = $reader.ReadLine()
if($line -eq $NULL){
$eof = $TRUE
break
}
$chunkArr.add($line) | Out-Null
}
While(#(Get-Job -state running).count -ge $MAX_THREADS){
Start-Sleep -Seconds .2
}
Start-Job -ArgumentList (,$chunkArr) -ScriptBlock $scriptBlock
}
While(#(Get-Job -state running).count -ge 1){
Start-Sleep -Seconds .2
}
Get-Job
$total = 0
foreach($job in Get-Job){
$tmp = Receive-job $job
Write-Output ("Recieved: " + $tmp)
$total += $tmp
Write-Output ("Total: " + $total)
Remove-Job $job
}
$reader.Close()
$total
EDIT: this image shows what the jobs should be returning, the last line showing the correct total..it seems my first job is never being received and my last one is being received twice
EDIT: Compare What I should be getting from each Job to What I actually get

Powershell Throttle Multi thread jobs via job completion

All the tuts I have found use a pre defined sleep time to throttle jobs.
I need the throttle to wait until a job is completed before starting a new one.
Only 4 jobs can be running at one time.
So The script will run up 4 and currently pauses for 10 seconds then runs up the rest.
What I want is for the script to only allow 4 jobs to be running at one time and as a job is completed a new one is kicked off.
Jobs are initialised via a list of servers names.
Is it possible to archive this?
$servers = Get-Content "C:\temp\flashfilestore\serverlist.txt"
$scriptBlock = { #DO STUFF }
$MaxThreads = 4
foreach($server in $servers) {
Start-Job -ScriptBlock $scriptBlock -argumentlist $server
While($(Get-Job -State 'Running').Count -ge $MaxThreads) {
sleep 10 #Need this to wait until a job is complete and kick off a new one.
}
}
Get-Job | Wait-Job | Receive-Job
You can test the following :
$servers = Get-Content "C:\temp\flashfilestore\serverlist.txt"
$scriptBlock = { #DO STUFF }
invoke-command -computerName $servers -scriptblock $scriptBlock -jobname 'YourJobSpecificName' -throttlelimit 4 -AsJob
This command uses the Invoke-Command cmdlet and its AsJob parameter to start a background job that runs a scriptblock on numerous computers. Because the command must not be run more than 4 times concurrently, the command uses the ThrottleLimit parameter of Invoke-Command to limit the number of concurrent commands to 4.
Be careful that the file contains the computer names in a domain.
In order to avoid inventing a wheel I would recommend to use one of the
existing tools.
One of them is the script
Invoke-Parallel.ps1.
It is written in PowerShell, you can see how it is implemented directly. It is
easy to get and it does not require any installation for using it.
Another one is the module SplitPipeline.
It may work faster because it is written in C#. It also covers some more use
cases, for example slow or infinite input, use of initialization and cleanup scripts.
In the latter case the code with 4 parallel pipelines will be
$servers | Split-Pipeline -Count 4 {process{ <# DO STUFF on $_ #> }}
I wrote a blog article which covers multithreading any given script via actual threads. You can find the full post here:
http://www.get-blog.com/?p=189
The basic setup is:
$ISS = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()
$RunspacePool = [runspacefactory]::CreateRunspacePool(1, $MaxThreads, $ISS, $Host)
$RunspacePool.Open()
$Code = [ScriptBlock]::Create($(Get-Content $FileName))
$PowershellThread = [powershell]::Create().AddScript($Code)
$PowershellThread.RunspacePool = $RunspacePool
$Handle = $PowershellThread.BeginInvoke()
$Job = "" | Select-Object Handle, Thread, object
$Job.Handle = $Handle
$Job.Thread = $PowershellThread
$Job.Object = $Object.ToString()
$Job.Thread.EndInvoke($Job.Handle)
$Job.Thread.Dispose()
Instead of sleep 10 you could also just wait on a job (-any job):
Get-Job | Wait-Job -Any | Out-Null
When there are no more jobs to kick off, start printing the output. You can also do this within the loop immediately after the above command. The script will receive jobs as they finish instead of waiting until the end.
Get-Job -State Completed | % {
Receive-Job $_ -AutoRemoveJob -Wait
}
So your script would look like this:
$servers = Get-Content "C:\temp\flashfilestore\serverlist.txt"
$scriptBlock = { #DO STUFF }
$MaxThreads = 4
foreach ($server in $servers) {
Start-Job -ScriptBlock $scriptBlock -argumentlist $server
While($(Get-Job -State Running).Count -ge $MaxThreads) {
Get-Job | Wait-Job -Any | Out-Null
}
Get-Job -State Completed | % {
Receive-Job $_ -AutoRemoveJob -Wait
}
}
While ($(Get-Job -State Running).Count -gt 0) {
Get-Job | Wait-Job -Any | Out-Null
}
Get-Job -State Completed | % {
Receive-Job $_ -AutoRemoveJob -Wait
}
Having said all that, I prefer runspaces (similar to Ryans post) or even workflows if you can use them. These are far less resource intensive than starting multiple powershell processes.
Your script looks good, try and add something like
Write-Host ("current count:" + ($(Get-Job -State 'Running').Count) + " on server:" + $server)
after your while loop to work out whether the job count is going down where you wouldn't expect it.
I noticed that every Start-Job command resulted in an additional conhost.exe process in the task manager. Knowing this, I was able to throttle using the following logic, where 5 is my desired number of concurrent threads (so I use 4 in my -gt statement since I am looking for a count greater than):
while((Get-Process conhost -ErrorAction SilentlyContinue).Count -gt 4){Start-Sleep -Seconds 1}

Powershell, Stop-Job takes too long

I created a powershell job, I want to limit it's running time to 10 seconds.
so I used the Wait-Job command, and if it times out I execute a Stop-Job command.
The problem is that the Stop-Job command takes about 2 minutes.
How can I fix it and stop the job immediately?
While($hasTimeFromNTP -eq $false)
{
Write-Host "Start get time from NTP" -ForegroundColor Yellow
Start-Job -Name GetNTPTime -ScriptBlock $getTimeFromNtp | Out-Null
$result = Wait-Job GetNTPTime -Timeout 10
if($result -ne $null)
{
$NTPTime = Receive-Job GetNTPTime
$hasTimeFromNTP = $true
}
else
{
Write-Host "GetTimeFromNTP timed out"
Stop-Job GetNTPTime
Remove-Job GetNTPTime -Force
}
}
Thanks
I don't see a -Force parameter on Stop-Job. One option would be to have the Job return the PowerShell process id it is running in $pid as the initial output. You could use that pid to Stop-Process on the Powershell.exe spun up for that background job. That's harsh but if you don't want to wait for 2 minutes, I'm not seeing other ways to force the job to stop quicker.

Powershell runspaces and color output

Short issue description:
I need non-interspersed output from additional powershell runspaces that my script spawns. I need to be able to set the color of individual lines in the output.
Long issue description:
I wrote a script to deploy updates to a vendor application on multiple remote servers. Originally the script was meant to only deploy a single update at a time, but I have a new requirement that I now process multiple updates interactively. So I've re-written my script to use runspaces so that I can open sessions on all the servers, initialize them, then for every update the user inputs a runspace is created for each of the initialized sessions and the deployment work is done. I previously used jobs, but had to go to runspaces because you cannot pass a PSSession to a job.
I now have my script working, but the output is less than desirable. The job version had nice output because I could call Receive-Job and get all all my output grouped by thread. Also I could use write-host to log output which allows for colored responses (ie. Red text when updates fail to apply). I have been able to get my runspace version of the job to group output by thread, but only by using write-output. If I use write-host output occurs immediately, causing interspersed output which is unacceptable. Unfortunately write-output does not allow colored output. Setting $host.UI.RawUI.ForegroundColor also does not work because if the output does not happen at the moment in time when the color has been set, the effect does not happen. Since my output won't happen until the end, the $host settings are no longer in play.
Below is a quick demo script to illustrate my woes:
#Runspacing output example. Fix interleaving
cls
#region Setup throttle, pool, iterations
$iterations = 5
$throttleLimit = 2
write-host ('Throttled to ' + $throttleLimit + ' concurrent threads')
$iss = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()
$Pool = [runspacefactory]::CreateRunspacePool(1,$throttleLimit,$iss,$Host)
$Pool.Open()
#endregion
#This works because the console color is set at the time output occurs
$OrigfC = $host.UI.RawUI.ForegroundColor
$host.UI.RawUI.ForegroundColor = 'red'
write-host 'THIS IS RED TEXT!'
start-sleep 2
$host.UI.RawUI.ForegroundColor = $OrigfC
#define code to run off the main thread
$scriptBlock = {
$nl = ([Environment]::NewLine.Chars(0)+[Environment]::NewLine.Chars(1))
#This does not work because output won't occur until after color has been reset
$OrigfC = $host.UI.RawUI.ForegroundColor
$host.UI.RawUI.ForegroundColor = 'yellow'
write-output ($nl + ' TEST: ' + $args[0])
Write-Output (' Some write-output: ' + $args[0])
Start-Sleep 1
write-host (' Some write-host: ' + $args[0]) -ForegroundColor Cyan # notice write-host occurs immediately
Start-Sleep 1
$host.UI.RawUI.ForegroundColor = $OrigfC
}
#Start new runspaces
$threads = #()
$handles = #(for($x = 1; $x -le $iterations; $x++)
{
$powerShell = [PowerShell]::Create().AddScript($scriptBlock).AddParameters(#($x))
$powershell.RunspacePool = $Pool
$powerShell.BeginInvoke()
$threads += $powerShell
})
#Wait for threads to complete
$completedCount = 0
$completed = ($handles | where-object {$_.IsCompleted -eq $true}).count
while($handles.IsCompleted.Contains($false))
{
if($completedCount -ne ($handles | where-object {$_.IsCompleted -eq $true}).count)
{
$completedCount = ($handles | where-object {$_.IsCompleted -eq $true}).count
write-host ('Threads Completed: ' + $completedCount + ' of ' + $iterations)
}
write-host '.' -nonewline
Start-Sleep 1
}
write-host ('Threads Completed: ' + ($handles | where-object {$_.IsCompleted -eq $true}).count + ' of ' + $iterations)
#output from threads
for($z = 0; $z -lt $handles.Count; $z++)
{
$threads[$z].EndInvoke($handles[$z]) #causes output
$threads[$z].Dispose()
$handles[$z] = $null
}
$Pool.Dispose()
The output is below, unless specified the output is in gray:
My goal is to be able to get the lines that say "Some write-output: X" set to one color, and the "TEST: X" set to a different color. Ideas? Please note I'm running from the powershell prompt. You will get different results if you run in the ISE.
Throttled to 2 concurrent threads
THIS IS RED TEXT! #Outputs in Red
. Some write-host: 1 #Outputs in Cyan
Some write-host: 2 #Outputs in Cyan
.Threads Completed: 2 of 5 #Outputs in Yellow
. Some write-host: 3 #Outputs in Cyan
Some write-host: 4 #Outputs in Cyan
.Threads Completed: 4 of 5 #Outputs in Yellow
. Some write-host: 5 #Outputs in Cyan
.Threads Completed: 5 of 5
TEST: 1
Some write-output: 1
TEST: 2
Some write-output: 2
TEST: 3
Some write-output: 3
TEST: 4
Some write-output: 4
TEST: 5
Some write-output: 5
Edit: Adding another example to address Mjolinor's Answer. M, you are correct I can pass sessions to a job; I have oversimplified my example above. Please consider this example below where I am sending a function to the job. If this line ( if(1 -ne 1){MyFunc -ses $args[0]} ) is commented out below, it will run. If the line is not commented out the session (type System.Management.Automation.Runspaces.PSSession) gets converted to a type of Deserialized.System.Management.Automation.Runspaces.PSSession, even though the MyFunc call cannot be hit. I have not been able to figure this out so I started moving towards runspaces. Do you think there is a job oriented solution?
cls
$ses = New-PSSession -ComputerName XXXX
Write-Host ('Outside job: '+$ses.GetType())
$func = {
function MyFunc {
param([parameter(Mandatory=$true)][PSSession]$ses)
Write-Host ('Inside fcn: '+$ses.GetType())
}
}
$scriptBlock = {
Write-Host ('Inside job: '+$args[0].GetType())
if(1 -ne 1){MyFunc -ses $args[0]}
}
Start-Job -InitializationScript $func -ScriptBlock $scriptBlock -Args #($ses) | Out-Null
While (Get-Job -State "Running") { }
Get-Job | Receive-Job
Remove-Job *
Remove-PSSession -Session $ses
I don't quite understand the statement that you can't pass a PSSession to a job. You can run Invoke-Command with both -Session and -AsJob parameters, creating at job targeted to the session.
As far as the coloring conundrum, have you considered using the Verbose or Debug streams for the output you want to be a different color? Powershell should automatically make it a different color depending on the stream it came from.
This gets a bunch easier in Powershell 7, which wasn't yet available when the question was originally asked.
In powershell 7 you can use the foreach-object -parallel {} -asjob command and then fetch the results when all the jobs are completed.
In this example:
There are 5 different jobs with 5 sub-steps
Each step has a 0.5 second delay to demonstrate interleaving
Each job get a color specific to the job
Each step gets a color specific to the step
The output is color coded, and grouped by job
$script = {
foreach ( $Step in 1..5 ) {
write-host "Job $_" -ForegroundColor #{'1'='red';'2'='yellow';'3'='cyan';'4'='magenta';'5'='green'}[[string]$_] -NoNewLine
write-host " Step $Step" -ForegroundColor #{'1'='red';'2'='yellow';'3'='cyan';'4'='magenta';'5'='green'}[[string]$Step]
sleep 0.5
} # next step
} # end script
$Job = 1..5 | ForEach-Object -Parallel $Script -ThrottleLimit 5 -AsJob
$job | Wait-Job | Receive-Job
It should be noted:
The challenge with $job | Wait-Job | Receive-Job is that if any one job hangs, then this command will never return. There are a plethora of posts which describe how to deal with this limitation.
Foreach-object and Foreach-Object -Parallel are two entirely different animals. The former can make calls to predefined functions whereas the later will only have access to what was defined inside the script block.

Resources