Powershell: Increment variable from multiple threads - multithreading

In Powershell, is there a way to increment a variable from multiple threads safely.
I thought the below code using System.Threading.Interlock::Add would work, but the output shows thread mashing is occurring.
# Threads will attempt to increment this number
$testNumber = 0
# Script will increment the test number 10 times
$script = {
Param(
[ref]$testNumber
)
1..10 | % {
[System.Threading.Interlocked]::Add($testNumber, 1) | Out-Null
}
}
# Start 10 threads to increment the same number
$threads = #()
1..10 | % {
$ps = [powershell]::Create()
$ps.AddScript($script) | Out-Null
$ps.RunspacePool = $pool
$ps.AddParameter('testNumber', [ref]$testNumber) | Out-Null
$threads += #{
ps = $ps
handle = $ps.BeginInvoke()
}
}
# Wait for threads to complete
while ($threads | ? {!$_.Handle.IsCompleted }) {
Start-Sleep -Milliseconds 100
}
# Print the output (should be 100=10*10, but is between 90 and 100)
echo $testNumber

As iRon mentions in the comments, you'll want to use a synchronized hashtable for reading and writing data across the runspace boundary (see inline comments for explanation):
# Threads will attempt to increment this number
$testNumber = 0
# Create synchronized hashtable to act as gatekeeper
$syncHash = [hashtable]::Synchronized(#{ testNumber = $testNumber })
# Create runspace pool with a proxy variable mapping back to $syncHash
$iss = [initialsessionstate]::CreateDefault2()
$iss.Variables.Add(
[System.Management.Automation.Runspaces.SessionStateVariableEntry]::new('syncHash', $syncHash,'')
)
$pool = [runspacefactory]::CreateRunspacePool($iss)
$pool.Open()
# Script will increment the test number 10 times
$script = {
Param(
[string]$targetValue
)
1..10 | % {
# We no longer need Interlocked.* and hairy [ref]-casts, $syncHash is already thread safe
$syncHash[$targetValue]++
}
}
# Start 10 threads to increment the same number
$threads = #()
1..10 | % {
$ps = [powershell]::Create()
$ps.AddScript($script) | Out-Null
$ps.RunspacePool = $pool
# We're no longer passing a [ref] to the variable in the calling runspace.
# Instead, we target the syncHash entry by name
$ps.AddParameter('targetValue', 'testNumber') | Out-Null
$threads += #{
ps = $ps
handle = $ps.BeginInvoke()
}
}
# Wait for threads to complete
while ($threads | ? {!$_.Handle.IsCompleted }) {
Start-Sleep -Milliseconds 100
}
$errorCount = 0
# End invocation lifecycle
$threads|%{
if($_.ps.HadErrors){
$errorCount++
}
$_.ps.EndInvoke($_.handle)
}
if($errorCount){
Write-Warning "${errorCount} thread$(if($errorCount -gt 1){'s'}) had errors"
}
# Et voila, $syncHash['testNumber'] is now 100
$syncHash['testNumber']

Related

Two objects of different sizes and $null

My code below works in every instance except for if one object is $null and the other object has one item. When that situation occurs the output becomes 1 letter like it is indexing and I am not sure why.
How do I combine the two objects to make a final report?
$ADGroups = Get-ADPrincipalGroupMembership -Identity $UserSam | Select-Object distinguishedName, name | Where-Object { ($_.distinguishedName -ne 'CN=Domain Users,CN=Users,DC=com') }
#record AD groups
$ADResult = #()
if ($null -eq $ADGroups) {
Write-Warning "No AD Groups"
$ADResult = [PSCustomObject]#{
ADGroups = #()
ADGroupsdistinguishedName = #()
}
}
Else {
$ADResult = $ADGroups | ForEach-Object {
[PSCustomObject]#{
ADGroups = $_.name
ADGroupsdistinguishedName = $_.distinguishedName
}
}
}
#============= Now Google, get user groups and record
$GoogleGroups = gam print groups member $email members managers owners | ConvertFrom-Csv
# Record Google Groups
$GResult = #()
If ($null -eq $GoogleGroups) {
Write-Warning "No Google Groups"
$GResult = [PSCustomObject]#{
GoogleGroups = #()
Role = #()
}
}
Else {
$group = $null
$GResult = ForEach ($group in $GoogleGroups) {
#this records what role the user had in the group(s)
$GoogleMember = gam print group-members group $group.email members | ConvertFrom-Csv | Select-Object -ExpandProperty email
$Role = $null
If ( $GoogleMember -contains $EMAIL) {
$Role = 'Member'
}
Else {
$GoogleManager = gam print group-members group $group.email managers | ConvertFrom-Csv | Select-Object -ExpandProperty email
If ($GoogleManager -contains $EMAIL) {
$Role = 'Manager'
}
Else {
$Role = 'Owner'
}
}
[PSCustomObject]#{
GoogleGroups = $group.email
Role = $role
}
$group = $null
}
}
# ---------now report that will be dropped off at end
[int]$max = $ADResult.count
if ([int]$GResult.count -gt $max) { [int]$max = $GResult.count }
If ($max -eq 1 -or $max -eq 0) {
$Result = [PSCustomObject]#{
PrimaryEmail = $email
Title = $UserInfo.title
Department = $UserInfo.Department
Manager = $Manager
ADGroupName = $ADResult.ADGroups
ADGroupNameDistinguishedName = $ADResult.ADGroupsdistinguishedName
GoogleGroup = $GResult.GoogleGroups
Role = $GResult.role
DateOfSeparation = (Get-Date).ToString("yyyy_MM_dd")
UserDistinguishedName = $UserInfo.distinguishedName
UserOU = $UserInfo.Ou
PrimaryGroup = $UserInfo.primaryGroup.Split('=').Split(',')
}
}
Else {
$Result = for ( $i = 0; $i -lt $max; $i++) {
[PSCustomObject]#{
PrimaryEmail = $email
Title = $UserInfo.title
Department = $UserInfo.Department
ADGroupName = $ADResult.ADGroups[$i]
ADGroupNameDistinguishedName = $ADResult.ADGroupsdistinguishedName[$i]
GoogleGroup = $GResult.GoogleGroups[$i]
Role = $GResult.role[$i]
DateOfSeparation = (Get-Date).ToString("yyyy_MM_dd")
UserDistinguishedName = $UserInfo.distinguishedName
UserOU = $UserInfo.Ou
PrimaryGroup = $UserInfo.primaryGroup.Split('=').Split(',')[$i]
}
}
}
$Result | Export-Csv 'C:\temp\Groups.csv' -NoTypeInformation
Going by the abstract description of your problem:
You're seeing an unfortunate asymmetry in PowerShell:
In the pipeline, a [string] instance is considered a single object.
PS> ('foo' | Measure-Object).Count
1
With respect to indexing, it is considered an array of characters.
PS> 'foo'[0]
f
A general feature of capturing a PowerShell pipeline's output is that if a command situationally outputs just a single object, that object is captured as-is, whereas two or more output objects result in a regular PowerShell array, of type [object[]].
Typically, this isn't a problem, because PowerShell's unified handling of scalars and collections allows you to index even into a scalar (single object), i.e. to implicitly treat a single object as if it were a single-element array:
PS> (Write-Output 42, 43)[0]
42
PS> (Write-Output 42)[0]
42 # still OK, even though only *one* object was output; same as: (42)[0]
However, with a single [string] instance as the output it becomes a problem, for the reasons stated above:
PS> (Write-Output 'foo', 'bar')[0]
foo # OK
PS> (Write-Output 'foo')[0]
f # !! Indexing into a *single string* treats it as *character array*
The same applies to values returned via member-access enumeration, perhaps surprisingly :
PS> (Get-Item $HOME, /).FullName[0]
C:\Users\Jdoe
PS> (Get-Item $HOME).FullName[0]
C # !! Indexing into a *single string* treats it as *character array*
Workarounds:
Enclose the command of interest in #(...), the array-subexpression operator so as to ensure that its output is always considered an array.
PS> #(Write-Output 'foo')[0]
foo # OK
Alternatively, when capturing a command's output in a variable, type-constrain that variable to [array] (same as [object[]]) or a strongly typed array, [string[]]:
PS> [array] $output = Write-Output 'foo'; $output[0]
foo # OK

Not able to move to the next loop of vm list

I have a code that starts VM's in batches. I am able to start vm's in batches. I am facing an issue here where ('*adds*','*DB*','*') are vm's containing the following wildcard names and need to be started in the following order.The script only starts the first list of vms with 'adds' and continues to restart it in loop it does not move to the next. Any help guys.
$bubbleName="VmList"
do{
$vname=#('*adds*','*DB*','*')
$i=0
Write-Host "Starting VM with "$vname[$i]
$vmList = Get-AzVM -ResourceGroupName $bubbleName -Name $vname[$i]
$batch = #{
Skip = 0
First = 2
}
do{
do{
foreach($vm in ($vmList | Select-Object #batch)){
$params = #($vm.Name, $vm.ResourceGroupName)
$job = Start-Job -ScriptBlock {
param($ComputerName,$serviceName)
Start-AzVM -Name $ComputerName -ResourceGroupName $serviceName
} -ArgumentList $params
}
Wait-Job -Job $job
Get-Job | Receive-Job
Write-Host $batch
$batch.Skip += 2
}
until($batch.skip -ge $vmList.count)
}while($job.state -ne "Completed")
$i++
}while($vname[$i] -ne $null)
You changed this a bit from an earlier (now deleted) version. However, I continued working on the old one and I'll share my thoughts.
If I understand correctly you want to start VMs in orders
*adds*
*db*
* (everything else)
Honestly, I think you're going about your loops are a little convoluted. If it were me, I'd sort the a list of all VMs according to the above order. Then code a single traditional for loop to start 2 at a time and use Wait-Job in the loop.
I don't have an Azure environment to test with but here are my notes from earlier:
Step 1: get the list in the order you need it. This will prevent the funky stuff you're doing with -First & -Skip:
# Example data, substituting for VM objects
$Objects =
#(
[PSCustomObject]#{ Name = "adds1" }
[PSCustomObject]#{ Name = "DB1" }
[PSCustomObject]#{ Name = "adds2" }
[PSCustomObject]#{ Name = "DB2" }
[PSCustomObject]#{ Name = "something" }
[PSCustomObject]#{ Name = "another" }
[PSCustomObject]#{ Name = "last" }
)
$Objects =
$Objects.Where( { $_.Name -match 'adds'} ) +
$Objects.Where( { $_.Name -match 'db'} ) +
$Objects.Where( { $_.Name -notmatch '(adds|db)'} )
Obviously this is a simulation that you'd have to adjust to your VMs. Maybe Objects will actually be defined by Get-AzVM -ResourceGroupName $bubbleName -Name *. At any rate, you should be able to get the VMs in the desired order.
Step 2: a loop to start 2 at a time, waiting for them to start before moving on to the next 2:
$Step = 1
# Note: 1 means 2 at a time, $i and $i+$Step, in this case 1.
# So the first iteration will work on index 0 & 1
# You can modify $Step to different size chunks.
For($i = 0; $i -lt $Objects.Count; ++$i )
{
$ii = $i + $Step
$Jobs =
$Objects[$i..$ii] |
ForEach-Object{
# Run the Start-Job commands against the appropriate machines
}
# Note: The assignment of $Jobs solves another problem. You were only waiting
# for the 2nd of the 2 jobs that were started. Now Wait-Job -Job $Job
# will be an array of jobs, which is allowed by the cmdlet without piping..
# Wait-Job -Job $Jobs
# ...
# I also didn't understand why one of the loops was conditioned on While
# $job.State -ne 'Completed'. Wait job should be enough and the above code
# assumed that.
# Reassign $i so when the loop increments naturally it's 1 past $ii.
# Effectively this chunks 2 elements per iteration.
$i = $ii
}
Again, I don't have the right environment to test all this. But conceptually this should be enough to get you through. I'm sorry about the variable renaming, but given the disadvantage, I had to hack my way through purely on concept.

Reduce threads to reduce CPU usage of powershell process through powershell command

I have used a Fake cpu usage script in powershell,when i run it CPU usage increases,i can see the thread count. Now i have to reduce/increase the threadcount inorder reduce the CPU usage.
Below is the script used for increasing CPU usage. Please help with any powershell command that would reduce/increase thread count for reducing CPU usage. or any other method to reduce CPU usage in this script.
Below is the main part of loop script,by running this in powershell CPU usage increases
# Code
$num = 2
If ([$num -gt 1])) {
$ThreadCount=(Get-Process -ProcessName Powershell | Select-Object -ExpandProperty Threads).Count
$ErrorMessage = "Before CPU Saturation threadcount:" + $ThreadCount
write-AppEventLog $ErrorMessage
$result = 1;
foreach ($number in 1..2147483647) {
$ThreadCount1=(Get-Process -ProcessName Powershell | Select-Object -ExpandProperty Threads).Count
foreach ($number in 1..2147483647) {
$result = $result * $number
$ErrorMessage1 = " CPU Saturation threadcount:" + $ThreadCount1
write-AppEventLog $ErrorMessage1
}
}
}
If you want multiple threads, you can use jobs
Start-Job -ScriptBlock {
# do the thing
}
# Example infinite loop
while($true)
{
Start-Job -ScriptBlock {
# do the thing
}
}

Better than Switch case in powershel?

I have an array like 5 system IDs for now like SID_1_name, SID_2_name......For each system ID .I need to add some details. So I have written powershell script of switch case .Which is working fine for now. But I wonder in future we may add more systems so that I need to add more cases which expands the script...Is there better way to do for future purpose aswell
$excel=#()
$list = Get-AznetworkInterface |where-Object {$_.ResourceGroupName -Clike '*$(givenVarible)'} |Select-Object
foreach ($i in $list) {
$x = " " | Select-Object SID1_name,SID1_VIP,SID2_name,SID2_VIP,SID3_name,SID3_VIP
$case =1
While ($case -1t $i.IpConfigurations.Count)
{
switch ($case){
1 {
$x.SID1_name = $i.IPconfigurations[$case];
$x.SID1_VIP = $i.IPconfigurations[$case].PrivateIpaddress;
break
}
2 {
$x.SID2_name = $i.IPconfigurations[$case];
$x.SID2_VIP = $i.IPconfigurations[$case].PrivateIpaddress;
break
}
3 {
$x.SID1_name = $i.IPconfigurations[$case];
$x.SID1_VIP = $i.IPconfigurations[$case].PrivateIpaddress;
break
}
$case =$case+1
$excel +=$x
$excel | Format-Table SID1_name,SID1_VIP,SID2_name,SID2_VIP,SID3_name,SID3_VIP
$excel |Export-Csv -NTI - Path "$(Build.ArtifactoryStagingDirectory)/report.csv"

Powershell runspaces and color output

Short issue description:
I need non-interspersed output from additional powershell runspaces that my script spawns. I need to be able to set the color of individual lines in the output.
Long issue description:
I wrote a script to deploy updates to a vendor application on multiple remote servers. Originally the script was meant to only deploy a single update at a time, but I have a new requirement that I now process multiple updates interactively. So I've re-written my script to use runspaces so that I can open sessions on all the servers, initialize them, then for every update the user inputs a runspace is created for each of the initialized sessions and the deployment work is done. I previously used jobs, but had to go to runspaces because you cannot pass a PSSession to a job.
I now have my script working, but the output is less than desirable. The job version had nice output because I could call Receive-Job and get all all my output grouped by thread. Also I could use write-host to log output which allows for colored responses (ie. Red text when updates fail to apply). I have been able to get my runspace version of the job to group output by thread, but only by using write-output. If I use write-host output occurs immediately, causing interspersed output which is unacceptable. Unfortunately write-output does not allow colored output. Setting $host.UI.RawUI.ForegroundColor also does not work because if the output does not happen at the moment in time when the color has been set, the effect does not happen. Since my output won't happen until the end, the $host settings are no longer in play.
Below is a quick demo script to illustrate my woes:
#Runspacing output example. Fix interleaving
cls
#region Setup throttle, pool, iterations
$iterations = 5
$throttleLimit = 2
write-host ('Throttled to ' + $throttleLimit + ' concurrent threads')
$iss = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()
$Pool = [runspacefactory]::CreateRunspacePool(1,$throttleLimit,$iss,$Host)
$Pool.Open()
#endregion
#This works because the console color is set at the time output occurs
$OrigfC = $host.UI.RawUI.ForegroundColor
$host.UI.RawUI.ForegroundColor = 'red'
write-host 'THIS IS RED TEXT!'
start-sleep 2
$host.UI.RawUI.ForegroundColor = $OrigfC
#define code to run off the main thread
$scriptBlock = {
$nl = ([Environment]::NewLine.Chars(0)+[Environment]::NewLine.Chars(1))
#This does not work because output won't occur until after color has been reset
$OrigfC = $host.UI.RawUI.ForegroundColor
$host.UI.RawUI.ForegroundColor = 'yellow'
write-output ($nl + ' TEST: ' + $args[0])
Write-Output (' Some write-output: ' + $args[0])
Start-Sleep 1
write-host (' Some write-host: ' + $args[0]) -ForegroundColor Cyan # notice write-host occurs immediately
Start-Sleep 1
$host.UI.RawUI.ForegroundColor = $OrigfC
}
#Start new runspaces
$threads = #()
$handles = #(for($x = 1; $x -le $iterations; $x++)
{
$powerShell = [PowerShell]::Create().AddScript($scriptBlock).AddParameters(#($x))
$powershell.RunspacePool = $Pool
$powerShell.BeginInvoke()
$threads += $powerShell
})
#Wait for threads to complete
$completedCount = 0
$completed = ($handles | where-object {$_.IsCompleted -eq $true}).count
while($handles.IsCompleted.Contains($false))
{
if($completedCount -ne ($handles | where-object {$_.IsCompleted -eq $true}).count)
{
$completedCount = ($handles | where-object {$_.IsCompleted -eq $true}).count
write-host ('Threads Completed: ' + $completedCount + ' of ' + $iterations)
}
write-host '.' -nonewline
Start-Sleep 1
}
write-host ('Threads Completed: ' + ($handles | where-object {$_.IsCompleted -eq $true}).count + ' of ' + $iterations)
#output from threads
for($z = 0; $z -lt $handles.Count; $z++)
{
$threads[$z].EndInvoke($handles[$z]) #causes output
$threads[$z].Dispose()
$handles[$z] = $null
}
$Pool.Dispose()
The output is below, unless specified the output is in gray:
My goal is to be able to get the lines that say "Some write-output: X" set to one color, and the "TEST: X" set to a different color. Ideas? Please note I'm running from the powershell prompt. You will get different results if you run in the ISE.
Throttled to 2 concurrent threads
THIS IS RED TEXT! #Outputs in Red
. Some write-host: 1 #Outputs in Cyan
Some write-host: 2 #Outputs in Cyan
.Threads Completed: 2 of 5 #Outputs in Yellow
. Some write-host: 3 #Outputs in Cyan
Some write-host: 4 #Outputs in Cyan
.Threads Completed: 4 of 5 #Outputs in Yellow
. Some write-host: 5 #Outputs in Cyan
.Threads Completed: 5 of 5
TEST: 1
Some write-output: 1
TEST: 2
Some write-output: 2
TEST: 3
Some write-output: 3
TEST: 4
Some write-output: 4
TEST: 5
Some write-output: 5
Edit: Adding another example to address Mjolinor's Answer. M, you are correct I can pass sessions to a job; I have oversimplified my example above. Please consider this example below where I am sending a function to the job. If this line ( if(1 -ne 1){MyFunc -ses $args[0]} ) is commented out below, it will run. If the line is not commented out the session (type System.Management.Automation.Runspaces.PSSession) gets converted to a type of Deserialized.System.Management.Automation.Runspaces.PSSession, even though the MyFunc call cannot be hit. I have not been able to figure this out so I started moving towards runspaces. Do you think there is a job oriented solution?
cls
$ses = New-PSSession -ComputerName XXXX
Write-Host ('Outside job: '+$ses.GetType())
$func = {
function MyFunc {
param([parameter(Mandatory=$true)][PSSession]$ses)
Write-Host ('Inside fcn: '+$ses.GetType())
}
}
$scriptBlock = {
Write-Host ('Inside job: '+$args[0].GetType())
if(1 -ne 1){MyFunc -ses $args[0]}
}
Start-Job -InitializationScript $func -ScriptBlock $scriptBlock -Args #($ses) | Out-Null
While (Get-Job -State "Running") { }
Get-Job | Receive-Job
Remove-Job *
Remove-PSSession -Session $ses
I don't quite understand the statement that you can't pass a PSSession to a job. You can run Invoke-Command with both -Session and -AsJob parameters, creating at job targeted to the session.
As far as the coloring conundrum, have you considered using the Verbose or Debug streams for the output you want to be a different color? Powershell should automatically make it a different color depending on the stream it came from.
This gets a bunch easier in Powershell 7, which wasn't yet available when the question was originally asked.
In powershell 7 you can use the foreach-object -parallel {} -asjob command and then fetch the results when all the jobs are completed.
In this example:
There are 5 different jobs with 5 sub-steps
Each step has a 0.5 second delay to demonstrate interleaving
Each job get a color specific to the job
Each step gets a color specific to the step
The output is color coded, and grouped by job
$script = {
foreach ( $Step in 1..5 ) {
write-host "Job $_" -ForegroundColor #{'1'='red';'2'='yellow';'3'='cyan';'4'='magenta';'5'='green'}[[string]$_] -NoNewLine
write-host " Step $Step" -ForegroundColor #{'1'='red';'2'='yellow';'3'='cyan';'4'='magenta';'5'='green'}[[string]$Step]
sleep 0.5
} # next step
} # end script
$Job = 1..5 | ForEach-Object -Parallel $Script -ThrottleLimit 5 -AsJob
$job | Wait-Job | Receive-Job
It should be noted:
The challenge with $job | Wait-Job | Receive-Job is that if any one job hangs, then this command will never return. There are a plethora of posts which describe how to deal with this limitation.
Foreach-object and Foreach-Object -Parallel are two entirely different animals. The former can make calls to predefined functions whereas the later will only have access to what was defined inside the script block.

Resources