Wait until all threads complete before running next task - multithreading

I would wrap everything inside foreach($computer in $computers) in a Start-Job to make them run simultaneously. The only problem is, I need to wait for all the jobs to complete before I do the ConvertTo-Json at the bottom.
$sb = "OU=some,OU=ou,DC=some,DC=domain"
$computers = Get-ADComputer -Filter {(Enabled -eq $true)} -SearchBase "$sb" -Properties *
$hasmanufacturer = New-Object System.Collections.Generic.List[System.Object]
foreach($computer in $computers)
{
$drives = try{#(Get-WMIObject -Class Win32_CDROMDrive -Property * -ComputerName $computer.Name -ErrorAction Stop)} catch {$null}
foreach($drive in $drives)
{
if($drive.Manufacturer)
{
$hasmanufacturer.Add($computer)
continue
}
} # inner foreach
}
ConvertTo-Json $hasmanufacturer

Use a Get-Job | Wait-Job before executing the ConvertTo-Json

How about using the array of computer names as a parameter to Invoke-Command. It will run, by default, 32 concurrent remote sessions. The number can be changed with the -Throttle parameter.
$computers = Get-ADComputer -Filter {(Enabled -eq $true)} -SearchBase "OU=Servers,DC=xxx,DC=com" -Properties Name |
Where-Object { $_.Name -match 'LAX_*' } |
ForEach-Object { $_.Name }
$computers
$j = Invoke-Command `
-ComputerName $computers `
-ScriptBlock { Get-WMIObject -Class Win32_CDROMDrive -Property * -ErrorAction Stop } `
-AsJob
while ( (Get-Job -Id $j.Id).Status -eq 'Running') {}
Get-Job -Id $j.Id | Wait-Job
$results = Receive-Job -Id $j.Id
$results

Related

My Powershell script is not explicitly calling for any specific resources but I am getting "ErrorCode: TargetResourceNotFound" error

My Powershell script is not explicitly calling for any specific resources but I am getting "ErrorCode: TargetResourceNotFound" error. I have attached the error in the image. What am I missing?
$subs = Get-AzSubscription | Where-Object {$_.Name -like "*-NonProd"}
foreach ($sub in $subs)
{
Select-AzSubscription -SubscriptionId $sub.Id
$RGs = Get-AzResourceGroup | Where-Object {$_.ResourceGroupName -like "*Infra"}
foreach ($RG in $RGs)
{
$NetworkWatchers = Get-AzNetworkWatcher
$NSGs = (Get-AzNetworkSecurityGroup).Id
foreach ($NSG in $NSGs)
{
foreach ($NetworkWatcher in $NetworkWatchers)
{
$Status = Get-AzNetworkWatcherFlowLogStatus -NetworkWatcherName $NetworkWatcher.Name
ResourceGroupName $RG.ResourceGroupName -TargetResourceId $NSG -Verbose
}
if (($Status).Enabled -eq $true)
{
Write-Output "$NSG in $(($sub).Name) has FlowLogs Enabled" | Tee-Object -FilePath 'C:\Users\user1\downloads\Output.txt' -Verbose -Append
}
if (($Status).Enabled -ne $true)
{
Write-Output "$NSG in $(($sub).Name) does not have FlowLogs Enabled" | Tee-Object -FilePath 'C:\Users\user1\downloads\Output.txt' -Verbose -Append
}
}
}
}
enter code here
Error Attached
Network Watchers are usually in a hidden resource group, and perhaps you are trying to find one in one of the available RGs. Try omitting the RG factor and use
$subs = Get-AzSubscription | Where-Object { $_.Name -like "*-NonProd" }
foreach ($sub in $subs) {
Select-AzSubscription -SubscriptionId $sub.Id
$NetworkWatchers = Get-AzNetworkWatcher
$NSGs = (Get-AzNetworkSecurityGroup).Id
foreach ($NSG in $NSGs) {
foreach ($NetworkWatcher in $NetworkWatchers) {
$Status = Get-AzNetworkWatcherFlowLogStatus -NetworkWatcher $NetworkWatcher -TargetResourceId $NSG -Verbose
}
if (($Status).Enabled -eq $true) {
Write-Output "$NSG in $(($sub).Name) has FlowLogs Enabled" | Tee-Object -FilePath 'C:\Users\user1\downloads\Output.txt' -Verbose -Append
}
if (($Status).Enabled -ne $true) {
Write-Output "$NSG in $(($sub).Name) does not have FlowLogs Enabled" | Tee-Object -FilePath 'C:\Users\user1\downloads\Output.txt' -Verbose -Append
}
}
}
I am able to get all flow logs configuration status.
Here is another approach:
I tried to reproduce the same in my environment and got the same error as below:
The error TargetResourceNotFound usually occurs if you are passing invalid resource group name or subscription name.
To confirm whether the resource group or subscription exists, execute the below code lines separately like below:
$subs = Get-AzSubscription | Where-Object {$_.Name -like "*-name"}
$subs
$RGs = Get-AzResourceGroup | Where-Object {$_.ResourceGroupName -like "*name"}
$RGs
The error states that Target resource identifier /subscriptions/subid/resourceGroups/RG/providers/Microsoft.Network/networkWatchers/*** not found in the region westeurope. Cross-verify whether the Network Watcher exists.
I am able to get the status of the Network Watcher successfully when I passed valid subscription and Resource group like below:
If the error still persists, try excluding the $RGs = Get-AzResourceGroup | Where-Object {$_.ResourceGroupName -like "*name"} and execute.
I appreciate your assistance.
I got it working by changing -NetworkWatcherName $NetworkWatcher.Name to -NetworkWatcherName $NetworkWatcher.ResourceGroupName
foreach ($NSG in $NSGs)
{
# $NSG.Id
# $NSGid = $NSG.Id
foreach ($NetworkWatcher in $NetworkWatchers)
{
$Status = Get-AzNetworkWatcherFlowLogStatus -NetworkWatcherName $NetworkWatcher.ResourceGroupName -ResourceGroupName $RG.ResourceGroupName -TargetResourceId $NSG -Verbose -ErrorAction SilentlyContinue
if (($Status).Enabled -eq $true)
{
Write-Output "$NSG in $(($sub).Name) has FlowLogs Enabled" | Tee-Object -FilePath 'C:\Users\A240379\downloads\OutEnabled.csv' -Verbose -Append
}
if (($Status).Enabled -ne $true)
{
Write-Output "$NSG in $(($sub).Name) does not have FlowLogs Enabled" | Tee-Object -FilePath 'C:\Users\A240379\downloads\OutNotEnabled.csv' -Verbose -Append
}
}
}

Need to run app on N remotes hosts and handle them all

Im using this code to run app on N remote hosts and need to wait until all of them are finished and get their results. But the execution just passes throuhg, all jobs are being marked Completed and code exits.
How to make it wait till apps finished their execution?
$procArray = [System.Collections.ArrayList]#()
foreach ($key in $simulatorServers.keys) {
$unitHost = $simulatorServers[$key][0]
$EXE="C:\app.exe"
Wr-DebugReg "Running $EXE on $unitHost "
$ScriptString="Start-Process -FilePath $EXE "
$ScriptBlock=[System.Management.Automation.ScriptBlock]::Create($ScriptString)
$Session=New-PSSession -ComputerName $unitHost -EnableNetworkAccess -Name "session$counter" -Credential $crNonProd
$rez2 = Invoke-Command -Session $Session -ScriptBlock $ScriptBlock -AsJob
$rez00=$procArray.Add($rez2)
Wr-DebugReg "Running process id=$($rez2.id) name=$($proc.Name)on $unitHost"
Wr-DebugReg ""
}
$procArray | Wait-Job
$procArray | Receive-Job
these jobs gone to status Completed even if launched processes still running
let invoke-command handle the the amount of jobs/sessions to open in parallel - you will receive 1 job with childs:
$scriptBlock = {
start-process -FilePath 'C:\app.exe'
}
$sessions = #(
foreach ($key in $simulatorServers.keys) {
$unitHost = $simulatorServers[$key][0]
New-PSSession -ComputerName $unitHost -EnableNetworkAccess -Name "session$counter" -Credential $crNonProd
}
)
$job = Invoke-Command -Session $Sessions -ScriptBlock $ScriptBlock -AsJob
$result = receive-job -Wait -Job $job
btw. I do not see, based on this sample, what you want to receive. you want to execute "start-process -FilePath 'C:\app.exe' " on the target machines but this won't give you anything back.
to get the information back modify the scriptblock like this:
$scriptBlock = {
$result = start-process -FilePath 'C:\app.exe' -wait -PassThru
return $result
}
This code is working. -Wait is the key to make it wait until all jobs are finished.
$procArray = [System.Collections.ArrayList]#()
foreach ($key in $hosts.keys) {
$unitHost = $hosts[$key][0]
$EXE="c:\app.exe"
$Session=New-PSSession -ComputerName $unitHost -EnableNetworkAccess -Name "session$counter" -Credential $crNonProd
$rez2 = Invoke-Command -Session $Session -ScriptBlock {Start-Process -FilePath $args[0] -Wait} -ArgumentList $EXE -AsJob
$rez00=$procArray.Add($rez2)
Wr-DebugReg "Starting process id=$($rez2.id) name=$($proc.Name)on $unitHost"
Wr-DebugReg ""
}
while ($procArray.State -contains 'Running')
{
Start-Sleep -Seconds 1
}

Execute SQLs in Parallel from directory

I am able to execute all the .SQL files from directory on to the server serially using below command. But, we want to execute all these files in parallel.
My code for serial execution -
$PSScriptRoot = 'Y:\test\'
$ServerName = $env:COMPUTERNAME
foreach ($f in Get-ChildItem -path $PSScriptRoot -Filter *.sql | sort-object
-desc )
{
invoke-sqlcmd -InputFile $f.fullname -ServerInstance $ServerName
}
You could use jobs to have them execute in the background:
foreach ($f in Get-ChildItem -path $PSScriptRoot -Filter *.sql | sort-object
-desc)
{
Start-Job -ScriptBlock { invoke-sqlcmd -InputFile $args[0].fullname -ServerInstance $args[1] } -ArgumentList $f, $ServerName
}
If there is any output you can collect it via:
Get-Job | Wait-Job | Receive-Job

PowerShell Mutex & Jobs

I'm trying to run a PowerShell file delete operation that deletes files based on specific parameters. So, the code goes like this:
$sb = {
Get-ChildItem -Path $SubFolder | Remove-ChildItem
if ($Error[0] -eq $null) {
$results = New-Object PSObject -Property #{
$Foldername = $Subfolder.Name
$TotalFiles = $SubFolder.(Count)
}
} else {
$errorresult = "Error deleting files from $($Subfolder.Name)"
}
#Error Mutex
$ErrorLogmutex = New-Object System.Threading.Mutex($false, "ErrorLogFileHandler")
$ErrorLogmutex.WaitOne()
Out-File -Append -InputObject $errorresult -FilePath $Errorlog
$ErrorLogmutex.ReleaseMutex()
#Success Mutex
$Successmutex = New-Object System.Threading.Mutex($false, "SuccessFileHandler")
$SuccessLogmutex.WaitOne()
Out-File -Append -InputObject $results -FilePath $successlog
$Successmutex.ReleaseMutex()
}
#Calling scriptblock in multi-thread count of 5
{
foreach ($Subfolder in $Folder) {
while ((Get-Job -State Running).Count -ge 5) {
Start-Sleep -Seconds 5
}
Start-Job -ScriptBlock $sb -ArgumentList $arguments | Out-Null
}
The script runs, able to see the results if I explicitly call out Receive-Job -Name JobID, but the output does not produce any log files as I would've thought it would.

Need to Multithread this script and log the results of each [duplicate]

I'm trying to run a virus scan on a list of servers in our environment. There are hundreds of machines, so we'd like to run the scan (using a command line prompt that we already have) around 10 at a time. We're totally new to PowerShell so any help would be really appreciated. We have a general idea of what commands we need to use -- here's how we think it might work for now:
$server = Get-Content "serverlist.txt"
$server | % {
$VirusScan = { Scan32.exe }
Invoke-Command -ScriptBlock { $VirusScan } -computerName $server -ThrottleLimit 10 -Authentication domain/admin
}
Does anyone have any suggestions on how we might orchestrate this?
I'm using something like this for running tasks in parallel on remote hosts:
$maxSlots = 10
$hosts = "foo", "bar", "baz", ...
$job = {
Invoke-Command -ScriptBlock { Scan32.exe } -Computer $ARGV[0] -ThrottleLimit 10 -Authentication domain/admin
}
$queue = [System.Collections.Queue]::Synchronized((New-Object System.Collections.Queue))
$hosts | ForEach-Object { $queue.Enqueue($_) }
while ( $queue.Count -gt 0 -or #(Get-Job -State Running).Count -gt 0 ) {
$freeSlots = $maxSlots - #(Get-Job -State Running).Count
for ( $i = $freeSlots; $i -gt 0 -and $queue.Count -gt 0; $i-- ) {
Start-Job -ScriptBlock $job -ArgumentList $queue.Dequeue() | Out-Null
}
Get-Job -State Completed | ForEach-Object {
Receive-Job -Id $_.Id
Remove-Job -Id $_.Id
}
Sleep -Milliseconds 100
}
# Remove all remaining jobs.
Get-Job | ForEach-Object {
Receive-Job -Id $_.Id
Remove-Job -Id $_.Id
}

Resources