Need to Multithread this script and log the results of each [duplicate] - multithreading

I'm trying to run a virus scan on a list of servers in our environment. There are hundreds of machines, so we'd like to run the scan (using a command line prompt that we already have) around 10 at a time. We're totally new to PowerShell so any help would be really appreciated. We have a general idea of what commands we need to use -- here's how we think it might work for now:
$server = Get-Content "serverlist.txt"
$server | % {
$VirusScan = { Scan32.exe }
Invoke-Command -ScriptBlock { $VirusScan } -computerName $server -ThrottleLimit 10 -Authentication domain/admin
}
Does anyone have any suggestions on how we might orchestrate this?

I'm using something like this for running tasks in parallel on remote hosts:
$maxSlots = 10
$hosts = "foo", "bar", "baz", ...
$job = {
Invoke-Command -ScriptBlock { Scan32.exe } -Computer $ARGV[0] -ThrottleLimit 10 -Authentication domain/admin
}
$queue = [System.Collections.Queue]::Synchronized((New-Object System.Collections.Queue))
$hosts | ForEach-Object { $queue.Enqueue($_) }
while ( $queue.Count -gt 0 -or #(Get-Job -State Running).Count -gt 0 ) {
$freeSlots = $maxSlots - #(Get-Job -State Running).Count
for ( $i = $freeSlots; $i -gt 0 -and $queue.Count -gt 0; $i-- ) {
Start-Job -ScriptBlock $job -ArgumentList $queue.Dequeue() | Out-Null
}
Get-Job -State Completed | ForEach-Object {
Receive-Job -Id $_.Id
Remove-Job -Id $_.Id
}
Sleep -Milliseconds 100
}
# Remove all remaining jobs.
Get-Job | ForEach-Object {
Receive-Job -Id $_.Id
Remove-Job -Id $_.Id
}

Related

Need to run app on N remotes hosts and handle them all

Im using this code to run app on N remote hosts and need to wait until all of them are finished and get their results. But the execution just passes throuhg, all jobs are being marked Completed and code exits.
How to make it wait till apps finished their execution?
$procArray = [System.Collections.ArrayList]#()
foreach ($key in $simulatorServers.keys) {
$unitHost = $simulatorServers[$key][0]
$EXE="C:\app.exe"
Wr-DebugReg "Running $EXE on $unitHost "
$ScriptString="Start-Process -FilePath $EXE "
$ScriptBlock=[System.Management.Automation.ScriptBlock]::Create($ScriptString)
$Session=New-PSSession -ComputerName $unitHost -EnableNetworkAccess -Name "session$counter" -Credential $crNonProd
$rez2 = Invoke-Command -Session $Session -ScriptBlock $ScriptBlock -AsJob
$rez00=$procArray.Add($rez2)
Wr-DebugReg "Running process id=$($rez2.id) name=$($proc.Name)on $unitHost"
Wr-DebugReg ""
}
$procArray | Wait-Job
$procArray | Receive-Job
these jobs gone to status Completed even if launched processes still running
let invoke-command handle the the amount of jobs/sessions to open in parallel - you will receive 1 job with childs:
$scriptBlock = {
start-process -FilePath 'C:\app.exe'
}
$sessions = #(
foreach ($key in $simulatorServers.keys) {
$unitHost = $simulatorServers[$key][0]
New-PSSession -ComputerName $unitHost -EnableNetworkAccess -Name "session$counter" -Credential $crNonProd
}
)
$job = Invoke-Command -Session $Sessions -ScriptBlock $ScriptBlock -AsJob
$result = receive-job -Wait -Job $job
btw. I do not see, based on this sample, what you want to receive. you want to execute "start-process -FilePath 'C:\app.exe' " on the target machines but this won't give you anything back.
to get the information back modify the scriptblock like this:
$scriptBlock = {
$result = start-process -FilePath 'C:\app.exe' -wait -PassThru
return $result
}
This code is working. -Wait is the key to make it wait until all jobs are finished.
$procArray = [System.Collections.ArrayList]#()
foreach ($key in $hosts.keys) {
$unitHost = $hosts[$key][0]
$EXE="c:\app.exe"
$Session=New-PSSession -ComputerName $unitHost -EnableNetworkAccess -Name "session$counter" -Credential $crNonProd
$rez2 = Invoke-Command -Session $Session -ScriptBlock {Start-Process -FilePath $args[0] -Wait} -ArgumentList $EXE -AsJob
$rez00=$procArray.Add($rez2)
Wr-DebugReg "Starting process id=$($rez2.id) name=$($proc.Name)on $unitHost"
Wr-DebugReg ""
}
while ($procArray.State -contains 'Running')
{
Start-Sleep -Seconds 1
}

Register trackingevent for all background jobs?

Good afternoon,
I've been working with trying to register an event based on when all jobs are completed. Im able to successfully register one, but id like to get a message pop-up once all background jobs are completed. Anyone familiar with how to do so?
I attempted the following, but it errors out saying jobs is null:
1..10 | ForEach-Object -Process {
Start-Job { Start-Sleep $_ } -Name "$_" | Out-Null} -OutVariable $jobs
Register-ObjectEvent $jobs StateChanged -Action {
[System.Windows.MessageBox]::Show('Done')
$eventSubscriber | Unregister-Event
$eventSubscriber.Action | Remove-Job
} | Out-Null
I feel like a Do{}Until() loop can do it but, im not sure how to register that to check until the job has completed. Also tried to follow along with some ways other people have done it using different languages, but, I cant pick it up.
I don't want to post everything ive tried so this post doesn't bore anyone. Searched on google as well but, I couldn't find much on registering an object for multiple jobs.
EDIT
Heres what does work:
$job = Start-Job -Name GetLogFiles { Start-Sleep 10 }
Register-ObjectEvent $job StateChanged -Action {
[System.Windows.MessageBox]::Show('Done')
$eventSubscriber | Unregister-Event
$eventSubscriber.Action | Remove-Job
} | Out-Null
Which is what id like to happened, but to evaluate all jobs, not just one.
This is what a personally use when monitoring running jobs:
$jobs= 1..10 | ForEach-Object -Process {
Start-Job { Start-Sleep $using:_ ; "job {0} done" -f $using:_ } -Name "$_"
}
do{
$i = (Get-Job -State Completed).count
$progress = #{
Activity = 'Jobs completed'
Status = "$i of {0}" -f $jobs.Count
PercentComplete = $i / $jobs.count * 100
}
Write-Progress #progress
Start-Sleep -Milliseconds 10
}
until($i -eq $jobs.Count)
$result = Get-Job | Receive-Job
$jobs | Remove-Job
Of course, under certain scenarios where I know some jobs might fail I change the until(...) condition for something different and the do {...} contains the logic for restarting failing jobs.
Edit 1:
It's worth mentioning that Start-Job is not worth your time if you're interested in multithreading, it has been proven to be slower than a linear loop in many scenarios. You should be looking at the ThreadJob Module
Edit 2:
After some testing, this worked for me:
# Clear the Event queue
Get-EventSubscriber|Unregister-Event
# Clear the Job queue
Get-Job|Remove-Job
1..10 | ForEach-Object -Process {
$job = Start-Job { Sleep -Seconds (1..20|Get-Random) } -Name "$_"
Register-ObjectEvent -InputObject $job -EventName StateChanged -Action {
$eventSubscriber | Unregister-Event
$eventSubscriber.Action | Remove-Job
if(-not (Get-EventSubscriber))
{
[System.Windows.MessageBox]::Show('Done')
}
} | Out-Null
}
At first I didn't even know this was possible so thanks for pointing this out. Great question :)

Struggling to export PowerShell job information to excel

I have a PowerShell script that is running a web request as part of a job. With the script block that is run by the job I am logging the Endpoint Uri and the response time. and then after all the jobs are finished but before I remove the jobs I am trying to export the result into excel.
PS Verion:
Major Minor Build Revision
----- ----- ----- --------
5 1 17763 503
Code:
$Code = {
Param(
[array]$Domain,
[array]$Services,
[array]$TestData,
[string]$Path
)
$Log = #()
## Get Random School ##
$Random = Get-Random -InputObject $TestData -Count 1
## Get Random Service ##
$RandService = Get-Random -InputObject $Services
## Get Random Endpoint ##
$ServiceEndpoints = Get-Content "$Path\Service.Endpoints.json" | Out-String | ConvertFrom-Json
$GatewayEndpoints = $ServiceEndpoints.Services.$RandService.Gateway
$RandomEndpoint = Get-Random -InputObject $GatewayEndpoints
$Headers = #{
"Authorization" = "Bearer" + ' ' + $Random.Token
}
$Uri = 'https://' + $Domain + $RandomEndpoint
Try {
$TimeTaken = Measure-Command -Expression {
$JsonResponse = Invoke-WebRequest -Uri $Uri -Headers $Headers -ContentType 'application/json' -Method Get -UseBasicParsing
}
}
Catch {
}
$ResponseTime = [Math]::Round($TimeTaken.TotalMilliseconds, 1)
$LogItem = New-Object PSObject
$LogItem | Add-Member -type NoteProperty -Name 'Endpoint' -Value $Uri
$LogItem | Add-Member -type NoteProperty -Name 'Time' -Value $ResponseTime
$Log += $LogItem
Write-Host $Log
}
#Remove all jobs
Get-Job | Remove-Job
#Start the jobs. Max 4 jobs running simultaneously.
foreach($Row in $TestData){
While ($(Get-Job -state running).count -ge $MaxThreads){
Start-Sleep -Milliseconds 3
}
Start-Job -Scriptblock $Code -ArgumentList $Domain, $Services, $TestData, $Path
}
#Wait for all jobs to finish.
While ($(Get-Job -State Running).count -gt 0) {
start-sleep 1
}
$Log | Export-XLSX -Path .\Test.Results\Performance\Performance.Test.Log.xlsx -ClearSheet
#Get information from each job.
foreach($Job in Get-Job) {
$Info = Receive-Job -Id ($Job.Id)
}
#Remove all jobs created.
Get-Job | Remove-Job
I cannot seem to get the endpoint uri and the response time out of the script block. When I try to export the $Log, all that happens is it creates an empty excel file.
Write-Host $Log
#{Endpoint=https://domain/customer/v1/years/2019/marks; Time=1233.3}
#{Endpoint=https://domain/customer/v1/years/2019/marks; Time=2131.7}
You've to return $Log in your script block, since $Log lives in another scope. You can return $Log in your $Code script block, and finally, fetch it via Receive-Job.
Change your code to:
$Code = {
...
$Log += $LogItem
Write-Host $Log
$Log # return via pipeline to the caller
}
As Niraj Gajjar's answer suggests you can use Export-Csv cmdlet to create an Excel file:
#Get information from each job.
foreach($Job in Get-Job) {
Receive-Job -Id ($Job.Id) | Export-CSV -Path .\Test.Results\Performance\Performance.Test.Log.xlsx -Append -NoTypeInformation
}
Global variables inside of jobs aren't visible in the main script since job is running in a new session with its own global space.
You can use Export-Csv to open file in excel.
sample code with multiple jobs to CSV format :
$jobs = #() # INITILIZING ARRAY
$jobs += Start-Job { appwiz.cpl } # START JOB 1 AND ADDING TO ARRAY
$jobs += Start-Job { compmgmt.msc } # START JOB 2 AND ADDING TO ARRAY
$jobs += Start-Job { notepad.exe } # START JOB 3 AND ADDING TO ARRAY
foreach ( $job in $jobs) # INTERATION OF JOBS
{
Export-Csv -InputObject $job "C:\result.csv" -Append # SAVING TO FILE
}
Note : There are some internal properties of job which are not converted by CSV because depth is 1 but some basic properties are available.

PowerShell Mutex & Jobs

I'm trying to run a PowerShell file delete operation that deletes files based on specific parameters. So, the code goes like this:
$sb = {
Get-ChildItem -Path $SubFolder | Remove-ChildItem
if ($Error[0] -eq $null) {
$results = New-Object PSObject -Property #{
$Foldername = $Subfolder.Name
$TotalFiles = $SubFolder.(Count)
}
} else {
$errorresult = "Error deleting files from $($Subfolder.Name)"
}
#Error Mutex
$ErrorLogmutex = New-Object System.Threading.Mutex($false, "ErrorLogFileHandler")
$ErrorLogmutex.WaitOne()
Out-File -Append -InputObject $errorresult -FilePath $Errorlog
$ErrorLogmutex.ReleaseMutex()
#Success Mutex
$Successmutex = New-Object System.Threading.Mutex($false, "SuccessFileHandler")
$SuccessLogmutex.WaitOne()
Out-File -Append -InputObject $results -FilePath $successlog
$Successmutex.ReleaseMutex()
}
#Calling scriptblock in multi-thread count of 5
{
foreach ($Subfolder in $Folder) {
while ((Get-Job -State Running).Count -ge 5) {
Start-Sleep -Seconds 5
}
Start-Job -ScriptBlock $sb -ArgumentList $arguments | Out-Null
}
The script runs, able to see the results if I explicitly call out Receive-Job -Name JobID, but the output does not produce any log files as I would've thought it would.

Wait until all threads complete before running next task

I would wrap everything inside foreach($computer in $computers) in a Start-Job to make them run simultaneously. The only problem is, I need to wait for all the jobs to complete before I do the ConvertTo-Json at the bottom.
$sb = "OU=some,OU=ou,DC=some,DC=domain"
$computers = Get-ADComputer -Filter {(Enabled -eq $true)} -SearchBase "$sb" -Properties *
$hasmanufacturer = New-Object System.Collections.Generic.List[System.Object]
foreach($computer in $computers)
{
$drives = try{#(Get-WMIObject -Class Win32_CDROMDrive -Property * -ComputerName $computer.Name -ErrorAction Stop)} catch {$null}
foreach($drive in $drives)
{
if($drive.Manufacturer)
{
$hasmanufacturer.Add($computer)
continue
}
} # inner foreach
}
ConvertTo-Json $hasmanufacturer
Use a Get-Job | Wait-Job before executing the ConvertTo-Json
How about using the array of computer names as a parameter to Invoke-Command. It will run, by default, 32 concurrent remote sessions. The number can be changed with the -Throttle parameter.
$computers = Get-ADComputer -Filter {(Enabled -eq $true)} -SearchBase "OU=Servers,DC=xxx,DC=com" -Properties Name |
Where-Object { $_.Name -match 'LAX_*' } |
ForEach-Object { $_.Name }
$computers
$j = Invoke-Command `
-ComputerName $computers `
-ScriptBlock { Get-WMIObject -Class Win32_CDROMDrive -Property * -ErrorAction Stop } `
-AsJob
while ( (Get-Job -Id $j.Id).Status -eq 'Running') {}
Get-Job -Id $j.Id | Wait-Job
$results = Receive-Job -Id $j.Id
$results

Resources