Im having trouble using deserialized file info object when running multiple threads
Main gets file info and starts a new thread. The thread will load my module file and "use" the file that I got in main
$file = Get-FileInfo -path "test.zip"
$Scriptblock = {
Import-Module ".\path\to\foo\module.psm1"
Foo -File $using:file
}
Start-Job -ScriptBlock $ScriptBlock
In my thread, Im not able to call $file.Basename because my object is deseralized when passing it into my thread. The Deseralized.IO.fileinfo class has less functionality than the original FileInfo class
I want to be able to have the same functionality with $file in my thread as I do in main
Is there a way for me to do this? Perhaps by reserializing my object..?
I think you'd be better served by passing the file into the scriptblock as a parameter and then calling Get-FileInfo inside the job:
$MyPath = "test.zip"
$Scriptblock = {
Import-Module ".\path\to\foo\module.psm1"
Foo -File $using:file
$file = Get-FileInfo -path $Using:MyPath
}
Start-Job -ScriptBlock $ScriptBlock
Alternative:
$MyPath = "test.zip"
$Scriptblock = {
param([String]$Path)
Import-Module ".\path\to\foo\module.psm1"
Foo -File $using:file
$file = Get-FileInfo -path $Path
}
Start-Job -ScriptBlock $ScriptBlock -ArgumentList $MyPath
Related
I am trying to search for a string in multiple text files to trigger an event. The log file is being actively added to by a program. The following script successfully achieves that goal, but it only works for one text file at a time:
$PSDefaultParameterValues = #{"Get-Date:format"="yyyy-MM-dd HH:mm:ss"}
Get-Content -path "C:\Log 0.txt" -Tail 1 -Wait | ForEach-Object { If ($_ -match 'keyword') {
Write-Host "Down : $_" -ForegroundColor Green
Add-Content "C:\log.txt" "$(get-date) down"
Unfortunately it means I have to run 3 instances of this script to search the 3 log files (C:\log 0.txt, C:\log 1.txt and C:'log 2.txt).
What I want to do is run one powershell script to search for that string across all three text files and not three.
I tried using a wildcard in the path ("C:\log*.txt)
I also tried adding a foreach loop:
$PSDefaultParameterValues = #{"Get-Date:format"="yyyy-MM-dd HH:mm:ss"}
$LogGroup = ('C:\log 0.txt', 'C:\Log 1.txt', 'C:\Log 2.txt')
ForEach ($log in $LogGroup) {
Get-Content $log -Tail 1 -Wait | ForEach-Object { If ($_ -match 'keyword') {
Write-Host "Down: $_" -ForegroundColor Green
Add-Content -path "C:\log.txt" "$(get-date) down"
Add-Content -path "C:\log.txt" "$(get-date) down"
}
}
}
This got me no errors but it also didn't work.
I saw others use Get-ChildItem instead of Get-Content but since this worked with one file... shouldn't it work with multiple? I assume it's my lack of scripting ability. Any help would be appreciated. Thanks.
This is how you can apply the same logic you already have for one file but for multiple logs at the same time, the concept is to spawn as many PowerShell instances as log paths there are in the $LogGroup array. Each instance is assigned and will be monitoring 1 log path and when the keyword is matched it will append to the main log file.
The instances are assigned the same RunspacePool, this help us initialize all with a SemaphoreSlim instance which help us ensure thread safety (only 1 thread can write to the main log at a time).
using namespace System.Management.Automation.Runspaces
using namespace System.Threading
# get the log files here
$LogGroup = ('C:\log 0.txt', 'C:\Log 1.txt', 'C:\Log 2.txt')
# this help us write to the main log file in a thread safe manner
$lock = [SemaphoreSlim]::new(1, 1)
# define the logic used for each thread, this is very similar to the
# initial script except for the use of the SemaphoreSlim
$action = {
param($path)
$PSDefaultParameterValues = #{ "Get-Date:format" = "yyyy-MM-dd HH:mm:ss" }
Get-Content $path -Tail 1 -Wait | ForEach-Object {
if($_ -match 'down') {
# can I write to this file?
$lock.Wait()
try {
Write-Host "Down: $_ - $path" -ForegroundColor Green
Add-Content "path\to\mainLog.txt" -Value "$(Get-Date) Down: $_ - $path"
}
finally {
# release the lock so other threads can write to the file
$null = $lock.Release()
}
}
}
}
try {
$iss = [initialsessionstate]::CreateDefault2()
$iss.Variables.Add([SessionStateVariableEntry]::new('lock', $lock, $null))
$rspool = [runspacefactory]::CreateRunspacePool(1, $LogGroup.Count, $iss, $Host)
$rspool.ApartmentState = [ApartmentState]::STA
$rspool.ThreadOptions = [PSThreadOptions]::UseNewThread
$rspool.Open()
$res = foreach($path in $LogGroup) {
$ps = [powershell]::Create($iss).AddScript($action).AddArgument($path)
$ps.RunspacePool = $rspool
#{
Instance = $ps
AsyncResult = $ps.BeginInvoke()
}
}
# block the main thread
do {
$id = [WaitHandle]::WaitAny($res.AsyncResult.AsyncWaitHandle, 200)
}
while($id -eq [WaitHandle]::WaitTimeout)
}
finally {
# clean all the runspaces
$res.Instance.ForEach('Dispose')
$rspool.ForEach('Dispose')
}
I have a rootDir with 3 dirs under it, I just want to compress each dir and listen the result by using multithreaded code, so I tried the following:
Set-Location "C:\test"
$sw = [Diagnostics.Stopwatch]::StartNew()
Get-Job | Remove-Job
$rootDirectory = $PWD
$dirs = $(Get-ChildItem -Path $rootDirectory -Directory).Name
# $dirs = d1, d2, d3
$sb = {
Param($init)
Compress-Archive -Path $init -DestinationPath "$init.zip" -CompressionLevel NoCompression
}
$jobs = #()
$dirs | ForEach-Object {
$jobs += Start-Job -ScriptBlock $sb -ArgumentList $_
}
Wait-Job -Job $jobs | Out-Null
Receive-Job -Job $jobs
I got
The path 'd1' either does not exist or is not a valid file system path.
+ CategoryInfo : InvalidArgument: (d1:String) [Compress-Archive], InvalidOperationException
+ FullyQualifiedErrorId : ArchiveCmdletPathNotFound,Compress-Archive
+ PSComputerName : localhost
if I run the compress command serially, not using the Start-Job, all seems to work.
The jobs don't inherit their working directory directory from your script. Put the Set-Location inside the scriptblock and pass the directory as a parameter:
$rootdir = 'C:\test'
...
$jobs = Get-ChildItem -Path $rootdir -Directory | ForEach-Object {
Start-Job -ScriptBlock {
Param($init, $dir)
Set-Location $dir
Compress-Archive -Path $init -DestinationPath "${init}.zip" -CompressionLevel NoCompression
} -ArgumentList $_, $rootdir
}
...
Better yet, pass the full path as the parameter, not just the name:
$rootdir = 'C:\test'
...
$jobs = Get-ChildItem -Path $rootdir -Directory | ForEach-Object {
Start-Job -ScriptBlock {
Param($init)
Compress-Archive -Path $init -DestinationPath "${init}.zip" -CompressionLevel NoCompression
} -ArgumentList $_.FullName
}
...
I'm trying to run a PowerShell file delete operation that deletes files based on specific parameters. So, the code goes like this:
$sb = {
Get-ChildItem -Path $SubFolder | Remove-ChildItem
if ($Error[0] -eq $null) {
$results = New-Object PSObject -Property #{
$Foldername = $Subfolder.Name
$TotalFiles = $SubFolder.(Count)
}
} else {
$errorresult = "Error deleting files from $($Subfolder.Name)"
}
#Error Mutex
$ErrorLogmutex = New-Object System.Threading.Mutex($false, "ErrorLogFileHandler")
$ErrorLogmutex.WaitOne()
Out-File -Append -InputObject $errorresult -FilePath $Errorlog
$ErrorLogmutex.ReleaseMutex()
#Success Mutex
$Successmutex = New-Object System.Threading.Mutex($false, "SuccessFileHandler")
$SuccessLogmutex.WaitOne()
Out-File -Append -InputObject $results -FilePath $successlog
$Successmutex.ReleaseMutex()
}
#Calling scriptblock in multi-thread count of 5
{
foreach ($Subfolder in $Folder) {
while ((Get-Job -State Running).Count -ge 5) {
Start-Sleep -Seconds 5
}
Start-Job -ScriptBlock $sb -ArgumentList $arguments | Out-Null
}
The script runs, able to see the results if I explicitly call out Receive-Job -Name JobID, but the output does not produce any log files as I would've thought it would.
I have functions in separate files I need to run as jobs in one main file.
I need to be able to pass these functions arguments.
Right now my problem is figuring out how to pass the path of the function files to the jobs in a way that is not completely awful.
I need to have the functions defined at the top of the file for readability (just having a static comment that says "script uses somefunc.ps1" is not adequate)
I also need to refer to the scripts relative path (they will all be in the same folder).
Right now I am using env: to store the path of the scripts, but doing this I need to refer to the script in like 5 places!
This is what I have:
testJobsMain.ps1:
#Store path of functions in env so jobs can find them
$env:func1 = "$PSScriptRoot\func1.ps1"
$env:func2 = "$PSScriptRoot\func2.ps1"
$arrOutput = #()
$Jobs = #()
foreach($i in ('aaa','bbb','ccc') ) {
$Import = {. $env:func1}
$Execute = {func1 -myArg $Using:i}
$Jobs += Start-Job -InitializationScript $Import -ScriptBlock $Execute
}
$JobsOutput = $Jobs | Wait-Job | Receive-Job
$JobsOutput
$Jobs | Remove-Job
#Clean up env
Remove-Item env:\func1
$arrOutput
func1.ps1
function func1( $myArg ) { write-output $myArg }
func2.ps1
function func2( $blah ) { write-output $blah }
You can simply make array of paths, and then pass one of paths/all of them in -ArgumentList param from Start-Job:
#func1.ps1
function add($inp) {
return $inp + 1
}
#func2.ps1
function add($inp) {
return $inp + 2
}
$paths = "$PSScriptRoot\func1.ps1", "$PSScriptRoot\func2.ps1"
$i = 0
ForEach($singlePath in $paths) {
$Execute = {
Param(
[Parameter(Mandatory=$True, Position=1)]
[String]$path
)
Import-Module $path
return add 1
}
Start-Job -Name "Job$i" -ScriptBlock $Execute -ArgumentList $singlePath
$i++
}
for ($i = 0; $i -lt 2; $i++) {
Wait-Job "Job$i"
[int]$result = Receive-Job "Job$i"
}
You can skip all those $i iterators with names, Powershell will name jobs automatically, and easly predictable: Job1, Job2.. So it would make code a lot prettier.
I am new to PowerShell scripting and would like to do the following:
Given a list of config names and servers, return the values for the configs from each server.
Transform them in such a way to group them by config name, and not server.
Currently, I have a script that spawns one job per server and calls a script remotely on the server to return the list of configs for that server.
However, I do not know how to aggregate and transform the output from these jobs so that instead of getting config names by server, I would like to sort them by config name first, then server.
Current output:
Server1:
Config1 = 'abc'
Config2 = 'def'
Server2:
Config1 = 'xyz'
Config2 = '123'
Desired output:
Config1:
Server1 : 'abc'
Server2 : 'xyz'
Config2:
Server1 : 'def'
Server2 : '123'
I don't want to iterate over the config names because that would waste time in connecting to the server for every call. Therefore I'd like to iterate over the servers and do some kind of transformation.
I'm wondering if this is a matter of having each job return some kind of dictionary, then iterate over them after all the threads finish to transform?
Here is the code that calls the jobs:
$all_servers = #('server1', 'server2')
$config_names = #('config1', 'config2')
foreach($servername in $all_servers) {
Start-Job -FilePath C:\scripts\get_config_from_servers.ps1
-ArgumentList $servername,$config_names
}
Get-Job | Wait-Job
Get-Job | Receive-Job | Out-GridView
Here is the job script:
Param($servername,$config_names)
$session = Get-Session -computername $servername
-username $$$$
-pwd ####
try {
$sb = {
param($servername,$config_names)
$output = #{}
foreach ($cfg in $config_names) {
$config_value = Get-Config -configname $cfg
$output.Add("$servername : $cfg", "($config_value)")
}
write-host $output | Out-String
return $output | Out-String
}
$out = Invoke-Command -session $session
-ScriptBlock $sb
-ArgumentList $servername,$config_names
write-host $out
return $out
}
finally {
Remove-PSSession $session
}
Instead of making a hash table and converting to a string you could create some custom object in you job script just like this SO Question
Instead of this:
$output = #{}
foreach ($cfg in $config_names) {
$config_value = Get-Config -configname $cfg
$output.Add("$servername : $cfg", "($config_value)")
}
write-host $output | Out-String
return $output | Out-String
You could try something like this:
$output = New-Object System.Object
Add-Member -MemberType NoteProperty -Name Server -Value $servername -InputObject $output
foreach ($cfg in $config_names) {
$config_value = Get-Config -configname $cfg
Add-Member -MemberType NoteProperty -Name "Config$cfg" -Value $config_value -InputObject $output
}
write-host $output
return $output
I can't test this accurately as i'm not sure what Get-Config is but hopefully it should be enough to get you thinking.