Rename-Item : Object reference not set to an instance of an object - string

I have a script that I've been working on, which reads a specified directory, locates multiple .CSV files, and executes some logic for each of the .CSV files and ultimately renames them .csv.archived . I'm trying to handle this as cleanly as possible, but I am making a mess.
The issue at hand, is that I cannot seem to figure out how to pass the individual file names through to strings for purposes of renaming the existing file. The process loops through fine, and the files ultimately get renamed but I get the following error:
#set the location where the .CSV files will be pulled from
$Filecsv = get-childitem "\\SERVER\Audit Test\" -recurse | where {$_.extension -eq ".csv"} | % {
$_.Name
}
In the code above, my thoughts are that $_.Name is where I (believe I) am pulling the file name of each file. At the end of this next block, the file is renamed with the file name.
#for each file found in the directory
ForEach ($item in $Filecsv) {
#count the times we've looped through
"Iterations : " + $iterations
# get the date and time from the system
$datetime = get-date -f MMddyy-hhmmtt
# rename the file
rename-item -path ("\\SERVER\Audit Test\" + $_.Name ) -newname ($filename + $datetime + ".csv.archived")
$iterations ++
}
I think the process is fubar'd here:
rename-item -path ("\\SERVER\Audit Test\" + $_.Name )
I've gutted the irrelevant code for testing purposes, and would be happy if someone could tell me that I am doing something wrong, and that I am not crazy.
I am not sure that I properly understand the way that the ForEach loop works, TechNet helps: http://social.technet.microsoft.com/Forums/en-US/e8da8249-ea91-4772-ae85-582a4b37425b/powershell-foreachobject-vs-foreach?forum=smallbusinessserver
But doesn't answer my particular question.
Anyone care to shed some light?
Thanks! Here's the full script:
$iterations = 1
#set the location where the .CSV files will be pulled from
$Filecsv = get-childitem "\\SERVER\Audit Test\" -recurse | where {$_.extension -eq ".csv"} | % {
$_.Name
}
#check to see if files exist, if not exit cleanly
#for each file found in the directory
ForEach ($item in $Filecsv) {
#count the times we've looped through
"Iterations : " + $iterations
# get the date and time from the system
$datetime = get-date -f MMddyy-hhmmtt
# rename the file
rename-item -path ("\\SERVER\Audit Test\" + $_.Name ) -newname ($_.Name + $datetime + ".csv.archived")
$iterations ++
}

If it were me I'd work with the files as objects and not just a string for their name. So I'd do something like this:
#set the location where the .CSV files will be pulled from
$Filecsv = get-childitem "\\SERVER\Audit Test\" -recurse | where {$_.extension -eq ".csv"}
#for each file found in the directory
ForEach ($item in $Filecsv) {
#count the times we've looped through
"Iterations : $iterations"
# get the date and time from the system
$datetime = get-date -f MMddyy-hhmmtt
# rename the file
$NewName = $item.fullname -replace ".csv$","$datetime.csv.archived"
$Item.MoveTo($NewName)
$iterations ++
}
That takes each file, sets up a new name by replacing the .csv at the end with the $datetime.csv.archived that you want, and then moves the file to the new name effectively renaming it.
Also, if you want the Name property for each item instead of doing a ForEach{$_.Name} you are probably better off doing Select -Expand Name

This works for me, I changed $_.Name to $item (which is a string at this point so doesn't have the name property) and it works
$iterations = 1
#set the location where the .CSV files will be pulled from
$Filecsv = get-childitem "c:\AuditTest\" -recurse | where {$_.extension -eq ".csv"} | % {
$_.Name
}
#check to see if files exist, if not exit cleanly
#for each file found in the directory
ForEach ($item in $Filecsv)
{
#count the times we've looped through
"Iterations : " + $iterations
# get the date and time from the system
$datetime = get-date -f MMddyy-hhmmtt
# rename the file
rename-item -path ("c:\AuditTest\" + $item ) -newname ("c:\AuditTest\" + $item + $datetime + ".csv.archived")
$iterations ++
}
Depending on exactly what you're doing though I'd change it like this
$Filecsv = get-childitem "c:\AuditTest\" -recurse | where {$_.extension -eq ".csv"}
This will give you the actual file object rather than a string which gives you more options in terms of parsing the names etc.

Related

How to use dash or hyphen to join a string array in Powershell

I wrote code like this:
$count = 0
$path = "C:\Videos\"
$oldvids = Get-ChildItem -Path $path -Include *.* -Recurse
foreach ($oldvid in $oldvids) {
$curpath = $oldvid.DirectoryName
$name = [System.IO.Path]::GetFileNameWithoutExtension($oldvid)
$names = $name.Split(" - ")
$names[0] = ""
$metadata_title = $names -join "-"
$ext = [System.IO.Path]::GetExtension($oldvid)
if ($name.StartsWith("new_") -eq $false)
{
$newvid = $curpath + "/new_" + $name + ".mp4"
if ([System.IO.File]::Exists($newvid) -eq $false)
{
$count++
Write-Output $metadata_title
}
}
}
But this code causes a file name like this:
Chapter 1 - New Video
to become:
Chapter 1---New Video
How can I make sure a single - is actually only one? Do I have to escape it?
The idea is to eliminate first part of the file names, so from:
01 - Chapter 1 - Video 1
to:
Chapter 1 - Video 1
So I wanted to split using " - " and then join everything back without the first element in the split array.
Looking at your example and your explanation of changing metadata with ffmpeg on each file, I guess this is what you need:
$count = 0
$path = 'C:\Videos'
# get a list of old video files (these do not start with 'new_')
$oldvids = Get-ChildItem -Path $path -Filter *.mp4 -File -Recurse |
Where-Object { $_.Name -notmatch '^new_' }
foreach ($oldvid in $oldvids) {
# if the file is called 'C:\Videos\01 - Chapter 1 - Video 1.mp4'
$tempName = $oldvid.Name -replace '^\d+\s*-\s*(.+)', 'new_$1' # --> new_Chapter 1 - Video 1.mp4
# or do
# $tempName = 'new_' + ($oldvid.Name -split '-', 2)[-1].Trim() # --> new_Chapter 1 - Video 1.mp4
# or
# $tempName = $oldvid.Name -replace '^\d+\s*-\s*', 'new_' # --> new_Chapter 1 - Video 1.mp4
# combine the current file path with the temporary name
$outputFile = Join-Path -Path $oldvid.DirectoryName -ChildPath $tempName
#######################################################################
# next do your ffmpeg command to change metadata
# for input you use $oldvid.FullName and for output you use $outputFile
Write-Host "Updated file $($oldvid.Name) as $tempName"
#######################################################################
# when done with ffmpeg, delete the original (or for safety move it to somewhere else)
Write-Host "Deleting file '$($oldvid.Name)'"
$oldvid | Remove-Item -WhatIf
# and rename the updated file by removing the 'new_' part from its name
$newName = ($tempName -replace '^new_').Trim()
Write-Host "Renaming updated file to '$newName'"
$tempName | Rename-Item -NewName $newName
# all done, proceed with the next file
$count++
}
Note: I have added switch -WhatIf to the Remove-Item line. This is a safety measure that will only display what file would be deleted without actually deleting it.
If you are sure the correct file should be deleted, then remove that -WhatIf switch so the original file gets destroyed after maipulating it with ffmpeg.
As per your comment, to send items to the Recycle bin instead of destroying them like Remove-Item does, here's two ways of achieving that:
Method 1: Use COM
function RemoveTo-RecycleBin {
[CmdletBinding()]
param (
[Parameter(Mandatory = $true, ValueFromPipeline = $true, ValueFromPipelineByPropertyName = $true)]
[Alias('FullName')]
[string[]]$Path
)
begin {
$shell = New-Object -ComObject 'Shell.Application'
$Recycler = $Shell.NameSpace(0xa)
}
process {
foreach ($item in $Path) {
[void]$Recycler.MoveHere($item)
}
}
end {
# clean-up the used COM objects
$null = [System.Runtime.Interopservices.Marshal]::ReleaseComObject($Recycler)
$null = [System.Runtime.Interopservices.Marshal]::ReleaseComObject($shell)
$null = [System.GC]::Collect()
$null = [System.GC]::WaitForPendingFinalizers()
}
}
# usage example, remove all files from the D:\Test directory
Get-ChildItem -Path 'D:\Test' -Filter '*.*' -File | RemoveTo-RecycleBin
# usage example, remove all files and subdirectories from the D:\Test directory
Get-ChildItem -Path 'D:\Test' | RemoveTo-RecycleBin
Method 2: Use the Microsoft.VisualBasic assembly
function RemoveTo-RecycleBin {
[CmdletBinding()]
param (
[Parameter(Mandatory = $true, ValueFromPipeline = $true, ValueFromPipelineByPropertyName = $true)]
[Alias('FullName')]
[string[]]$Path,
[switch]$ShowConfirmationDialog
)
begin {
Add-Type -AssemblyName Microsoft.VisualBasic
$showUI = if ($ShowConfirmationDialog) { 'AllDialogs' } else { 'OnlyErrorDialogs' }
}
process {
foreach ($item in $Path) {
Write-Host $item
# detect if this is a file or a directory
if ((Get-Item -Path $item) -is [System.IO.DirectoryInfo]) {
# first parameter: the absolute full path
# second parameter: one of Microsoft.VisualBasic.FileIO.UIOption values: OnlyErrorDialogs or AllDialogs
# third parameter: one of Microsoft.VisualBasic.FileIO.RecycleOption values: DeletePermanently or SendToRecycleBin
[Microsoft.VisualBasic.FileIO.FileSystem]::DeleteDirectory($item, $showUI, 'SendToRecycleBin')
}
else {
# first parameter: the absolute full path and file name
# second parameter: one of Microsoft.VisualBasic.FileIO.UIOption values: OnlyErrorDialogs or AllDialogs
# third parameter: one of Microsoft.VisualBasic.FileIO.RecycleOption values: DeletePermanently or SendToRecycleBin
[Microsoft.VisualBasic.FileIO.FileSystem]::DeleteFile($item,$showUI, 'SendToRecycleBin')
}
}
}
}
# usage example, remove all files from the D:\Test directory
Get-ChildItem -Path 'D:\Test' -Filter '*.*' -File | RemoveTo-RecycleBin
# usage example, remove all files and subdirectories from the D:\Test directory
Get-ChildItem -Path 'D:\Test' | RemoveTo-RecycleBin
Just choose any of the above functions, put it on top of your script and then change line
$oldvid | Remove-Item -WhatIf
into
$oldvid | RemoveTo-RecycleBin

Referring to Files as variables in multiple ForEach loops [PowerShell]

Im am making some progress on my script that automatically updates links in excel files without opening them. I have successfully made the function work on a single file with inputs of file name and text to replace. Now I am trying to scale this so that it does the same actions for all files in the script directory.
Here is how the script goes with comments on steps:
# This part will be responsible from fetching the week number to replace from the script directory name, currently I am testing with pre-determined number
# $wk = Get-Item -Path $PWD | Select-Object -Property BaseName
# $wknn = "$wk".Substring(13,2) -as [int]
$wknn = 41
$wkold = $wknn-1
$wkprev = $wknn-2
$DefaultFiles = Get-ChildItem | Where-Object {$_.Name -like "*.xls*"}
ForEach($File in $DefaultFiles)
{
# Build ZIP file name
$zipFile = $_ -ireplace [regex]::Escape(".xlsb"), ".zip"
# Create temporary folder
$parent = [System.IO.Path]::GetTempPath();
[string] $guid = [System.Guid]::NewGuid();
$tempFolder = Join-Path $parent $guid;
New-Item -ItemType Directory -Path $tempFolder;
# Rename file to ZIP
Rename-Item -Path $_ -NewName $zipFile
# Not using Expand-Archive because it changes the ZIP format
C:\7z\7za.exe x "$zipFile" -o"$tempFolder"
# Replace old text with new text. First replace wk-1 to wk and then wk-2 to wk-1
$fileNames = Get-ChildItem -Path $tempFolder -Recurse -Include *.rels
foreach ($file in $fileNames)
{
(Get-Content -ErrorAction SilentlyContinue $file.FullName) |
Foreach-Object { $_ -replace $wkold, $wknn } |
Foreach-Object { $_ -replace $wkprev, $wkold } |
Set-Content $file.FullName
}
# Changing working folder because 7Zip option -w doesn't work
Set-Location -Path $tempfolder
# Update archive with new files. Not using Compress-Archive because it changes the ZIP format
C:\7z\7za.exe u -r "$zipFile" *.*
# Rename file back to XLSB
Rename-Item -Path $zipFile -NewName $_
#Move the final .xlsb file back to the script root
move-Item -Path $_ -destination $PSScriptRoot
#Set location to script root to start over
Set-Location -Path $PSScriptRoot
}
}
I am running into problems with the forEach loop. I am unsure on how do I refer to the file name within the first loop at the Build Zip File Name step. And how do I refer to the output file when i Want to move it to the script root afterwards. Also I suspect that stacking of forEach loops may be not as simple and require extra steps within the code, but due to me just starting out in C I dont have the experience and could not find a simple answer to my problem.
I would really appreciate some assistance with the syntax in my code :)
I would create a temporary folder outside the main loop and set the working directory to that folder. Then remove the folder and reset the working directory when all is done.
Also, there is no need to rename the finished zip file first and then move it back to its original location, because you can do that with the Move-Item cmdlet at the same time.
Try:
$wknn = 41
$wkold = $wknn - 1
$wkprev = $wknn - 2
$7zipExe = 'C:\7z\7za.exe' # path to 7za.exe
$sourcePath = $PSScriptRoot # path to where the Excel files are
# Create temporary folder
$tempFolder = Join-Path -Path ([System.IO.Path]::GetTempPath()) -ChildPath ((New-Guid).Guid)
$null = New-Item -ItemType Directory -Path $tempFolder -Force
# retrieve a collection of Excel files (FullNames only).
# If you ONLY want .xlsb files, set the Filter to '*.xlsb'
$excelFiles = (Get-ChildItem -Path $sourcePath -Filter '*.xls*' -File).FullName
# Changing working folder because 7Zip option -w doesn't work
Set-Location -Path $tempfolder
foreach($File in $excelFiles) {
# Build ZIP file name
$zipFile = [System.IO.Path]::ChangeExtension($File, '.zip')
# Rename file to ZIP
Rename-Item -Path $File -NewName $zipFile
# Not using Expand-Archive because it changes the ZIP format
& $7zipExe x "$zipFile" -o"$tempFolder" | Out-Null
# Replace old text with new text. First replace wk-1 to wk and then wk-2 to wk-1
Get-ChildItem -Path $tempFolder -Recurse -Filter '*.rels' -File | ForEach-Object {
(Get-Content -Path $_.FullName -Raw) -replace $wkold, $wknn -replace $wkprev, $wkold |
Set-Content $_.FullName
}
# Update archive with new files. Not using Compress-Archive because it changes the ZIP format
& $7zipExe u -r "$zipFile" *.* | Out-Null
# Rename and Move the updated file back to the original location (overwrite)
Move-Item -Path $zipFile -Destination $File -Force
# remove all files from the temporary folder to start fresh
Remove-Item -Path "$tempfolder\*" -Recurse -Force
}
# Set location back to script root
Set-Location -Path $PSScriptRoot
# remove the temporary folder
Remove-Item -Path $tempfolder -Recurse -Force

Powershell script to extract data from multiple text files into an excel spreadsheet

I'm pretty new to PS and been struggling for a few days.
I have multiple text files in a folder with specific data that I would like to extract into an excel spreadsheet.
each files look like this :
Client n° : xxx Client name : xxx
Computer status
pc group 1 :
n°1 OK n°2 Disconnected n°3 Unresponsive
n°4 Unreachable host n°5 Unresponsive
Data read 11/11/20 12:50:07
Version: x.x.x
I would like to have an output file that looks like this :
Client name and n° OK Disconnected Unresponsive Unreachable host version
xxx/xxx 1 1 2 1 x.x.x
For the status columns it's the sum number of pc with that status and not the pc n° that I would like to display.
At the moment I'm working with multiple .bat files that searches for the status and output one file per status
find /c "Disconnected" *.* > disconnected.txt
find /c "Unresponsive" *.* > unresponsive.txt
And then I sort every single output in an excel which takes me too much time, I was wondering if it was possible to automate this task with a script.
I really don't have any knowledge of PS, only basic batch commands.
Let's assume your files are all in one folder and all of them have the .txt extension.
Then you need to loop through these files and parse the data you need from it:
# create a Hashtable to add the different status values in
$status = #{'OK' = 0; 'Disconnected'= 0; 'Unresponsive' = 0; 'Unreachable host'= 0}
# loop through the files in your path and parse the information out
$result = Get-ChildItem -Path 'D:\Test' -Filter '*.txt' -File | ForEach-Object {
switch -Regex -File $_.FullName {
'^Client n°\s*:\s*([^\s]+)\s+Client name\s*:\s*(.+)$' {
# start collecting data for this client
$client = '{0}/{1}' -f $matches[2], $matches[1]
# reset the Hashtable to keep track of the status values
$status = #{'OK' = 0; 'Disconnected'= 0; 'Unresponsive' = 0; 'Unreachable host'= 0 }
}
'^\d+' {
# increment the various statuses in the Hahstable
($_ -split '\d+').Trim() | ForEach-Object { $status[$_]++ }
}
'^Version:\s(.+)$' {
$version = $matches[1]
# since this is the last line for this client, output the collected data as object
[PsCustomObject]#{
'Client name and n°' = $client
'OK' = $status['OK']
'Disconnected' = $status['Disconnected']
'Unresponsive' = $status['Unresponsive']
'Unreachable host' = $status['Unreachable host']
'Version' = $version
}
}
}
}
# output on screen
$result | Format-Table -AutoSize
# output to CSV file
$result | Export-Csv -Path 'D:\Test\clientdata.csv' -UseCulture -NoTypeInformation
Result on screen:
Client name and n° OK Disconnected Unresponsive Unreachable host Version
------------------ -- ------------ ------------ ---------------- -------
xxx/xxx 1 1 2 1 x.x.x
I used this as an exercise to test my abilities. I created three of the same files, with different data, and tested this script. As long as they are text files in the directory the script will iterate through each file and pull the data from each as you stated it needs to be. If a stray text file gets added the script does not know nor care and will treat it like the others. If there is data it can find it will, and it will output that data to the excel file. Lastly the file is set to save itself and then immediately close.
It starts by Creating the Excel file, then Workbook. (I commented out the naming of the workbook. If you like you can add it back.) Finds all text files in a directory, then searches the text for the specific content within the text you specified above.
During the script I commented as much as I thought might be needed to assist with modification later on.
Output formatted like this:
Excel Output
#Create An Excel File
$excel = New-Object -ComObject excel.application
$excel.visible = $True
#Add Workbook
$workbook = $excel.Workbooks.Add()
<#Rename Workbook
$workbook= $workbook.Worksheets.Item(1)
$workbook.Name = 'Client name and #'#>
#create the column headers
$workbook.Cells.Item(1,1) = 'Client name and n°'
$workbook.Cells.Item(1,2) = 'OK'
$workbook.Cells.Item(1,3) = 'Disconnected'
$workbook.Cells.Item(1,4) = 'Unresponsive'
$workbook.Cells.Item(1,5) = 'Unreachable'
$workbook.Cells.Item(1,6) = 'Version'
$workbook.Cells.Item(1,7) = 'Date Gathered'
$move = "C:\Users\iNet\Desktop\Testing"
$root = "C:\Users\iNet\Desktop\Testing"
$files = Get-ChildItem -Path $root -Filter *.txt
#Starting on Row 2
[int]$i = 2
ForEach ($file in $files){
$location = $root+"\"+$file
#Format your client data to output what you want to see.
$ClientData = select-string -path "$location" -pattern "Client"
$ClientData = $ClientData.line
$ClientData = $ClientData -replace "Client n° :" -replace ""
$ClientData = $ClientData -replace "Client name :" -replace "|"
$row = $i
$Column = 1
$workbook.Cells.Item($row,$column)= "$ClientData"
#Data Read Date
$DataReadDate = select-string -path "$location" -pattern "Data read"
$DataReadDate = $DataReadDate.line
$DataReadDate = $DataReadDate -replace "Data read " -replace ""
#Data Read Date, you asked for everything but this.
$row = $i
$Column = 7
$workbook.Cells.Item($row,$column)= "$DataReadDate"
#Version
$Version = select-string -path "$location" -pattern "Version:"
$Version = $Version.line
$Version = $Version -replace "Version: " -replace ""
$row = $i
$Column = 6
$workbook.Cells.Item($row,$column)= "$Version"
#How Many Times Unresponsive Shows Up
$Unresponsive = (Get-Content "$location" | select-string -pattern "Unresponsive").length
$row = $i
$Column = 4
$workbook.Cells.Item($row,$column)= "$Unresponsive"
#How Many Times Disconnected Shows Up
$Disconnected = (Get-Content "$location" | select-string -pattern "Disconnected").length
$row = $i
$Column = 3
$workbook.Cells.Item($row,$column)= "$Disconnected"
#How Many Times Unreachable host Shows Up
$Unreachable = (Get-Content "$location" | select-string -pattern "Unreachable host").length
$row = $i
$Column = 5
$workbook.Cells.Item($row,$column)= "$Unreachable"
#How Many Times OK Shows Up
$OK = (Get-Content "$location" | select-string -pattern "OK").length
$row = $i
$Column = 2
$workbook.Cells.Item($row,$column)= "$OK"
#Iterate by one so each text file goes to its own line.
$i++
}
#Save Document
$output = "\Output.xlsx"
$FinalOutput = $move+$output
#saving & closing the file
$workbook.SaveAs($move)
$excel.Quit()

PowerShell does not replace string although you can see it in cmd

I normally find the answer to my problem by going through the site, but this time I have read every question yet still I am in despair and really need an experienced eye.
What I have is basically a structural health monitoring system. I measure strains and receive raw data. This raw data is processed by a MATLAB executable that I wrote myself and then uploaded to an ftp-server. We had a student that automated this with a PowerShell script which was working perfectly until I changed literally one small line in MATLAB and recompiled the code.
I do not understand much about PowerShell, so please be patient with me. The error I receive is you cannot call a method on a null-valued expression. This occurs when I try to replace a set of strings (just called xxx_xxx) with a date that exists as a variable in PowerShell. I can see xxx_xxx in the command window (see attached image), I can print out the date that I want to use as replacement, but somehow it does not work.
I cannot provide a working code snippet because you would need the DAQ to generate data, and as I said, I don't understand the language much. But below is the code. For easier reading, the line that I am receiving the error is the following:
$outData = $cmdOutput.Replace("xxx_xxx",$snaps[$i].Substring(6,4)+"-"+$snaps[$i].Substring(3,2)+"-"+$snaps[$i].Substring(0,2)+" "+$snaps[$i].Substring(11,8)+";")
If anyone could help me with this, I would be eternally grateful!
$retry=3
while(1){
#$dir = "C:\Users\Petar\Documents\Zoo\PetarData\INPUT DATA\New folder\"
$dir = "C:\Users\Yunus\Documents\Micron Optics\ENLIGHT\Data\" + $(get-date -f yyyy) + "\" + $(get-date -f MM) + "\"
#$outdir = "C:\Users\Petar\Documents\Zoo\PetarData\OUTPUT DATA\New folder\"
$archivedirin = "C:\Users\Yunus\Documents\Elefantenhaus\Archive\IN\"
$archivedirout = "C:\Users\Yunus\Documents\Elefantenhaus\Archive\OUT\"
$tempdir = "C:\Users\Yunus\Documents\Elefantenhaus\Archive\TEMP\"
$prefix = "EHZZ";
$filecount=(Get-ChildItem $dir).Count
$latest = Get-ChildItem -Path $dir | Sort-Object LastAccessTime -Descending | Select-Object -First 1
if($filecount -gt 1){
$exclude = $latest.name
$Files = GCI -path $dir | Where-object {$_.name -ne $exclude}
$dest = $archivedirin + "batch_"+$(get-date -f MM-dd-yyyy_HH_mm_ss)+"\"
new-item -type directory $dest
foreach ($file in $Files){move-item -path ($dir+$file) -destination $dest}
$latest = Get-ChildItem -Path $dest | Sort-Object LastAccessTime -Descending | Select-Object -First 1
$filename = $dest + $latest.name
$s=Get-Content $filename
while($s -eq $null){
if($retry -lt 0){break}
write-host "could not read file"
$retry = $retry -1
$s=Get-Content $filename
}
#read content of input file
$snaps = $s
#loop through the lines in the file until the first occurence of a timestamp, that is our desired line
for ($i = 0; $i -lt $snaps.length; $i++)
{
$ismatch =[regex]::Matches($snaps[$i], '^(\d\d.\d\d.\d\d\d\d\s\d\d+)')
if ( $ismatch -ne $null -and $ismatch[0].Groups[1].Value)
{
$temp=Get-Content $filename | select -skip $i
$filenametemp = $tempdir+"\temp.txt" #temp file path, don't change the filename "temp.txt"
#$filename3 = $tempdir+"\test.txt"
Add-Content $filenametemp $temp
$filename = $archivedirout+$prefix+"_"+$snaps[$i].Substring(8,2)+$snaps[$i].Substring(3,2)+$snaps[$i].Substring(0,2)+"_"+$snaps[$i].Substring(11,2)+$snaps[$i].Substring(14,2)+$snaps[$i].Substring(17,2)+".txt"
$cmdOutput = (cmd /c new_modified.exe $tempdir) | Out-String
write-output $cmdOutput #"$cmdOutput is:"
#IF ([string]::IsNullOrWhitespace($cmdOutput)){
# break
#}
$outData = $cmdOutput.Replace("xxx_xxx",$snaps[$i].Substring(6,4)+"-"+$snaps[$i].Substring(3,2)+"-"+$snaps[$i].Substring(0,2)+" "+$snaps[$i].Substring(11,8)+";")
Add-Content $filename $outData
remove-item -path $filenametemp
break
}
}
#break
}
else
{
write-host "waiting for file"
}
Start-Sleep -s 30
}
I think what is happening is that the output of the external program isn't being piped into a variable correctly. I haven't had a chance to test this but Tee-Object looks like the appropriate method for you.
I would suggest you try replacing...
$cmdOutput = (cmd /c new_modified.exe $tempdir) | Out-String
with...
cmd /c new_modified.exe $tempdir | Tee-Object -variable $cmdOutput

Powershell - Optimizing a very, very large csv and text file search and replace

I have a directory with ~ 3000 text files in it, and I'm doing periodic search and replaces on those text files as I transition a program to a new server.
Each text file may have an average of ~3000 lines, and I need to search the files for maybe 300 - 1000 terms at a time.
I'm replacing the server prefix which is related to the string I'm searching for. So for every one of the csv entries, I'm looking for Search_String, \\Old_Server\"Search_String" and making sure that after the program completes, the result is "\\New_Server\Search_String".
I cobbled together a powershell program, and it works. But it's so slow I've never seen it complete.
Any suggestions for making it faster?
EDIT 1:
I changed get-content as suggested, but it still took 3 minutes to search two files (~8000 lines) for 9 separate search terms. I must still be screwing up; a notepad++ search and replace would still be way faster if done manually 9 times.
I'm not sure how to get rid of the first (Get-Content) because I want to make a copy of the file for backup before I make any changes to it.
EDIT 2:
So this is an order of magnitude faster; it's searching a file in maybe 10 seconds. But now it doesn't write changes to files, and it only searches the first file in the directory! I didn't change that code, so I don't know why it broke.
EDIT 3:
Success! I adapted a solution posted below to make it much, much faster. It's searching each file in a couple of seconds now. I may reverse the loop order, so that it loads the file into the array and then searches and replaces each entry in the CSV rather than the other way around. I'll post that if I get it to work.
Final script is below for reference.
#get input from the user
$old = Read-Host 'Enter the old cimplicity qualifier (F24, IRF3 etc'
$new = Read-Host 'Enter the new cimplicity qualifier (CB3, F24_2 etc)'
$DirName = Get-Date -format "yyyy_MM_dd_hh_mm"
New-Item -ItemType directory -Path $DirName -force
New-Item "$DirName\log.txt" -ItemType file -force -Value "`nMatched CTX files on $dirname`n"
$logfile = "$DirName\log.txt"
$VerbosePreference = "SilentlyContinue"
$points = import-csv SearchAndReplace.csv -header find #Import CSV File
#$ctxfiles = Get-ChildItem . -include *.ctx | select -expand fullname #Import local directory of CTX Files
$points | foreach-object { #For each row of points in the CSV file
$findvar = $_.find #Store column 1 as string to search for
$OldQualifiedPoint = "\\\\"+$old+"\\" + $findvar #Use escape slashes to escape each invidual bs so it's not read as regex
$NewQualifiedPoint = "\\"+$new+"\" + $findvar #escape slashes are NOT required on the new string
$DuplicateNew = "\\\\" + $new + "\\" + "\\\\" + $new + "\\"
$QualifiedNew = "\\" + $new + "\"
dir . *.ctx | #Grab all CTX Files
select -expand fullname | #grab all of those file names and...
foreach {#iterate through each file
$DateTime = Get-Date -Format "hh:mm:ss"
$FileName = $_
Write-Host "$DateTime - $FindVar - Checking $FileName"
$FileCopied = 0
#Check file contents, and copy matching files to newly created directory
If (Select-String -Path $_ -Pattern $findvar -Quiet ) {
If (!($FileCopied)) {
Copy $FileName -Destination $DirName
$FileCopied = 1
Add-Content $logfile "`n$DateTime - Found $Findvar in $filename"
Write-Host "$DateTime - Found $Findvar in $filename"
}
$FileContent = Get-Content $Filename -ReadCount 0
$FileContent =
$FileContent -replace $OldQualifiedPoint,$NewQualifiedPoint -replace $findvar,$NewQualifiedPoint -replace $DuplicateNew,$QualifiedNew
$FileContent | Set-Content $FileName
}
}
$File.Dispose()
}
If I'm reading this correctly, you should be able to read a 3000 line file into memory, and do those replaces as an array operation, eliminating the need to iterate through each line. You can also chain those replace operations into a single command.
dir . *.ctx | #Grab all CTX Files
select -expand fullname | #grab all of those file names and...
foreach {#iterate through each file
$DateTime = Get-Date -Format "hh:mm:ss"
$FileName = $_
Write-Host "$DateTime - $FindVar - Checking $FileName"
#Check file contents, and copy matching files to newly created directory
If (Select-String -Path $_ -Pattern $findvar -Quiet ) {
Copy $FileName -Destination $DirName
Add-Content $logfile "`n$DateTime - Found $Findvar in $filename"
Write-Host "$DateTime - Found $Findvar in $filename"
$FileContent = Get-Content $Filename -ReadCount 0
$FileContent =
$FileContent -replace $OldQualifiedPoint,$NewQualifiedPoint -replace $findvar,$NewQualifiedPoint -replace $DuplicateNew,$QualifiedNew
$FileContent | Set-Content $FileName
}
}
On another note, Select-String will take the filepath as an argument, so you don't have to do a Get-Content and then pipe that to Select-String.
Yes, you can make it much faster by not using Get-Content... Use Stream Reader instead.
$file = New-Object System.IO.StreamReader -Arg "test.txt"
while (($line = $file.ReadLine()) -ne $null) {
# $line has your line
}
$file.dispose()
i wanted to use PowerShell for this and created a script like the one below:
$filepath = "input.csv"
$newfilepath = "input_fixed.csv"
filter num2x { $_ -replace "aaa","bbb" }
measure-command {
Get-Content -ReadCount 1000 $filepath | num2x | add-content $newfilepath
}
It took 19 minutes on my laptop to process 6.5Gb file. The code below is reading file in a batch (using ReadCount) and uses filter that should optimize performance.
But then I tried FART and it did the same thing in 3 minutes! quite a difference!

Resources