Importing large csv file into Excel using PowerShell - excel

I'm writing a script which imports a large csv file in Excel document.
I try to use a faster way to enter the data and pass the array directly to Excel without looping it.
$p = Import-Csv -Path "C:\Report.csv" -Delimiter "`t"
$Excel01 = New-Object -ComObject Excel.Application
$Excel01.Visible = $True
$Workbook01 = $Excel01.Workbooks.Add()
$Worksheet01 = $Workbook01.Sheets.Item(1)
$Worksheet01.Activate()
$Worksheet01.Range("A1:D1").EntireColumn.Value() = $p | select field1,field2...
But when I run this it hungs...How can I do that?

OpenText() already exists in Excel. Note, however, that you MUST change the extension of the text file to something other than .csv, because Excel has its own mind about how files with that particular extension should be handled.
New-Variable -Option Constant -Name xlDelimited -Value 1
New-Variable -Option Constant -Name xlTextQualifierNone -Value -4142
New-Variable -Option Constant -Name xlWorkbookDefault -Value 51
$csv = 'C:\path\to\your.csv'
$txt = $csv -replace '\.csv$','.txt'
$xls = $csv -replace '\.csv$','.xlsx'
Rename-Item $csv $txt
$xl = New-Object -COM 'Excel.Application'
$xl.Workbooks.OpenText($txt, [Type]::Missing, [Type]::Missing, $xlDelimited, $xlTextQualifierNone, $false, $true)
$wb = $xl.Workbooks | ? { $_.FullName -eq $txt }
$wb.SaveAs($xls, $xlWorkbookDefault)
$wb.Close()
$xl.Quit()
The [Type]::Missing values are required for parameters that should retain their default value.

Quick and dirty. Maybe you can optimize it :-)
$p = Import-Csv -Path "C:\Report.csv" -Delimiter "`t"
$Excel01 = New-Object -ComObject Excel.Application
$Excel01.Visible = $True
$Workbook01 = $Excel01.Workbooks.Add()
$Worksheet01 = $Workbook01.Sheets.Item(1)
$Worksheet01.Activate()
#Add csv header to excel
For ($i = 0; $i -lt ($p | Get-Member | Where-Object -FilterScript {$_.MemberType -eq "NoteProperty"}).Count; $i ++) {
$Worksheet01.Cells.Item(1,(1+$i)) = "$(($p | Get-Member | Where-Object -FilterScript {$_.MemberType -eq "NoteProperty"})[$i].Name)"
}
#Add csv data to ecxel
$startRow = 2
For ($i = 0; $i -lt ($p | Measure-Object).Count; $i ++) {
For ($i2 = 0; $i2 -lt ($p[$i] | Get-Member | Where-Object -FilterScript {$_.MemberType -eq "NoteProperty"}).Count; $i2 ++) {
$PropertyName = ($p[$i2] | Get-Member | Where-Object -FilterScript {$_.MemberType -eq "NoteProperty"})[$i2].Name
$Worksheet01.Cells.Item($startRow,(1+$i2)) = "$($p[$i].$PropertyName)"
}
$startRow ++
}

Related

Clipboard access denied, csv to xlsx conversion

I have a most peculiar issue with clipboard. Below is the code I've written that in essence gathers info about many many thousands of files, compares hashes, compares filnames and lists zero length files and then writes them all to an xlsx file in separate worksheets.
Everything works fine if the scope is relatively small (i.e. ~20k files), but if the scope becomes greater (i.e. ~200k files) I get an error Clipboard access denied. Initially I believed that the issue was clipboard capacity, as I use clipboard and .pastespecial method. But when intermediate csv files are created and their contents copied, everything seems to work fine. Ant thoughts?
Original script
$excel = New-Object -com excel.application
$excel.SheetsInNewWorkbook = 2
$excel.displayalerts = $false
$workbook = $excel.workbooks.add()
$worksheet = $workbook.worksheets
$True_Dups = $worksheet.item(1)
$True_Dups.name = "True Duplicates"
$Dupes = $worksheet.item(2)
$Dupes.name = "Name Duplicates"
$wbPersonalXLSB = $excel.workbooks.open("$env:USERPROFILE\Application Data\Microsoft\Excel\XLSTART\PERSONAL_2.XLSB")
$path = "PATH"
$GCI = GCI $path -file -Recurse -Ea 0
$hashes =
$GCI|
Get-FileHash -Algorithm MD5 -Ea 0|
select Algorithm, Hash, #{l="File";e={$_.Path.split("\")|select -Last 1}},#{l="Path";e={$_.Path.Substring(0,$_.Path.LastIndexof('\'))}}, #{l="Link";e={$_.Path}}|
Group -property "Hash"|
Where {$_.Count -ge 2}|
select -Expand Group
$hashes|ConvertTo-Csv -NoTypeInformation -Delimiter "`t"|scb
$True_Dups.Cells.Item(1).pastespecial()|out-null
$True_Dups.activate()
$excel.run("PERSONAL_2.XLSB!Empty_Row_Dupes")
$filenames =
$GCI|
Select #{l='File';e={$_.PSChildName}}, #{l='Compare Filename';e={$_.BaseName.replace('_','*').replace(' ','*').replace('-','*')}}, Directory, FullName, #{l="Extension";e={$_.Extension}}|
group -Property 'Compare Filename'|
Where {#($_.Group.Extension |Sort -Unique).Count -ge 2}|
select -expand Group
$filenames|ConvertTo-Csv -NoTypeInformation -Delimiter "`t"|scb
$Dupes.Cells.Item(1).pastespecial()|out-null
$Dupes.Activate()
$excel.run("PERSONAL_2.XLSB!Empty_Row_Dupes")
$wbPersonalXLSB.Close()
$Empty = $worksheet.add([System.Reflection.Missing]::Value,$worksheet.Item($worksheet.count))
$Empty.Name = "Zero Lenght"
$zero_length =
$GCI|
? {$_.Length -eq 0}|
Select #{l='File';e={$_.PSChildName}}, Length, Directory, FullName
$zero_length|ConvertTo-Csv -NoTypeInformation -Delimiter "`t"|scb
$zero_length.cells.item(1).pastespecial()|out-null
$zero_length.range("A1:D1").Interior.Color = 8454080
$save = "CSV_PATH"
$workbook.saveas($save)
$workbook.close()
$excel.quit()
[System.Runtime.Interopservices.Marshal]::ReleaseComObject($excel)|out-null
Remove-Variable excel
Interdemiate CSVs added
$start = Get-Date
$path = "Path"
$GCI = GCI $path -file -Recurse -Ea 0
$hashes =
$GCI|
Get-FileHash -Algorithm MD5 -Ea 0|
select Algorithm, Hash, #{l="File";e={$_.Path.split("\")|select -Last 1}},#{l="Path";e={$_.Path.Substring(0,$_.Path.LastIndexof('\'))}}, #{l="Link";e={$_.Path}}|
Group -property "Hash"|
Where {$_.Count -ge 2}|
select -Expand Group
$hashes|export-Csv "PATH_1 csv" -NoTypeInformation
$filenames =
$GCI|
Select #{l='File';e={$_.PSChildName}}, #{l='Compare Filename';e={$_.BaseName.replace('_','*').replace(' ','*').replace('-','*')}}, Directory, FullName, #{l="Extension";e={$_.Extension}}|
group -Property 'Compare Filename'|
Where {#($_.Group.Extension |Sort -Unique).Count -ge 2}|
select -expand Group
$filenames|export-Csv "PATH_2.csv" -NoTypeInformation
$zero_length =
$GCI|
? {$_.Length -eq 0}|
Select #{l='File';e={$_.PSChildName}}, Length, Directory, FullName
$zero_length|export-Csv "PATH_3.csv" -NoTypeInformation
$span = ((get-date) - $start).ToString("hh\:mm\:ss")
Write "Span lasted $span"
and conversion script (credit goes to the original creator linked at the end)
$path = "CSV_FOLDER"
$csvs = Get-ChildItem $path -filter *.csv
$y = $csvs.Count
Write-Host "Detected the following CSV files: ($y)"
Write-Host " "$csvs.Name"`n"
$outputfilename = "Final Registry Results"
Write-Host Creating: $outputfilename
$excelapp = New-Object -ComObject Excel.Application
$excelapp.SheetsInNewWorkbook = $csvs.Count
$xlsx = $excelapp.Workbooks.Add()
for($i=1;$i -le $y;$i++) {
$worksheet = $xlsx.Worksheets.Item($i)
$worksheet.Name = $csvs[$i-1].Name
$file = (Import-Csv $csvs[$i-1].FullName)
$file | ConvertTo-Csv -Delimiter "`t" -NoTypeInformation | Clip
$worksheet.Cells.Item(1).PasteSpecial()|out-null
}
$output = "XLSX_OUTPUT"
$xlsx.SaveAs($output)
$excelapp.Quit()
Merging CSV Files into a XLSX with Tabs

Powershell, close Excel

I have the following code and it works perfectly except it's not closing Excel properly. It's leaving an Excel process running.
Is there a way to close Excel properly without killing the process?
Since i'm using other Excel files while running this script i can not kill all active Excel processes.
I think i tried everything i found online.
$WorkDir = "D:\Test\QR_ES\RG_Temp"
$BGDir = "D:\Test\QR_ES\3_BG"
$File = "D:\Test\QR_ES\4_Adr_Excel\KD_eMail.xlsx"
$SentDir = "D:\Test\QR_ES\RG_Temp\Sent\Dunning"
chdir $WorkDir
$firstPageList = Get-ChildItem "$WorkDir\1*.pdf" -File -Name
ForEach ($firstPage in $firstPageList)
{
$secondPage = "$BGDir\BG_RG.pdf"
$output = "Dunn-$firstPage"
invoke-command {pdftk $firstPage background $secondPage output $output}}
del 1*.pdf
gci $WorkDir\Dunn-*.pdf | rename-item -newname {$_.Name.Substring(5)} -Force
$Excel = New-Object -ComObject Excel.Application
$Excel.visible = $false
$Workbook = $Excel.workbooks.open($file)
$DunnList = Get-ChildItem "$WorkDir\1*.pdf" -File -Name
ForEach ($Dunn in $DunnList)
{
$Worksheets = $Workbooks.worksheets
$Worksheet = $Workbook.Worksheets.Item("KD_eMail")
$Range = $Worksheet.Range("A1").EntireColumn
$DunnSearch = $Dunn.Substring(0,5)
$SearchString = $DunnSearch
$Search = $Range.find($SearchString)
$Recipient = $Worksheet.Cells.Item($Search.Row, $Search.Column + 1)
$Msg = "<span style='font-family:Calibri;font-size:12pt;'>Test</span>"
$Outlook = New-Object -ComObject Outlook.Application
$namespace = $Outlook.GetNameSpace("MAPI")
$namespace.Logon($null, $null, $false, $true)
$EmailFrom = ('test#test.com')
$account = $outlook.Session.Accounts.Item($EmailFrom)
$Mail = $Outlook.CreateItem(0)
$Mail.HTMLBody = $Msg
$Mail.Subject = "OP - $SearchString"
$Mail.To = $Recipient
function Invoke-SetProperty {
param(
[__ComObject] $Object,
[String] $Property,
$Value
)
[Void] $Object.GetType().InvokeMember($Property,"SetProperty",$NULL,$Object,$Value)
}
Invoke-SetProperty -Object $mail -Property "SendUsingAccount" -Value $account
$Mail.Attachments.Add("$WorkDir\$Dunn")
$Mail.Save()
$Mail.close(1)
$Mail.Send()}}
$workbook.close($false)
$Excel.Quit()
chdir $WorkDir
del 1*.pdf
See this post:
https://stackoverflow.com/a/35955339/5329137
which is not accepted as an answer, but I believe is the full, correct way to close Excel.
This is what did it for me:
$FilePID = (Get-Process -name Excel | Where-Object { $_.MainWindowTitle -like 'FileName.xlsx*' }).Id
$Workbook.Save()
$Workbook.close($false)
Stop-Process $FilePID
Elaborating on #ASD's answer, since the MainWindowTitle doesn't (always) include the file suffix (.xlsx) you may have to strip that when comparing it to the filename. I'm using -replace to use a Regex match of everything before the last dot.
$excelPID = (Get-Process -name Excel | Where-Object { $_.MainWindowTitle -eq $fileName -replace '\.[^.]*$', '' }).Id
$workbook.Close()
Stop-Process $excelPID

PowerShell - Open Excel and Unlock Certain Cells Based On Value

I have a report that I am running via SSRS to brings down some data that I need people to give comments on. I am attempting to write a PowerShell script that will open the file and unprotect certain cells based on their value.
This is what the report looks like:
The default for SSRS is that all cells are automatically protected. However, I would like to unprotect all the cells in column A (the column with the yellow text boxes) that do not have the value of "Manager Comments".
I am not really sure how to go about this. I have this so far for my script:
Param(
[string] $FolderPath,
[string] $FileName
)
Write-Output $FolderPath
Write-Output $FileName
$Files = Dir $FolderPath -Recurse | ? {$_.Name -eq $FileName} | Select -ExpandProperty FullName
$excl=New-Object -ComObject "Excel.Application"
foreach ($file in $Files)
{
$wrkb=$excl.Workbooks.Open($file, 0, $false)
$wrkb.Worksheets("Comments").Range("A1:A100").Locked = $False
$wrkb.Save()
$wrkb.Close()
}
$excl.Quit()
But this will just unprotect all of column A. Does anyone know how to have it look at the values and only unprotect those blank values? Bonus points if it can just look for certain colors (the yellow).
And yes, I know that this is a terrible way of doing this...but I work with what I have.
A little progress update...I now can pull the value and cell color index in a loop, but I am not sure how to write the IF statement to lock the cell if the $Color value is equal to "36".
Param(
[string] $FolderPath = "C:\Root\Data\VisualCron\Development\",
[string] $FileName = "Combined Master Report.xlsx"
)
Write-Output $FolderPath
Write-Output $FileName
$Files = Dir $FolderPath -Recurse | ? {$_.Name -eq $FileName} | Select -ExpandProperty FullName
$SheetName = "Comments"
$Excel = New-Object -ComObject "Excel.Application"
foreach ($File in $Files)
{
$Workbook = $Excel.Workbooks.Open($File, 0, $False)
$Sheet = $Workbook.Worksheets.Item("Comments")
$MaxRow = ($Sheet.UsedRange.Rows).Count
$Row,$Column = 1,1
for ($i = 0; $i -le $MaxRow - 1; $i++)
{
$Color = $Sheet.Cells.Item($Row+$i,$Column).Interior.ColorIndex
$Value = $Sheet.Cells.Item($Row+$i,$Column).Text
Write-Host ("Color Value:" + $Color + " " + $i + " " + $Value)
}
#$Workbook.Worksheets("Comments").Range("A3:A50").Locked = $False
#$Workbook.Worksheets("Comments").Protect('Test',1,1,1,1,0,0,0,0,0,0,0,0,0,0,0)
$Workbook.Save()
$Workbook.Close()
}
$excl.Quit()
Another update:
I have gotten it to work, but it is a little strange. When I first go through my loop, the color index appears to be "36", but once I change one value, the yellow index changes to "5". I am not really sure why it does that, but I wrote this script, and hopefully, it is consistent.
Param(
[string] $FolderPath = "C:\Root\Data\VisualCron\Development\",
[string] $FileName = "Combined Master Report.xlsx"
)
Write-Output $FolderPath
Write-Output $FileName
$Files = Dir $FolderPath -Recurse | ? {$_.Name -eq $FileName} | Select -ExpandProperty FullName
$SheetName = "Comments"
$Excel = New-Object -ComObject "Excel.Application"
foreach ($File in $Files)
{
$Workbook = $Excel.Workbooks.Open($File, 0, $False)
$Sheet = $Workbook.Worksheets.Item("Comments")
$MaxRow = ($Sheet.UsedRange.Rows).Count
$Row,$Column = 1,1
for ($i = 1; $i -le $MaxRow - 1; $i++)
{
$Color = $Sheet.Cells.Item($Row+$i,$Column).Interior.ColorIndex
$Value = $Sheet.Cells.Item($Row+$i,$Column).Text
#Write-Output ($Color)
$Range = "A" + ($Row + $i) + ":A" + ($Row + $i)
if ($Color -eq 36 -or $Color -eq 5)
{
$Range = "A" + ($Row + $i) + ":A" + ($Row + $i)
$Workbook.Worksheets("Comments").Range($Range).Locked = $False
#Write-Host $Range
}
}
$Workbook.Worksheets("Comments").Protect('Test',1,1,1,1,0,0,0,0,0,0,0,0,0,0,0)
$Workbook.Save()
$Workbook.Close()
}
$Excel.Quit()
Does anyone have any ideas on why the color index would be changing?

Passing CSV to Excel Workbook (Not From File)

I have a folder of CSV files that contain log entries. For each entry of the CSV, if the Risk property is not Low and not None then I put it in an accumulation CSV object. From there, I want to import it into an Excel Workbook directly WITHOUT having to save the CSV to file.
$CSVPaths = (Split-Path $PSCommandPath)
$AccumulateExportPath = (Split-Path $PSCommandPath)
$FileName="Accumulate"
$Acc=#()
Foreach ($csv in (Get-ChildItem C:\Scripts\Nessus\Sheets |? {$_.Extension -like ".csv" -and $_.BaseName -notlike "$FileName"}))
{
$Content = Import-CSV $csv.FullName
Foreach ($Log in $Content)
{
If ($Log.Risk -ne "None" -and $Log.Risk -ne "Low")
{
$Acc+=$Log
}
}
}
$CSV = $ACC |ConvertTo-CSV -NoTypeInformation
Add-Type -AssemblyName Microsoft.Office.Interop.Excel
$Script:Excel = New-Object -ComObject Excel.Application
$Excel.Visible=$True
#$Excel.Workbooks.OpenText($CSV) What should replace this?
Is there a Method like OpenText() that lets me pass a CSV object instead of a filepath to a CSV file or am I going to have to write my own conversion function?
Interesting question. I'm not aware of a method that allows you to pass a CSV Object.
However, if your result CSV is not too big and you are using PowerShell 5.0+ you could convert the object to a string and leverage Set-Clipboard (more info)
$headers = ($csv | Get-Member | Where-Object {$_.MemberType -eq "NoteProperty"}).Name
$delim = "`t"
# headers
foreach($header in $headers){
$myString += $header + $delim
}
# trim delimiter at the end, and add new line
$myString = $myString.TrimEnd($delim)
$myString = $myString + "`n"
# loop over each line and repeat
foreach($line in $csv){
foreach($header in $headers){
$myString += $line.$header + $delim
}
$myString = $myString.TrimEnd($delim)
$myString = $myString + "`n"
}
# copy to clipboard
Set-Clipboard $myString
# paste into excel from clipboard
$Excel.Workbooks.Worksheets.Item(1).Paste()
Here is another way to create an Excel spreadsheet from PowerShell without writing a .csv file.
$dirs = 'C:\src\t', 'C:\src\sql'
$records = $()
$records = foreach ($dir in $dirs) {
Get-ChildItem -Path $dir -File '*.txt' -Recurse |
Select-Object #{Expression={$_.FullName}; Label="filename"}
}
#open excel
$excel = New-Object -ComObject excel.application
$excel.visible = $false
#add a default workbook
$workbook = $excel.Workbooks.Add()
#remove worksheet 2 & 3
$workbook.Worksheets.Item(3).Delete()
$workbook.Worksheets.Item(2).Delete()
#give the remaining worksheet a name
$uregwksht = $workbook.Worksheets.Item(1)
$uregwksht.Name = 'File Names'
# Start on row 1
$i = 1
# the .appendix to $record refers to the column header in the csv file
foreach ($record in $records) {
$excel.cells.item($i,1) = $record.filename
$i++
}
#adjusting the column width so all data's properly visible
$usedRange = $uregwksht.UsedRange
$usedRange.EntireColumn.AutoFit() | Out-Null
#saving & closing the file
$outputpath = Join-Path -Path $Env:USERPROFILE -ChildPath "desktop\exceltest.xlsx"
$workbook.SaveAs($outputpath)
$excel.Quit()

Merging CSV Files into a XLSX with Tabs

I currently have 5 Registry CSV files which are created during a PowerShell script:
HKCC
HKCR
HKCU
HKLM
HKU
I need these CSV files to open at the end of the script however would like if all of them were contained within one XLSX file with 5 different headings
Is there a way to combine the files through PowerShell?
I understand how to get the data of the CSV files but don't understand how to merge them or convert. Some of the variables I believe which may be helpful.
$Date = Get-Date -Format "d.MMM.yyyy"
$DIR = $WPFlistview.Selecteditem.Ransomware
$path = "F:\Registry_Export\Results\$DIR\$Date\*"
$csvs = Get-ChildItem $path -Include *.csv
$output = "F:\Registry_Export\Results\$DIR\$Date\Results.Xlsx"
Paths to the CSV files if needed:
F:\Registry_Export\Results\$DIR\$Date\HKCR.CSV
F:\Registry_Export\Results\$DIR\$Date\HKCU.CSV
F:\Registry_Export\Results\$DIR\$Date\HKLM.CSV
F:\Registry_Export\Results\$DIR\$Date\HKU.CSV
F:\Registry_Export\Results\$DIR\$Date\HKCC.CSV
This is what I have tried prior. However, it completly scrambles my data into the wrong lines and cells:
function MergeCSV {
$Date = Get-Date -Format "d.MMM.yyyy"
$DIR = $WPFlistview.Selecteditem.Ransomware
$path = "F:\Registry_Export\Results\$DIR\$Date\*"
$csvs = Get-ChildItem $path -Include *.csv
$y = $csvs.Count
Write-Host "Detected the following CSV files: ($y)"
foreach ($csv in $csvs) {
Write-Host " "$csv.Name
}
$outputfilename = "Final Registry Results"
Write-Host Creating: $outputfilename
$excelapp = New-Object -ComObject Excel.Application
$excelapp.SheetsInNewWorkbook = $csvs.Count
$xlsx = $excelapp.Workbooks.Add()
$sheet = 1
foreach ($csv in $csvs) {
$row = 1
$column = 1
$worksheet = $xlsx.Worksheets.Item($sheet)
$worksheet.Name = $csv.Name
$file = (Get-Content $csv)
foreach ($line in $file) {
$linecontents = $line -split ',(?!\s*\w+")'
foreach ($cell in $linecontents) {
$worksheet.Cells.Item($row,$column) = $cell
$column++
}
$column = 1
$row++
}
$sheet++
}
$output = "F:\Registry_Export\Results\$DIR\$Date\Results.Xlsx"
$xlsx.SaveAs($output)
$excelapp.Quit()
}
How the CSV looks
https://gyazo.com/177c7c3bb21ddf06d0ebacbb7f4d537b
How the XLSX looks
https://gyazo.com/cd5fb48d61f93aac5ec3034d81811094
So, using the Excel.Application ComObject still, what I would suggest is loading each CSV as a CSV, not using Get-Content like you are. Then use the ConvertTo-CSV cmdlet, specifying to use tab as the delimiter, and copy that to the clipboard. Then just paste into Excel, and it will paste in fairly nicely. You may want to adjust column size, but the data will show up just as you would expect it to. I would also use a For loop instead of a ForEach loop, since Excel plays nice with numbers for the tabs (though it is 1 based instead of PowerShell's 0 base). Here's what I would end up with after making those modifications:
function MergeCSV {
$Date = Get-Date -Format "d.MMM.yyyy"
$DIR = $WPFlistview.Selecteditem.Ransomware
$path = "F:\Registry_Export\Results\$DIR\$Date\*"
$csvs = Get-ChildItem $path -Include *.csv
$y = $csvs.Count
Write-Host "Detected the following CSV files: ($y)"
Write-Host " "$csvs.Name"`n"
$outputfilename = "Final Registry Results"
Write-Host Creating: $outputfilename
$excelapp = New-Object -ComObject Excel.Application
$excelapp.SheetsInNewWorkbook = $csvs.Count
$xlsx = $excelapp.Workbooks.Add()
for($i=1;$i -le $y;$i++) {
$worksheet = $xlsx.Worksheets.Item($i)
$worksheet.Name = $csvs[$i-1].Name
$file = (Import-Csv $csvs[$i-1].FullName)
$file | ConvertTo-Csv -Delimiter "`t" -NoTypeInformation | Clip
$worksheet.Cells.Item(1).PasteSpecial()|out-null
}
$output = "F:\Registry_Export\Results\$DIR\$Date\Results.Xlsx"
$xlsx.SaveAs($output)
$excelapp.Quit()
}
You could use ImportExcel by Doug Finke And then replace your Export-CSV in the original script with Export-Excel -WorksheetName
Install-Module ImportExcel
Export-Excel "F:\Registry_Export\Results\$DIR\$Date\Results.xlsx" -worksheetname "HKCR"
Export-Excel "F:\Registry_Export\Results\$DIR\$Date\Results.xlsx" -worksheetname "HKCU"

Resources