Speed up reading an Excel File in Powershell - excel

I wonder if there is any way to speed up reading an Excel file with powershell. Many would say I should stop using the do until, but the problem is I need it badly, because in my Excel sheet there can be 2 rows or 5000 rows. I understand that 5000 rows needs some time. But 2 rows shouldn't need 90sec+.
$Excel = New-Object -ComObject Excel.Application
$Excel.Visible = $true
$Excel.DisplayAlerts = $false
$Path = EXCELFILEPATH
$Workbook = $Excel.Workbooks.open($Path)
$Sheet1 = $Workbook.Worksheets.Item(test)
$URows = #()
Do {$URows += $Sheet1.Cells.Item($Row,1).Text; $row = $row + [int] 1} until (!$Sheet1.Cells.Item($Row,1).Text)
$URows | foreach {
$MyParms = #{};
$SetParms = #{};
And i got this 30 times in the script too:
If ($Sheet1.Cells.Item($Row,2).Text){$var1 = $Sheet1.Cells.Item($Row,2).Text
$MyParms.Add("PAR1",$var1)
$SetParms.Add("PAR1",$var1)}
}
I have the idea of running the $MyParms stuff contemporarily, but I have no idea how. Any suggestions?
Or
Increase the speed of reading, but I have no clue how to achieve that without destroying the "read until nothing is there".
Or
The speed is normal and I shouldn't complain.

Don't use Excel.Application in the first place if you need speed. You can use an Excel spreadsheet as an ODBC data source - the file is analogous to a database, and each worksheet a table. The speed difference is immense. Here's an intro on using Excel spreadsheets without Excel

Appending to an array with the += operator is terribly slow, because it will copy all elements from the existing array to a new array. Use something like this instead:
$URows = for ($row = 1; !$Sheet1.Cells.Item($row, 1).Text; $row++) {
if ($Sheet1.Cells.Item($Row,2).Text) {
$MyParms['PAR1'] = $Sheet1.Cells.Item($Row, 2).Text)
$SetParms['PAR1'] = $Sheet1.Cells.Item($Row, 2).Text)
}
$Sheet1.Cells.Item($Row,1).Text
}
Your Do loop is basically a counting loop. The canonical form for such loops is
for (init counter; condition; increment counter) {
...
}
so I changed the loop accordingly. Of course you'd achieve the same result like this:
$row = 1
$URows = Do {
...
$row += 1
}
but that would just mean more code without any benefits. This modification doesn't have any performance impact, though.
Relevant in terms of performance are the other two changes:
I moved the code filling the hashtables inside the first loop, so the code won't loop twice over the data. Using index and assignment operators instead of the Add method for assigning values to the hashtable prevents the code from raising an error when a key already exists in the hashtable.
Instead of appending to an array (which has the abovementioned performance impact) the code now simply echoes the cell text in the loop, which PowerShell automatically turns into a list. The list is then assigned to the variable $URows.

Related

Pull data from a specified row in Excel spreadsheet

I'm working on a PS script to take a row of data from an Excel spreadsheet and populate that data in certain places in a Word document. To elaborate, we have a contract tracking MASTER worksheet that among other things contains data such as name of firm, address, services, contact name. Additionally, we have another TASK worksheet in the same workbook that tracks information such as project owner, project name, contract number, task agree number.
I'm writing a script that does the following:
Ask the user through a message box what kind of contract is being written ("Master", or "Task")
Opens the workbook with the appropriate worksheet opened ("Master" tab or "Task" tab)
Asks the user through a VB InputBox from which Excel row of data they want to use to populate the Word contract
Extracts that row of data from Excel
Outputs certain portions of that row of data to certain location in a Word document
Saves the Word document
Opens the Word document so the user can continue editing it
My question is this - using something like PSExcel, how do I extract that row of data out to variables that can be placed in a Word document. For reference, in case you're going to reply with a snippet of code, here are what the variables are defined as for the Excel portion my script:
$Filepath = "C:\temp\ContractScript\Subconsultant Information Spreadsheet.xlsx"
$Excel = New-Object -ComObject Excel.Application
$Workbook = $Excel.Workbooks.Open($Filepath)
$Worksheet = $Workbook.sheets.item($AgreementType)
$Excel.Visible = $true
#Choosing which row of data
[int]$RowNumber = [Microsoft.VisualBasic.Interaction]::InputBox("Enter the row of data from $AgreementType worksheet you wish to use", "Row")
Additionally, the first row of data in the excel worksheets are the column headings, in case it matters.
I've gotten this far so far:
import-module psexcel
$Consultant = new-object System.Collections.Arraylist
foreach ($data in (Import-XLSX -path $Filepath -Sheet $AgreementType -RowStart $RowNumber))
{
$Consultant.add($data)'
But I'm currently stuck because I can't figure out how to reference the data being added to $consultant.$data. Somehow I need to read in the column headings first so the $data variable can be defined in some way, so when I add the variable $consultant.Address in Word it finds it. Right now I think the variable name is going to end up "$Consultant.1402 S Broadway" which obviously won't work.
Thanks for any help. I'm fairly new to powershell scripting, so anything is much appreciated.
I have the same issue and searching online for solutions in a royal PITA.
I'd love to find a simple way to loop through all of the rows like you're doing.
$myData = Import-XLSX -Path "path to the file"
foreach ($row in $myData.Rows)
{
$row.ColumnName
}
But sadly something logical like that doesn't seem to work. I see examples online that use ForEach-Object and Where-Object which is cumbersome. So any good answers to the OP's question would be helpful for me too.
UPDATE:
Matthew, thanks for coming back and updating the OP with the solution you found. I appreciate it! That will help in the future.
For my current project, I went about this a different way since I ran into lack of good examples for Import-XLSX. It's just quick code to do a local task when needed, so it's not in a production environment. I changed var names, etc. to show an example:
$myDataField1 = New-Object Collections.Generic.List[String]
$myDataField2 = New-Object Collections.Generic.List[String]
# ...
$myDataField10 = New-Object Collections.Generic.List[String]
# PSExcel, the third party library, might want to install it first
Import-Module PSExcel
# Get spreadsheet, workbook, then sheet
try
{
$mySpreadsheet = New-Excel -Path "path to my spreadsheet file"
$myWorkbook = $mySpreadsheet | Get-Workbook
$myWorksheet = $myWorkbook | Get-Worksheet -Name Sheet1
}
catch { #whatever error handling code you want }
# calculate total number of records
$recordCount = $myWorksheet.Dimension.Rows
$itemCount = $recordCount - 1
# specify column positions
$r, $my1stColumn = 1, 1
$r, $my2ndColumn = 1, 2
# ...
$r, $my10thColumn = 1, 10
if ($recordCount -gt 1)
{
# loop through all rows and get data for each cell's value according to column
for ($i = 1; $i -le $recordCount - 1; $i++)
{
$myDataField1.Add($myWorksheet.Cells.Item($r + $i, $my1stColumn).text)
$myDataField2.Add($myWorksheet.Cells.Item($r + $i, $my2ndColumn).text)
# ...
$myDataField10.Add($myWorksheet.Cells.Item($r + $i, $my10thColumn).text)
}
}
#loop through all imported cell values
for ([int]$i = 0; $i -lt $itemCount; $i++)
{
# use the data
$myDataField1[$i]
$myDataField2[$i]
# ...
$myDataField10[$i]
}

Powershell Excel Automating adding worksheet

I am attempting to automate the process of adding a worksheet (with data) per clientname in excel for a monthly report type workbook
I thought it should be straight forward... but the method I am using isnt working.... it doesn't even get to the sheet making mode... can you help me figure out what I did wrong?
The following is the function I made
function Excelclientstatstemplate ($clients) {
$Exl = New-Object -ComObject "Excel.Application"
$Exl.Visible = $false
$Exl.DisplayAlerts = $false
$WB = $Exl.Workbooks.Open($excelmonthlysummary)
$clientws = $WB.worksheets | where {$_.name -like "*$clients*"}
#### Check if Clients worksheet exists, if no then make one with client name ###
$sheetcheck = if (($clientws)) {} Else {
$WS = $WB.worksheets.add
$WS.name = "$clients"
}
$sheetcheck
$WB.Save
# Enter stat labels
$clientws.cells.item(1,1) = "CPU Count"
$clientws.cells.item(2,1) = "RAM"
$clientws.cells.item(3,1) = "Reserved CPU"
$clientws.cells.item(4,1) = "Reserved RAM"
### Put in Values in the next column ###
$clientws.cells.item(1,2) = [int]($cstats.cpuAllocationGHz/2)
$clientws.cells.item(2,2) = [decimal]$cstats.memoryLimitGB
$clientws.cells.item(3,2) = [int]($cstats.rescpuAllocationGHz/2)
$clientws.cells.item(4,2) = [decimal]$cstats.resmemoryLimitGB
$WB.save
$Exl.quit()
Stop-Process -processname EXCEL
Start-Sleep -Seconds 1
Echo "$clients excel sheet in monthly summary is done.."
}
and then I tried to make a Foreach thing for it
$clientxlmonthlywrite = Foreach ($client in $clientlist){
$cstats = $Combinedstats | Where {$_.Group -eq "$client"}
Excelclientstatstemplate -clients $client
}
The entire Process of the function goes
Take client name
Open a particular excel workbook
Check if there are any sheets with client name
If there are NO sheets with client name, make one with client name
Fill The first column Cells with labels
Fill the second column cells with data (data works I already write CSVs withem)
Save and exit
The Foreach variable just does the function for each of Clients names from a clientlist (nothing wrong with clientlist)
Am I messing something up?
Thanks for the help.
You are not calling the .Add() method correctly. You are missing the parenthesis at the end of it. To fix it you should be able to simply modify the line to this:
$WS = $WB.worksheets.add()
Also, the cells have properties that you should refer to, so I would also modify the part that sets your cell values to something like this:
# Enter stat labels
$clientws.cells.item(1,1).value2 = "CPU Count"
$clientws.cells.item(2,1).value2 = "RAM"
$clientws.cells.item(3,1).value2 = "Reserved CPU"
$clientws.cells.item(4,1).value2 = "Reserved RAM"
### Put in Values in the next column ###
$clientws.cells.item(1,2).value2 = [int]($cstats.cpuAllocationGHz/2)
$clientws.cells.item(2,2).value2 = [decimal]$cstats.memoryLimitGB
$clientws.cells.item(3,2).value2 = [int]($cstats.rescpuAllocationGHz/2)
$clientws.cells.item(4,2).value2 = [decimal]$cstats.resmemoryLimitGB
I'm fairly sure that defining the type is pointless, since to Excel they're all strings until you set the cell's formatting settings to something else. I could be wrong, but that is the behavior that I have observed.
Now, for other critiques that you didn't ask for... Don't launch Excel, open the book, save the book, and close Excel for each client. Open Excel once at the beginning, open the book, make your updates for each client, and then save, and close.
Test to see if the client has a sheet, and add it if needed, then select the client's sheet afterwords. Right now there's nothing there to set $clientws if you have to add one for that client.
Adding a worksheet by default places it before the active worksheet. This was a poor choice in design in my opinion, but it is what it is. If it were me I'd add new sheets specifying the last worksheet in the workbook, which will add the new worksheet before the last one, making it the second to the last worksheet. Then I'd move the last worksheet up in front of the new one, effectively adding the new worksheet as the last one listed. Is it possible to add the new worksheet as the last one when you make it? Yes, but it's was too complicated for my taste. See here if you are interested in doing that.
When testing for an existing client worksheet to make one if it is missing, do that, don't tell it to test for something, and do nothing, and put everything you want in an Else statement. That just complicates things. All that said, here's some of those suggestions put into practice:
function Excelclientstatstemplate ($clients) {
#### Check if Clients worksheet exists, if no then make one with client name ###
if (($clients -notin $($WB.worksheets).Name)){
#Find the current last sheet
$LastSheet = $WB.Worksheets|Select -Last 1
#Make a new sheet before the current last sheet so it's near the end
$WS = $WB.worksheets.add($LastSheet)
#Name it
$WS.name = "$clients"
#Move the last sheet up one spot, making the new sheet the new effective last sheet
$LastSheet.Move($WS)
}
#Find the current client sheet regardless of if it existed before or not
$clientws = $WB.worksheets | where {$_.name -like "*$clients*"}
# Enter stat labels
$clientws.cells.item(1,1).value2 = "CPU Count"
$clientws.cells.item(2,1).value2 = "RAM"
$clientws.cells.item(3,1).value2 = "Reserved CPU"
$clientws.cells.item(4,1).value2 = "Reserved RAM"
### Put in Values in the next column ###
$clientws.cells.item(1,2).value2 = [int]($cstats.cpuAllocationGHz/2)
$clientws.cells.item(2,2).value2 = [decimal]$cstats.memoryLimitGB
$clientws.cells.item(3,2).value2 = [int]($cstats.rescpuAllocationGHz/2)
$clientws.cells.item(4,2).value2 = [decimal]$cstats.resmemoryLimitGB
Start-Sleep -Seconds 1
Echo "$clients excel sheet in monthly summary is done.."
}
$Exl = New-Object -ComObject "Excel.Application"
$Exl.Visible = $false
$Exl.DisplayAlerts = $false
$WB = $Exl.Workbooks.Open($excelmonthlysummary)
$clientxlmonthlywrite = Foreach ($client in $clientlist){
$cstats = $Combinedstats | Where {$_.Group -eq "$client"}
Excelclientstatstemplate -clients $client
}
$WB.save
$Exl.quit()
Stop-Process -processname EXCEL

Adding Same Column Header Names to all worksheets in an Excel Book

I have a script that will create worksheets based on the number of files that it finds in a directory. From there it changes the name of the sheets to the file name. During that process, I am attempting to add two Column header values of "Hostname" and "IP Address" to every sheet. I can achieve this by activating each sheet individually but this becomes rather cumbersome as the amount of sheets goes past 20+ and thus I am trying to find a dynamic way of doing this regardless the amount of sheets that are present.
This is the code that I have to do everything up to the column header portion:
$WorksheetCount = (Get-ChildItem $Path\Info*.txt).count
$TabNames = Get-ChildItem $Path\Info*.txt
$NewTabNames = Foreach ($file IN $TabNames.Name){$file.Substring(0,$file.Length-4)}
$Break = 0
$excel = New-Object -ComObject Excel.Application
$excel.Visible = $true
$Workbook = $Excel.Workbooks.Add()
$null = $Excel.Worksheets.Add($MissingType, $Excel.Worksheets.Item($Excel.Worksheets.Count),
$WorksheetCount - $Excel.Worksheets.Count, $Excel.Worksheets.Item(1).Type)
1..$WorksheetCount
Start-Sleep -s 1
ForEach ($Name In $NewTabNames){
$Break++
$Excel.Worksheets.Item($Break).Name = $Name
}
I have attempted to insert my code as such:
ForEach ($Name In $NewTabNames){
$Break++
$Excel.Worksheets.Item($Break).Name = $Name
$cells=$Name.Cells
$cells.item(1,1)="Hostname"
$cells.item(1,2)="IP Address"
}
When I attempt to run the script, I get the following error..
You cannot call a method on a null-valued expression.
And then it proceeds to list each line of the code that I had put in. I thought that since I created a variable during the operation, that it was the issue:
$cells=$Name.Cells
I thought That perhaps if I moved it before the ForEach command that it would resolve it but I still receive the same issue. I have looked through various ways of trying to select ranges of sheets within excel via powershell but have not found anything helpful.
Would appreciate any assistance on this.
This is actually my first post in StackOverflow ever and I feel pretty excited to finally help out. I made some small modifications to your code and seems to work fine. I noticed some odd behavior when I removed the $null variable that was getting assigned because it seemed strange to me why it was being done, but after removing that assignment my outlook application open by itself automatically every time I ran the script. I found the site where you got the code from just to see if there were any changes to the original code.
I found this Microsoft documentation very helpful to figure this out.
This is what I modified
ForEach ($Name In $NewTabNames){
$Break++
$Excel.Worksheets($Break).Name = $Name
$Excel.Worksheets($Break).Cells(1,1).Font.Bold = $true
$Excel.Worksheets($Break).Cells(1,1) = "Hostname"
$Excel.Worksheets($Break).Cells(1,2).Font.Bold = $true
$Excel.Worksheets($Break).Cells(1,2) = "IP Address"
}

powershell excel Deleting group of rows

I have an xlsx file with thousands of entries
I can within a second filter a column to show only certain information with $workbook.AutoFilter("DATA")
This filter only takes a second however deleting all rows whos first column = "DATA" takes forever with a loop.
Is there a way to capture an array of the hidden rows or a range... or anything that I could .DELETE()
I tried this
[void] [Reflection.Assembly]::LoadWithPartialName( 'System.Windows.Forms' )
$Excel = New-Object -Com Excel.Application
$WorkBook = $Excel.Workbooks.Open($filename)
$Excel.visible = $true
$Excel.selection.autofilter(1,"DATA")
$sheet = $workbook.Sheets.Item(1)
$max = $sheet.UsedRange.Rows.Count
for ($i=2; $i -le $max; $i++)
{
$row = $sheet.Cells.Item($i,1).EntireRow
if ($row.hidden -eq $false)
{
$row.Delete()
}
}
FIXED.. loop backwards $i-- *
However This failed me misserably because for some reason it leaves roughly 10% of the visabled rows undeleted. If I run it twice it works however scaling up this would become a bigger issue.
In a perfect world I would like something like this
$Excel.selection.autofilter(1,"DATA").DELETE()
Thanks in advance for any hints or tricks you geniuses may have.
Update: Thanks Graimer, you are right I have to loop in the other directions, this still takes quite some time with 10,000+ entries... I am looking for a way to do it without the manual loop.
If I go $Excel.visible = $true, and then $Excel.selection.autofilter(1,"DATA")... then as a user I ctrl+A and delete the selected rows... its quicker manually then the looping process... I cant help but think there MUST be some way to script that action.
Turned out to be pretty easy
after applying a fiter select a range from row1 to Lastrow, delete range.
Because the filter is only showing that one value the range cannot select hidden cells

Why is writing Excel cell values fast in VBScript but slow in PowerShell?

Why is it that writing cell values to Excel is a lot faster in VBScript than in PowerShell?
Isn't PowerShell the new thing, and VBScript the deprecated MS scripting language?
VBScript example (save to filename.vbs)
This runs in a split second.
Set objExcel = CreateObject("Excel.Application")
objExcel.Visible = false
Set objWorkbook = objExcel.Workbooks.Add()
' Edit: increased number of writes to 500 to make speed difference more noticeable
For row = 1 To 500
'Edit: using .cells(row,1) instead of .cells(50,1) - this was a mistake
objWorkbook.workSheets(1).cells(row,1).value = "test"
Next
objWorkbook.SaveAs(CreateObject("Scripting.FileSystemObject").GetParentFolderName(WScript.ScriptFullName) & "\test.xlsx")
objExcel.Quit
msgbox "Done."
PowerShell example (save to filename.ps1) This takes multiple seconds to run (problematic on thousands of records)
#need this to work around bug if you use a non-US locale: http://support.microsoft.com/default.aspx?scid=kb;en-us;320369
[System.Threading.Thread]::CurrentThread.CurrentCulture = "en-US"
$excel = New-Object -ComObject Excel.Application
$excel.Visible = $False
$xls_workbook = $excel.Workbooks.Add()
# Edit: using foreach instead of for
# Edit: increased number of writes to 500 to make speed difference more noticeable
foreach ($row in 1..500) {
# Edit: Commented out print-line, slows down the script
#"Row " + $row
# This is very slow! - http://forums.redmondmag.com/forums/forum_posts.asp?tid=4037&pn=7
$xls_workbook.sheets.item(1).cells.item($row,1) = "test"
}
$xls_workbook.SaveAs($MyInvocation.MyCommand.Definition.Replace($MyInvocation.MyCommand.Name, "") + "test.xlsx")
$excel.Quit()
[System.Runtime.Interopservices.Marshal]::ReleaseComObject($excel)
I want to use this for thousands of records. If there is no fast way to do this, PowerShell is not an option. Are there better alternatives?
You can speed things up by not looping through individual cells:
$excel = New-Object -ComObject Excel.Application
$excel.Visible = $True
$xls_workbook = $excel.Workbooks.Add()
$range = $xls_workbook.sheets.item(1).Range("A1:A100")
$range.Value2 = "test"
If you want to write an array of values to a range, here is a nice blog post that demonstrates similar technique:
How to Get Data into an Excel Spreadsheet Very Quickly with PowerShell
some things don't add up here:
your VBScript, writes on ONE cell over and over, while your PowerShell code writes into 100 cells
objWorkbook.workSheets(1).cells(50,1).value = "test"
$xls_workbook.sheets.item(1).cells.item($row,1) = "test"
you are executing "Row " + $row on PowerShell - this might offset comparison too.
If you want to write into multiple cells, you should think about using arrays and wrinting onto whole ranges, because this has better performance.
You can shave a little time off the PowerShell version by eliminating the for loop test and using a foreach.
for ($row = 1; $row -le 100; $row++)
goes to:
foreach ($row in 1..100)
By doing this you eliminate the comparison and increment.
But aside from that, my observations match yours (see my comments on Jook's answer).
You're still interfacing with Excel through COM though. That's adding some overhead due to COMInterop processing.
PowerShell, by its very design and use of cmdlets is a non-standard mess, at least for basic things. VBScript, which any programmer should be able to use and understand, has a general way of doing basic things that does not require special cmdlets to be installed or included with the deployed code. I believe this is a step backwards in many respects.
Before anyone trashes me and says I just don't PowerShell, I must mention I have a long history of UNIX shell scripting behind me. PowerShell is similar, obviously, but to me its not nearly as well implemented.
I do know that reality dictates that I will end up using PowerShell sooner or later - I just hope it evolves into a more "standard" replacement in the future.

Resources