I'm using the Export-Excel cmdlet to export the output of a function into Excel. My function is as follows:
function SQLQuery($ServerName, $DBName, $Query)
{
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Server=$ServerName;Database=$DBName;Integrated Security=True"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = $Query
$SqlCmd.Connection = $SqlConnection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
$Output = $DataSet.Tables[0]
$SqlConnection.Close()
return $Output
}
$ResultCost = SQLQuery -ServerName $SName -DBName $DBName -Query (Get-Content -Path $CostQueryPath)
$ResultCost.Table | Export-Excel -Path $ReportPath
The direct output doesn't get exported to Excel so I used $ResultCost.Table to export. However, I see many duplicates being exported (if the result has 10 records, the $ResultCost.Table has 10 to the power of 10 i.e., 100 records). How can I export only the direct output? And how do I remove the last 5 unwanted columns?
If you are trying to export a dataset to a .xlsx file try exporting the rows, not the table.
So instead of $ResultCost.Tables | Export-Excel -Path $ReportPath
Try $ResultCost.Tables.Rows | Export-Excel -Path $ReportPath
Related
Actually, this is a version of question here:
How to use powershell to select and copy columns and rows in which data is present in new workbook.
The goal is to grab certain columns from multiple Excel workbooks and dump everything to one csv file. Columns are always the same.
I'm doing that manually:
$xl = New-Object -ComObject Excel.Application
$xl.Visible = $false
$xl.DisplayAlerts = $false
$counter = 0
$input_folder = "C:\Users\user\Documents\excelfiles"
$output_folder = "C:\Users\user\Documents\csvdump"
Get-ChildItem $input_folder -File |
Foreach-Object {
$counter++
$wb = $xl.Workbooks.Open($_.FullName, 0, 1, 5, "")
try {
$ws = $wb.Worksheets.item('Calls') # => This specific worksheet
$rowMax = ($ws.UsedRange.Rows).count
for ($i=1; $i -le $rowMax-1; $i++) {
$newRow = New-Object -Type PSObject -Property #{
'Type' = $ws.Cells.Item(1+$i,1).text
'Direction' = $ws.Cells.Item(1+$i,2).text
'From' = $ws.Cells.Item(1+$i,3).text
'To' = $ws.Cells.Item(1+$i,4).text
}
$newRow | Export-Csv -Path $("$output_folder\$ESO_Output") -Append -noType -Force
}
}
} catch {
Write-host "No such workbook" -ForegroundColor Red
# Return
}
}
Question:
This works, but is extremely slow because Excel has to select every cell, copy that, then Powershell has to create array and save row by row in output csv file.
Is there a method to select a range in Excel (number of columns times ($ws.UsedRange.Rows).count), cut header line and just append this range (array?) to csv file to make everything much faster?
So that's the final solution
Script is 22 times faster!!! than original solution.
Hope somebody will find that useful :)
PasteSpecial is to filter out empty rows. There is no need to save them into csv
$xl = New-Object -ComObject Excel.Application
$xl.Visible = $false
$xl.DisplayAlerts = $false
$counter = 0
$input_folder = "C:\Users\user\Documents\excelfiles"
$output_folder = "C:\Users\user\Documents\csvdump"
Get-ChildItem $input_folder -File |
Foreach-Object {
$counter++
try {
$new_ws1 = $wb.Worksheets.add()
$ws = $wb.Worksheets.item('Calls')
$rowMax = ($ws.UsedRange.Rows).count
$range = $ws.Range("A1:O$rowMax")
$x = $range.copy()
$y = $new_ws1.Range("A1:O$rowMax").PasteSpecial([System.Type]::Missing,[System.Type]::Missing,$true,$false)
$wb.SaveAs("$($output_folder)\$($_.Basename)",[Microsoft.Office.Interop.Excel.XlFileFormat]::xlCSVWindows)
} catch {
Write-host "No such workbook" -ForegroundColor Red
# Return
}
}
$xl.Quit()
Part above will generate a bunch of csv files.
Part below will read these files in separate loop and combine them together into one.
-exclude is an array of something I want to omit
Remove-Item to remove temporary files
Answer below is based on this post: https://stackoverflow.com/a/27893253/6190661
$getFirstLine = $true
Get-ChildItem "$output_folder\*.csv" -exclude $excluded | foreach {
$filePath = $_
$lines = Get-Content $filePath
$linesToWrite = switch($getFirstLine) {
$true {$lines}
$false {$lines | Select -Skip 1}
}
$getFirstLine = $false
Add-Content "$($output_folder)\MERGED_CSV_FILE.csv" $linesToWrite
Remove-Item $_.FullName
}
I have the following code and it works perfectly except it's not closing Excel properly. It's leaving an Excel process running.
Is there a way to close Excel properly without killing the process?
Since i'm using other Excel files while running this script i can not kill all active Excel processes.
I think i tried everything i found online.
$WorkDir = "D:\Test\QR_ES\RG_Temp"
$BGDir = "D:\Test\QR_ES\3_BG"
$File = "D:\Test\QR_ES\4_Adr_Excel\KD_eMail.xlsx"
$SentDir = "D:\Test\QR_ES\RG_Temp\Sent\Dunning"
chdir $WorkDir
$firstPageList = Get-ChildItem "$WorkDir\1*.pdf" -File -Name
ForEach ($firstPage in $firstPageList)
{
$secondPage = "$BGDir\BG_RG.pdf"
$output = "Dunn-$firstPage"
invoke-command {pdftk $firstPage background $secondPage output $output}}
del 1*.pdf
gci $WorkDir\Dunn-*.pdf | rename-item -newname {$_.Name.Substring(5)} -Force
$Excel = New-Object -ComObject Excel.Application
$Excel.visible = $false
$Workbook = $Excel.workbooks.open($file)
$DunnList = Get-ChildItem "$WorkDir\1*.pdf" -File -Name
ForEach ($Dunn in $DunnList)
{
$Worksheets = $Workbooks.worksheets
$Worksheet = $Workbook.Worksheets.Item("KD_eMail")
$Range = $Worksheet.Range("A1").EntireColumn
$DunnSearch = $Dunn.Substring(0,5)
$SearchString = $DunnSearch
$Search = $Range.find($SearchString)
$Recipient = $Worksheet.Cells.Item($Search.Row, $Search.Column + 1)
$Msg = "<span style='font-family:Calibri;font-size:12pt;'>Test</span>"
$Outlook = New-Object -ComObject Outlook.Application
$namespace = $Outlook.GetNameSpace("MAPI")
$namespace.Logon($null, $null, $false, $true)
$EmailFrom = ('test#test.com')
$account = $outlook.Session.Accounts.Item($EmailFrom)
$Mail = $Outlook.CreateItem(0)
$Mail.HTMLBody = $Msg
$Mail.Subject = "OP - $SearchString"
$Mail.To = $Recipient
function Invoke-SetProperty {
param(
[__ComObject] $Object,
[String] $Property,
$Value
)
[Void] $Object.GetType().InvokeMember($Property,"SetProperty",$NULL,$Object,$Value)
}
Invoke-SetProperty -Object $mail -Property "SendUsingAccount" -Value $account
$Mail.Attachments.Add("$WorkDir\$Dunn")
$Mail.Save()
$Mail.close(1)
$Mail.Send()}}
$workbook.close($false)
$Excel.Quit()
chdir $WorkDir
del 1*.pdf
See this post:
https://stackoverflow.com/a/35955339/5329137
which is not accepted as an answer, but I believe is the full, correct way to close Excel.
This is what did it for me:
$FilePID = (Get-Process -name Excel | Where-Object { $_.MainWindowTitle -like 'FileName.xlsx*' }).Id
$Workbook.Save()
$Workbook.close($false)
Stop-Process $FilePID
Elaborating on #ASD's answer, since the MainWindowTitle doesn't (always) include the file suffix (.xlsx) you may have to strip that when comparing it to the filename. I'm using -replace to use a Regex match of everything before the last dot.
$excelPID = (Get-Process -name Excel | Where-Object { $_.MainWindowTitle -eq $fileName -replace '\.[^.]*$', '' }).Id
$workbook.Close()
Stop-Process $excelPID
I have a CSV file that has similar products within it and quantities of each product beside it.
Sample from CSV file
Qty Ordered Product/Item Description Top row (header)
7 Product1
3 Product2
5 Product1
3 Product3
I need a method to find all the similar product#s, add up their Quantities, and place the total of each similar product in a new row.
Add-Type -AssemblyName System.Windows.Forms
$FileBrowser = New-Object System.Windows.Forms.OpenFileDialog -Property
#{
Multiselect = $false # Multiple files can be chosen
Filter = 'Excel (*.csv, *.xlxs)|*.csv;*.xlsx' # Specified file types
}
[void]$FileBrowser.ShowDialog()
$file = $FileBrowser.FileNames;
[Reflection.Assembly]::LoadWithPartialName
("Microsoft.Office.Interop.Excel")|Out-Null
$excel = New-Object Microsoft.Office.Interop.Excel.ApplicationClass
$excel.Visible = $true
$wb = $excel.Workbooks.Open($file)
$ws = $wb.ActiveSheet
$c = $ws.Columns
$c.Item(2).hidden = $true
This code, asks the user to select the csv file, hides useless columns and auto-sizes the important columns as well.
Rather than using Excel as a COM Object you could use Import-CSV and then Group-Object. Then loop through the groups for the information you need.
Add-Type -AssemblyName System.Windows.Forms
$FileBrowser = New-Object System.Windows.Forms.OpenFileDialog -Property #{
Multiselect = $false # Multiple files can be chosen
Filter = 'Excel (.csv, *.xlxs)|.csv;*.xlsx' # Specified file types
}
[void]$FileBrowser.ShowDialog()
ForEach ($file in $FileBrowser.FileNames) {
$CSV = Import-CSV $file | Add-Member -Name Total -Value 0 -MemberType NoteProperty
$Groups = $CSV | Group-Object "Product/Item Description"
$NewCSV = Foreach ($Group in $Groups) {
$Count = 0
$Group.Group."Qty Ordered" | ForEach-Object {$Count += $_}
Foreach ($value in $CSV) {
If ($value."Product/Item Description" -eq $Group.Name) {
$value.Total = $Count
$value
}
}
}
Export-CSV "$filenew" -NoTypeInformation
}
I wrote a function Get-oracleresultDa which have oracle connection properties.Through which I can query my DB.
But,the problem is which I try to export the data to the excel sheet it only returns the result of the second query i.e)no status
and not no type
$results = Get-OracleResultDa -conString $connectionString -sqlString $query
-Verbose
$results | SELECT no, type| Export-CSV "H:\Book2.csv" -Force
$rows++
$results1 = Get-OracleResultDa -conString $connectionString -sqlString
$created -Verbose
$results1 | SELECT no, status| Export-CSV "H:\Book2.csv" -
NoTypeInformation
The below mentioned block was in the first 10 linesof the script
$file="H:\Book2.csv"
$excel = New-Object -ComObject excel.application
#Makes Excel Visable
$excel.Application.Visible = $true
$excel.DisplayAlerts = $false
#Creates Excel workBook
$book = $excel.Workbooks.Add()
#Adds worksheets
#gets the work sheet and Names it
$sheet = $book.Worksheets.Item(1)
$sheet.name = 'Created'
#Select a worksheet
$sheet.Activate() | Out-Null
I have few more query's which also as to be exported
If you use powershell 3.0 or better, you can use -Append modificator
$results = Get-OracleResultDa -conString $connectionString -sqlString $query
-Verbose
$results | SELECT no, type| Export-CSV "H:\Book2.csv" -Force
$rows++
$results1 = Get-OracleResultDa -conString $connectionString -sqlString
$created -Verbose
$results1 | SELECT no, status| Export-CSV "H:\Book2.csv" -
NoTypeInformation -Append
-Apologies for the back and forth question!
I pieced together the following PowerShell script which runs two SQL queries, exports each query to a CSV file then moves the CSV files into an Excel workbook.
The code works as expected when the two CSV files are already created. But the script fails when it is run the first time when the CSV files get created.
Function Run-Query {
param([string[]]$queries,[string[]]$sheetnames,[string[]]$filenames)
$Excel = New-Object -ComObject Excel.Application
$Excel.Visible = 0
$dest = $Excel.Workbooks.Add(1)
for ($i = 0; $i -lt $queries.Count; $i++){
$query = $queries[$i]
$sheetname = $sheetnames[$i]
$filename = $filenames[$i]
### SQL query results sent to Excel
$SQLServer = 'Server'
$Database = 'Database'
## - Connect to SQL Server using non-SMO class 'System.Data':
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Server = $SQLServer; Database = $Database; Integrated Security = True"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = $query
$SqlCmd.Connection = $SqlConnection
## - Extract and build the SQL data object '$Table2':
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
$SqlConnection.Close()
$DataSet.Tables[0] | Export-Csv -NoTypeInformation -Path "C:\Scripts\Organize\ExcelStuff\$sheetname.csv"
}#End For.
#Begin excel test, loop over each CSV.
$loopy = (Resolve-Path $filename).ProviderPath
$Book = $Excel.Workbooks.Open($loopy)
foreach ($item in $loopy){
$next = $Excel.workbooks.Open($item)
$next.ActiveSheet.Move($dest.ActiveSheet)
$xlsRng = $dest.ActiveSheet.UsedRange
$xlsRng.EntireColumn.AutoFit() | Out-Null
}# END ForEach
#$Excel.Visible = 1 #For debugging.
$dest.sheets.item('Sheet1').Delete()
$xlsFile = "C:\Scripts\MonthlyReboots.xlsx"
$Excel.ActiveWorkbook.SaveAs($xlsFile) | Out-Null
$Excel.Quit()
While ([System.Runtime.Interopservices.Marshal]::ReleaseComObject($xlsRng)) {'cleanup xlsRng'}
While ([System.Runtime.Interopservices.Marshal]::ReleaseComObject($next)) {'cleanup xlsSh'}
While ([System.Runtime.Interopservices.Marshal]::ReleaseComObject($Book)) {'cleanup xlsWb'}
While ([System.Runtime.Interopservices.Marshal]::ReleaseComObject($Excel)) {'cleanup xlsObj'}
[gc]::collect() | Out-Null
[gc]::WaitForPendingFinalizers() | Out-Null
}#End Function
$queries = #()
$queries += #'
'#
$queries += #'
'#
$sheetnames = #('Cert','Prod')
$filenames = #(".\prod.csv", ".\cert.csv")
Run-Query -queries $queries -sheetnames $sheetnames -filenames $filenames
Ok, we've got a few lessons to work with here I think. First, functions, what they should do, and what they shouldn't do, and structure. Later we'll touch on organizing your script so that it runs a bit more optimally.
So let's look at that massive function you've got there. That's a lot of stuff in there, and I'm willing to bet that it probably shouldn't all be in there. What is in there will benefit from using the Begin, Process, and End scriptblock sections. For the time being, we're going to ignore Excel, and have the function actually just work with your SQL queries. Right now your function (remember, ignoring Excel for the time being) takes a collection of strings for queries, connects to the SQL server, runs a query, disconnects from the server, reconnects to the server, runs a query, disconnects from the server, and keeps doing that until it runs out of queries. I think a better option would be to use the Begin scriptblock to connect to the server once, then the Process scriptblock to run each query, and the End block to close the connection and return the query results. That stops us from having to open and close the connection a bunch, and keeps the function focused on doing one thing, but doing it well.
Function Run-Query {
param([string[]]$queries)
Begin{
$SQLServer = 'Server'
$Database = 'Database'
## - Connect to SQL Server using non-SMO class 'System.Data':
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Server = $SQLServer; Database = $Database; Integrated Security = True"
}
Process{
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = $queries
$SqlCmd.Connection = $SqlConnection
## - Extract and build the SQL data object '$Table2':
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
$DataSet.Tables[0]
}
End{
$SqlConnection.Close()
}
}#End Run-Query Function
That will put out an array of objects for however many queries you feed it. So then we just assign a variable to that and we have two datasets in an array. That part is simple:
#Define Queries
$Queries = #()
$Queries += #'
Select * From TableA;
Where Stuff = 'Cert'
'#
$Queries += #'
Select * From TableB;
Where Stuff = 'Prod'
'#
#Get data from SQL
$Data = Run-Query -queries $Queries
Now that we have our datasets we will launch Excel, create a new workbook, name the sheet it starts with, make a second sheet and name that, then just paste the data directly into Excel. There is no reason to export to CSV files, load them into Excel, and copy the data around within Excel when we can just paste the data directly into Excel.
#Launch Excel and add a workbook
$Excel = New-Object -ComObject Excel.Application
$Workbook = $Excel.Workbooks.Add()
#Set the current worksheet at Cert, and add a new one as Prod, then name them appropriately
$Cert = $Workbook.ActiveSheet
$Prod = $Workbook.Worksheets.Add()
$Cert.Name = 'Cert'
$Prod.Name = 'Prod'
#Copy the data from the first query to the clipboard as a tab delimited CSV, then paste it into the Cert sheet
$Data[0] | ConvertTo-Csv -notype -Delimiter "`t" | Clip
[Void]$Cert.Cells.Item(1).PasteSpecial()
#Do the same with the second query and paste it into the Prod sheet
$Data[1] | ConvertTo-Csv -notype -Delimiter "`t" | Clip
[Void]$Prod.Cells.Item(1).PasteSpecial()
You should now have an open workbook with two sheets, each containing the results of one SQL query. Now to just perform the autofit to make it look nice, save the workbook, close it, exit Excel, and perform garbage collection...
#Autofit the columns to make it all look nice
$Prod.UsedRange.EntireColumn.AutoFit()
$Cert.UsedRange.EntireColumn.AutoFit()
#Save the workbook
$Workbook.SaveAs("C:\Scripts\MonthlyReboots.xlsx")
#Close the worbook, and Excel
$Workbook.Close()
$Excel.Quit()
[System.Runtime.Interopservices.Marshal]::ReleaseComObject($Excel)|Out-Null
[gc]::collect() | Out-Null
[gc]::WaitForPendingFinalizers() | Out-Null
That should do it. No more opening Excel a whole bunch and working with a bunch of files, the SQL connection just gets opened once, and closed once, with queries performed during the session. If the script takes a long time to run at this point I'd be willing to bet it's the SQL queries that are taking the bulk of the time because once you have the data out of SQL bringing up Excel, and getting the data into the sheets should be really fast.
Edit: Well, it sounds like you aren't getting back results from all of the queries that you are submitting, so I have restructured the function a little and hopefully this will work better.
Function Run-Query {
param([string[]]$queries)
Begin{
$SQLServer = 'Server'
$Database = 'Database'
$Results = #()
}
Process{
## - Connect to SQL Server using non-SMO class 'System.Data':
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Server = $SQLServer; Database = $Database; Integrated Security = True"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = $queries
$SqlCmd.Connection = $SqlConnection
## - Extract and build the SQL data object '$Table2':
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
$Results += $DataSet.Tables[0]
}
End{
$SqlConnection.Close()
$Results
}
}#End Run-Query Function
If it doesn't, you could always go back to your old way of doing things, and instead of outputting to CSV files you can start pasting to Excel directly like I've shown you how to do. That should speed things up at the least. Like, open Excel, run the old function (except take out the part that opens Excel), and have the old function paste into sheets in Excel.
I do wish I had a SQL server I could test against. Everything should have worked as far as I could tell, but obviously didn't work like I had anticipated.
Major thanks given to TheMadTechnician for the guidance on using a function.
Here is what I've cobbled together which does work and it creates an Excel file with two worksheets in under 2 seconds. Additionally, the code correctly cleans up the Excel ComObject I'm boasting here but I'd love to see someone come up with a faster way of accomplising this!
Function Run-Query {
param([string[]]$queries,[string[]]$sheetnames,[string[]]$filenames)
Begin{
$SQLServer = 'ServerName'
$Database = 'DataBase'
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Server = $SQLServer; Database = $Database; Integrated Security = True"
$Excel = New-Object -ComObject Excel.Application
$Excel.Visible = 0
$dest = $Excel.Workbooks.Add(1)
}#End Begin
Process{
For($i = 0; $i -lt $queries.Count; $i++){
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = $queries[$i]
$SqlCmd.Connection = $SqlConnection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
$DataSet.Tables[0] | Export-Csv -NoTypeInformation -Path "C:\Scripts\$($sheetnames[$i]).csv" -Force
}#end for loop.
}#End Process
End{
$SqlConnection.Close()
#Excel magic test!
For($i = 0; $i -lt $queries.Count; $i++){
$loopy = (Resolve-Path -Path $filenames[$i]).ProviderPath
$Book = $Excel.Workbooks.Open($loopy)
$next = $Excel.workbooks.Open($loopy)
$next.ActiveSheet.Move($dest.ActiveSheet)
$xlsRng = $dest.ActiveSheet.UsedRange
$xlsRng.EntireColumn.AutoFit() | Out-Null
}
$dest.sheets.item('Sheet1').Delete()
$xlsFile = "C:\Scripts\MonthlyReboots.xlsx"
[void] $Excel.ActiveWorkbook.SaveAs($xlsFile)
$Excel.Quit()
While ([System.Runtime.Interopservices.Marshal]::ReleaseComObject($xlsRng)) {'cleanup xlsRng'}
While ([System.Runtime.Interopservices.Marshal]::ReleaseComObject($next)) {'cleanup xlsSh'}
While ([System.Runtime.Interopservices.Marshal]::ReleaseComObject($Book)) {'cleanup xlsWb'}
While ([System.Runtime.Interopservices.Marshal]::ReleaseComObject($Excel)) {'cleanup xlsObj'}
[gc]::collect() | Out-Null
[gc]::WaitForPendingFinalizers() | Out-Null
}#End end block.
}#End function run-query.