I've a script in Powershell that update Excel files with connections to SQL databases. The script works fine but the problem is if one connection doesn't work the script can't continue. Is there a way to put like a timeout or something so that can continue after a wile?
Here is the script that I have:
$libraryPath = "C:\temp\Excel\"
$excel = new-object -comobject Excel.Application
# Give delay to open
Start-Sleep -s 5
$allExcelfiles = Get-ChildItem $libraryPath -recurse -include “*.xlsx”
foreach ($file in $allExcelfiles)
{
$workbookpath = $file.fullname
Write-Host "Updating " $workbookpath
# Open the Excel file
$excelworkbook = $excel.workbooks.Open($workbookpath)
$connections = $excelworkbook.Connections
foreach ($c in $connections)
{
if ($c.DataFeedConnection -ne $null)
{
$conn = $c.DataFeedConnection.Connection
# Use regex to search and replace part of connection string
$new = $conn -replace 'ProjectName eq ''(.*)''', "ProjectName eq '$title'"
$c.DataFeedConnection.Connection = $new
Write-Host "Connection replaced."
}
}
Start-Sleep -s 5
# This will Refresh All the pivot tables data.
$excelworkbook.RefreshAll()
# The following script lines will Save the file.
Start-Sleep -s 50
$excelworkbook.Save()
$excelworkbook.Close()
}
$excel.quit()
Thanks
You could run the your script as a job, and if the job exceeds a given period terminate it.
$timeout_in_sec = 10
$excel_update = {
$libraryPath = "C:\temp\Excel\"
$excel = new-object -comobject Excel.Application
# Give delay to open
Start-Sleep -s 5
$allExcelfiles = Get-ChildItem $libraryPath -recurse -include “*.xlsx”
foreach ($file in $allExcelfiles)
{
$workbookpath = $file.fullname
Write-Host "Updating " $workbookpath
# Open the Excel file
$excelworkbook = $excel.workbooks.Open($workbookpath)
$connections = $excelworkbook.Connections
foreach ($c in $connections)
{
if ($c.DataFeedConnection -ne $null)
{
$conn = $c.DataFeedConnection.Connection
# Use regex to search and replace part of connection string
$new = $conn -replace 'ProjectName eq ''(.*)''', "ProjectName eq '$title'"
$c.DataFeedConnection.Connection = $new
Write-Host "Connection replaced."
}
}
Start-Sleep -s 5
# This will Refresh All the pivot tables data.
$excelworkbook.RefreshAll()
# The following script lines will Save the file.
Start-Sleep -s 50
$excelworkbook.Save()
$excelworkbook.Close()
}
$excel.quit()
}
$job = Start-Job -Name 'thing' -ScriptBlock $excel_update
Wait-Job -Timeout $timeout_in_sec -Job $job
if ($job.State -eq 'Running') { Stop-Job -Job $job }
Remove-Job -Job $job
Change $timeout_in_sec to whatever suites your needs.
Related
I'm trying to create an Excel workbook, then populate the cells with data found from searching many txt files.
I read a file and extract all comments AFTER I find "IDENTIFICATION DIVISION" and BEFORE I find "ENVIRONMENT DIVISION"
I then populate two cells in my excel workbook. cell one if the file and cell two is the comments extracted.
I have 256GB of memory on the work server. less than %5 is being used before Powershell throws the memory error.
Can anyone see where I'm going wrong?
Thanks,
-Ron
$excel = New-Object -ComObject excel.application
$excel.visible = $False
$workbook = $excel.Workbooks.Add()
$diskSpacewksht= $workbook.Worksheets.Item(1)
$diskSpacewksht.Name = "XXXXX_Desc"
$col1=1
$diskSpacewksht.Cells.Item(1,1) = 'Program'
$diskSpacewksht.Cells.Item(1,2) = 'Description'
$CBLFileList = Get-ChildItem -Path 'C:\XXXXX\XXXXX' -Filter '*.cbl' -File -Recurse
$Flowerbox = #()
ForEach($CBLFile in $CBLFileList) {
$treat = $false
Write-Host "Processing ... $CBLFile" -foregroundcolor green
Get-content -Path $CBLFile.FullName |
ForEach-Object {
if ($_ -match 'IDENTIFICATION DIVISION') {
# Write-Host "Match IDENTIFICATION DIVISION" -foregroundcolor green
$treat = $true
}
if ($_ -match 'ENVIRONMENT DIVISION') {
# Write-Host "Match ENVIRONMENT DIVISION" -foregroundcolor green
$col1++
$diskSpacewksht.Cells.Item($col1,1) = $CBLFile.Name
$diskSpacewksht.Cells.Item($col1,2) = [String]$Flowerbox
$Flowerbox = #()
continue
}
if ($treat) {
if ($_ -match '\*(.{62})') {
Foreach-Object {$Flowerbox += $matches[1] + "`r`n"}
$treat = $false
}
}
}
}
$excel.DisplayAlerts = 'False'
$ext=".xlsx"
$path="C:\Desc.txt"
$workbook.SaveAs($path)
$workbook.Close
$excel.DisplayAlerts = 'False'
$excel.Quit()
Not knowing what the contents of the .CBL files could be, I would suggest not to try and do all of this using an Excel COM object, but create a CSV file instead to make things a lot easier.
When finished, you can simply open that csv file in Excel.
# create a List object to collect the 'flowerbox' strings in
$Flowerbox = [System.Collections.Generic.List[string]]::new()
$treat = $false
# get a list of the .cbl files and loop through. Collect all output in variable $result
$CBLFileList = Get-ChildItem -Path 'C:\XXXXX\XXXXX' -Filter '*.cbl' -File -Recurse
$result = foreach ($CBLFile in $CBLFileList) {
Write-Host "Processing ... $($CBLFile.FullName)" -ForegroundColor Green
# using switch -File is an extremely fast way of testing a file line by line.
# instead of '-Regex' you can also do '-WildCard', but then add asterikses around the strings
switch -Regex -File $CBLFile.FullName {
'IDENTIFICATION DIVISION' {
# start collecting Flowerbox lines from here
$treat = $true
}
'ENVIRONMENT DIVISION' {
# stop colecting Flowerbox lines and output what we already have
# output an object with the two properties you need
[PsCustomObject]#{
Program = $CBLFile.Name # or $CBLFile.FullName
Description = $Flowerbox -join [environment]::NewLine
}
$Flowerbox.Clear() # empty the list for the next run
$treat = $false
}
default {
# as I have no idea what these lines may look like, I have to
# assume your regex '\*(.{62})' is correct..
if ($treat -and ($_ -match '\*(.{62})')) {
$Flowerbox.Add($Matches[1])
}
}
}
}
# now you have everything in an array of PSObjects so you can save that as Csv
$result | Export-Csv -Path 'C:\Desc.csv' -UseCulture -NoTypeInformation
Parameter -UseCulture ensures you can double-click the file so it will open correctly in your Excel
You can also create an Excel file from this csv programmatically like:
$excel = New-Object -ComObject Excel.Application
$excel.Visible = $false
$workbook = $excel.Workbooks.Open('C:\Desc.csv')
$worksheet = $workbook.Worksheets.Item(1)
$worksheet.Name = "XXXXX_Desc"
# save as .xlsx
# 51 ==> [Microsoft.Office.Interop.Excel.XlFileFormat]::xlWorkbookDefault
# see: https://learn.microsoft.com/en-us/office/vba/api/excel.xlfileformat
$workbook.SaveAs('C:\Desc.xlsx', 51)
# quit Excel and remove all used COM objects from memory
$excel.Quit()
$null = [System.Runtime.Interopservices.Marshal]::ReleaseComObject($worksheet)
$null = [System.Runtime.Interopservices.Marshal]::ReleaseComObject($workbook)
$null = [System.Runtime.Interopservices.Marshal]::ReleaseComObject($excel)
[System.GC]::Collect()
[System.GC]::WaitForPendingFinalizers()
I have the following code and it works perfectly except it's not closing Excel properly. It's leaving an Excel process running.
Is there a way to close Excel properly without killing the process?
Since i'm using other Excel files while running this script i can not kill all active Excel processes.
I think i tried everything i found online.
$WorkDir = "D:\Test\QR_ES\RG_Temp"
$BGDir = "D:\Test\QR_ES\3_BG"
$File = "D:\Test\QR_ES\4_Adr_Excel\KD_eMail.xlsx"
$SentDir = "D:\Test\QR_ES\RG_Temp\Sent\Dunning"
chdir $WorkDir
$firstPageList = Get-ChildItem "$WorkDir\1*.pdf" -File -Name
ForEach ($firstPage in $firstPageList)
{
$secondPage = "$BGDir\BG_RG.pdf"
$output = "Dunn-$firstPage"
invoke-command {pdftk $firstPage background $secondPage output $output}}
del 1*.pdf
gci $WorkDir\Dunn-*.pdf | rename-item -newname {$_.Name.Substring(5)} -Force
$Excel = New-Object -ComObject Excel.Application
$Excel.visible = $false
$Workbook = $Excel.workbooks.open($file)
$DunnList = Get-ChildItem "$WorkDir\1*.pdf" -File -Name
ForEach ($Dunn in $DunnList)
{
$Worksheets = $Workbooks.worksheets
$Worksheet = $Workbook.Worksheets.Item("KD_eMail")
$Range = $Worksheet.Range("A1").EntireColumn
$DunnSearch = $Dunn.Substring(0,5)
$SearchString = $DunnSearch
$Search = $Range.find($SearchString)
$Recipient = $Worksheet.Cells.Item($Search.Row, $Search.Column + 1)
$Msg = "<span style='font-family:Calibri;font-size:12pt;'>Test</span>"
$Outlook = New-Object -ComObject Outlook.Application
$namespace = $Outlook.GetNameSpace("MAPI")
$namespace.Logon($null, $null, $false, $true)
$EmailFrom = ('test#test.com')
$account = $outlook.Session.Accounts.Item($EmailFrom)
$Mail = $Outlook.CreateItem(0)
$Mail.HTMLBody = $Msg
$Mail.Subject = "OP - $SearchString"
$Mail.To = $Recipient
function Invoke-SetProperty {
param(
[__ComObject] $Object,
[String] $Property,
$Value
)
[Void] $Object.GetType().InvokeMember($Property,"SetProperty",$NULL,$Object,$Value)
}
Invoke-SetProperty -Object $mail -Property "SendUsingAccount" -Value $account
$Mail.Attachments.Add("$WorkDir\$Dunn")
$Mail.Save()
$Mail.close(1)
$Mail.Send()}}
$workbook.close($false)
$Excel.Quit()
chdir $WorkDir
del 1*.pdf
See this post:
https://stackoverflow.com/a/35955339/5329137
which is not accepted as an answer, but I believe is the full, correct way to close Excel.
This is what did it for me:
$FilePID = (Get-Process -name Excel | Where-Object { $_.MainWindowTitle -like 'FileName.xlsx*' }).Id
$Workbook.Save()
$Workbook.close($false)
Stop-Process $FilePID
Elaborating on #ASD's answer, since the MainWindowTitle doesn't (always) include the file suffix (.xlsx) you may have to strip that when comparing it to the filename. I'm using -replace to use a Regex match of everything before the last dot.
$excelPID = (Get-Process -name Excel | Where-Object { $_.MainWindowTitle -eq $fileName -replace '\.[^.]*$', '' }).Id
$workbook.Close()
Stop-Process $excelPID
This question already has answers here:
Excel, save and close after run
(6 answers)
Closed 4 years ago.
We're migrating databases so I'm using powershell to modify the hundreds of excel files that reference to old DB instance to the new one. This is all fine and works as intended. The problem I have is that the Excel application will not exit when I'm done. The process will just hang around as a background process and I need to go into task manager to kill it. Not a huge deal but it is annoying. Here is my script. I'm on powershell v5 and Office 2016.
param(
[string]$search_root=$(throw "missing search root parameter"),
[boolean]$test=$true
)
echo $search_root
$NAMEPOSTFIX = '-Updated'
$OLDCONN = 'Data Source=abc;'
$NEWCONN = 'Data Source=xyz;'
$filelist = Get-ChildItem -Path $search_root *.xls* -Recurse -Exclude '*Updated.*'
$Excel = New-Object -Com Excel.Application
$Excel.DisplayAlerts = $False
function update_con_xls{
param($file)
$Workbook = $Excel.Workbooks.Open($file)
foreach($con in $Workbook.Connections){
if ($con.OLEDBConnection -ne $null){
$con.OLEDBConnection.Connection = $con.OLEDBConnection.Connection.Replace($OLDCONN,$NEWCONN)
}
if ($con.ODBCConnection -ne $null){
$con.ODBCConnection.Connection = $con.ODBCConnection.Connection.Replace($OLDCONN,$NEWCONN)
}
}
$Workbook.Save()
$Workbook.saved = $true
$Excel.Workbooks.Close()
}
foreach ($file in $filelist) {
echo $file
if ($file.Extension -eq '.xls' -or $file.Extension -eq '.xlsx') {
$newfile = ($file.DirectoryName + '\' + $file.BaseName + $NAMEPOSTFIX + $file.Extension)
if($test){
echo $test
$newfile = ('C:\test\' + $file.Name) #for test run to copy coppy locally
}
Copy-Item $file.FullName -Destination $newfile
update_con_xls($newfile)
}
}
$Excel.Quit()
$Excel = $null
Maybe change:
$Excel.Workbooks.Close()
to :
$Excel= New-Object -ComObject Excel.Application;
$Workbook = $Excel.Workbooks.Open($file);
$Workbook.Save();
$Workbook.Close(); # <--- try
$Excel.Quit();
Remove-Variable -Name Excel;
I do not see anything here that could close your ODB and OLEDb connection trying to add:
$con.OLEDBConnection.Connection.Close();
$con.ODBCConnection.Connection.Close();
or
$con.OLEDBConnection.Close();
$con.ODBCConnection.Close();
after your work is finished.
So i have this script that i coded on my laptop that works just fine, the job is to combine two .csv-files into one .xls-file.
And running the script with two .csv-files containing a couple of thousand rows takes a few seconds max.
But when i try to run it on the server where it should be located, it takes... hours. I haven't done a full run, but writing one line in the .xls-file takes maybe 2-3 seconds.
So what im wondering is what is causing the huge increase in runtime. I'm monitoring the CPU-load while the script is running, and it's at 50-60% load.
The server has loads of Ram, and two CPU-core.
How can i speed this up?
The script looks like this:
$path = "C:\test\*"
$path2 = "C:\test"
$date = Get-Date -Format d
$csvs = Get-ChildItem $path -Include *.csv | Sort-Object LastAccessTime -Descending | Select-Object -First 2
$y = $csvs.Count
Write-Host "Detected the following CSV files: ($y)"
foreach ($csv in $csvs) {
Write-Host " "$csv.Name
}
$outputfilename = "regSCI " + $date
Write-Host Creating: $outputfilename
$excelapp = New-Object -ComObject Excel.Application
$excelapp.sheetsInNewWorkbook = $csvs.Count
$xlsx = $excelapp.Workbooks.Add()
$sheet = 1
$xlleft = -4131
foreach ($csv in $csvs) {
$row = 1
$column = 1
$worksheet = $xlsx.Worksheets.Item($sheet)
$worksheet.Name = $csv.Name
$worksheet.Rows.HorizontalAlignment = $xlleft
$file = (Get-Content $csv)
Write-Host Worksheet created: $worksheet.Name
foreach($line in $file) {
Write-Host Writing Line
$linecontents = $line -split ',(?!\s*\w+")'
foreach($cell in $linecontents) {
Write-Host Writing Cell
$cell1 = $cell.Trim('"')
$worksheet.Cells.Item($row, $column) = $cell1
$column++
}
$column = 1
$row++
$WorkSheet.UsedRange.Columns.Autofit() | Out-Null
}
$sheet++
$headerRange = $worksheet.Range("a1", "q1")
$headerRange.AutoFilter() | Out-Null
}
$output = $path2 + "\" + $outputfilename
Write-Host $output
$xlsx.SaveAs($output)
$excelapp.Quit()
To speed up your existing code, add these just after creating Excel object:
$excelapp.ScreenUpdating = $false
$excelapp.DisplayStatusBar = $false
$excelapp.EnableEvents = $false
$excelapp.Visible = $false
And these just before SaveAs:
$excelapp.ScreenUpdating = $true
$excelapp.DisplayStatusBar = $true
$excelapp.EnableEvents = $true
This causes excel not to render the worksheet in realtime and fire events every time you change the contets. Most probably DisplayStatusBar and ScreenUpdating doesn't matter if you make an application invisible, but I included it just in case.
Also, you're running Autofit() after every line. This certainly doesn't help with performance.
Basic script idea:
Hello. I've created a powershell script which I use to check the filesizes of certain executables, and then keep them in a text file. Next time the script runs, if a filesize differs it will replace the one in the text file with the new one.
The structure:
I have a main script and a folder which contains many scripts, each for every executable of which I want to check the filesize. So the scripts in the folder will return a string containing the link to the executable, which will be fed to the main script.
The code:
$progdir = "C:\script\programms"
$items = Get-ChildItem -filter *.ps1 -Path $progdir
$webclient = New-Object System.Net.WebClient
$filesizes = get-content C:\updatechecker\programms\filesizes
if ($filesizes.length -ne $items.length) {
if ($filesizes.length -eq $null) {
Write-Host ("Building filesize database...") -nonewline
}
else {
Write-Host ("Rebuilding filesize database...") -nonewline
}
clear-content C:\programms\filesizes
for ($i=0; $i -le $items.length-1; $i++) {
$command = "c:\programms\" + $items[$i].name
$link = & $command
$webclient.OpenRead($link) | Out-Null
$filesize = $webclient.ResponseHeaders["Content-Length"]
$filesize >> C:\programms\filesizes
}
echo "Done."
}
else {
...
Question:
This for loop is the one I want to run in parallel. I need your advice on how to do this since I'm new to powershell. I tried to implement a few things I found but they didn't work correctly (took very long to finish, output errors, multiple entries of filesizes in my filesizes file). I suspect it's a synchronization issue and somehow I need to lock the critical parts. Isn't there anything like omp parallel for in powershell? :P
Any help,advice on how to achieve this would be appreciated :)
edit:
Get-Job | Remove-Job -Force
$progdir = "C:\programms"
$items = Get-ChildItem -filter *.ps1 -Path $progdir
$webclient = New-Object System.Net.WebClient
$filesizes = get-content C:\programms\filesizes
$jobWork = {
param ($MyInput)
$command = "c:\programms\" + $MyInput
$link = & $command
$webclient.OpenRead($link) | Out-Null
$filesize = $webclient.ResponseHeaders["Content-Length"]
$filesize >> C:\programms\filesizes
}
foreach ($item in $items) {
Start-Job -ScriptBlock $jobWork -ArgumentList $item.name | out-null
}
Get-Job | Wait-Job
Get-Job | Receive-Job | Out-GridView | out-null
echo "Done."
Edit 2: Used code I found here: http://ryan.witschger.net/?p=22
$mutex = new-object -TypeName System.Threading.Mutex -ArgumentList $false, “RandomGlobalMutexName”;
$MaxThreads = 4
$SleepTimer = 500
$jobWork = {
param ($MyInput)
$webclient = New-Object System.Net.WebClient
$command = "c:\programms\" + $MyInput
$link = & $command
$webclient.OpenRead($link) | Out-Null
$result = $mutex.WaitOne();
$file = $webclient.ResponseHeaders["Content-Length"]
$file >> C:\programms\filesizes
$mutex.ReleaseMutex();
}
$progdir = "C:\programms"
$items = Get-ChildItem -filter *.ps1 -Path $progdir
$webclient = New-Object System.Net.WebClient
$filesizes = get-content C:\programms\filesizes
Get-Job | Remove-Job -Force
$i = 0
ForEach ($item in $items){
While ($(Get-Job -state running).count -ge $MaxThreads){
Start-Sleep -Milliseconds $SleepTimer
}
$i++
Start-Job -ScriptBlock $jobWork -ArgumentList $item.name | Out-Null
}
You can run each iteration of the loop in a background job which is not the same a seperate thread in that it is a whole other PowerShell.exe process. Data is passed from the background processes through serialization.
To approach it using background jobs you'll need to define a script block that will do that actual work and then call the script block with parameters in each iteration of the loop. The script block can report back status via Write-Output or by throwing an exception.
You'll probably want to throttle how many concurrent background jobs are running. Here's an example of how to throttle:
$jobItems = "a", "b", "c", "d", "e"
$jobMax = 2
$jobs = #()
$jobWork = {
param ($MyInput)
if ($MyInput -eq "d") {
throw "an example of an error"
} else {
write-output "Processed $MyInput"
}
}
foreach ($jobItem in $jobItems) {
if ($jobs.Count -le $jobMax) {
$jobs += Start-Job -ScriptBlock $jobWork -ArgumentList $jobItem
} else {
$jobs | Wait-Job -Any
}
}
$jobs | Wait-Job
As an alternative you might try eventing. Take a look at this thread for some examples of how to implement concurrency using events.
PowerShell: Runspace problem with DownloadFileAsync
You might be able to replace DownloadFileAsync with OpenReadAsync