Count occurrence of strings in a log.file with PowerShell - string

I need to get the occurrence of "failure" in a logfile.
The thing is I need to get the occurrence of "failure" for each session block.
What the log looks like:
---Session 1
check
check
failure
failure
check
----Session 2
check
failure
check
check
----Session 3
failure
failure
What I've got so far is this:
$rows = Get-Childitem -Path E:\shell\lot.log |
Select-String -Pattern failure
$i = 0
foreach ($row in $rows) {
$i++
}
echo $i
With that script I only get the total of the occurrences.

I would start a new counter whenever a line beginning with 3 or more consecutive hyphens occurs and collect the results in a hashtable.
$failcount = #{}
Get-Content 'E:\shell\lot.log' | ForEach-Object {
if ($_ -match '^-{3,}(.*)') {
$section = $matches[1].Trim()
$failcount[$section] = 0
} elseif ($_ -like '*failure*') {
$failcount[$section]++
}
}

I believe this will do it. Key part is to reset your counter after each session.
$rows = Get-Childitem -Path E:\shell\lot.log
$i = 0 # failure counter
$j = 1 # session counter
foreach($row in $rows){
if($row -like "*Session 1"){
# skip the first line. Edit: removed * as would skip session 10, 11 etc. assumes line ends with "session 1"
continue
}elseif($row -eq "failure){
# if the row is equal to failure, count it
$i++
}elseif($row -like "*Session*"){
# when you get to the end of a session, print the number of failures
Write-Host "Session $j had $i failures"
# ... and reset the failure counter
$i = 0
# increment the session counter
$j++
}
}

I'll add another option. Read the whole log file in with -Raw to get a multi-line string. Remove the ---- from the first line, and then split on 3 or more hyphens at the beginning of a line, this gets you each session as a multi-line string, then you could just output text or custom objects, or whatever you wanted with it. Split the multi-line string on new line characters, filter for 'failure', and do a count to get failures per session.
(GC E:\shell\lot.log -Raw) -replace '^-+' -split '(?<=[\r\n])---+'|%{
'{0} had {1} failure(s)' -f ($_.split("`n")[0].Trim()),($_ -split '[\r\n]+'|?{$_ -match 'failure'}).Count
}
That would (given the sample provided) output:
Session 1 had 2 failure(s)
Session 2 had 1 failure(s)
Session 3 had 2 failure(s)

Related

Powershell script to extract data from multiple text files into an excel spreadsheet

I'm pretty new to PS and been struggling for a few days.
I have multiple text files in a folder with specific data that I would like to extract into an excel spreadsheet.
each files look like this :
Client n° : xxx Client name : xxx
Computer status
pc group 1 :
n°1 OK n°2 Disconnected n°3 Unresponsive
n°4 Unreachable host n°5 Unresponsive
Data read 11/11/20 12:50:07
Version: x.x.x
I would like to have an output file that looks like this :
Client name and n° OK Disconnected Unresponsive Unreachable host version
xxx/xxx 1 1 2 1 x.x.x
For the status columns it's the sum number of pc with that status and not the pc n° that I would like to display.
At the moment I'm working with multiple .bat files that searches for the status and output one file per status
find /c "Disconnected" *.* > disconnected.txt
find /c "Unresponsive" *.* > unresponsive.txt
And then I sort every single output in an excel which takes me too much time, I was wondering if it was possible to automate this task with a script.
I really don't have any knowledge of PS, only basic batch commands.
Let's assume your files are all in one folder and all of them have the .txt extension.
Then you need to loop through these files and parse the data you need from it:
# create a Hashtable to add the different status values in
$status = #{'OK' = 0; 'Disconnected'= 0; 'Unresponsive' = 0; 'Unreachable host'= 0}
# loop through the files in your path and parse the information out
$result = Get-ChildItem -Path 'D:\Test' -Filter '*.txt' -File | ForEach-Object {
switch -Regex -File $_.FullName {
'^Client n°\s*:\s*([^\s]+)\s+Client name\s*:\s*(.+)$' {
# start collecting data for this client
$client = '{0}/{1}' -f $matches[2], $matches[1]
# reset the Hashtable to keep track of the status values
$status = #{'OK' = 0; 'Disconnected'= 0; 'Unresponsive' = 0; 'Unreachable host'= 0 }
}
'^\d+' {
# increment the various statuses in the Hahstable
($_ -split '\d+').Trim() | ForEach-Object { $status[$_]++ }
}
'^Version:\s(.+)$' {
$version = $matches[1]
# since this is the last line for this client, output the collected data as object
[PsCustomObject]#{
'Client name and n°' = $client
'OK' = $status['OK']
'Disconnected' = $status['Disconnected']
'Unresponsive' = $status['Unresponsive']
'Unreachable host' = $status['Unreachable host']
'Version' = $version
}
}
}
}
# output on screen
$result | Format-Table -AutoSize
# output to CSV file
$result | Export-Csv -Path 'D:\Test\clientdata.csv' -UseCulture -NoTypeInformation
Result on screen:
Client name and n° OK Disconnected Unresponsive Unreachable host Version
------------------ -- ------------ ------------ ---------------- -------
xxx/xxx 1 1 2 1 x.x.x
I used this as an exercise to test my abilities. I created three of the same files, with different data, and tested this script. As long as they are text files in the directory the script will iterate through each file and pull the data from each as you stated it needs to be. If a stray text file gets added the script does not know nor care and will treat it like the others. If there is data it can find it will, and it will output that data to the excel file. Lastly the file is set to save itself and then immediately close.
It starts by Creating the Excel file, then Workbook. (I commented out the naming of the workbook. If you like you can add it back.) Finds all text files in a directory, then searches the text for the specific content within the text you specified above.
During the script I commented as much as I thought might be needed to assist with modification later on.
Output formatted like this:
Excel Output
#Create An Excel File
$excel = New-Object -ComObject excel.application
$excel.visible = $True
#Add Workbook
$workbook = $excel.Workbooks.Add()
<#Rename Workbook
$workbook= $workbook.Worksheets.Item(1)
$workbook.Name = 'Client name and #'#>
#create the column headers
$workbook.Cells.Item(1,1) = 'Client name and n°'
$workbook.Cells.Item(1,2) = 'OK'
$workbook.Cells.Item(1,3) = 'Disconnected'
$workbook.Cells.Item(1,4) = 'Unresponsive'
$workbook.Cells.Item(1,5) = 'Unreachable'
$workbook.Cells.Item(1,6) = 'Version'
$workbook.Cells.Item(1,7) = 'Date Gathered'
$move = "C:\Users\iNet\Desktop\Testing"
$root = "C:\Users\iNet\Desktop\Testing"
$files = Get-ChildItem -Path $root -Filter *.txt
#Starting on Row 2
[int]$i = 2
ForEach ($file in $files){
$location = $root+"\"+$file
#Format your client data to output what you want to see.
$ClientData = select-string -path "$location" -pattern "Client"
$ClientData = $ClientData.line
$ClientData = $ClientData -replace "Client n° :" -replace ""
$ClientData = $ClientData -replace "Client name :" -replace "|"
$row = $i
$Column = 1
$workbook.Cells.Item($row,$column)= "$ClientData"
#Data Read Date
$DataReadDate = select-string -path "$location" -pattern "Data read"
$DataReadDate = $DataReadDate.line
$DataReadDate = $DataReadDate -replace "Data read " -replace ""
#Data Read Date, you asked for everything but this.
$row = $i
$Column = 7
$workbook.Cells.Item($row,$column)= "$DataReadDate"
#Version
$Version = select-string -path "$location" -pattern "Version:"
$Version = $Version.line
$Version = $Version -replace "Version: " -replace ""
$row = $i
$Column = 6
$workbook.Cells.Item($row,$column)= "$Version"
#How Many Times Unresponsive Shows Up
$Unresponsive = (Get-Content "$location" | select-string -pattern "Unresponsive").length
$row = $i
$Column = 4
$workbook.Cells.Item($row,$column)= "$Unresponsive"
#How Many Times Disconnected Shows Up
$Disconnected = (Get-Content "$location" | select-string -pattern "Disconnected").length
$row = $i
$Column = 3
$workbook.Cells.Item($row,$column)= "$Disconnected"
#How Many Times Unreachable host Shows Up
$Unreachable = (Get-Content "$location" | select-string -pattern "Unreachable host").length
$row = $i
$Column = 5
$workbook.Cells.Item($row,$column)= "$Unreachable"
#How Many Times OK Shows Up
$OK = (Get-Content "$location" | select-string -pattern "OK").length
$row = $i
$Column = 2
$workbook.Cells.Item($row,$column)= "$OK"
#Iterate by one so each text file goes to its own line.
$i++
}
#Save Document
$output = "\Output.xlsx"
$FinalOutput = $move+$output
#saving & closing the file
$workbook.SaveAs($move)
$excel.Quit()

How do I check a string exist in a file using PowerShell?

I have a 1st text file looks like this : 12AB34.US. The second text file is CD 34 EF.
I want to find my 2nd text file exist or not in the 1st text file.
I tried to cut 3 characters last in the first text file (.US). Then I split to each 2 characters (because the 2nd text file consist of 2 characters). Then, I tried this code, and it always return "Not Found".
$String = Get-Content "C:\Users\te2.txt"
$Data = Get-Content "C:\Users\Fixed.txt"
$Split = $Data -split '(..)'
$Cut = $String.Substring(0,6)
$String_Split = $Cut -split '(..)'
$String_Split
$Check= $String_Split | %{$_ -match $Split}
if ($Check-contains $true) {
Write-Host "0"
} else {
Write-Host "1"
}
There are a number of problems with your current approach.
The 2-char groups don't align:
# strings split into groups of two
'12' 'AB' '34' # first string
'CD' ' 3' '4 ' # second string
When you test multiple strings with -match, you need to
escape the input string to avoid matchings on meta characters (like .), and
place the collection on the left-hand side of the operator, the pattern on the right:
$Compare = $FBString_Split | % {$Data_Split -match [regex]::Escape($_)}
if ($Compare -contains $true) {
Write-Host "Found"
} else {
Write-Host "Not Found"
}
For a more general solution to find out if any substring of N chars of one string is also a substring of another, you could probably do something like this instead:
$a = '12AB34.US'
$b = 'CD 34 EF'
# we want to test all substrings of length 2
$n = 2
$possibleSubstrings = 0..($n - 1) | ForEach-Object {
# grab substrings of length $n at every offset from 0 to $n
$a.Substring($_) -split "($('.'*$n))" | Where-Object Length -eq $n |ForEach-Object {
# escape the substring for later use with `-match`
[regex]::Escape($_)
}
} |Sort-Object -Unique
# We can construct a single regex pattern for all possible substrings:
$pattern = $possibleSubstrings -join '|'
# And finally we test if it matches
if($b -match $pattern){
Write-Host "Found!"
}
else {
Write-Host "Not found!"
}
This approach will give you the correct answer, but it'll become extremely slow on large inputs, at which point you may want to look at non-regex based strategies like Boyer-Moore

Return duplicate names (including partial matches)

Excel guy here that occasionally turns to automating powershell via vba.
I tried to solve https://stackoverflow.com/q/36538022/641067 (now closed) and couldn't get there with my basic powershell knowledge and googlefu alone.
In essence the problem the OP presented is:
There are a list of names in a text file.
Aim is to capture only those names that occurr at least once (so discard unique names, see point (3)).
Names occurring at least once include partial matches, ie Will and William can be considered duplicates and should be retained. Whereas Bill is not a duplicate of William.
I tried various approaches including
Group
Compare-Object see example below
But I was stymied by part (3). I suspect that a loop is required to do this but am curious whether there is a direct Powershellapproach,
Looking forward to hearing from the experts.
what I tried
$a = Get-Content "c:\temp\in.txt"
$b = $a | select -unique
[regex] $a_regex = ‘(?i)(‘ + (($a |foreach {[regex]::escape($_)}) –join “|”) + ‘)’
$c = $b -match $a_regex
Compare-object –referenceobject $c -IncludeEqual $a
Following testscript using a loop would work for the rules you outlined and looks foolproof to me
$t = ('first', 'will', 'william', 'williamlong', 'unique', 'lieve', 'lieven')
$s = $t | sort-object
[String[]]$r = #()
$i = 0;
while ($i -lt $s.Count - 1) {
if ($s[$i+1].StartsWith($s[$i])) {
$r += $s[$i]
$r += $s[$i+1]
}
$i++
}
$r | Sort-Object -Unique
and following testscript using a regex might get you started.
$content = "nomatch`nevenmatch1`nevenmatch12`nunevenmatch1`nunevenmatch12`nunevenmatch123"
$string = (($content.Split("`n") | Sort-Object -Unique) -join "`n")
$regex = [regex] '(?im)^(\w+)(\n\1\w+)+'
$matchdetails = $regex.Match($string)
while ($matchdetails.Success) {
$matchdetails.Value
$matchdetails = $matchdetails.NextMatch()
}

PowerShell script to monitor IIS logs for 500 errors every 10 minutes

I'm trying to set up a script to monitor IIS 7.5 logs fro 500 errors. Now I can get it to do that OK but I would like it to check every 30 minutes. Quite naturally I don't want it to warn me about the previous 500 errors it has already reported.
As you can see from the script below I have added a $time variable to take this into account, however I can't seem to find a way to use this variable. Any help would be appreciated.
#Set Time Variable -30
$time = (Get-Date -Format hh:mm:ss (Get-Date).addminutes(-30))
# Location of IIS LogFile
$File = "C:\Users\here\Documents\IIS-log\"+"u_ex"+(get-date).ToString("yyMMdd")+".log"
# Get-Content gets the file, pipe to Where-Object and skip the first 3 lines.
$Log = Get-Content $File | where {$_ -notLike "#[D,S-V]*" }
# Replace unwanted text in the line containing the columns.
$Columns = (($Log[0].TrimEnd()) -replace "#Fields: ", "" -replace "-","" -replace "\(","" -replace "\)","").Split(" ")
# Count available Columns, used later
$Count = $Columns.Length
# Strip out the other rows that contain the header (happens on iisreset)
$Rows = $Log | where {$_ -like "*500 0 0*"}
# Create an instance of a System.Data.DataTable
#Set-Variable -Name IISLog -Scope Global
$IISLog = New-Object System.Data.DataTable "IISLog"
# Loop through each Column, create a new column through Data.DataColumn and add it to the DataTable
foreach ($Column in $Columns) {
$NewColumn = New-Object System.Data.DataColumn $Column, ([string])
$IISLog.Columns.Add($NewColumn)
}
# Loop Through each Row and add the Rows.
foreach ($Row in $Rows) {
$Row = $Row.Split(" ")
$AddRow = $IISLog.newrow()
for($i=0;$i -lt $Count; $i++) {
$ColumnName = $Columns[$i]
$AddRow.$ColumnName = $Row[$i]
}
$IISLog.Rows.Add($AddRow)
}
$IISLog | select time,csuristem,scstatus
OK With KevinD's help and PowerGUI with a fair bit of trial and error, I got it working as I expected. Here's the finished product.
#Set Time Variable -30
$time = (Get-Date -Format "HH:mm:ss"(Get-Date).addminutes(-30))
# Location of IIS LogFile
$File = "C:\Users\here\Documents\IIS-log\"+"u_ex"+(get-date).ToString("yyMMdd")+".log"
# Get-Content gets the file, pipe to Where-Object and skip the first 3 lines.
$Log = Get-Content $File | where {$_ -notLike "#[D,S-V]*" }
# Replace unwanted text in the line containing the columns.
$Columns = (($Log[0].TrimEnd()) -replace "#Fields: ", "" -replace "-","" -replace "\(","" -replace "\)","").Split(" ")
# Count available Columns, used later
$Count = $Columns.Length
# Strip out the other rows that contain the header (happens on iisreset)
$Rows = $Log | where {$_ -like "*500 0 0*"}
# Create an instance of a System.Data.DataTable
#Set-Variable -Name IISLog -Scope Global
$IISLog = New-Object System.Data.DataTable "IISLog"
# Loop through each Column, create a new column through Data.DataColumn and add it to the DataTable
foreach ($Column in $Columns) {
$NewColumn = New-Object System.Data.DataColumn $Column, ([string])
$IISLog.Columns.Add($NewColumn)
}
# Loop Through each Row and add the Rows.
foreach ($Row in $Rows) {
$Row = $Row.Split(" ")
$AddRow = $IISLog.newrow()
for($i=0;$i -lt $Count; $i++) {
$ColumnName = $Columns[$i]
$AddRow.$ColumnName = $Row[$i]
}
$IISLog.Rows.Add($AddRow)
}
$IISLog | select #{n="Time"; e={Get-Date -Format "HH:mm:ss"("$($_.time)")}},csuristem,scstatus | ? { $_.time -ge $time }
Thanks again Kev you're a good man. Hope this code helps someone else out there.
Here's
Try changing your last line to:
$IISLog | select #{n="DateTime"; e={Get-Date ("$($_.date) $($_.time)")}},csuristem,scstatus | ? { $_.DateTime -ge $time }
In the select, we're concatenating the date and time fields, and converting them to a date object, then selecting rows where this field is greater than your $time variable.
You'll also need to change your $time variable:
$time = (Get-Date).AddMinutes(-30)
You want a DateTime object here, not a string.

Replacing value on one line in a text file

I am currently working on editing one line of a text file. When I try to overwrite the text file, I only get one line back in the text file. I am trying to call the function with
modifyconfig "test" "100"
config.txt:
check=0
test=1
modifyConfig() function:
Function modifyConfig ([string]$key, [int]$value){
$path = "D:\RenameScript\config.txt"
((Get-Content $path) | ForEach-Object {
Write-Host $_
# If '=' is found, check key
if ($_.Contains("=")){
# If key matches, replace old value with new value and break out of loop
$pos = $_.IndexOf("=")
$checkKey = $_.Substring(0, $pos)
if ($checkKey -eq $key){
$oldValue = $_.Substring($pos+1)
Write-Host 'Key: ' $checkKey
Write-Host 'Old Value: ' $oldValue
$_.replace($oldValue,$value)
Write-Host "Result:" $_
}
} else {
# Do nothing
}
}) | Set-Content ($path)
}
The result I receive in my config.txt:
test=100
I am missing "check=0".
What have I missed?
$_.replace($oldValue,$value) in your innermost conditional replaces $oldValue with $value and then prints the modified string, but you don't have code printing non-matching strings. Because of that only the modified string are written back to $path.
Replace the line
# Do nothing
with
$_
and also add an else branch with a $_ to the inner conditional.
Or you could assign $_ to another variable and modify your code like this:
Foreach-Object {
$line = $_
if ($line -like "*=*") {
$arr = $line -split "=", 2
if ($arr[0].Trim() -eq $key) {
$arr[1] = $value
$line = $arr -join "="
}
}
$line
}
or a one liner.. (not exactly pin pointed answer, but to the question title)
(get-content $influxconf | foreach-object {$_ -replace "# auth-enabled = false" , "auth-enabled = true" }) | Set-Content $influxconf

Resources