How do i count number of occurence - python-3.x

i need to create a top 5 of interfaces that went up/down based on %LINK-3-UPDOWN from a log file.
and also need to count the amount of ICMP packets that are stopped based on amount of %SEC-6-IPACCESSLOGDP.
log file looks like this:
Sep 22 15:12:09 145.89.109.1 : %SEC-6-IPACCESSLOGP: list 120 denied tcp 80.82.77.33(0) -> 145.89.109.49(0), 1 packet
Sep 22 16:11:15 145.89.109.11 28w6d: %LINK-3-UPDOWN: Interface GigabitEthernet1/20, changed state to up
Sep 22 16:11:15 145.89.109.11 28w6d: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/20, changed state to up
Sep 22 15:16:09 145.89.109.1 : %SEC-6-IPACCESSLOGP: list 120 denied tcp 216.158.238.186(0) -> 145.89.109.49(0), 1 packet
Sep 22 15:17:10 145.89.109.1 : %SEC-6-IPACCESSLOGP: list 120 denied tcp 184.105.139.98(0) -> 145.89.109.49(0), 1 packet
Sep 22 15:22:10 145.89.109.1 : %SEC-6-IPACCESSLOGS: list 78 denied 145.89.110.15 1 packet
Sep 22 16:20:46 145.89.109.11 28w6d: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/20, changed state to down
Sep 22 16:20:46 145.89.109.11 28w6d: %LINK-3-UPDOWN: Interface GigabitEthernet1/20, changed state to down
My code is as followed but i am not getting the result i want:
infile = open("router1.log","r") #Open log bestand in "read modus"
dictionary = {} #Maak lege dictionary aan
for line in infile: #For-loop die elke regel afgaat in log-bestand
try:
naam = line.split(":")[3] #variable naam die regel split naar een lijst met index 3
naam2 = line.split(":")[4] #variable naam die regel split naar een lijst met index 4
if naam.strip()in dictionary.keys(): #"Als" naam zich bevindt in dictionary voer onderstaande uit:
dictionary[naam.strip()]+=1
else: #Anders voer onderstaan uit:
dictionary[naam.strip()]=0
except:
continue

If I had interpreted your question correctly, your problem here is that you are unable to obtain the correct count values.
To resolve that, you want set your first occurrence of a particular log issue to value 1 instead of 0.
So your else statement should be:
else:
dictionary[naam.strip()]=1
If you do not do that your counts will always be lesser by 1, hope that helps!

Related

SED style Multi address in Python?

I have an app that parses multiple Cisco show tech files. These files contain the output of multiple router commands in a structured way, let me show you an snippet of a show tech output:
`show clock`
20:20:50.771 UTC Wed Sep 07 2022
Time source is NTP
`show callhome`
callhome disabled
Callhome Information:
<SNIPET>
`show module`
Mod Ports Module-Type Model Status
--- ----- ------------------------------------- --------------------- ---------
1 52 16x10G + 32x10/25G + 4x100G Module N9K-X96136YC-R ok
2 52 16x10G + 32x10/25G + 4x100G Module N9K-X96136YC-R ok
3 52 16x10G + 32x10/25G + 4x100G Module N9K-X96136YC-R ok
4 52 16x10G + 32x10/25G + 4x100G Module N9K-X96136YC-R ok
21 0 Fabric Module N9K-C9504-FM-R ok
22 0 Fabric Module N9K-C9504-FM-R ok
23 0 Fabric Module N9K-C9504-FM-R ok
<SNIPET>
My app currently uses both SED and Python scripts to parse these files. I use SED to parse the show tech file looking for a specific command output, once I find it, I stop SED. This way I don't need to read all the file (these can get to be very big files). This is a snipet of my SED script:
sed -E -n '/`show running-config`|`show running`|`show running config`/{
p
:loop
n
p
/`show/q
b loop
}' $1/$file
As you can see I am using a multi address range in SED. My question specifically is, how can I achieve something similar in python? I have tried multiple combinations of flags: DOTALL and MULTILINE but I can't get the result I'm expecting, for example, I can get a match for the command I'm looking for, but python regex wont stop until the end of the file after the first match.
I am looking for something like this
sed -n '/`show clock`/,/`show/p'
I would like the regex match to stop parsing the file and print the results, immediately after seeing `show again , hope that makes sense and thank you all for reading me and for your help
You can use nested loops.
import re
def process_file(filename):
with open(filename) as f:
for line in f:
if re.search(r'`show running-config`|`show running`|`show running config`', line):
print(line)
for line1 in f:
print(line1)
if re.search(r'`show', line1):
return
The inner for loop will start from the next line after the one processed by the outer loop.
You can also do it with a single loop using a flag variable.
import re
def process_file(filename):
in_show = False
with open(filename) as f:
for line in f:
if re.search(r'`show running-config`|`show running`|`show running config`', line):
in_show = True
if in_show
print(line)
if re.search(r'`show', line1):
return

Can't paste value of re.findall in cell using openpyxl

I'm unable to print the result of re.findall in an Excel cell, which is the result of another re.findall which is captured from a text file with multi-line data.
Below is the section of code from the entire code where I am facing a problem.
import openpyxl,os,re # -- Import
pyro_asr920_dir = '{}\\'.format(os.getcwd()) # -- MARK DIRECTORIES
input_asr920_robot_pid_wb = openpyxl.load_workbook('Data_Workbook.xlsm', keep_vba = True) # -- OPEN EXCEL
input_asr920_robot_pid_tpd_sheet = input_asr920_robot_pid_wb['Topology-Ports_Details']
wduplinka = open(os.path.join(pyro_asr920_dir, 'DELSNACAG01C7606-logs.txt'),'r') # -- OPEN MULTILINE TEXT FILE
uplinkacontent = wduplinka.read()
PreBBa = re.findall( r'{}[\s\S]*?!\ni'.format('interface TenGigabitEthernet4/2'), uplinkacontent) # -- GET REQUIRED SUBSTRING WITH MATCH CRITERIA IN BETWEEN
print(PreBBa)
output01 = '''
['interface TenGigabitEthernet4/2\n isis metric 10 level-2\n!\ni']'''
for line in PreBBa: # - 01 > I CAN PRINT THIS ON EXCEL CELL
input_asr920_robot_pid_tpd_sheet['H27'] = line[:-1]
print(line[:-1])
print('-----------')
output02 = '''
interface TenGigabitEthernet4/2
isis metric 10 level-2
!'''
# ----------------------------------------------------------------- UNABLE TO GET VALUES IN CELL
for line in PreBBa: # - 02 > I CAN'T PRINT THIS ON EXCEL CELL {THIS IS WHERE I AM STUCK}
if 'ospf' in line:
theOSPF = re.findall(r'{}[\s\S]*?{}'.format(' ip ospf','\n c'), line)
input_asr920_robot_pid_tpd_sheet['C47'] = 'Yes'
else:
input_asr920_robot_pid_tpd_sheet['C47'] = 'No' # UNABLE TO GETRESULT IN EXCEL
output03 = '''No'''
# -----------------------------------------------------------------
metric = re.findall(r'{}[\s\S]*?{}'.format('metric ',' '), str(line))
metric = re.findall(r'\d+',str(metric))
input_asr920_robot_pid_tpd_sheet['C46'].value = metric[0] # UNABLE TO GETRESULT IN EXCEL
print(metric)
output04 = '''10'''
# -----------------------------------------------------------------
input_asr920_robot_pid_wb.save('Data_Workbook.xlsm')
wduplinka.close()
input_asr920_robot_pid_wb.close()
print('TEST COMPLETED')
Some content of Text file is as below:
DELSNACAG01C7606#sh running-config
Load for five secs: 18%/1%; one minute: 26%; five minutes: 26%
Time source is NTP, 13:43:23.718 IST Fri Aug 16 2019
Building configuration...
Current configuration : 228452 bytes
!
! Last configuration change at 21:15:56 IST Thu Aug 15 2019
! NVRAM config last updated at 06:42:52 IST Sat Aug 10 2019 by cor382499
!
interface TenGigabitEthernet4/2
isis metric 10 level-2
!
interface TenGigabitEthernet4/3
!
end

python3: Counting repeated occurrence in a list

Each line contains a special timestamp, the caller number, the receiver number, the duration of the call in seconds and the rate per minute in cents at which this call was charged, all separated by ";”. The file contains thousands of calls looks like this. I created a list instead of a dictionary to access the elements but I'm not sure how to count the number of calls originating from the phone in question
timestamp;caller;receiver;duration;rate per minute
1419121426;7808907654;7807890123;184;0.34
1419122593;7803214567;7801236789;46;0.37
1419122890;7808907654;7809876543;225;0.31
1419122967;7801234567;7808907654;419;0.34
1419123462;7804922860;7809876543;782;0.29
1419123914;7804321098;7801234567;919;0.34
1419125766;7807890123;7808907654;176;0.41
1419127316;7809876543;7804321098;471;0.31
Phone number || # |Duration | Due |
+--------------+-----------------------
|(780) 123 4567||384|55h07m53s|$ 876.97|
|(780) 123 6789||132|17h53m19s|$ 288.81|
|(780) 321 4567||363|49h52m12s|$ 827.48|
|(780) 432 1098||112|16h05m09s|$ 259.66|
|(780) 492 2860||502|69h27m48s|$1160.52|
|(780) 789 0123||259|35h56m10s|$ 596.94|
|(780) 876 5432||129|17h22m32s|$ 288.56|
|(780) 890 7654||245|33h48m46s|$ 539.41|
|(780) 987 6543||374|52h50m11s|$ 883.72|
list =[i.strip().split(";") for i in open("calls.txt", "r")]
print(list)
I have very simple solution for your issue:
First of all use with when opening file - it's a handy shortcut and it provides sames functionality as wrap this funtion into try...except. Consider this:
lines = []
with open("test.txt", "r") as f:
for line in f.readlines():
lines.append(line.strip().split(";"))
print(lines)
counters = {}
# you browse through lists and later through numbers inside lists
for line in lines:
for number in line:
# very basic way to count occurences
if number not in counters:
counters[number] = 1
else:
counters[number] += 1
# in this condition you can tell what number of digits you accept
counters = {elem: counters[elem] for elem in counters.keys() if len(elem) > 5}
print(counters)
This should get you started
import csv
import collections
Call = collections.namedtuple("Call", "duration rate time")
calls = {}
with open('path/to/file') as infile:
for time, nofrom, noto, dur, rate in csv.reader(infile):
calls.get(nofrom, {}).get(noto,[]).append(Call(dur, rate, time))
for nofrom, logs in calls.items():
for noto, callist in logs.items():
print(nofrom, "called", noto, len(callist), "times")

Search multiline error log for error code and then some of it's parameters on Linux

What command would give me the output I need for each instance of an error code in a very large log file? The file has records marked by a begin and end with number of characters. Such as:
SR 120
1414760452 0 1 Fri Oct 31 13:00:52 2014 2218714 4
GROVEMR2 scn
../SrxParamIF.m 284
New Exam Started
EN 120
The 5th field is the error code, 2218714 in previous example.
I thought of just grep'ing for the error code and outputting -A lines afterwards; then picking what I needed from that rather than parsing the entire file. That seems easy but my grep/awk/sed usage isn't to that level.
ONLY when error 2274021 is encountered as in the following example I'd like some output as shown.
Show me output such as: egrep ‘Coil:|Connector:|Channels faulted:| First channel:’ ERRORLOG|less
Part of input file of interest:
Mon Nov 24 13:43:37 2014 2274021 1
AWHMRGE3T NSP
SCP:RfHubCanHWO::RfBias 4101
^MException Class: Unknown Severity: Unknown
Function: RF: RF Bias
PSD: VIBRANT Coil: Breast SMI Scan: 1106/14
Coil Fault - Short Circuit
A multicoil bias fault was detected.
.
Connector: Port 1 (P1)
Channels faulted: 0x200
First channel: 10 of 32, counting from 1
Fault value: -2499 mV, Channel: 10->
Output:
Coil: Breast SMI
Connector: Port 1 (P1)
Channels faulted: 0x200
First channel: 10 of 32, counting from 1
Thanks in advance for any pointers!
Try the following (with the convenient adaptations)
#!/usr/bin/perl
use strict;
$/="\nEN "; # register separated by "\nEN "
my $error=2274021; # the error!
while(<>){ # for all registers
next unless /\b$error\b/; # ignore unless error
for my $line ( split(/\n/,$_)){
print "$line\n" if ($line =~ /Coil:|Connector:|Channels faulted:|First channel:/);
}
print "====\n"
}
Is this what you need?

How to grep all the data in a specified col from standard output?

Now i have a standard output which is printed by my program and displayed on the screen. The outputs looks like the following part:
Parsing command line string 'InputFile = foreman.qcif'.
Parsing command line string 'NumberReferenceFrames = 1'.
Parsing command line string 'QPISlice = 24'.
Parsing command line string 'QPPSlice = 24'.
------------------------------- JM 11.0 (FRExt) --------------------------------
Input YUV file : foreman.qcif
Output H.264 bitstream : test.264
Output YUV file : test_rec.yuv
YUV Format : YUV 4:2:0
Frames to be encoded I-P/B : 150/0
PicInterlace / MbInterlace : 0/0
Transform8x8Mode : 1
-------------------------------------------------------------------------------
Frame Bit/pic QP SnrY SnrU SnrV Time(ms) MET(ms) Frm/Fld Ref
-------------------------------------------------------------------------------
0000(NVB) 168
0000(IDR) 34280 24 39.724 41.720 43.998 286 0 FRM 1
0002(P) 7432 24 38.857 41.565 43.829 402 99 FRM 1
0004(P) 8976 24 38.642 41.275 43.698 409 97 FRM 1
0006(P) 8344 24 38.427 41.266 43.515 407 99 FRM 1
0008(P) 8224 24 38.609 41.082 43.524 413 94 FRM 1
0010(P) 7784 24 38.655 40.991 43.235 406 95 FRM 1
0012(P) 7136 24 38.534 40.687 43.273 411 95 FRM 1
0014(P) 6688 24 38.464 40.756 43.146 410 92 FRM 1
0016(P) 7720 24 38.516 40.585 42.851 410 91 FRM 1
0018(P) 6864 24 38.474 40.631 42.958 411 101 FRM 1
0020(P) 8392 24 38.433 40.607 42.646 415 99 FRM 1
0022(P) 9744 24 38.371 40.554 42.498 416 94 FRM 1
0024(P) 8368 24 38.362 40.531 42.380 417 93 FRM 1
0026(P) 7904 24 38.414 40.586 42.415 418 95 FRM 1
0028(P) 8688 24 38.403 40.523 42.366 418 96 FRM 1
0030(P) 9128 24 38.545 40.390 42.661 416 89 FRM 1
0032(P) 9664 24 38.399 40.538 42.740 413 88 FRM 1
0034(P) 8928 24 38.394 40.590 42.852 414 95 FRM 1
0036(P) 10024 24 38.423 40.562 42.697 415 92 FRM 1
0038(P) 9320 24 38.442 40.389 42.689 414 94 FRM 1
0040(P) 7304 24 38.404 40.487 42.884 410 90 FRM 1
0042(P) 8560 24 38.447 40.590 42.673 411 95 FRM 1
.......
Now I only need to process the 4th col of SnrY.
So, my question is how to grep this rol and to store them in a data.txt file?
Then I can use plot tool or Matlab to plot the data trends with these data..
Any advices? Many thanks for your kind help!
Addition:
Since it needs to pick those data from standard output, do I need to use(add) the command provided by you in the command line to make it works during outputs those data
?
$ awk '/FRM/ { print $4 }' < in.txt > data.txt
Note: I'm just guessing that the interesting lines contain FRM. If this is not the case, you must come up with a different way to identify them.
tail -n+17 in.txt | cut -c 23-31
./cmd_that_generates_results | awk '/^[0-9]/{print $4}' > data.txt
Quick breakdown of awk statement:
/^[0-9]/ : match lines that start with a number
{print $4} : print out fourth column
(Updated): to address question "Since it needs to pick those data from standard output, do I need to use(add) the command provided by you in the command line to make it works during outputs those data ?"
You can use the commands given in answers by piping to them the standard output of your program using the pipe (|) command. The results can then be store into "data.txt" using >.
see example above. Similar techniques can be used for other solutions in this page.
$ ./command | awk 'BEGIN{RS="--+";FS="\n"} END{ for(i=1;i<=NF;i++){if($i ~ /^[0-9]/){m=split($i,a," ");if(a[4]) print a[4]}}}'
39.724
38.857
38.642
38.427
38.609
38.655
38.534
38.464
38.516
38.474
38.433
38.371
38.362
38.414
38.403
38.545
38.399
38.394
38.423
38.442
38.404
38.447
I am anticipating that the data you want to get
need not be after the 16th line, may be after the 17th , or 18th ,etc
or that FRM/Fld field may have different indicators,
or the 4th field may not be at the exact character position every time.
So the above just say: set record separator as the line with dashes "-" , so the last record is the whole data you need. then using the newline as field separator (FS="\n") , each field will be each line of data. Split them up with spaces and get element 4 of the splitted line. that will be your desired column.
This will match numbers like 38.4* in the fourth column:
grep -E '^([^ ]+[ ]+){3}38.4'

Resources