Regex multiline - Python3 - Match everything inside curly braces [duplicate] - python-3.x

This question already has an answer here:
Regular expression works on regex101.com, but not on prod
(1 answer)
Closed 2 years ago.
I tried this code but it doesn't work; I cannot catch anything. I need to get a multiline match and have been working 3 days on it now. Thanks for your help!!
My regex:
print(re.findall(r'^ltm\s+pool\s+/Common/[0-9-A-Z_.-]+\s+\{([\s\S]*?)^\}',file.read(), re.MULTILINE))
print(re.findall(r'^ltm\s+pool\s+/Common/[0-9-A-Z_.-]+\s+\{(.*?)^\}',file.read(), re.DOTALL))
My code:
#!/usr/bin/env python3
import re, os, sys
### We create a new file
f = open("bigip.txt", "w")
### Default stdout value copied to a variable
orig_stdout = sys.stdout
### Stdout transfered to a file in write mode
sys.stdout = open("bigip.txt", "w")
file = open("bigiptemp", "r")
#for line in file:
#if re.findall(r'^ltm\spool\s\/Common\/([A-Z-a-z]+)', line):
#print(line)
print(re.findall(r'^ltm\s+pool\s+/Common/[0-9-A-Z_.-]+\s+\{([\s\S]*?)^\}',file.read(), re.MULTILINE))
### Default stdout reset
sys.stdout = orig_stdout
The file below is an extract:
ltm pool /Common/GEOG.GD {
members {
/Common/
address
}
/Common/
address
}
monitor
}
ltm pool /Common/HAP_NAODE_DEV {
members {
/Common
address
}
/Common
address
}
}
monitor
}
The expected behavior is the following but I cannot share the content of bigiptemp ... But my previous answer was tagged as duplicate ... Regular expression works on regex101.com, but not on prod
Expected result

try this
#!/usr/bin/env python3
import re, os, sys
### We create a new file
f = open("bigip.txt", "w")
### Default stdout value copied to a variable
orig_stdout = sys.stdout
### Stdout transfered to a file in write mode
sys.stdout = open("bigip.txt", "w")
file = open("bigiptemp", "r")
#for line in file:
#if re.findall(r'^ltm\spool\s\/Common\/([A-Z-a-z]+)', line):
#print(line)
print(*re.findall(r'^ltm\s+pool\s+/Common/[0-9-A-Z_.-]+\s+\{([\s\S]*?)^\}',file.read(), re.MULTILINE))
### Default stdout reset
sys.stdout = orig_stdout
it seems work and match excepted results

My solution is this (three days to approximately understand how regex works):
regex = r'(^ltm\s+pool\s+/Common/[0-9-A-Z_.-]+\s+\{[\s\S]*?^\}\n?)'
print(*(re.findall(regex, file.read(), re.MULTILINE)))
ltm pool /Common/GEOG.GD {
members {
/Common/
address
}
/Common/
address
}
monitor
}
ltm pool /Common/HAP_NAODE_DEV {
members {
/Common
address
}
/Common
address
}
}
monitor
}
Never use like me 'for line in file' if you want to catch several lines in a grasp (one-shot) ... logic!!!
A very good regex-topic book for the insomniacs ... lol:
=> https://www.princeton.edu/~mlovett/reference/Regular-Expressions.pdf
Thanks to you all!!
Thanks to #Wiktor Stribiżew
#gueug

Related

Python Error: Cannot extract data from dictionary

I have a json output (y) like this below.
{
"WebACL":{
"Name":"aBlockKnownBadInputs-WebAcl",
"Id":"4312a5d0-9878-4feb-a083-09d7a9cfcfbb",
"ARN":"arn:aws:wafv2:us-east-1:100467320728:regional/webacl/aBlockKnownBadInputs-WebAcl/4312a5d0-9878-4feb-a083-09d7a9cfcfbb",
"DefaultAction":{
"Allow":{
}
},
"Description":"",
"Rules":[
{
"Name":"AWS-AWSManagedRulesKnownBadInputsRuleSet",
"Priority":500,
"Statement":{
"ManagedRuleGroupStatement":{
"VendorName":"AWS",
"Name":"AWSManagedRulesKnownBadInputsRuleSet"
}
},
"OverrideAction":{
"None":{
}
},
"VisibilityConfig":{
"SampledRequestsEnabled":true,
"CloudWatchMetricsEnabled":true,
"MetricName":"AWS-AWSManagedRulesKnownBadInputsRuleSet"
}
}
]
}
}
I want to extract "AWS-AWSManagedRulesKnownBadInputsRuleSet" from the section:-
"Name":"AWS-AWSManagedRulesKnownBadInputsRuleSet",
"Priority":500,
"Statement":{
"ManagedRuleGroupStatement":{
"VendorName":"AWS",
"Name":"AWSManagedRulesKnownBadInputsRuleSet"*
At the minute my code is returning a key error:
KeyError: 'Rules[].Statement[].ManagedRuleGroupStatement[].Name'
The format of this line is clearly wrong, but I don't know why.
ruleset = y['Rules[].Statement[].ManagedRuleGroupStatement[].Name']
My code block:
respons = client.get_web_acl(Name=(acl),Scope='REGIONAL',Id=(ids))
for y in response['WebACLs']:
ruleset = y['Rules[].Statement[].ManagedRuleGroupStatement[].Name']
Does anyone know what I'm doing wrong here?
UPDATE :- In case anyone else comes up against this problem, I fixed this by doing it a slightly different way.
#Requesting the info from AWS via get_web_acl request
respons = client.get_web_acl(Name=(acl),Scope='REGIONAL',Id=(ids))
#Converting the dict output to a string to make it searchable
result = json.dumps(respons)
#Defining what I want to search for
fullstring = "AWS-AWSManagedRulesKnownBadInputsRuleSet"
#Searching the output & printing the result: if = true / else = false
if fullstring in result:
print("Found WAF ruleset: AWS-AWSManagedRulesKnownBadInputsRuleSet!")
else:
print("WAF ruleset not found!")
Also, as part of my research I found a cool thing called jello.
(https://github.com/kellyjonbrazil/jello).
jello is similar to jq in that it processes JSON and JSON Lines data except it uses standard python dict and list syntax.
So, I copied my json into a file called file.json
Ran cat file.json | jello -s to print a grep-able schema by using the -s option
Found the bit I was interested in - in my case the name of the ManagedRuleGroupStatement and ran the following:
cat file.json | jello -s _.WebACL.Rules[0].Statement.ManagedRuleGroupStatement.Name
This gave me exactly what I wanted!
It doesn't work inside a python script but was something new and interesting to learn!

How to open syslog files in Python

I am trying an assignment to open a syslog for a server program (called ticky) that creates logs and errors and then assign the errors to a dictionary and export to a csv file to sort and host to a webpage,
I am unsure of how to access syslog files, as the course only went into sys.argv and I don't know if this can be used or if I need to figure out how to use syslog module. Once the log is opened, the regex will pull the error message and add it to the dictionary, either creating a new entry or adding value to an existing key.
Am I on the right track?
#!/usr/bin/env python3
import re
import sys
errors = {}
# log line format
# Jun 1 11:06:48 ubuntu.local ticky: ERROR: Connection to DB failed (username)
logfile = sys.argv[1]
# NOTE: Check to find correct log file
with open(logfile) as f:
for line in f:
if "ERROR:" not in line:
continue
regex_error = r"ERROR: (\d+) "
"""searches for error messages"""
error = re.search(regex_error, line)
if error is None:
continue
name = error[1]
errors[name] = errors.get(name, 0) + 1

From SSH not decoded from bytes to ASCII?

Good afternoon.
I get the example below from SSH:
b"rxmop:moty=rxotg;\x1b[61C\r\nRADIO X-CEIVER ADMINISTRATION\x1b[50C\r\nMANAGED OBJECT DATA\x1b[60C\r\n\x1b[79C\r\nMO\x1b[9;19HRSITE\x1b[9;55HCOMB FHOP MODEL\x1b[8C\r\nRXOTG-58\x1b[10;19H54045_1800\x1b[10;55HHYB"
I process ssh.recv (99999) .decode ('ASCII')
but some characters are not decoded for example:
\x1b[61C
\x1b[50C
\x1b[9;55H
\x1b[9;19H
The article below explains that these are ANSI escape codes that appear since I use invoke_shell. Previously everything worked until it moved to another server.
Is there a simple way to get rid of junk values that come when you SSH using Python's Paramiko library and fetch output from CLI of a remote machine?
When I write to the file, I also get:
rxmop:moty=rxotg;[61C
RADIO X-CEIVER ADMINISTRATION[50C
MANAGED OBJECT DATA[60C
[79C
MO[9;19HRSITE[9;55HCOMB FHOP MODEL[8C
RXOTG-58[10;19H54045_1800[10;55HHYB
If you use PuTTY everything is clear and beautiful.
I can't get away from invoke_shell because the connection is being thrown from one server to another.
Sample code below:
# coding:ascii
import paramiko
port = 22
data = ""
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(hostname=host, username=user, password=secret, port=port, timeout=10)
ssh = client.invoke_shell()
ssh.send("rxmop:moty=rxotg;\n")
while data.find("<") == -1:
time.sleep(0.1)
data += ssh.recv(99999).decode('ascii')
ssh.close()
client.close()
f = open('text.txt', 'w')
f.write(data)
f.close()
The normal output is below:
MO RSITE COMB FHOP MODEL
RXOTG-58 54045_1800 HYB BB G12
SWVERREPL SWVERDLD SWVERACT TMODE
B1314R081D TDM
CONFMD CONFACT TRACO ABISALLOC CLUSTERID SCGR
NODEL 4 POOL FLEXIBLE
DAMRCR CLTGINST CCCHCMD SWVERCHG
NORMAL UNLOCKED
PTA JBSDL PAL JBPTA
TGFID SIGDEL BSSWANTED PACKALG
H'0001-19B3 NORMAL
What can you recommend in order to return normal output, so that all characters are processed?
Regular expressions do not help, since the structure of the record is shifted, then characters from certain positions are selected in the code.
PS try to use ssh.invoke_shell (term='xterm') don't work.
There is an answer here:
How can I remove the ANSI escape sequences from a string in python
There are other ways...
https://unix.stackexchange.com/questions/14684/removing-control-chars-including-console-codes-colours-from-script-output
Essentially, you are 'screen-scraping' input, and you need to strip the ANSI codes. So, grab the input, and then strip the codes.
import re
... (your ssh connection here)
data = ""
while data.find("<") == -1:
time.sleep(0.1)
chunk = ssh.recv(99999)
data += chunk
... (your ssh connection cleanup here)
ansi_escape = re.compile(r'\x1B(?:[#-Z\\-_]|\[[0-?]*[ -/]*[#-~])')
data = ansi_escape.sub('', data)

Groovy: Read and then Write to interactive process

I am beginning to think my search skills are lacking.
I trying to find any articles on how with Groovy, to open an interactive process, read its output and then write to the process depending on the output text. All I can find is how printing, reading and writing with files. Nothing about how to Write to a interactive process.
The process is asking for a password
Write the password to process
Something like this if possible:
def process = "some-command.sh".execute()
process.in.eachLine { line ->
if (line.contains("enter password")) {
process.out.write("myPassword")
}
}
This here works reading from the process output:
def process = "some-command.sh".execute()
process.in.eachLine { line ->
println line
}
Though it stops when the process is asking for input. It does not print out the line with the question.
Edit: Found out why it did not print the line with the ask password. It was not a new line. The question was a simple print (not println). How do I read when there is not yet a new line?
I have been told expect can be used, but I am looking for a solution which does not require a dependency.
1.bat
#echo off
echo gogogo
set /P V=input me:
echo V=%V%
this script waits for input just after :
gogogo
input me:
this means that eachLine not triggered for input me because no new line after it
however the previous line gogogo could be caught
and following script works for gogogo but does not work for input me
groovy
def process = "1.bat".execute()
process.in.eachLine { line ->
if (line.contains("gogogo")) {
process.out.write("myPassword\n".getBytes("UTF-8"))
process.out.flush()
}
}
groovy2
probably this could be optimized.. following script works without new line:
def process = "1.bat".execute()
def pout = new ByteArrayOutputStream()
def perr = new ByteArrayOutputStream()
process.consumeProcessOutput(pout, perr) //starts listening threads and returns immediately
while(process.isAlive()){
Thread.sleep(1234)
if(pout.toString("UTF-8").endsWith("input me:")){
process.out.write("myPassword\n".getBytes("UTF-8"))
process.out.flush()
}
}

how to replace a string/word in a text file in groovy

Hello I am using groovy 2.1.5 and I have to write a code which show the contens/files of a directory with a given path then it makes a backup of the file and replace a word/string from the file.
here is the code I have used to try to replace a word in the file selected
String contents = new File( '/geretd/resume.txt' ).getText( 'UTF-8' )
contents = contents.replaceAll( 'visa', 'viva' )
also here is my complete code if anyone would like to modify it in a more efficient way, I will appreciate it since I am learning.
def dir = new File('/geretd')
dir.eachFile {
if (it.isFile()) {
println it.canonicalPath
}
}
copy = { File src,File dest->
def input = src.newDataInputStream()
def output = dest.newDataOutputStream()
output << input
input.close()
output.close()
}
//File srcFile = new File(args[0])
//File destFile = new File(args[1])
File srcFile = new File('/geretd/resume.txt')
File destFile = new File('/geretd/resumebak.txt')
copy(srcFile,destFile)
x = " "
println x
def dire = new File('/geretd')
dir.eachFile {
if (it.isFile()) {
println it.canonicalPath
}
}
String contents = new File( '/geretd/resume.txt' ).getText( 'UTF-8' )
contents = contents.replaceAll( 'visa', 'viva' )
As with nearly everything Groovy, AntBuilder is the easiest route:
ant.replace(file: "myFile", token: "NEEDLE", value: "replacement")
As an alternative to loading the whole file into memory, you could do each line in turn
new File( 'destination.txt' ).withWriter { w ->
new File( 'source.txt' ).eachLine { line ->
w << line.replaceAll( 'World', 'World!!!' ) + System.getProperty("line.separator")
}
}
Of course this (and dmahapatro's answer) rely on the words you are replacing not spanning across lines
I use this code to replace port 8080 to ${port.http} directly in certain file:
def file = new File('deploy/tomcat/conf/server.xml')
def newConfig = file.text.replace('8080', '${port.http}')
file.text = newConfig
The first string reads a line of the file into variable. The second string performs a replace. The third string writes a variable into file.
Answers that use "File" objects are good and quick, but usually cause following error that of course can be avoided but at the cost of loosen security:
Scripts not permitted to use new java.io.File java.lang.String.
Administrators can decide whether to approve or reject this signature.
This solution avoids all problems presented above:
String filenew = readFile('dir/myfile.yml').replaceAll('xxx','YYY')
writeFile file:'dir/myfile2.yml', text: filenew
Refer this answer where patterns are replaced. The same principle can be used to replace strings.
Sample
def copyAndReplaceText(source, dest, Closure replaceText){
dest.write(replaceText(source.text))
}
def source = new File('source.txt') //Hello World
def dest = new File('dest.txt') //blank
copyAndReplaceText(source, dest) {
it.replaceAll('World', 'World!!!!!')
}
assert 'Hello World' == source.text
assert 'Hello World!!!!!' == dest.text
other simple solution would be following closure:
def replace = { File source, String toSearch, String replacement ->
source.write(source.text.replaceAll(toSearch, replacement))
}

Resources