I am doing text analysis on SEC filings (e.g., 10-K), and the documents I have are the complete submission. The complete filing submission includes the 10-K, plus several other documents. Each document resides within the tags ‘<DOCUMENT>’ and ‘</DOCUMENT>’.
What I want: To count the number of words in the 10-K only before the first instance of ‘</DOCUMENT>’
How I want to accomplish it: I want to use a for loop, with a regex (regex_end10k) to indicate where to stop the for loop.
What is happening: No matter where I put my regex match break, the program counts all of the words in the entire document. I have no error, however I cannot get the desired results.
How I know this: I have manually trimmed one filing, while retaining the full document (results below). When I manually remove the undesired documents after the first instance of ‘</DOCUMENT>’, I yield about 750,000 fewer words.
Current output
Note: Apparently I don't have enough SO reputation to embed a screenshot in my post; it defaults to a link.
What I have tried: several variations of where to put the regex match break. No matter what, it almost always counts the entire document. I believe that the two functions may be performed over the entire document. I have tried putting the break statement within get_text_from_html() so that count_words() only performs on the 10-K, but I have had no luck.
The code below is a snippet from a larger function. It's purpose is to (1) strip html tags and (2) count the number of words in the text. If I can provide any additional information, please let me know and I'll update my post.
The remaining code (not shown) extracts firm and report identifiers, (e.g., ‘file’ or ‘cik’) from the header section between tags ‘<SEC-HEADER>’ and ‘</SEC-HEADER>’. Using the same logic, when extracting header information, I use a regex match break logic and it works perfectly. I need help trying to understand why this same logic isn’t working when I try to count the number of words and how to correct my code. Any help is appreciated.
regex_end10k = re.compile(r'</DOCUMENT>', re.IGNORECASE)
for line in f:
def get_text_from_html(html:str):
doc = lxml.html.fromstring(html)
for table in doc.xpath('.//table'): # optional: removes tables from HTML source code
table.getparent().remove(table)
for tag in ["a", "p", "div", "br", "h1", "h2", "h3", "h4", "h5"]:
for element in doc.findall(tag):
if element.text:
element.text = element.text + "\n"
else:
element.text = "\n"
return doc.text_content()
to_clean = f.read()
clean = get_text_from_html(to_clean)
#print(clean[:20000])
def count_words(clean):
words = re.findall(r"\b[a-zA-Z\'\-]+\b",clean)
word_count = len(words)
return word_count
header_vars["words"] = count_words(clean)
match = regex_end10k.search(line) # This should do it, but it doesn't.
if match:
break
You dont need regx, just split your orginal string, and then in the part before count the words, simple example above:
text = 'Text before <DOCUMENT> text after'
splited_text = text.split('<DOCUMENT>')
splited_text_before = splited_text[0]
count_words = len(splited_text_before.split())
print(splited_text_before)
print(count_words)
output
Text before
2
Related
I ran a PDF through a series of processes to extra the text from it. I was successful in that regard. However, now I want to extract specific text from documents.
The document is set up as a multi lined string (I believe. when I paste it into Word the paragraph character is at the end of each line):
Send Unit: COMPLETE
NOA Selection: 20-0429.07
#for some reason, in this editor, despite the next line having > infront of it, the following line (Pni/Trk) keeps wrapping up to the line above. This doesn't exist in the actual doc.
Pni/Trk: 3 Panel / 3 Track
Panel Stack: STD
Width: 142.0000
The information is want to extract are the numbers following "NOA Selection:".
I know I can do a regex something to the effect of:
pattern = re.compile(r'NOA\sSelection:\s\d*-\d*\.\d*)
but I only want the numbers after the NOA selection, especially because NOA Selection will always be the same but the format of the numbers/letters/./-/etc. can vary pretty wildly. This looked promising but it is in Java and I haven't had much luck recreating it in Python.
I think I need to use (?<=...), but haven't been able to implement it.
Also, several of the examples show the string stored in the python file as a variable, but I'm trying to access it from a .txt file, so I might be going wrong there. This is what I have so far.
with open('export1.txt', 'r') as d:
contents = d.read()
p = re.compile('(?<=NOA)')
s = re.search(p, contents)
print(s.group())
Thank you for any help you can provide.
With your shown samples, you could try following too. For sample 20-0429.07 I have kept .07 part optional in regex in case you have values 20-0429 only it should work for those also.
import re
val = """Send Unit: COMPLETE
NOA Selection: 20-0429.07"""
matches = re.findall(r'NOA\s+Selection:\s+(\d+-\d+(?:\.\d+)?)', val)
print(matches)
['20-0429.07']
Explanation: Adding detailed explanation(only for explanation purposes).
NOA\s+Selection:\s+ ##matching NOA spaces(1 or more occurrences) Selection: spaces(1 or more occurrences)
(\d+-\d+(?:\.\d+)?) ##Creating capturing group matching(1 or more occurrences) digits-digits(1 or more occurrences)
##and in a non-capturing group matching dot followed by digits keeping it optional.
Keeping it simple, you could use re.findall here:
inp = """Send Unit: COMPLETE
NOA Selection: 20-0429.07"""
matches = re.findall(r'\bNOA Selection: (\S+)', inp)
print(matches) # ['20-0429.07']
I would like to separate my text into a list of records using the timestamp as the separator. My current code captures the first record, but not the second. How should I modify my code to capture both?
s = """13:45:09 HEY HOW ARE YOU
I AM FINE
13:50:10 OK THEN
Bye"""
import re
text_regex = r'^\d\d:\d\d:\d\d(.*?)(?=\d\d:\d\d:\d\d)'
pattern = re.compile(text_regex,re.DOTALL)
records = []
for match in pattern.findall(s):
match = match.rstrip()
records.append(match)
My current code captures the first record, but not the second. How should I modify my code to capture both?
Your code doesn't capture the second record because due to the trailing (?=\d\d:\d\d:\d\d) lookahead it matches only if a timestamp follows, which isn't so for the last record.
Wiktor Stribiżew's suggestion works if corrected to
records = re.split(r'(?!\A)(?=^\d\d:\d\d:\d\d)', s, flags=re.M)
- otherwise the multiline flag would be taken for the third positional argument maxsplit.
A slightly simplified variant of Wiktor Stribiżew's suggestion which drops extra text before the first timestamp:
records = re.split(r'(?=\d\d:\d\d:\d\d)', s)[1:]
Another variant which (as in your original code) doesn't capture the timestamps:
records = re.split(r'\d\d:\d\d:\d\d', s)[1:]
I'm designing a Mastermind game, which basically compares 2 lists and marks the similarities. When a colour is found at the right place, a flag making the correct position is added and the item found on the reference list is marked off. The reference list is feeding off an array from another function. The problem is at the mark off, as any changes done to the reference list is changing also the original array, which i don't want it to happen
tempCode = mCode #mCode is the array combination randomly generated from another function
for i in range (len(uCode)): #user input array
for j in range (len(tempCode)): #temp array
if uCode[i] == tempCode[j]: # compare individual chars
if i == j: #compare position
flagMark = "*"
tempCode.insert(j+1, "x") #problem starts here
tempCode.remove(tempCode[j])
fCode.append(flagMark)
When the insert is reached both the tempCode and mCode change which it is not intended.
The code is written in a way should the user enter a combination of the same colours, thus checking the chras(the colours are just letters) and the position, and then mark them of with "x"
As it stands, when it gets to
tempCode.insert(j+1, "x")
the arrays will change to
mCode = ["B","R","x","G","Y"]
tempCode = ["B","R","x","G","Y"]
when I would just want
mCode = ["B","R","G","Y"]
tempCode = ["B","R","x","G","Y"]
See also this answer, which is a different presentation of the same problem.
Essentially, when you do tempCode = mCode, you're not making a copy of mCode, you're actually making another reference to it. Anything you do to tempCode thereafter affects the original as well, so at any given time the condition tempCode == mCode will be true (as they're the same object).
You probably want to make a copy of mCode, which could be done in either of the following ways:
tempCode = mCode.copy()
tempCode = mCode[:]
which produces a different list with the same elements, rather than the same list
I am trying to find specific word in a div (id="Test") that starts with "a04" (no case). I can find and replace the words found. But I am unable to correctly use the word found in a "href" link.
I am trying the following working code that correctly identifies my search criteria. My current code is working as expected but I would like help as i do not know how to used the found work as the url id?
var test = document.getElementById("test").innerHTML
function replacetxt(){
var str_rep = document.getElementById("test").innerHTML.replace(/a04(\w)+/g,'TEST');
var temp = str_rep;
//alert(temp);
document.getElementById("test").innerHTML = temp;
}
I would like to wrap the found word in an href but i do not know how to use the found word as the url id (url.com?id=found word).
Can someone help point out how to reference the found work please?
Thanks
If you want to use your pattern with the capturing group, you could move the quantifier + inside the group or else you would only get the value of the last iteration.
\ba04(\w+)
\b word boundary to prevent the match being part of a longer word
a04 Match literally
(\w+) Capture group 1, match 1+ times a word character
Regex demo
Then you could use the first capturing group in the replacement by referring to it with $1
If the string is a04word, you would capture word in group 1.
Your code might look like:
function replacetxt(){
var elm = document.getElementById("test");
if (elm) {
elm.innerHTML = elm.innerHTML.replace(/\ba04(\w+)/g,'TEST');
}
}
replacetxt();
<div id="test">This is text a04word more text here</div>
Note that you don't have to create extra variables like var temp = str_rep;
I'm making a game of hangman. I use a list to keep track of the word that you are guessing for, and a list of blanks that you fill in. But I can't figure out what to do if for example someone's word was apple, and I guessed p.
My immediate thought was to just find if a letter is in the word twice, then figure out where it is, and when they guess that letter put it in both the first and second spot where that letter is. But I can't find
How to test if two STRINGS are duplicates in a list, and
If I were to use list.index to test where the duplicate letters are how to I find both positions instead of just one.
Create a string for your word
Create a string for user input
Cut your string into letters and keep it on a list/array
Get input
Cut input into letters and keep it on another array
Create a string = "--------" as displayed message
Using a for loop check every position in both array lists and compare them
If yourArray[i] == inputArray[i]
Then change displayedString[i] = inputArray[i] and display message then get another input
If it doesnt match leave "-" sings
Displayed the "---a--b" string
One way to do it would be to go through the list one by one and check if something comes up twice.
def isDuplicate(myList):
a = []
index = 0
for item in myList:
if type(item) == str:
if item in a:
return index
else:
a.append(item)
index += 1
return False
This function goes through the list and adds what it has seen so far into another list. Each time it also checks if the item it is looking at is already in that list, meaning it has already been seen before. If it gets through the whole list without any duplicates, it returns False.
It also keeps track of the index it is on, so it can return that index if it does find a duplicate.
Alternately, If you want to find multiple occurrences of a given string, you would use the same structure with some modifications.
def isDuplicate(myList, query):
index = 0
foundIndexes = []
for item in myList:
if item == query:
foundIndexes.append(index)
index += 1
return foundIndexes
This would return a list of the indexes of all instances of query in myList.