using regex in python to return line and some lines are skipped - python-3.x

Question

I am new to Python and programming in general. My code gets data from a pdf. Then iterates over the returned lines to pull only lines meeting a regex expression. Most lines have data in the next line that need to be added to the current line which is being iterated over. The code I wrote does this successfully except that lines that do not have data in the next line to be added, returns the current line but skips the very next line.
What can I add or change to make sure that it does not skip the line.
An example of the lines that need to be iterated over are:
#Input text
CATS THIRD PARTY PAYMENT 1,664.58 0320 2,130.05
MUTUAL/IBS /GAL /0000010318908
IB TRANSFER TO 2,000.00- 0323 130.05
578441575425 10H32 28662338
FEE-INTER ACCOUNT TRANSFER ## 5.50- 0323 124.55
8419752
IB PAYMENT FROM 9,000.00 0325 9,124.55
JENNIFER LIVINGSTONE
IB PAYMENT FROM 1,000.00 0401 10,124.55
JENNIFER LIVINGSTONE
MONTHLY MANAGEMENT FEE ## 21.00- 0331 10,103.55 (This line has no description in the following line)
CREDIT TRANSFER 9,000.00 0401 19,103.55 (This line gets skipped)
ABSA BANK rent
IB TRANSFER TO 19,000.00- 0403 103.55
578441575425 11H45 286623383
bpdf = 'test pdf.pdf'
with pdfplumber.open(bpdf) as pdf:
page = pdf.pages[0]
text = page.extract_text()
print(text)
new_trn_line = re.compile(r'(\D+)(\d.*) (\d.*) (\d.*\.\d{2})')
def transactions(sentences):
for lines in sentences.split('\n'):
yield lines
my_list = transactions(text)
my_data = []
for each_line in my_list:
if new_trn_line.search(each_line):
my_next_line = next(my_list)
if not new_trn_line.search(my_next_line):
my_data.append(new_trn_line.search(each_line).group(1) + my_next_line + " " +
new_trn_line.search(each_line).group(2) + " " + new_trn_line.search(each_line).group(3))
elif re.search(new_trn_line,text):
my_data.append(each_line)
else:
continue
my_data
#Output
['CATS THIRD PARTY PAYMENT MUTUAL/IBS /GAL /0000010318908 1,664.58 0320',
'IB TRANSFER TO 578441575425 10H32 286623383 2,000.00- 0323',
'FEE-INTER ACCOUNT TRANSFER ## 8419752 5.50- 0323',
'IB PAYMENT FROM JENNIFER LIVINGSTONE 9,000.00 0325',
'IB PAYMENT FROM JENNIFER LIVINGSTONE 1,000.00 0401',
'MONTHLY MANAGEMENT FEE ## 21.00- 0331 10,103.55',
'IB TRANSFER TO 578441575425 11H45 286623383 19,000.00- 0403'\]
If you compare it to the input you will see CREDIT TRANSFER 9,000.00 0401 19,103.55 gets skipped

Answer 1

Your current pattern would not match (\D+)(\d.*) (\d.*) (\d.\*.\d{2}) as \* matches an asterix which is not in the example data.
If you have read the whole text from the .pdf, perhaps you could use a single pattern with capture groups, and assemble the results.
The pattern matches
^ Start of string
([^\d\n]+) Capture group 1 Match 1+ chars other than digits or newlines
(\d[\d,.-]* \d+) Capture group 2 Match a digit followed by optional repetitions of digits, comma's, dots or a hyphen. Then match a space and 1+ digits
\d[\d,.]*\.\d{2} match a digit followed by digits, comma's and dots and then match . followed by 2 digits
( Capture group 3
(?: Non capture group to repeat as a whole part
\n(?!.*\b\d[\d,.]*\.\d{2}$) Assert that the line does not end on a digit with a decimal part having a dot and 2 digits
.* Match the whole line
)* close the non capture group and optionally repeat to also match just a single previous line
) Close group 3
See a regex101 demo
import re
pattern = r"^([^\d\n]+)(\d[\d,.-]* \d+) \d[\d,.]*\.\d{2}((?:\n(?!.*\b\d[\d,.]*\.\d{2}$).*)*)"
my_data = []
text = ("CATS THIRD PARTY PAYMENT 1,664.58 0320 2,130.05\n"
"MUTUAL/IBS /GAL /0000010318908\n"
"IB TRANSFER TO 2,000.00- 0323 130.05\n"
"578441575425 10H32 28662338\n"
"FEE-INTER ACCOUNT TRANSFER ## 5.50- 0323 124.55\n"
"8419752\n"
"IB PAYMENT FROM 9,000.00 0325 9,124.55\n"
"JENNIFER LIVINGSTONE\n"
"IB PAYMENT FROM 1,000.00 0401 10,124.55\n"
"JENNIFER LIVINGSTONE\n"
"MONTHLY MANAGEMENT FEE ## 21.00- 0331 10,103.55\n"
"CREDIT TRANSFER 9,000.00 0401 19,103.55\n"
"ABSA BANK rent\n"
"IB TRANSFER TO 19,000.00- 0403 103.55\n"
"578441575425 11H45 286623383")
matches = re.finditer(pattern, text, re.MULTILINE)
for _, match in enumerate(matches):
my_data.append(f"{match.group(1)}{match.group(3).strip()} {match.group(2)}")
print(my_data)
Note sure why in your example output only MONTHLY MANAGEMENT FEE ends with 10,103.55 because all the other output lines seems to end with 4 digits.
Output
[
'CATS THIRD PARTY PAYMENT MUTUAL/IBS /GAL /0000010318908 1,664.58 0320',
'IB TRANSFER TO 578441575425 10H32 28662338 2,000.00- 0323',
'FEE-INTER ACCOUNT TRANSFER ## 8419752 5.50- 0323',
'IB PAYMENT FROM JENNIFER LIVINGSTONE 9,000.00 0325',
'IB PAYMENT FROM JENNIFER LIVINGSTONE 1,000.00 0401',
'MONTHLY MANAGEMENT FEE ## 21.00- 0331',
'CREDIT TRANSFER ABSA BANK rent 9,000.00 0401',
'IB TRANSFER TO 578441575425 11H45 286623383 19,000.00- 0403'
]