PyPDF2/PDFminner text extracting issue when PDF is created using InDesign - python-3.x

Question

We have a large number of PDFs which have been created with InDesign and not all the text was being extracted by PyPDF2. Here is the code:-
for pageNum in range(0, pdfReader.numPages):
pageObj = pdfReader.getPage(pageNum)
# note text is a bytes object not string.
text = pageObj.extractText().encode('utf-8')
search_text = text.lower()
if search_word in search_text.decode("utf-8"):
search_word = search_word.strip()
search_word_count += 1
print("Pattern Found on Page: " + str(pageNum+1))
search_word_count_list.append(search_word_count)
print("The word:- '{}' was found:- {} times\n".format(search_word, search_word_count))
I did some testing with PDFminner and found I had the same results i.e. the same bits of text were extracted/not extracted. So I figured there must be something going on with the PDF.
Off the back of this, I worked with a Typesetter doing some testing and discovered when text boxes are locked in InDesign (Crtl+L) the PDF exported has its text locked and is not extractable, I mean the bits that are locked are not extractable via PyPDF2 or PDFminner.
While going forward I can ask the typesetters to unlock text before exporting PDFs. BUT with the thousands to existing PDFs, I want to be able to extract the locked text, asking the typesetters to unlock thousands of files is not an option. Does anyone have experience of this? Any ideas on how to access the locked text?
Edit 1
So doing some testing with the Adobe Acrobat pro 11. When Saving-As Plain Text the locked text does not save to the text file. But the unlocked text does save to the .txt file.
Checking the Security tab in Acrobat:-
With all tested documents, open in Acrobat I pick File -> Properties, switch to the Security tab of the Document Properties dialog, and there I read "Security Method: No Security", and under the restrictions everything is 'Allowed' (Printing, Changing, Copying ...). So I think these are all valid PDFs, which are unprotected.
Edit 2
I have tried to install pdf2txt but my machine does not meet requirements as I am missing "Microsoft Visual C++ 14.0" and as it's a work machine it is locked down.
Edit 3
Acrobat says the PDF Version 1.7. PDF Producer Adobe PDF Library 15.0
I can copy-paste the locked text so I do not think rasterisation is the problem.
Edit 4 - possible solution
So I have tested using the https://pdftotext.com/ and it was able to access the locked text. So I will talk to the IT department to get "Microsoft Visual C++ 14.0" installed so I can use the pdf2txt library.
Edit 5
Have not had much luck with installing PDFtotext due to problems installing poppler which is nothing short of a nightmare to install.
Off the back of usr2564301 input, I have done some more testing with PDFminner. Here is the code I am using to test with:-
from pdfminer.pdfinterp import PDFResourceManager,PDFPageInterpreter
from pdfminer.converter import TextConverter
from pdfminer.layout import LAParams
from pdfminer.pdfpage import PDFPage
from io import BytesIO
def pdf_to_text(path):
manager = PDFResourceManager()
retstr = BytesIO()
layout = LAParams(all_texts=True)
device = TextConverter(manager, retstr, laparams=layout)
filepath = open(path, 'rb')
interpreter = PDFPageInterpreter(manager, device)
for page in PDFPage.get_pages(filepath, check_extractable=True):
interpreter.process_page(page)
text = retstr.getvalue()
filepath.close()
device.close()
retstr.close()
return text
As for versions I have pdfminer==20191125 installed. On Github it says "Supports PDF-1.7. (well, almost)" so maybe the problem is PDF 1.7?
Edit 6
Just tried to use PDFminer (PDF2txt.py) via Command Prompt. Using this code python C:\Users\my_name\AppData\Local\Programs\Python\Python37-32\Scripts\pdf2txt.py -o output.txt file_name.pdf but I get the same result i.e. locked text does not come through.
Edit 7
So did some testing with a designer and we proved it is when the text is on the master page that the text is not accessible to PDFminer. The designers have multiple preset master pages so that they can drag the required one onto the frontcover. If a designer wrongly works directly on the master page the content is lock and can not be accessed via PDFminer.
Note there is not a problem when text is on the frontcover and locked. Only when on the master page.