Recursive data scraping in excel sheets within nested folder structure - python-3.x

Question

Help me out please. I would like to traverse through a directory structure that looks like this:
Topdir > subdir 1 > excel 1/2/3
Topdir > subdir 2 > excel 4
etc
I am scraping the excel's column B for a string, and that is working nicely. However, my script only goes through the topdir, and doesn't go into the subdirs. Below is my code:
import openpyxl, os, sys, warnings, glob
warnings.simplefilter("ignore")
targetString = str("Sample Error")
scriptPath = os.path.abspath(__file__)
outputFile = open('logging.txt', "w+")
def scrapeSheets():
for i in os.listdir(path='.'):
if i.endswith("data-eval.xlsm"):
print("Working on:", i)
wb = openpyxl.load_workbook(i, data_only=True)
sheet = wb["data-sheet"]
outputFile.write("{}\n".format(i))
for cellObj in sheet["B"]:
if cellObj.value == targetString:
print(cellObj.row, cellObj.value)
outputFile.write("\t{}\t{}\n".format(cellObj.row, cellObj.value))
def mainLoop():
for filename in glob.iglob('**/*.xlsm', recursive=True):
scrapeSheets()
if __name__ == "__main__":
mainLoop()
As I said, the scraping works, but I cannot go into the subfolders. I have a hunch it has to do with the line
for i in os.listdir(path='.')
however, I don't know the solution to make the loop's variable increment.

Answer 1

You can try like this:
for dirname in os.listdir(path='.'):
for main_dir, dirs, files in os.walk(dirname):
for f in files:
if f.endswith("data-eval.xlsm"):
print("Working on:", f)
wb = openpyxl.load_workbook(f, data_only=True)
sheet = wb["data-sheet"]
outputFile.write("{}\n".format(i))
for cellObj in sheet["B"]:
if cellObj.value == targetString:
print(cellObj.row, cellObj.value)
outputFile.write("\t{}\t{}\n".format(cellObj.row, cellObj.value))
Explanation:
Using listdir iterate over the directories.
for dirname in os.listdir(path='.')
Iterate over the sub-directories and the files inside those using :
for main_dir, dirs, files in os.walk(dirname):
Iterate over the files and continue your logic.
for f in files:
if f.endswith("data-eval.xlsm"):
print("Working on:", f)
wb = openpyxl.load_workbook(f, data_only=True)
sheet = wb["data-sheet"]
outputFile.write("{}\n".format(i))
for cellObj in sheet["B"]:
if cellObj.value == targetString:
print(cellObj.row, cellObj.value)
outputFile.write("\t{}\t{}\n".format(cellObj.row, cellObj.value))

Answer 2

For future reference, I figured out that implementing the for filename in glo.iglob part in the scraping function instead the os.listdir line works perfectly and loops through the content of the script's folder and its subfolders.