Iterate through files in sub-folders inside a main folder in bash [duplicate] - linux

Question

This question already has answers here:
Interacting with files from multiple directories via bash script
(2 answers)
Closed 3 years ago.
The question is really simple, I know how to do it in Python but I want to do it in Linux shell (bash).
I have a main folder Dataset inside which there are multiple sub-folders Dataset_FinalFolder_0_10 all the way up to Dataset_FinalFolder_1090_1100 each with 10 files.
I want to run a program on each of those files. In Python I would do this with something like:
for folder in /path/to/folders:
for file in folder:
run program
Is there any way so mimic this in Shell / bash?
I have this code which I have used for more direct iterations:
for i in /path/to/folder/*;
do program "$i";
done
Thanks in advance

Answer 1

If you are sure that there are no files mixed in with the folders, and no folders mixed in with the files:
for folder in /path/to/Dataset/*; do
for file in "$folder"/*; do
program "$file"
done
done
Alternatively, it is possible to give more than one *:
for file in /path/to/Dataset/*/*; do
program "$file"
done
If you aren't sure about the folder contents, then find can help. This example selects files in just the first-level subdirectories of the given folder and xargs calls program for each one:
find /path/to/Dataset/ -mindepth 2 -maxdepth 2 -type f |\
xargs -n1 program
The find method may also be useful if .../*/*/*/... could expand to a huge number of paths. On linux, the commandline length limit is shown by:
getconf ARG_MAX
On my machine that is 2^21 (~2 million) characters. So the limit is high, but worth keeping at the back of your mind that there is one.

Answer 2

From the Linux perspective, you have to watch out for properly escaping spaces, new lines, etc which can get kinda funky. There are multiple references for why not to do it - see
http://mywiki.wooledge.org/ParsingLs
And
https://unix.stackexchange.com/questions/128985/why-not-parse-ls-and-what-do-to-instead
That said...
You can always use the find command wiht the -exec option -
find /path/to/top/level -type f -exec /path/to/processing/program {} \;
The \; at the end is required to indicate the end of the exec

Answer 3

You don't need nested loops in either Python or the shell unless you have so many files that you are running up against "argument list too long" errors.
for file in /path/to/folders/*/*; do
program "$file"
done
This is equivalent to the Python code
from glob import glob
from subprocess import run
for file in glob('/path/to/folders/*/*'):
run(['program', file])
Of course, if program is at all competently written, you can probably simply do
program /path/to/folders/*/*
This corresponds to
run(['program'] + glob('/path/to/folders/*/*')
If program accepts a list of file name arguments, but you do need to break up the command line to avoid "argument list too long" errors, try
printf '%s\0' /path/to/folders/*/* |
xargs -r0 program
(The zero-terminator pattern is a GNU find extension, as is the -r option.)

Answer 4

for dir in ./* ./**/* # list directories in the current directory
do
python $dir
done
./* are files in dir and ./**/* are files in subfolders.
Make sure you have only python files in your directory it will run all the files in that directory
Actually I have already answered it here
Iterate shell script over list of subdirectories