Check if files exist in multiple directories using wildcards - linux

Question

I have ~10,000 directories. Most of them have a similarly named text file.
I would like to take these .txt files and copy them to a folder in the main directory, ALL_RESULTS. How can i accomplish this? What I have below
for d in *_directories/; do
#go into directory
cd "$d"
#check if file exists using wildcard, then copy it into ALL_RESULTS and print the name of
#directory out
if ls *SCZ_PGC3_GWAS.sumstats.gz*.txt 1> /dev/null 2>&1; then
cp *SCZ_PGC3_GWAS.sumstats.gz*.txt ../ALL_RESULTS && echo "$d"
#if file does not exist, print the name of the directory we're
#in
else
echo "$d"
echo "files do not exist"
cd ..
fi
done
I keep getting errors saying the directories themselves don't exist. What am I doing wrong?

Answer 1

All relative paths are interpreted relative to the directory you are in (the "current working directory"). So, imagine, you cd into the first directory. So now you are in that directory. Then you loop executes, and you try to cd into the second directory. But that directory is no longer then, you need to go "up" and then cd into the directory. That is the reason the directory does not exists - you have to go "up" a directory for each directory you cd into.
So you need to cd .. on the end of your loop to go back to the directory you started from.
I have ~10,000 directories. ... I would like to take these .txt files and move them to a folder in the main directory, ALL_RESULTS
If you don't need to output anything, just use find for that with a proper regex. Doing ls and cd and a loop will be very slow. Something along:
find . -maxdepth 2 -type f -regex '\./.*_directories/.*SCZ_PGC3_GWAS.sumstats.gz.*\.txt' -exec cp {} ALL_RESULTS \;
You can also add -v to cp to see what it copies.

Answer 2

You miss
shopt -s nullglob
and don't parse ls output :
#!/bin/bash
shopt -s nullglob
for d in *_directories/; do
# check if file exists using wildcard, then copy it into ALL_RESULTS and print
# the name of directory
files=$( $d/*SCZ_PGC3_GWAS.sumstats.gz*.txt )
if [[ ${files[#]} ]]; then
cp "${files[#]}" ALL_RESULTS && echo "$d"
#if file does not exist, print the name of the directory we're
#in
else
echo "$d"
echo "files do not exist"
fi
done