How do we concatenate all files in Linux excluding the directories? - linux

Question

I am trying to get the total number of lines in all files in a directory.
I tried to do the following:
cat * | wc -1
to get the total number of lines in the directory, but it gives me a message that some of the files are directories. ('cat : some_dir: Is a directory')
How can I exclude directories when concatenating all files?

Answer 1

To get just sum you can try something like below, get count of each file and sum it
find . -type f -exec wc -l {} \; | awk '{ SUM += $1} END { print SUM }'
add -maxdepth 1 which skips scanning further
-type f for filtering only files
Here is test results :
$ seq 1 4 >file1
$ seq 1 5 >file2
$ cat file1
1
2
3
4
$ cat file2
1
2
3
4
5
$ find . -type f -exec wc -l {} \;
5 ./file2
4 ./file1
$ find . -type f -exec wc -l {} \; | awk '{ SUM += $1} END { print SUM }'
9
$ find . -type f -exec wc -l {} +
5 ./file2
4 ./file1
9 total
$ find . -type f -exec wc -l {} + | awk 'END{print $1}'
9

Answer 2

Under bash:
shopt -s globstar
wc -l **/*
From bash's man page:
globstar
If set, the pattern ** used in a pathname expansion con‐
text will match all files and zero or more directories
and subdirectories. If the pattern is followed by a /,
only directories and subdirectories match.
But care: this will take symlinks too!
Some tricks about find
If you want to read files only:
find . -type f -exec wc -l {} +
and, for total only:
find . -type f -exec wc -l {} + | tail -n 1
Huge dirs:
Syntax find ... -exec ... + will limit groupping args to maximum line length. So if your tree is really big, the previous command will generate more than only one fork to wc.
-exec command {} +
This variant of the -exec action runs the specified command on
the selected files, but the command line is built by appending
each selected file name at the end; the total number of invoca‐
tions of the command will be much less than the number of
matched files. The command line is built in much the same way
that xargs builds its command lines. Only one instance of `{}'
is allowed within the command, and (when find is being invoked
from a shell) it should be quoted (for example, '{}') to protect
it from interpretation by shells. The command is executed in
the starting directory. If any invocation returns a non-zero
value as exit status, then find returns a non-zero exit status.
If find encounters an error, this can sometimes cause an immedi‐
ate exit, so some pending commands may not be run at all. This
variant of -exec always returns true.
find . -type f -exec wc -l {} + | awk 'BEGIN{t=0};/total/{t+=$1};END{print t}'
This will compute sum of total lines.
Alternative: using bash for summarize
For fun, as there is no real improvment:
tot=0
while read val nam;do
[ "$nam" = "total" ] && ((tot+=val))
done < <( find . -type f -exec wc -l {} + )
echo $tot