Accumulating different lines into different files in awk - linux

Question

I have a huge .txt file (15 GB) and having almost 30 million lines.
I want to put its lines to different files based on the 4th column. And the unique number of 4th column is around 2 million.
file1.txt
1 10 ABC KK-LK
1 33 23 KK-LK
2 34 32 CK-LK,LK
11 332 2 JK#
11 23 2 JK2
Right now, I can separate these lines to different files in the same folder as following:
awk '{ print $0 >> $4"_sep.txt" }' file1.txt
And it results in 4 different files as:
KK-LK_sep.txt
1 10 ABC KK-LK
1 33 23 KK-LK
and
CK-LK,LK_sep.txt
2 34 32 CK-LK,LK
and
JK#_sep.txt
11 332 2 JK#
and finally,
JK2_sep.txt
11 23 2 JK2
What I want is, to not to put 2 million files in one folder, to separate them into 20 different folders. I can make folders as folder1,2,3....:
mkdir folder{1..20}
With the answers below, I suppose something like broken below code would work:
#!/bin/env bash
shopt -s nullglob
numfiles=(*)
numfiles=${#numfiles[#]}
numdirs=(*/)
numdirs=${#numdirs[#]}
(( numfiles -= numdirs ))
echo $numfiles
var1=$numfiles
awk -v V1=var1 '{
if(V1 <= 100000)
{
awk '{ print $0 >> $4"_sep.txt" }' file1.txt
}
else if(V1 => 100000)
{
cd ../folder(cnt+1)
awk '{ print $0 >> $4"_sep.txt" }' file1.txt
}
}'
But then, how can I make this a loop and stop adding up to the folder1 once it has 100.000 files in it, and start adding files to folder2 and so on?

Answer 1

Maybe this is what you want (untested since your question doesn't include an example we can test against):
awk '
!($4 in key2out) {
if ( (++numKeys % 100000) == 1 ) {
dir = "dir" ++numDirs
system("mkdir -p " dir)
}
key2out[$4] = dir "/" $4 "_sep.txt"
}
{ print > key2out[$4] }
' file1.txt
That relies on GNU awk for managing the number of open files internally. With other awks you'd need to change that last line to { print >> key2out[$4]; close(key2out[$4]) } or otherwise handle how many concurrently open files you have to avoid getting a "too many open files" error, e.g. if your $4 values are usually grouped together then more efficiently than opening and closing the output file on every single write, you could just do it when the $4 value changes:
awk '
$4 != prevKey { close(key2out[prevKey]) }
!($4 in key2out) {
if ( (++numKeys % 100000) == 1 ) {
dir = "dir" ++numDirs
system("mkdir -p " dir)
}
key2out[$4] = dir "/" $4 "_sep.txt"
}
{ print >> key2out[$4]; prevKey=$4 }
' file1.txt

Answer 2

something like this?
count the unique keys and increment bucket after threshold.
count += !keys[$4]++;
bucket=count/100000;
ibucket=int(bucket);
ibucket=ibucket==bucket?ibucket:ibucket+1;
folder="folder"ibucket