Python: Combine multiple zipped files and output to multiple csv files - python-3.x

Question

Edit 2:
Adding in a few sample lines for reference. The first line is the column names.
field 1|field 2|field3|id
123|xxx|aaa|118
123|xxx|aaa|56
124|xxx|aaa|184
124|yyy|aaa|156
Edit:
Open to non-Python solutions (grep/awk etc are ok)
The csv files are pipe-delimited "|"
I need to retain the headers
I have 20 .gz files (each ~100MB, zipped). Within each .gz file is a csv file, with many columns, including an index column 'id'. There are around 250 unique ids across all the files.
I need to output all the rows for each unique id to each csv (i.e. there should be 250 csv files generated).
How should I best do this?
I am currently using Python but it takes around 1 minute to generate each csv, I would like to know if there is any faster solution please.
output_folder = 'indiv_ids/'
# get list of files
list_of_files = [filename for filename in os.listdir() if filename.endswith(".gz")]
# get list of unique ids
for i in range(len(list_of_files)):
df = pd.read_csv(list_of_files[i], sep='|', usecols=['id'], dtype=str, engine='c')
id_list = df['id'].unique()
if len(id_list) == 250:
break
# load into a list for each id
list_df = {id:[] for id in id_list}
for filename in list_of_files:
df = pd.read_csv(filename, sep='|', dtype=str, engine='c')
for id in id_list:
df_id = df[df['id'] == id]
list_df[id].append(df_id)
for id in id_list:
# join into one big df
df_full = pd.concat(list_df[id], axis=0)
df_full.to_csv(f'{output_folder}{id}.csv', sep="|", index=False)

Answer 1

Updated Answer
Now that I have seen how your data looks, I think you want this:
gunzip -c *gz | awk -F'|' '$4=="id"{hdr=$0;next} hdr{f=$4; print hdr > f ".csv"; hdr=""} {print > f ".csv"}'
Original Answer
I presume your asking for "any faster solution" permits non-Python solutions, so I would suggest awk.
I generated 4 files of 1000 lines of dummy data like this:
for ((i=0;i<4;i++)) ; do
perl -E 'for($i=0;$i<1000;$i++){say "Line $i,field2,field3,",int rand 250}' | gzip > $i.gz
done
Here are the first few lines of one of the files. The fourth field varies between 0..250 and is supposed to be like your id field.
Line 0,field2,field3,81
Line 1,field2,field3,118
Line 2,field2,field3,56
Line 3,field2,field3,184
Line 4,field2,field3,156
Line 5,field2,field3,87
Line 6,field2,field3,118
Line 7,field2,field3,59
Line 8,field2,field3,119
Line 9,field2,field3,183
Line 10,field2,field3,90
Then you can process like this:
gunzip -c *gz | awk -F, '{ id=$4; print > id ".csv" }'
That says... "Unzip all the .gz files without deleting them and pass the results to awk. Within awk the field separator is the comma. The id should be picked up from the 4th field of each line. Each line should be printed to an output file whose name is id followed by .csv".
You should get 250 CSV files... pretty quickly.
Note: If you run out of open file descriptors, you may need to raise the limit. Try running the following commands:
help ulimit
ulimit -n 500