python 3.X concatenate zipped csv files to one non-zipped csv file - python-3.x

Question

here is my python 3 code:
import zipfile
import os
import time
from timeit import default_timer as timer
import re
import glob
import pandas as pd
# local variabless
# pc version
# the_dir = r'c:\ImpExpData'
# linux version
the_dir = '/home/ralph/Documents/lulumcusb/ImpExpData/Exports/92-95'
def main():
"""
this is the function that controls the processing
"""
start_time = timer()
for root, dirs, files in os.walk(the_dir):
for file in files:
if file.endswith(".zip"):
print("working dir is ...", the_dir)
zipPath = os.path.join(root, file)
z = zipfile.ZipFile(zipPath, "r")
for filename in z.namelist():
if filename.endswith(".csv"):
# print filename
if re.match(r'^Trade-Geo.*\.csv$', filename):
pass # do somethin with geo file
# print " Geo data: " , filename
elif re.match(r'^Trade-Metadata.*\.csv$', filename):
pass # do something with metadata file
# print "Metadata: ", filename
else:
try:
with zipfile.ZipFile(zipPath) as z:
with z.open(filename) as f:
# print("send to test def...", filename)
# print(zipPath)
with zipfile.ZipFile(zipPath) as z:
with z.open(filename) as f:
frame = pd.DataFrame()
# EmptyDataError: No columns to parse from file -- how to deal with this error
train_df = read_csv(f, index_col=None, header=0, skiprows=1, encoding="cp1252")
# train_df = pd.read_csv(f, header=0, skiprows=1, delimiter=",", encoding="cp1252")
list_ = []
list_.append(train_df)
# print(list_)
frame = pd.concat(list_, ignore_index=True)
frame.to_csv('/home/ralph/Documents/lulumcusb/ImpExpData/Exports/concat_test.csv', encoding='cp1252') # works
except: # catches EmptyDataError: No columns to parse from file
print("EmptyDataError...." ,filename, "...", zipPath)
# GetSubDirList(the_dir)
end_time = timer()
print("Elapsed time was %g seconds" % (end_time - start_time))
if __name__ == '__main__':
main()
it mostly works -- only it does not concatenate all zipped csv files into one. there is one empty file and all csv files have the same field structure with all csv files varing in number of rows.
here is what spyder reports when i run it:
runfile('/home/ralph/Documents/lulumcusb/Sep15_cocncatCSV.py', wdir='/home/ralph/Documents/lulumcusb')
working dir is ... /home/ralph/Documents/lulumcusb/ImpExpData/Exports/92-95
EmptyDataError.... Trade-Exports-Chp-77.csv ... /home/ralph/Documents/lulumcusb/ImpExpData/Exports/92-95/Trade-Exports-Yr1992-1995.zip
/home/ralph/anaconda3/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py:688: DtypeWarning: Columns (1) have mixed types. Specify dtype option on import or set low_memory=False.
execfile(filename, namespace)
Elapsed time was 104.857 seconds
the final csvfile is the last zipped csv file processed; the csv file changes in size as it processes the files
there are 99 csv files in the zipped file that i wish to concat into one non-zipped csv file
the field or column names are:
colmNames = ["hs_code", "uom", "country", "state", "prov", "value", "quatity", "year", "month"]
the csvfiles are labled: chp01.csv, cht02.csv, etc to chp99.csv with the "uom" (unit of measure) being either empty, or an integer or a string depending on the hs_code
Question: how do I get the zipped csv files to get concatenated into one large(estimated 100 mb uncompressed) csv file?
added details:
i am trying not to unzip the csv files, i would then have to go an delete them. I need to concat files because i have additional processing to do. The extracting of the zipped csv files is a viable option, i was hoping not having to do that

Answer 1

Is there any reason you don't want to do this with your shell?
Assuming the order in which you concatenate is irrelevant:
cd "/home/ralph/Documents/lulumcusb/ImpExpData/Exports/92-95"
unzip "Trade-Exports-Yr1992-1995.zip" -d unzipped && cd unzipped
for f in Trade-Exports-Chp*.csv; do tail --lines=+2 "$f" >> concat.csv; done
This removes the first line (column names) from each csv file before appending to concat.csv.
If you just did:
tail --lines=+2 "Trade-Exports-Chp*.csv" > concat.csv
You'd end up with:
==> Trade-Exports-Chp-1.csv <==
...
==> Trade-Exports-Chp-10.csv <==
...
==> Trade-Exports-Chp-2.csv <==
...
etc.
If you care about the order, change Trade-Exports-Chp-1.csv .. Trade-Exports-Chp-9.csv to Trade-Exports-Chp-01.csv .. Trade-Exports-Chp-09.csv.
Although it's doable in Python I don't think it's the right tool for the job in this case.
If you want to do the job in place without actually extracting the zip file:
for i in {1..99}; do
unzip -p "Trade-Exports-Yr1992-1995.zip" "Trade-Exports-Chp$i.csv" | tail --lines=+2 >> concat.csv
done