How to read big csv file? - haskell

Question

I trying to read a big csv file by haskell, and generate the word count by each column.
This more than 4M rows in the file.
So I choice read a block and get the word count each time(5k rows one block).
And than sum it together.
When I test the function with 12000 rows and 120000 rows the time increase almost linear.
But When read 180000 rows, run time exceeds more than four times.
I think it because the memory is not enough，swap with disk make the function much slower.
I had write my code as map/reduce style,But how to make the haskell don't hold all data in memory?
The blow is my code and profiling result.
import Data.Ord
import Text.CSV.Lazy.String
import Data.List
import System.IO
import Data.Function (on)
import System.Environment
splitLength = 5000
mySplit' [] = []
mySplit' xs = [x] ++ mySplit' t
where
x = take splitLength xs
t = drop splitLength xs
getBlockCount::Ord a => [[a]] -> [[(a,Int)]]
getBlockCount t = map
(map (\x -> ((head x),length x))) $
map group $ map sort $ transpose t
foldData::Ord a=> [(a,Int)]->[(a,Int)]->[(a,Int)]
foldData lxs rxs = map combind wlist
where
wlist = groupBy ((==) `on` fst) $ sortBy (comparing fst) $ lxs ++ rxs
combind xs
| 1==(length xs) = head xs
| 2 ==(length xs) = (((fst . head) xs ), ((snd . head) xs)+((snd . last) xs))
loadTestData datalen = do
testFile <- readFile "data/test_csv"
let cfile = fromCSVTable $ csvTable $ parseCSV testFile
let column = head cfile
let body = take datalen $ tail cfile
let countData = foldl1' (zipWith foldData) $ map getBlockCount $ mySplit' body
let output = zip column $ map ( reverse . sortBy (comparing snd) ) countData
appendFile "testdata" $ foldl1 (\x y -> x ++"\n"++y)$ map show $tail output
main = do
s<-getArgs
loadTestData $ read $ last s
profiling result
loadData +RTS -p -RTS 12000
total time = 1.02 secs (1025 ticks # 1000 us, 1 processor)
total alloc = 991,266,560 bytes (excludes profiling overheads)
loadData +RTS -p -RTS 120000
total time = 17.28 secs (17284 ticks # 1000 us, 1 processor)
total alloc = 9,202,259,064 bytes (excludes profiling overheads)
loadData +RTS -p -RTS 180000
total time = 85.06 secs (85059 ticks # 1000 us, 1 processor)
total alloc = 13,760,818,848 bytes (excludes profiling overheads)

Answer 1

So first, a few suggestions.
Lists aren't fast. Okay, okay, cons is constant time, but in general, lists aren't fast. You're using lists. (Data.Sequence would've been faster for two-ended cons'ing and consumption)
Strings are slow. Strings are slow because they're [Char] (List of Char). The library you're currently using is written in terms of lists of Strings. Usually linked-lists of linked-lists of characters aren't what you want for text processing. This is no bueno. Use Text (for, uh, text) or ByteString (for bytes) instead of String in future unless it's something small and not performance sensitive.
The library you are using is just lazy, not streaming. You'd have to handle getting streaming behavior overlaid onto the lazy semantics to get constant memory use. Streaming libraries solve the problem of incrementally processing data and limiting memory use. I'd suggest learning Pipes or Conduit for that general class of problems. Some problem-specific libraries will also offer an iteratee API which can be used for streaming. Iteratee APIs can be used directly or hooked up to Pipes/Conduit/etc.
I don't think the library you're using is a good idea.
I suggest you use one of the following libraries:
http://hackage.haskell.org/package/pipes-csv (Pipes based)
https://hackage.haskell.org/package/cassava-0.4.2.0/docs/Data-Csv-Streaming.html (Common CSV library, not based on a particular streaming library)
https://hackage.haskell.org/package/csv-conduit (Conduit based)
These should give you good performance and constant memory use modulo whatever you might be accumulating.

Answer 2

There are a couple of things to be aware of:
You want to stream the data so that you are only holding in memory a small portion of the input file at any time. You might be able to accomplish this with lazy IO and the lazy-csv package. However, it still is easy to inadvertently hold on to references which keep all of your input in memory. A better option is to use a streaming library like csv-conduit or pipes-csv.
Use ByteString or Text when processing large amounts of string data.
You want to make sure to use strict operations when reducing your data. Otherwise you will just be building up thunks of unevaluated expressions in memory until the very end when you print out the result. One place where thunks could be building up is your foldData function - the word count expressions do not appear to be getting reduced.
Here is an example of a program which will compute the total length of all of the words in each column of a CSV file and does it in constant memory. The main features are:
uses lazy IO
uses the lazy-csv package with (lazy) ByteString instead of String
uses BangPatterns to strictify the computation of the number of lines
uses an unboxed array to hold the column counters
The code:
{-# LANGUAGE BangPatterns #-}
import qualified Data.ByteString.Lazy.Char8 as BS
import Data.ByteString.Lazy (ByteString)
import Text.CSV.Lazy.ByteString
import System.Environment (getArgs)
import Data.List (foldl')
import Data.Int
import Data.Array.IO
import Data.Array.Unboxed
import Control.Monad
type Length = Int64 -- use Int on 32-bit systems
main = do
(arg:_) <- getArgs
(line1:lns) <- fmap BS.lines $ BS.readFile arg
-- line1 contains the header
let (headers:_) = [ map csvFieldContent r | r <- csvTable (parseCSV line1) ]
ncols = length headers :: Int
arr <- newArray (1,ncols) 0 :: IO (IOUArray Int Length)
let inc i a = do v <- readArray arr i; writeArray arr i (v+a)
let loop !n [] = return n
loop !n (b:bs) = do
let lengths = map BS.length $ head [ map csvFieldContent r | r <- csvTable (parseCSV b) ]
forM_ (zip [1..] lengths) $ \(i,a) -> inc i a
loop (n+1) bs
print headers
n <- loop 0 lns
putStrLn $ "n = " ++ show (n :: Int)
arr' <- freeze arr :: IO (UArray Int Length)
putStrLn $ "totals = " ++ show arr'

Answer 3

I have had this issue before in another language. The trick is not to read the data into memory, but rather just read it in one line at a time. When you read the next line just overwrite your variables as you are only looking for a word count.
Just test for an EOF end of file condition in your io stream and exit then. That way you don;t have to split the file.
Hope that helps