How to number lines read from a file using conduits? - haskell

Question

I'm a Haskell beginner trying to wrap my head around the conduit library.
I've tried something like this, but it does not compile:
import Data.Conduit
import Data.Conduit.Binary as CB
import Data.ByteString.Char8 as BS
numberLine :: Monad m => Conduit BS.ByteString m BS.ByteString
numberLine = conduitState 0 push close
where
push lno input = return $ StateProducing (lno + 1) [BS.pack (show lno ++ BS.unpack input)]
close state = return state
main = do
runResourceT $ CB.sourceFile "wp.txt" $= CB.lines $= numberLine $$ CB.sinkFile "test.txt"
It seems that the state in conduitState must be of the same type as the conduit's input type. Or at least that's what I understand from the error message:
$ ghc --make exp.hs
[1 of 1] Compiling Main ( exp.hs, exp.o )
exp.hs:8:27:
Could not deduce (Num [ByteString]) arising from the literal `0'
from the context (Monad m)
bound by the type signature for
numberLine :: Monad m => Conduit ByteString m ByteString
at exp.hs:(8,1)-(11,30)
Possible fix:
add (Num [ByteString]) to the context of
the type signature for
numberLine :: Monad m => Conduit ByteString m ByteString
or add an instance declaration for (Num [ByteString])
In the first argument of `conduitState', namely `0'
In the expression: conduitState 0 push close
In an equation for `numberLine':
numberLine
= conduitState 0 push close
where
push lno input
= return
$ StateProducing (lno + 1) [pack (show lno ++ unpack input)]
close state = return state
How can this be done using conduits? I want to read lines from a file and append a line number to each line.

Answer 1

Yes, it can be done. I prefer to use the helper functions in Data.Conduit.List and also avoid Data.ByteString.Char8 if at all possible. I'm assuming your file is UTF-8 encoded.
import Data.Conduit
import Data.Conduit.Binary as CB
import Data.Conduit.List as Cl
import Data.Conduit.Text as Ct
import Data.Monoid ((<>))
import Data.Text as T
numberLine :: Monad m => Conduit Text m Text
numberLine = Cl.concatMapAccum step 0 where
format input lno = T.pack (show lno) <> T.pack " " <> input <> T.pack "\n"
step input lno = (lno+1, [format input lno])
main :: IO ()
main =
runResourceT
$ CB.sourceFile "wp.txt"
$$ Ct.decode Ct.utf8
=$ Ct.lines
=$ numberLine
=$ Ct.encode Ct.utf8
=$ CB.sinkFile "test.txt"

Answer 2

close state = return state
Herein lies the type error. Your close function should have type (state -> m [output]) (as per the docs). In your case state = Int (you may want to add type annotations to make sure it selects Int) and output = BS.ByteString, so probably just return the empty list, since at the point of closing the conduit, you haven't really saved any ByteStrings to produce or anything like that.
close _ = return []
Especially note from the docs for that argument:
The state need not be returned, since it will not be used again

Answer 3

An alternative solution with pipes 3.0, though it does use string instead of ByteString. The main advantage in my mind is being able to use the normal state monad methods get and put. Another benefit is that the starting line number is not hidden in the addLineNumber(numberLine) so it is easier to start at any given line number.
import System.IO
import Data.Monoid ((<>))
import Control.Proxy
import qualified Control.Proxy.Trans.State as S
addLineNumber r = forever $ do
n <- S.get
line <- request r -- request line from file
respond $ show n <> " " <> line
S.put (n + 1) -- increments line counter
main =
withFile "wp.txt" ReadMode $ \fin ->
withFile "test.txt" WriteMode $ \fout ->
runProxy $ S.execStateK 1 -- start at line number at 1
$ hGetLineS fin >-> addLineNumber >-> hPutStrLnD fout
Find out how to do more fine grained resource management at the announce blog post of pipes-safe..