I can't really figure out whether some of these other questions are similar enough to mine but I couldn't extract a solution out of them so I'm posting. Feel free to indicate to me otherwise.
I have a flow where I need to download a large CSV file, and 1) save it to disk, and 2) process it. I'd like to use Haskell pipes, with the pipes-http and pipes-csv packages to do this.
The obvious way is to have two separate pipes: 1) web -> disk, and then 2) disk -> process. Is it possible to do another topology where the output from the web splits into two consumers, one that saves and the other that processes? I feel that this could be more elegant and possibly more efficient.
If so, how is the splitting done? Splitting of pipes is not mentioned anywhere in the documentation.
The expression "splitting the content between consumers" might be a little misleading; you want to send all bytes to each of two consumers. But Pipes.Prelude.tee turns any consumer into a Pipe, thus
producer >-> tee consumer1 >-> consumer2
feeds the producer to both of the consumers. But the particular case of writing to a file might be simplest with Pipes.Prelude.chain, rather than a consumer. tee and chain allow you to do something with each incoming value, before forwarding it along the pipeline. In this case I just write each successive chunk to a handle, before passing it along:
import Pipes
import Pipes.HTTP
import qualified Pipes.ByteString as PB
import qualified Pipes.Prelude as P
import qualified System.IO as IO
import qualified Data.ByteString as B
main = do
req <- parseUrl "https://www.example.com"
m <- newManager tlsManagerSettings
withHTTP req m $ \resp ->
IO.withFile "file.txt" IO.WriteMode $ \h ->
runEffect $ responseBody resp >-> P.chain (B.hPut h) >-> PB.stdout
I ended the pipeline with PB.stdout where you would use pipes-csv materials. Using tee, I could as well have written
runEffect $ responseBody resp >-> P.tee (PB.toHandle h) >-> PB.stdout
for the last line. Where the 'consumers' can be viewed as folds, there is the apparatus of Control.Foldl for combining many folds together - and any number of other devices.
Related
I really like the concept of conduit/pipes for applying operations to a streaming IO source. I am interested in building tools that work on very large log files. One of the attractions of moving to Haskell from Python/Ruby is the easier way of writing parallel code, but I can't find any documentation of this. How could I set up a conduit-flow which reads lines from a file and works on them in parallel (ie. with 8 cores, it should read eight lines, and hand them off to eight different threads to be processed, and then collected again etc), ideally with as little "ceremony" as possible...
Optionally it could be noted whether the lines need to be rejoined in order or not, if that could influence the speed of the process?
I am sure it would be possible to cobble together something myself using ideas from the Parallel Haskell book, but it seems to me that running a pure function in parallel (parmap etc) in the middle of a Conduit workflow should be very easy?
As an example of the "internal parallelism" mentioned by Petr Pudlák in his comment, consider this function (I'm using pipes, but could be implemented with conduit just as easily):
import Control.Monad
import Control.Lens (view)
import Control.Concurrent.Async (mapConcurrently)
import Pipes
import qualified Pipes.Group as G
import qualified Control.Foldl as L
concProd :: Int -> (a -> IO b) -> Producer a IO r -> Producer b IO r
concProd groupsize action producer =
L.purely G.folds L.list (view (G.chunksOf groupsize) producer)
>->
forever (await >>= liftIO . mapConcurrently action >>= mapM G.yield)
This function takes as parameters a group size, an action we want to run for each value of type a, and a Producer of a values.
It returns a new Producer. Internally, the producer reads a values in batches of groupsize, processes them concurrently, and yields the results one by one.
The code uses Pipes.Group to "partition" the original producer into sub-producers of size groupsize, and then Control.Foldl to "fold" each sub-producer into a list.
For more sophisticated tasks, you could turn to the asynchronous channels provided by pipes-concurrency or stm-conduit. But these yank you out somewhat of the "single pipeline" worldview of vanilla pipes/conduits.
I'm in the process of wrapping a C library for some encoding in a pipes interface, but I've hit upon some design decisions that need to be made.
After the C library is set up, we hold on to an encoder context. With this, we can either encode, or change some parameters (let's call the Haskell interface to this last function tune :: Context -> Int -> IO ()). There are two parts to my question:
The encoding part is easily wrapped up in a Pipe Foo Bar IO (), but I would also like to expose tune. Since simultaneous use of the encoding context must be lock protected, I would need to take a lock at every iteration in the pipe, and protect tune with taking the same lock. But now I feel I'm forcing hidden locks on the user. Am I barking up the wrong tree here? How is this kind of situation normally resolved in the pipes ecosystem? In my case I expect the pipe that my specific code is part of to always run in its own thread, with tuning happening concurrently, but I don't want to force this point of view upon any users. Other packages in the pipes ecosystem do not seem to force their users like either.
An encoding context that is no longer used needs to be properly de-initialized. How does one, in the pipes ecosystem, ensure that such things (in this case performing som IO actions) are taken care of when the pipe is destroyed?
A concrete example would be wrapping a compression library, in which case the above can be:
The compression strength is tunable. We set up the pipe and it runs along merrily. How should one best go about allowing the compression strength setting to be changed while the pipe keeps running, assuming that concurrent access to the compression codec context must be serialized?
The compression library allocated a bunch of memory off the Haskell heap when set up, and we'll need to call some library function to clean this up when the pipe is torn down.
Thanks… this might all be obvious, but I'm quite new to the pipes ecosystem.
Edit: Reading this after posting, I'm quite sure it's the vaguest question I've ever asked here. Ugh! Sorry ;-)
Regarding (1), the general solution is to change your Pipe's type to:
Pipe (Either (Context, Int) Foo) Bar IO ()
In other words, it accepts both Foo inputs and tune requests, which it processes internally.
So let's then assume that you have two concurrent Producers corresponding to inputs and tune requests:
producer1 :: Producer Foo IO ()
producer2 :: Producer (Context, Int) IO ()
You can use pipes-concurrency to create a buffer that they both feed into, like this:
example = do
(output, input) <- spawn Unbounded
-- input :: Input (Either (Context, Int) Foo)
-- output :: Output (Either (Context, Int) Foo)
let io1 = runEffect $ producer1 >-> Pipes.Prelude.map Right >-> toOutput output
io2 = runEffect $ producer2 >-> Pipes.Prelude.map Left >-> toOutput output
as <- mapM async [io1, io2]
runEffect (fromInput >-> yourPipe >-> someConsumer)
mapM_ wait as
You can learn more about the pipes-concurrency library by reading this tutorial.
By forcing all tune requests to go through the same single-threaded Pipe you can ensure that you don't accidentally have two concurrent invocations of the tune function.
Regarding (2) there are two ways you can acquire a resource using pipes. The more sophisticated approach is to use the pipes-safe library, which provides a bracket function that you can use within a Pipe, but that is probably overkill for your purpose and only exists for acquiring and releasing multiple resources over the lifetime of a pipe. A simpler solution is just to use the following with idiom to acquire the pipe:
withEncoder :: (Pipe Foo Bar IO () -> IO r) -> IO r
withEncoder k = bracket acquire release $ \resource -> do
k (createPipeFromResource resource)
Then a user would just write:
withEncoder $ \yourPipe -> do
runEffect (someProducer >-> yourPipe >-> someConsumer)
You can optionally use the managed package, which simplifies the types a bit and makes it easier to acquire multiple resources. You can learn more about it from reading this blog post of mine.
Can IO actions in IO blocks call within unsafePerformIO be reordered?
I have effectively the IO function.
assembleInsts :: ... -> IO S.ByteString
assembleInsts ... = do
tmpInputFile <- generateUniqueTmpFile
writeFile tmpInputFile str
(ec,out,err) <- readProcessWithExitCode asm_exe [tmpInputFile] ""
-- asm generates binary output in tmpOutputFile
removeFile tmpInputFile
let tmpOutputFile = replaceExtension tmpIsaFile "bits" -- assembler creates this
bs <- S.readFile tmpOutputFile -- fails due to tmpOutputFile not existing
removeFile tmpOutputFile
return bs
where S.ByteString is a strict byte string.
Sadly, I need to call this in a tree of pure code far from the IO monad,
but since I the assembler behaves as a referentially transparent
(given unique files) tool, I figured for the time being I could make
an unsafe interface for the time being.
{-# NOINLINE assembleInstsUnsafe #-}
assembleInstsUnsafe :: ... -> S.ByteString
assembleInstsUnsafe args = unsafePerformIO (assembleInsts args)
In addition I added to the top of the module the following annotation
as per the documentation's (System.IO.Unsafe's) instructions.
{-# OPTIONS -fno-cse #-}
module Gen.IsaAsm where
(I tried to also add -fnofull-laziness as well, as per a reference that
I consulted, but this was rejected by the compiler. I don't think that
case applies here though.)
Running in ghci it reports the following error.
*** Exception: C:\Users\trbauer\AppData\Local\Temp\tempfile_13516_0.dat: openBinaryFile: does not exist (No such file or directory)
But if I remove removeFile tmpOutputFile, then it magically works.
Hence, it seems like the removeFile is executing ahead of the process termination.
Is this possible? The bytestring is strict, and I even tried to force the output at one point with a:
S.length bs `seq` return ()
before the removeFile.
Is there a way to dump intermediate code to find out what's going on?
(Maybe I can trace this with Process Monitor or something to find out.)
Unfortunately, I'd like to clean up within this operation (remove the file).
I think the exe version might work, but under ghci it fails (interpreted).
I am using GHC 7.6.3 from the last Haskell Platform.
I know unsafePerformIO is a really big hammer and has other risks associated with it, but it would really limit the complexity of my software change.
This may not be applicable, since it is based on assumptions unspecified in your question. In particular, this answer is based on the following two assumptions. S, which is unspecified, is Data.ByteString.Lazy and tmpDatFile, which is undefined, is tmpOutputFile.
import qualified Data.ByteString.Lazy as S
...
let tmpDatFile = tmpOutputFile
Possible Cause
If these assumptions are true, removeFile will run too early, even without the use of unsafePerformIO. The following code
import System.Directory
import qualified Data.ByteString.Lazy as S
assembleInsts = do
-- prepare a file, like asm might have generated
let tmpOutputFile = "dataFile.txt"
writeFile tmpOutputFile "a bit of text"
-- read the prepared file
let tmpDatFile = tmpOutputFile
bs <- S.readFile tmpOutputFile
removeFile tmpDatFile
return bs
main = do
bs <- assembleInsts
print bs
Results in the error
lazyIOfail.hs: DeleteFile "dataFile.txt": permission denied (The process cannot access the file because it is being used by another process.)
Removing the line removeFile tmpDatFile will make this code execute correctly, just like you describe, but leaving behind the temporary file isn't what is desired.
Possible Solution
Changing the import S to
import qualified Data.ByteString as S
instead results in the correct output,
"a bit of text".
Explanation
The documentation for Data.ByteSting.Lazy's readFile states that it will
Read an entire file lazily into a ByteString. The Handle will be held open until EOF is encountered.
Internally, readfile accomplishes this by calling unsafeInterleaveIO. unsafeInterleaveIO defers execution of the IO code until the term it returns is evaluated.
hGetContentsN :: Int -> Handle -> IO ByteString
hGetContentsN k h = lazyRead -- TODO close on exceptions
where
lazyRead = unsafeInterleaveIO loop
loop = do
c <- S.hGetSome h k -- only blocks if there is no data available
if S.null c
then do hClose h >> return Empty
else do cs <- lazyRead
return (Chunk c cs)
Because nothing tries to look at the constructor of the bs defined in the example above until it is printed, which doesn't happen until after removeFile has been executed, no chunks are read from the file (and the file is not closed) before removeFile is executed. Therefore, when removeFile is executed, the Handle opened by readFile is still open, and the file can't be removed.
Even if you are using unsafePerformIO, IO actions should not be reordered. If you want to be sure of that, you can use the -ddump-simpl flag to see the intermediate Core language which GHC produces, or even one of the other -dump-* flags showing all the compilation intermediate steps up to assembly.
I am aware that this answers what you asked, and not what you actually need, but you can rule out GHC bugs at least. It seems unlikely there's a bug affecting this in GHC, though.
Totally my fault.... sorry everyone. GHC does not reorder IO actions in an IO block under the above stated conditions as mentioned by those above. The assembler was just failing to assemble the output and create the assumed file. I simply forgot to check the exit code or the output stream of the assembler. I assumed the input to be syntactically correct since it is generated, the assembler rejected it and simply failed to create the file. It gave a valid error code and error diagnostic too, so that was really bad on my part. I may have been using readProcess the first time around, which raises an exception on a non-zero exit, but must have eventually changed this. I think the assembler had a bug where it didn't correctly indicate a failing exit code for some cases, and I had to change from readProcessWithExitCode.
I am still not sure why the error went away when I elided the removeFile.
I thought about deleting the question, but I a hoping the suggestions above help others debug similar (more valid) problems as well. I've been burned by the lazy IO thing Cirdec mentioned, and the -ddump-simpl flag mentioned by chi is good to know as well.
Is it safe to perform multiple actions using the same conduit value? Something like
do
let sink = sinkSocket sock
something $$ sink
somethingElse $$ sink
I recall that in the early versions of conduit there were some dirty hacks that made this unsafe. What's the current status?
(Note that sinkSocket doesn't close the socket.)
That usage is completely safe. The issue in older versions had to do with blurring the line between resumable and non-resumable components. With modern versions (I think since 0.4), the line is very clear between the two.
It might be safe to reuse sinks in the sense that the semantics for the "used" sink doesn't change. But you should be aware of another threat: space leaks.
The situation is analogous to lazy lists: you can consume a huge list lazily in a constant space, but if you process the list twice it will be kept in memory. The same thing might happen with a recursive monadic expression: if you use it once it's constant size, but if you reuse it the structure of the computation is kept in memory, resulting in space leak.
Here's an example:
import Data.Conduit
import Data.Conduit.List
import Control.Monad.Trans.Class (lift)
consumeN 0 _ = return ()
consumeN n m = do
await >>= (lift . m)
consumeN (n-1) m
main = do
let sink = consumeN 1000000 (\i -> putStrLn ("Got one: " ++ show i))
sourceList [1..9000000::Int] $$ sink
sourceList [1..22000000::Int] $$ sink
This program uses about 150M of ram on my machine, but if you remove the last line or repeat the definition of sink in both places, you get a nice constant space usage.
I agree that this is a contrived example (this was the first that came to my mind), and this is not very likely to happen with most Sinks. For example this will not happen with your sinkSocket. (Why is this contrived: because the control structure of the sink doesn't depend on the values it gets. And that is also why it can leak.) But, for example, for sources this would be much more common. (Many of the common Sources exhibit this behavior. The sourceList would be an obvious example, because it would actually keep the source list in memory. But, enumFromTo is no different, although there is no data to keep in memory, just the structure of the monadic computation.)
So, all in all, I think it's important to be aware of this.
I'm creating a small program to use with an irc bot that should take a string and then evaluate the string. For this I'm using the hint package, which work very well for my needs. The problem that I now have is that I want to be able to prevent evaluation of expressions that take a vary long to calculate e.g. 2^1000000000.
I tried using the System.Timeout package like this:
import Data.Maybe
import Language.Haskell.Interpreter
import System.Timeout
import System.Environment (getArgs)
main :: IO()
main = do
r <- timeout 500000 $ runInterpreter $ hEval arg
case r of
Nothing -> putStrLn "Timed out!"
Just x ->
case x of
Left err -> putStrLn (show err)
Right a -> putStrLn a
hEval e = do
setImportsQ [("Prelude", Nothing),("Data.List",Nothing)]
a <- eval e
return $ take 200 a
But it's not working, the timeout does not fire unless I put in such a short time that nothing can be evaluated. I read on the page for the Timeout package that it could have problems with some modules and have to let theme finish but my understanding is not good enough to know if Hint is such a module.
So any help on this would be appreciated, even if it's just to tell me that this isn't going to work.
GHC threads are cooperative. They can only yield or be terminated by asynchronous exceptions when they perform a memory allocation. This normally works fine, but someone malicious can write a tight loop that runs for a significant time without allocating.
The mueval package was created to deal with things like this. It's implemented in terms of hint, but with a lot of extra safety added in various ways.