I want something like
f :: [forall m. (Mutable v) (PrimState m) r -> m ()] -> v r -> v r -- illegal signature
f gs x = runST $ do
y <- thaw x
foldM_ (\_ g -> g y) undefined gs -- you get the idea
unsafeFreeze y
I'm essentially in the same position I was in this question where Vitus commented:
[I]f you want keep polymorphic functions inside some structure, you need either specialized data type (e.g. newtype I = I (forall a. a -> a)) or ImpredicativeTypes.
Also, see this question. The problem is, these are both really ugly solutions. So I've come up with a third alternative, which is to avoid the polymorphism altogether by running what "should" be a ST computation in IO instead. Thus f becomes:
f :: [(Mutable v) RealWorld r -> IO ()] -> v r -> v r
f gs x = unsafePerformIO $ do
y <- thaw x
foldM_ (\_ g -> g y) undefined gs -- you get the idea
unsafeFreeze y
I feel slightly dirty for going the unsafe IO route compared the to the "safe" ST route, but if my alternative is a wrapper or impredicative types... Apparently, I'm not alone.
Is there any reason I shouldn't use unsafePerformIO here? In this case, is it really unsafe at all? Are there performance considerations or anything else I should be aware of?
--------------EDIT----------------
An answer below shows me how to get around this problem altogether, which is great. But I'm still interested in the original question (implicaitons of runST vs unsafePerformIO when using mutable vectors) for educational purposes.
I can't say I understand the problem statement completely yet, but the following file compiles without error under GHC 7.6.2. It has the same body as your first example (and in particular doesn't call unsafePerformIO at all); the primary difference is that the forall is moved outside of all type constructors.
{-# LANGUAGE RankNTypes #-}
import Control.Monad
import Control.Monad.Primitive (PrimState)
import Control.Monad.ST
import Data.Vector.Generic hiding (foldM_)
f :: Vector v r => (forall m. [Mutable v (PrimState m) r -> m ()]) -> v r -> v r
f gs x = runST $ do
y <- thaw x
foldM_ (\_ g -> g y) undefined gs
unsafeFreeze y
Now let's tackle the the ST vs IO question. The reason it's called unsafePerformIO and not unusablePerformIO is because it comes with a proof burden that can't be checked by the compiler: the thing you are running unsafePerformIO on must behave as if it is referentially transparent. Since ST actions come with a (compiler-checked) proof that they behave transparently when executed with runST, this means there is no more danger in using unsafePerformIO on code that would typecheck in ST than there is in using runST.
BUT: there is danger from a software engineering standpoint. Since the proof is no longer compiler-checked, it's much easier for future refactoring to violate the conditions under which it's safe to use unsafePerformIO. So if it is possible to avoid it (as it seems to be here), you should take efforts to do so. (Additionally, "there is no more danger" doesn't mean "there is no danger": the unsafeFreeze call you are making has its own proof burden that you must satisfy; but then you already had to satisfy that proof burden for the ST code to be correct.)
Related
I'm going through The Monad Challenges.
At the section A Missed Generalization, I'm supposed to have at least this code (I've removed parts not relevant to the question), where Gen looks a lot like the State Monad,
-- Imports and other stuff that hide Prelude
-- For instance, the do notation is NOT available at this stage of the challenges
type Gen a = Seed -> (a,Seed)
genTwo :: Gen a -> (a -> Gen b) -> Gen b
genTwo g f s = let (a,s') = g s
in f a s'
mkGen :: a -> Gen a
mkGen a s = (a,s)
generalB :: (a -> b -> c) -> Gen a -> Gen b -> Gen c
-- I've implemented it as follows and it works
generalB f a b s = let (x,s') = a s
(y,s'') = b s'
in (f x y,s'')
The text of the "assignment" reads
[…] you might not have implemented generalB in terms of genTwo. Go back and look at your generalB implementation and if you didn’t write it in terms of genTwo, do that now and call it generalB2. Doing this should get rid of the state threading between generators.
Is unclear to me what the solution to this should be, especially in view of the fact that the paragraph above doesn't mention mkGen. Assuming I'm able to apply f to the inside of a and b, I would still get something of type c, which I have to shove in a Gen, and I don't see how I can do that without mkGen or, alternatively, without using (,) explicitly (as I did in the above implementation).
Even assuming that the text implies that I should use mkGen, how should I go about it to get rid of the state threading?
With some editing I was able to come up with this
generalB2' f a b = genTwo a (genTwo b . (mkGen .) . f)
but I hardly believe this is the intended solution, because it's far from being readable, in my opinion. Also, getting to this form was a bit harder than anything else so far in the challenges, but it was after all just mechanical, so it didn't really pose a difficulty from the point of view of understanding monads, I believe, so I really think I took a wrong turn here, and I'd like some help.
I wonder whether the authors of the challenges hang out here on StackOverflow.
Your solution is probably close to the intended solution, although you might be able to make it more readable by eta-expanding it. You might even consider writing it using do notation, but still use genTwo and mkGen.
As far as I can tell, mkGen is a 'disguised' return function, and genTwo likewise is a 'disguised' monadic bind (i.e. >>=).
The type of generalB (and generalB2) is equivalent to liftM2, which is implemented like this:
liftM2 :: (Monad m) => (a1 -> a2 -> r) -> m a1 -> m a2 -> m r
liftM2 f m1 m2 = do { x1 <- m1; x2 <- m2; return (f x1 x2) }
That is, in terms of return and >>= (which you don't see, because it's using do syntax).
As claimed in the documentation <$!> is the strict version of <$>, but surprisingly
<$!> :: Monad m => (a -> b) -> m a -> m b
f <$!> m = do
x <- m
let z = f x
z `seq` return z
instead of the more natural (in my opinion; because it keeps the weaker constraint and mimics $!)
<$!> :: Functor f => (a -> b) -> f a -> f b
f <$!> x = x `seq` (f <$> x)
I guess that appliying seq after the binding is different than the "natural" approach, but I don't know how different it is. My question is: Is there any reason which makes the "natural" approach useless, and that's why the implementation is constraint to Monad?
GHC's commit message includes the following two links which sheds more light on this function:
https://mail.haskell.org/pipermail/libraries/2013-November/021728.html
https://mail.haskell.org/pipermail/libraries/2014-April/022864.html
This was the reason which is mentioned by Johan Tibell for it (quoting from the linked mailing list):
It works on Monads instead of Functors as required by us inspecting
the argument.
This version is highly convenient if you want to work with
functors/applicatives in e.g. parser and avoid spurious thunks at the
same time. I realized that it was needed while fixing large space
usage (but not space-leak) issues in cassava.
I guess that appliying seq after the binding is different than the "natural" approach, but I don't know how different it is
Since haskell is functional, seq must work through data dependencies; it sets up a relationship: "when seq x y is evaluated to WHNF, a will have been as well".
The idea here is to pin the evaluation of a to the outer m a which we know must be evaluated for each >>= or <*> to proceed.
In your version:
Prelude> f <$!> x = x `seq` (f <$> x)
Prelude> let thunk = error "explode"
Prelude> case (+) <$!> Just thunk <*> Just thunk of ; Just _ -> "we can easily build up thunks"
"we can easily build up thunks"
I do wonder if there's a better solution possible though
Here's the code:
{-# LANGUAGE FlexibleContexts #-}
import Data.Int
import qualified Data.Vector.Unboxed as U
import qualified Data.Vector.Generic as V
{-# NOINLINE f #-} -- Note the 'NO'
--f :: (Num r, V.Vector v r) => v r -> v r -> v r
--f :: (V.Vector v Int64) => v Int64 -> v Int64 -> v Int64
--f :: (U.Unbox r, Num r) => U.Vector r -> U.Vector r -> U.Vector r
f :: U.Vector Int64 -> U.Vector Int64 -> U.Vector Int64
f = V.zipWith (+) -- or U.zipWith, it doesn't make a difference
main = do
let iters = 100
dim = 221184
y = U.replicate dim 0 :: U.Vector Int64
let ans = iterate ((f y)) y !! iters
putStr $ (show $ U.sum ans)
I compiled with ghc 7.6.2 and -O2, and it took 1.7 seconds to run.
I tried several different versions of f:
f x = U.zipWith (+) x
f x = (U.zipWith (+) x) . id
f x y = U.zipWith (+) x y
Version 1 is the same as the original while versions 2 and 3 run in in under 0.09 seconds (and INLINING f doesn't change anything).
I also noticed that if I make f polymorphic (with any of the three signatures above), even with a "fast" definition (i.e. 2 or 3), it slows back down...to exactly 1.7 seconds. This makes me wonder if the original problem is perhaps due to (lack of) type inference, even though I'm explicitly giving the types for the Vector type and element type.
I'm also interested in adding integers modulo q:
newtype Zq q i = Zq {unZq :: i}
As when adding Int64s, if I write a function with every type specified,
h :: U.Vector (Zq Q17 Int64) -> U.Vector (Zq Q17 Int64) -> U.Vector (Zq Q17 Int64)
I get an order of magnitude better performance than if I leave any polymorphism
h :: (Modulus q) => U.Vector (Zq q Int64) -> U.Vector (Zq q Int64) -> U.Vector (Zq q Int64)
But I should at least be able to remove the specific phantom type! It should be compiled out, since I'm dealing with a newtype.
Here are my questions:
Where is the slowdown coming from?
What is going on in versions 2 and 3 of f that affect performance in any way? It seems like a bug to me that (what amounts to) coding style can affect performance like this. Are there other examples outside of Vector where partially applying a function or other stylistic choices affect performance?
Why does polymorphism slow me down an order of magnitude independent of where the polymorphism is (i.e. in the vector type, in the Num type, both, or phantom type)? I know polymorphism makes code slower, but this is ridiculous. Is there a hack around it?
EDIT 1
I filed a issue with the Vector library page. I found a GHC
issue relating to this problem.
EDIT2
I rewrote the question after gaining some insight from #kqr's answer.
Below is the original for reference.
--------------ORIGINAL QUESTION--------------------
Here's the code:
{-# LANGUAGE FlexibleContexts #-}
import Control.DeepSeq
import Data.Int
import qualified Data.Vector.Unboxed as U
import qualified Data.Vector.Generic as V
{-# NOINLINE f #-} -- Note the 'NO'
--f :: (Num r, V.Vector v r) => v r -> v r -> v r
--f :: (V.Vector v Int64) => v Int64 -> v Int64 -> v Int64
--f :: (U.Unbox r, Num r) => U.Vector r -> U.Vector r -> U.Vector r
f :: U.Vector Int64 -> U.Vector Int64 -> U.Vector Int64
f = V.zipWith (+)
main = do
let iters = 100
dim = 221184
y = U.replicate dim 0 :: U.Vector Int64
let ans = iterate ((f y)) y !! iters
putStr $ (show $ U.sum ans)
I compiled with ghc 7.6.2 and -O2, and it took 1.7 seconds to run.
I tried several different versions of f:
f x = U.zipWith (+) x
f x = (U.zipWith (+) x) . U.force
f x = (U.zipWith (+) x) . Control.DeepSeq.force)
f x = (U.zipWith (+) x) . (\z -> z `seq` z)
f x = (U.zipWith (+) x) . id
f x y = U.zipWith (+) x y
Version 1 is the same as the original, version 2 runs in 0.111 seconds, and versions 3-6 run in in under 0.09 seconds (and INLINING f doesn't change anything).
So the order-of-magnitude slowdown appears to be due to laziness since force helped, but I'm not sure where the laziness is coming from. Unboxed types aren't allowed to be lazy, right?
I tried writing a strict version of iterate, thinking the vector itself must be lazy:
{-# INLINE iterate' #-}
iterate' :: (NFData a) => (a -> a) -> a -> [a]
iterate' f x = x `seq` x : iterate' f (f x)
but with the point-free version of f, this didn't help at all.
I also noticed something else, which could be just a coincidence and red herring:
If I make f polymorphic (with any of the three signatures above), even with a "fast" definition, it slows back down...to exactly 1.7 seconds. This makes me wonder if the original problem is perhaps due to (lack of) type inference, even though everything should be inferred nicely.
Here are my questions:
Where is the slowdown coming from?
Why does composing with force help, but using a strict iterate doesn't?
Why is U.force worse than DeepSeq.force? I have no idea what U.force is supposed to do, but it sounds a lot like DeepSeq.force, and seems to have a similar effect.
Why does polymorphism slow me down an order of magnitude independent of where the polymorphism is (i.e. in the vector type, in the Num type, or both)?
Why are versions 5 and 6, neither of which should have any strictness implications at all, just as fast as a strict function?
As #kqr pointed out, the problem doesn't seem to be strictness. So something about the way I write the function is causing the generic zipWith to be used rather than the Unboxed-specific version. Is this just a fluke between GHC and the Vector library, or is there something more general that can be said here?
While I don't have the definitive answer you want, there are two things that might help you along.
The first thing is that x `seq` x is, both semantically and computationally, the same thing as just x. The wiki says about seq:
A common misconception regarding seq is that seq x "evaluates" x. Well, sort of. seq doesn't evaluate anything just by virtue of existing in the source file, all it does is introduce an artificial data dependency of one value on another: when the result of seq is evaluated, the first argument must also (sort of; see below) be evaluated.
As an example, suppose x :: Integer, then seq x b behaves essentially like if x == 0 then b else b – unconditionally equal to b, but forcing x along the way. In particular, the expression x `seq` x is completely redundant, and always has exactly the same effect as just writing x.
What the first paragraph says is that writing seq a b doesn't mean that a will magically get evaluated this instant, it means that a will get evaluated as soon as b needs to be evaluated. This might occur later in the program, or maybe never at all. When you view it in that light, it is obvious that seq x x is a redundancy, because all it says is, "evaluate x as soon as x needs to be evaluated." Which of course is what would happen anyway if you had just written x.
This has two implications for you:
Your "strict" iterate' function isn't really any stricter than it would be without the seq. In fact, I have a hard time imagining how the iterate function could become any stricter than it already is. You can't make the tail of the list strict, because it is infinite. The main thing you can do is force the "accumulator", f x, but doing so doesn't give any significant performance increase on my system.[1]
Scratch that. Your strict iterate' does exactly the same thing as my bang pattern version. See the comments.
Writing (\z -> z `seq` z) does not give you a strict identity function, which I assume is what you were going for. In fact, the common identity function is as strict as you'll get – it will evaluate its result as soon as it is needed.
However, I peeked at the core GHC generates for
U.zipWith (+) y
and
U.zipWith (+) y . id
and there is only one big difference that my untrained eye can spot. The first one uses just a plain Data.Vector.Generic.zipWith (here's where your polymorphism coincidence might come into play – if GHC chooses a generic zipWith it will of course perform as if the code was polymorphic!) while the latter has exploded this single function call into almost 90 lines of state monad code and unpacked machine types.
The state monad code looks almost like the loops and destructive updates you would write in an imperative language, so I assume it's tailored pretty well to the machine it's running on. If I wasn't in such a hurry, I would take a longer look to see more exactly how it works and why GHC suddenly decided it needed a tight loop. I have attached the generated core as much for myself as anyone else who wants to take a look.[2]
[1]: Forcing the accumulator along the way: (This is what you already do, I misunderstood the code!)
{-# LANGUAGE BangPatterns #-}
iterate' f !x = x : iterate f (f x)
[2]: What core U.zipWith (+) y . id gets translated into.
Unlike other unsafe* operations, the documentation for unsafeInterleaveIO is not very clear about its possible pitfalls. So exactly when is it unsafe? I would like to know the condition for both parallel/concurrent and the single threaded usage.
More specifically, are the two functions in the following code semantically equivalent? If not, when and how?
joinIO :: IO a -> (a -> IO b) -> IO b
joinIO a f = do !x <- a
!x' <- f x
return x'
joinIO':: IO a -> (a -> IO b) -> IO b
joinIO' a f = do !x <- unsafeInterleaveIO a
!x' <- unsafeInterleaveIO $ f x
return x'
Here's how I would use this in practice:
data LIO a = LIO {runLIO :: IO a}
instance Functor LIO where
fmap f (LIO a) = LIO (fmap f a)
instance Monad LIO where
return x = LIO $ return x
a >>= f = LIO $ lazily a >>= lazily . f
where
lazily = unsafeInterleaveIO . runLIO
iterateLIO :: (a -> LIO a) -> a -> LIO [a]
iterateLIO f x = do
x' <- f x
xs <- iterateLIO f x' -- IO monad would diverge here
return $ x:xs
limitLIO :: (a -> LIO a) -> a -> (a -> a -> Bool) -> LIO a
limitLIO f a converged = do
xs <- iterateLIO f a
return . snd . head . filter (uncurry converged) $ zip xs (tail xs)
root2 = runLIO $ limitLIO newtonLIO 1 converged
where
newtonLIO x = do () <- LIO $ print x
LIO $ print "lazy io"
return $ x - f x / f' x
f x = x^2 -2
f' x = 2 * x
converged x x' = abs (x-x') < 1E-15
Although I would rather avoid using this code in serious applications because of the terrifying unsafe* stuff, I could at least be lazier than would be possible with the stricter IO monad in deciding what 'convergence' means, leading to (what I think is) more idiomatic Haskell. And this brings up another question:why is it not the default semantics for Haskell's (or GHC's?) IO monad? I've heard some resource management issues for lazy IO (which GHC only provides by a small fixed set of commands), but the examples typically given somewhat resemble like a broken makefile:a resource X depends on a resource Y, but if you fail to specify the dependency, you get an undefined status for X. Is lazy IO really the culprit for this problem? (On the other hand, if there is a subtle concurrency bug in the above code such as deadlocks I would take it as a more fundamental problem.)
Update
Reading Ben's and Dietrich's answer and his comments below, I have briefly browsed the ghc source code to see how the IO monad is implemented in GHC. Here I summerize my few findings.
GHC implements Haskell as an impure, non-referentially-transparent language. GHC's runtime operates by successively evaluating impure functions with side effects just like any other functional languages. This is why the evaluation order matters.
unsafeInterleaveIO is unsafe because it can introduce any kind of concurrency bugs even in a sigle-threaded program by exposing the (usually) hidden impurity of GHC's Haskell. (iteratee seems to be a nice and elegant solution for this, and I will certainly learn how to use it.)
the IO monad must be strict because a safe, lazy IO monad would require a precise (lifted) representation of the RealWorld, which seems impossible.
It's not just the IO monad and unsafe functions that are unsafe. The whole Haskell (as implemented by GHC) is potentially unsafe, and 'pure' functions in (GHC's) Haskell are only pure by convention and the people's goodwill. Types can never be a proof for purity.
To see this, I demonstrate how GHC's Haskell is not referentially transparent regardless of the IO monad, regardless of the unsafe* functions,etc.
-- An evil example of a function whose result depends on a particular
-- evaluation order without reference to unsafe* functions or even
-- the IO monad.
{-# LANGUAGE MagicHash #-}
{-# LANGUAGE UnboxedTuples #-}
{-# LANGUAGE BangPatterns #-}
import GHC.Prim
f :: Int -> Int
f x = let v = myVar 1
-- removing the strictness in the following changes the result
!x' = h v x
in g v x'
g :: MutVar# RealWorld Int -> Int -> Int
g v x = let !y = addMyVar v 1
in x * y
h :: MutVar# RealWorld Int -> Int -> Int
h v x = let !y = readMyVar v
in x + y
myVar :: Int -> MutVar# (RealWorld) Int
myVar x =
case newMutVar# x realWorld# of
(# _ , v #) -> v
readMyVar :: MutVar# (RealWorld) Int -> Int
readMyVar v =
case readMutVar# v realWorld# of
(# _ , x #) -> x
addMyVar :: MutVar# (RealWorld) Int -> Int -> Int
addMyVar v x =
case readMutVar# v realWorld# of
(# s , y #) ->
case writeMutVar# v (x+y) s of
s' -> x + y
main = print $ f 1
Just for easy reference, I collected some of the relevant definitions
for the IO monad as implemented by GHC.
(All the paths below are relative to the top directory of the ghc's source repository.)
-- Firstly, according to "libraries/base/GHC/IO.hs",
{-
The IO Monad is just an instance of the ST monad, where the state is
the real world. We use the exception mechanism (in GHC.Exception) to
implement IO exceptions.
...
-}
-- And indeed in "libraries/ghc-prim/GHC/Types.hs", We have
newtype IO a = IO (State# RealWorld -> (# State# RealWorld, a #))
-- And in "libraries/base/GHC/Base.lhs", we have the Monad instance for IO:
data RealWorld
instance Functor IO where
fmap f x = x >>= (return . f)
instance Monad IO where
m >> k = m >>= \ _ -> k
return = returnIO
(>>=) = bindIO
fail s = failIO s
returnIO :: a -> IO a
returnIO x = IO $ \ s -> (# s, x #)
bindIO :: IO a -> (a -> IO b) -> IO b
bindIO (IO m) k = IO $ \ s -> case m s of (# new_s, a #) -> unIO (k a) new_s
unIO :: IO a -> (State# RealWorld -> (# State# RealWorld, a #))
unIO (IO a) = a
-- Many of the unsafe* functions are defined in "libraries/base/GHC/IO.hs":
unsafePerformIO :: IO a -> a
unsafePerformIO m = unsafeDupablePerformIO (noDuplicate >> m)
unsafeDupablePerformIO :: IO a -> a
unsafeDupablePerformIO (IO m) = lazy (case m realWorld# of (# _, r #) -> r)
unsafeInterleaveIO :: IO a -> IO a
unsafeInterleaveIO m = unsafeDupableInterleaveIO (noDuplicate >> m)
unsafeDupableInterleaveIO :: IO a -> IO a
unsafeDupableInterleaveIO (IO m)
= IO ( \ s -> let
r = case m s of (# _, res #) -> res
in
(# s, r #))
noDuplicate :: IO ()
noDuplicate = IO $ \s -> case noDuplicate# s of s' -> (# s', () #)
-- The auto-generated file "libraries/ghc-prim/dist-install/build/autogen/GHC/Prim.hs"
-- list types of all the primitive impure functions. For example,
data MutVar# s a
data State# s
newMutVar# :: a -> State# s -> (# State# s,MutVar# s a #)
-- The actual implementations are found in "rts/PrimOps.cmm".
So, for example, ignoring the constructor and assuming referential transparency,
we have
unsafeDupableInterleaveIO m >>= f
==> (let u = unsafeDupableInterleaveIO)
u m >>= f
==> (definition of (>>=) and ignore the constructor)
\s -> case u m s of
(# s',a' #) -> f a' s'
==> (definition of u and let snd# x = case x of (# _,r #) -> r)
\s -> case (let r = snd# (m s)
in (# s,r #)
) of
(# s',a' #) -> f a' s'
==>
\s -> let r = snd# (m s)
in
case (# s, r #) of
(# s', a' #) -> f a' s'
==>
\s -> f (snd# (m s)) s
This is not what we would normally get from binding usual lazy state monads.
Assuming the state variable s carries some real meaning (which it does not), it looks more like a concurrent IO (or interleaved IO as the function rightly says) than a lazy IO as we would normally mean by 'lazy state monad' wherein despite the laziness the states are properly threaded by an associative operation.
I tried to implement a truely lazy IO monad, but soon realized that in order to define a lazy monadic composition for the IO datatype, we need to be able to lift/unlift the RealWorld. However this seems impossible because there is no constructor for both State# s and RealWorld. And even if that were possible, I would then have to represent the precise, functional represenation of our RealWorld which is impossible,too.
But I'm still not sure whether the standard Haskell 2010 breaks referential transparency or the lazy IO is bad in itself. At least it seems entirely possible to build a small model of the RealWorld on which the lazy IO is perfectly safe and predictable. And there might be a good enough approximation that serves many practical purposes without breaking the referential transparency.
At the top, the two functions you have are always identical.
v1 = do !a <- x
y
v2 = do !a <- unsafeInterleaveIO x
y
Remember that unsafeInterleaveIO defers the IO operation until its result is forced -- yet you are forcing it immediately by using a strict pattern match !a, so the operation is not deferred at all. So v1 and v2 are exactly the same.
In general
In general, it is up to you to prove that your use of unsafeInterleaveIO is safe. If you call unsafeInterleaveIO x, then you have to prove that x can be called at any time and still produce the same output.
Modern sentiment about Lazy IO
...is that Lazy IO is dangerous and a bad idea 99% of the time.
The chief problem that it is trying to solve is that IO has to be done in the IO monad, but you want to be able to do incremental IO and you don't want to rewrite all of your pure functions to call IO callbacks to get more data. Incremental IO is important because it uses less memory, allowing you to operate on data sets that don't fit in memory without changing your algorithms too much.
Lazy IO's solution is to do IO outside of the IO monad. This is not generally safe.
Today, people are solving the problem of incremental IO in different ways by using libraries like Conduit or Pipes. Conduit and Pipes are much more deterministic and well-behaved than Lazy IO, solve the same problems, and do not require unsafe constructs.
Remember that unsafeInterleaveIO is really just unsafePerformIO with a different type.
Example
Here is an example of a program that is broken due to lazy IO:
rot13 :: Char -> Char
rot13 x
| (x >= 'a' && x <= 'm') || (x >= 'A' && x <= 'M') = toEnum (fromEnum x + 13)
| (x >= 'n' && x <= 'z') || (x >= 'N' && x <= 'Z') = toEnum (fromEnum x - 13)
| otherwise = x
rot13file :: FilePath -> IO ()
rot13file path = do
x <- readFile path
let y = map rot13 x
writeFile path y
main = rot13file "test.txt"
This program will not work. Replacing the lazy IO with strict IO will make it work.
Links
From Lazy IO breaks purity by Oleg Kiselyov on the Haskell mailing list:
We demonstrate how lazy IO breaks referential transparency. A pure
function of the type Int->Int->Int gives different integers depending
on the order of evaluation of its arguments. Our Haskell98 code uses
nothing but the standard input. We conclude that extolling the purity
of Haskell and advertising lazy IO is inconsistent.
...
Lazy IO should not be considered good style. One of the common
definitions of purity is that pure expressions should evaluate to the
same results regardless of evaluation order, or that equals can be
substituted for equals. If an expression of the type Int evaluates to
1, we should be able to replace every occurrence of the expression with
1 without changing the results and other observables.
From Lazy vs correct IO by Oleg Kiselyov on the Haskell mailing list:
After all, what could be more against
the spirit of Haskell than a `pure' function with observable side
effects. With Lazy IO, one indeed has to choose between correctness
and performance. The appearance of such code is especially strange
after the evidence of deadlocks with Lazy IO, presented on this list
less than a month ago. Let alone unpredictable resource usage and
reliance on finalizers to close files (forgetting that GHC does not
guarantee that finalizers will be run at all).
Kiselyov wrote the Iteratee library, which was the first real alternative to lazy IO.
Laziness means that when (and whether) exactly a computation is actually carried out depends on when (and whether) the runtime implementation decides it needs the value. As a Haskell programmer you completely relinquish control over the evaluation order (except by the data dependencies inherent in your code, and when you start playing with strictness to force the runtime to make certain choices).
That's great for pure computations, because the result of a pure computation will be exactly the same whenever you do it (except that if you carry out computations that you don't actually need, you might encounter errors or fail to terminate, when another evaluation order might allow the program to terminate successfully; but all non-bottom values computed by any evaluation order will be the same).
But when you're writing IO-dependent code, evaluation order matters. The whole point of IO is to provide a mechanism for building computations whose steps depend on and affect the world outside the program, and an important part of doing that is that those steps are explicitly sequenced. Using unsafeInterleaveIO throws away that explicit sequencing, and relinquishes control of when (and whether) the IO operation is actually carried out to the runtime system.
This is unsafe in general for IO operations, because there may be dependencies between their side-effects which cannot be inferred from the data dependencies inside the program. For example, one IO action might create a file with some data in it, and another IO action might read the same file. If they're both executed "lazily", then they'll only get run when the resulting Haskell value is needed. Creating the file is probably IO () though, and it's quite possible that the () is never needed. That could mean that the read operation is carried out first, either failing or reading data that was already in the file, but not the data that should have been put there by the other operation. There's no guarantee that the runtime system will execute them in the right order. To program correctly with a system that always did this for IO you'd have to be able to accurately predict the order in which the Haskell runtime will choose to perform the various IO actions.
Treat unsafeInterlaveIO as promise to the compiler (which it cannot verify, it's just going to trust you) that it doesn't matter when the IO action is carried out, or whether it's elided entirely. This is really what all the unsafe* functions are; they provide facilities that are not safe in general, and for which safety cannot be automatically checked, but which can be safe in particular instances. The onus is on you to ensure that your use of them is in fact safe. But if you make a promise to the compiler, and your promise is false, then unpleasant bugs can be the result. The "unsafe" in the name is to scare you into thinking about your particular case and deciding whether you really can make the promise to the compiler.
Basically everything under "Update" in the question is so confused it's not even wrong, so please try to forget it when you're trying to understand my answer.
Look at this function:
badLazyReadlines :: Handle -> IO [String]
badLazyReadlines h = do
l <- unsafeInterleaveIO $ hGetLine h
r <- unsafeInterleaveIO $ badLazyReadlines h
return (l:r)
In addition to what I'm trying to illustrate: the above function also doesn't handle reaching the end of the file. But ignore that for now.
main = do
h <- openFile "example.txt" ReadMode
lns <- badLazyReadlines h
putStrLn $ lns ! 4
This will print the first line of "example.txt", because the 5th element in the list is actually the first line that's read from the file.
Your joinIO and joinIO' are not semantically equivalent. They will usually be the same, but there's a subtlety involved: a bang pattern makes a value strict, but that's all it does. Bang patterns are implemented using seq, and that does not enforce a particular evaluation order, in particular the following two are semantically equivalent:
a `seq` b `seq` c
b `seq` a `seq` c
GHC can evaluate either b or a first before returning c. Indeed, it can evaluate c first, then a and b, then return c. Or, if it can statically prove a or b are non-bottom, or that c is bottom, it doesn't have to evaluate a or b at all. Some optimisations do genuinely make use of this fact, but it doesn't come up very often in practice.
unsafeInterleaveIO, by contrast, is sensitive to all or any of those changes – it does not depend on the semantic property of how strict some function is, but the operational property of when something is evaluated. So all of the above transformations are visible to it, which is why it's only reasonable to view unsafeInterleaveIO as performing its IO non-deterministically, more or less whenever it feels appropriate.
This is, in essence, why unsafeInterleaveIO is unsafe - it is the only mechanism in normal use that can detect transformations that ought to be meaning-preserving. It's the only way you can detect evaluation, which by rights ought to be impossible.
As an aside, it's probably fair to mentally prepend unsafe to every function from GHC.Prim, and probably several other GHC. modules as well. They're certainly not ordinary Haskell.
I'm trying to wrap my head around the enumerator library and ran into a situation where I want to build a new Enumeratee in terms of two existing Enumeratees. Let's say I have the enumeratees:
e1 :: Enumeratee x y m b
e2 :: Enumeratee y z m b
I feel that I should be able to combine them into one enumeratee
e3 :: Enumeratee x z m b
but I couldn't find an existing function to do this in the package. I tried to write such a function myself, but my understanding of iteratees is still so limited that I couldn't figure out a way to get all the complex types to match.
Did I just miss some basic combinator, or are Enumeratees even supposed to be composable with each other?
In theory they are composable, but the types are a bit tricky. The difficulty is that the final parameter b of the first enumeratee isn't actually b; it's another iteratee!. Here's the type of the ><> operator from iteratee, which composes enumeratees:
Prelude Data.Iteratee> :t (><>)
(><>)
:: (Monad m, Nullable s1) =>
(forall x. Enumeratee s1 s2 m x)
-> Enumeratee s2 s3 m a -> Enumeratee s1 s3 m a
Note the extra forall in the first enumeratee; this indicates that a Rank-2 type is at work. If the enumerator author wants to maintain H98 compatibility (I believe this was one of the original goals), this approach is unavailable.
It is possible to write this type signature in a form which doesn't require Rank-2 types, but it's either longer, not clear from the type that it's actually two enumeratee's that are being composed, or both. For example, this is ghc's inferred type for (><>):
Prelude Data.Iteratee> :t (><>>)
(><>>)
:: (Monad m, Nullable s) =>
(b -> Iteratee s m (Iteratee s' m a1))
-> (a -> b) -> a -> Iteratee s m a1
Although these types are for iteratee combinators, hopefully it's enough information you'll be able to apply them to enumerator.
I ran with this problem a while ago, you need to have an Iteratee first (or an Enumerator) in order to make the composition of Enumeratees.
You can either start by doing this:
module Main where
import Data.Enumerator
import qualified Data.Enumerator.List as EL
main :: IO ()
main = run_ (enum $$ EL.consume) >>= print
where
enum = (enumList 5 [1..] $= EL.isolate 100) $= EL.filter pairs
pairs = (==0) . (`mod` 2)
The previous code composes a list of enumeratees together to create a new enumerator, and then it is applied to the consume Iteratee.
The ($=) serves to compose an Enumerator and an Enumeratee to create a new enumerator, while the (=$) can be used to compose an Iteratee with an Enumeratee to create a new Iteratee. I recommend the latter given that types won't bust your balls when composing a list of Enumeratees using (=$):
module Main where
import Data.Enumerator
import qualified Data.Enumerator.List as EL
main :: IO ()
main = run_ (enumList 5 [1..] $$ it) >>= print
where
it = foldr (=$)
EL.consume
[ EL.isolate 100
, EL.filter ((==0) . (`mod` 2))
]
If you would try to implement the same function above by creating an Enumerator instead of an Iteratee, you will get an infinite recursive type error when using foldl' ($=) (enumList 5 [1..]) [list-of-enumeratees].
Hope this helps.