I came up with the following code for the raw-input problem, as discussed in SO Haskell read raw keyboard input. Unfortunately, when I go to ghci, run getAllInput, and hit the right arrow key, it never returns. Unless I kill it pretty quickly, it seems to eat all my memory so that other applications stop responding and I have to restart the OS. In the Activity Monitor I can see the memory for the ghc process go quickly into the gigabytes.
(1) I think the problem is in the recursive call of go, which is evaluated lazily by calling hReady before getChar; this means hReady keeps returning true and the stack grows forever. Is that plausible?
(2) I'm used to languages in which this would soon cause a stack overflow exception, so it wouldn't prevent me from working. Is there any general way to protect against this massive memory leak? Maybe starting ghci with a hard limit on memory use?
import System.IO
-- For example, should get "\ESC[C" from the user hitting the right arrow key.
getAllInput :: IO [Char]
getAllInput =
let
go :: IO [Char] -> IO [Char]
go chars = do
more <- hReady stdin
if more then go (added chars getChar) else chars
added :: IO [Char] -> IO Char -> IO [Char]
added chars char = do
chars1 <- chars
char1 <- char
return (chars1 ++ [char1])
in do
hSetBuffering stdin NoBuffering
firstChar <- getChar
go (return [firstChar])
I'm running ghci 7.10.3 in OS X 10.11.6. I cleaned up the code in some obvious ways, basically following the similar SO answer: putting the getChar call in its own line fixes the problem. But I'd like to understand this better in case it bites me again.
You have an infinite loop in go. If hReady returns true, then you call go again, which immediately calls hReady again, which will of course return true, and so on. You probably assume that added will be run because of go (added chars getChar), but it will not; it is just building an IO action and passing it to go as an argument, but that argument is only used when hReady returns False. The name chars is misleading -- chars is actually an I/O procedure which will return a list of characters when it is eventually run.
In general, when using monads, a "normal" function signature looks like:
:: Foo -> Bar -> Baz -> IO Quuz
That is, the monad (IO) is only present on the return value, not the arguments. A signature like
:: IO Foo -> IO Bar
usually indicates that something higher-order is going on, for example that this function will perhaps execute its argument multiple times or in a new context.
I recommend the signatures
go :: [Char] -> IO [Char]
added :: [Char] -> Char -> IO [Char]
and trying to get the program to compile from there.
You should also try changing added to be a pure function
added :: [Char] -> Char -> [Char]
because it doesn't actually have any side effects. The implementation and usage will need to be altered a little, though.
Summarizing the discussion...
(1) As explained by luqui's answer, the memory leak is caused by an infinite loop in which lazy evaluation prevents getChar from being called. Adding a line like c <- getChar forces the call to happen, so that go (added chars c) is now safe to call.
(2) Starting GHCi with a limited heap size, as in ghci getchars.hs +RTS -M100m, will interrupt the memory leak before it eats up all memory. See the GHC Users Guide for further details.
Related
My code uses a resource that can be described as a pointer; I'll use a void pointer here for simplicity. The resource must be closed after the computation with it finishes, so the Control.Exception.bracket function is a natural choice to make sure the code won't leak if an error occurs:
run :: (Ptr () -> IO a) -> IO a
run action = bracket acquireResource closeResource action
-- no eta reduction for clarity
The downside of this pattern is that the resource will always be closed after action completes. AFAIU this means that it isn't possible to do something like
cont <- run $ \ptr -> do
a <- someAction ptr
return (\x -> otherActionUsingResource ptr a x)
cont ()
The resource will already be close by the time cont is executed. Now my approach is to use a ForeignPtr instead:
run' :: (ForeignPtr () -> IO a) -> IO a
run' action = do
ptr <- acquireResource
foreignPtr <- newForeignPtr closeResourceFunPtr ptr
action foreignPtr
Now it seems that this is roughly equivalent to the first version, minor typing differences and resource closing latency aside. However, I do wonder whether this is true, or if I miss something. Can some error conditions can lead to different outcomes with those two versions? Are ForeignPtr safe to use in this way?
If you want to do this, I'd recommend avoiding that run', which makes it look like you're going to close the resource. Do something like this instead.
acquire :: IO (ForeignPtr ())
acquire action = mask $ \unmask -> do
ptr <- unmask acquireResource
newForeignPtr closeResourceFunPtr ptr
As Carl pointed out in a comment, it's important that exceptions be masked between acquiring the resource and installing the finalizer to close it; otherwise an asynchronous exception could be delivered in between and cause a resource leak.
The challenge with anything of this sort is that you're leaving it up to the user and/or garbage collector to make sure the resource gets freed. The original Ptr-based code made the lifespan explicit. Now it's not. Many people believe that explicit lifespans are better for critical resources. What ForeignPtr gives you, automatic finalization by the GC, these people consider poor design. So think carefully! Is this a cheap resource (like a little bit of malloced memory) that you just want to free eventually? Or is it something expensive (like a file descriptor) that you really want to be sure about?
Side note: Ptr () and ForeignPtr () aren't very idiomatic. Usually the type argument should be a Haskell type representing whatever is being pointed to.
For example
main = do
let ls = [0..10000000]
print ls
This will create the array 'inplace', using O(1) memory.
The following edit causes the program to run out of memory while executing.
main = do
let ls = [0..10000000]
print ls
print ls
ls in this case must be kept in memory to be printed again. It would actually be heaps more memory efficient to recalculate the array again 'inplace' than to try to keep this in place. That's an aside though. My real question is "how and when does GHC communicate to the runtime system that ls can be destroyed while it's created in O(1) time?" I understand that liveness analysis can find this information, I'm just wondering where the information is used. Is it the garbage collector that is passed this info? Is it somehow compiled away? (If I look at the compiled core from GHC then both examples use eftInt, so if it's a compiler artifact then it must happen at a deeper level).
edit: My question was more about finding where this optimization took place. I thought maybe it was in the GC, which was fed some information from some liveness check in the compilation step. Due to the answers so far I'm probably wrong. This is most likely then happening at some lower level before core, so cmm perhaps?
edit2: Most of the answers here assume that the GC knows that ls is no longer referenced in the first example, and that it is referenced again in the second example. I know the basics of GC and I know that arrays are linked lists, etc. My question is exactly HOW the GC knows this. The answer could probably be only (a) it is getting extra information from the compiler, or (b) it doesn't need to know this, that this information is handled 100% by the compiler
ls here is a lazy list, not an array. In practice, it's closer to a stream or generator in another language.
The reason the first code works fine is that it never actually has the whole list in memory. ls is defined lazily and then consume element-by-element by print. As print is going along, there are no other references to the beginning of ls so list items can be garbage collected immediately.
In theory, GHC could realize that it's more efficient to not store the list in memory between the two prints but instead recompute it. However, this is not always desirable—a lot of code is actually faster if things are only evaluated once—and, more importantly, would make the execution model even more confusing for programmers.
This explanation is probably a lie, especially because I'm making it up as I go, but that shouldn't be a problem.
The essential mistake you're making is assuming that a value is live if a variable bound to it is in scope in a live expression. This is simply wrong. A value bound to a variable is only live as a result if it is actually mentioned in a live expression.
The job of the runtime is very simple
Execute the expression bound to main.
There is no 2.
We can think of this execution as involving a couple different steps that repeat over and over:
Figure out what to do now.
Figure out what to do next.
So we start with some main expression. From the start, the "root set" for GC consists of those names that are used in that main expression, not the things that are in scope in that expression. If I write
foo = "Hi!"
main = print "Bye!"
then since foo is not mentioned in main, it is not in the root set at the beginning, and since it is not even mentioned indirectly by anything mentioned by main, it is dead right from the start.
Now suppose we take a more interesting example:
foo = "Hi!"
bar = "Bye!"
main = print foo >> print bar
Now foo is mentioned in main, so it starts out live. We evaluate main to weak head normal form to find out what to do, and we get, approximately,
(primitive operation that prints out "Hi!") >> print bar
Note that foo is no longer mentioned, so it is dead!
Now we execute that primitive operation, printing "Hi!", and our "to do list" is reduced to
print bar
Now we evaluate that to WHNF, and get, roughly,
(primitive operation to print "Bye!")
Now bar is dead. We print "Bye!" and exit.
Consider, now, the first program you described:
main = do
let ls = [0..10000000]
print ls
This desugars to
main =
let ls = [0..10000000]
in print ls
This is where we start. The "root set" at the beginning is everything mentioned in the in clause of the expression. So we conceptually have ls and print to start out. Now we can imagine that print, specialized to [Integer], looks something vaguely like the following (this is greatly simplified, and will print out the list differently, but that really doesn't matter).
print xs = case xs of
[] -> return ()
(y:ys) = printInteger y >> print ys
So when we start executing main (What do we do now? What will we do afterwards?), we are trying to calculate print ls. To do this, we pattern match on the first constructor of ls, which forces ls to be evaluated to WHNF. We find the second pattern, y:ys, matches, so we replace print ls with print Integer y >> print ys, where y points to the first element of ls and ys points to the thunk representing the second list constructor of ls. But note that ls itself is now dead! Nothing is pointing to it! So as print forces bits of the list, the bits it has already passed become dead.
In contrast, when you have
main =
let ls = ...
in print ls >> print ls
and you start executing, you start by calculating the thing to do first (print ls). You get
(printInteger y >> print ys) >> print ls
Everything is the same, except that the second part of the expression now points to ls. So even though the first part will be dropping pieces of the list as it goes, the second part will keep holding on to the beginning, keeping it all live.
Edit
Let me try explaining with something a little simpler than IO. Pretend that your program is an expression of type [Int], and the job of the runtime system is to print each element on its own line. So we can write
countup m n = if m == n then [] else m : countup (m+1)
main = countup 0 1000
The runtime system holds a value representing everything that it should print. Let's call the "current value" whatPrint. The RTS needs to follow a process:
Set whatPrint to main.
Is whatPrint empty? If so, I'm done, and can exit the program. If not, it is a cons, printNow : whatPrint'.
Calculate printNow and print it.
Set whatPrint to whatPrint'
Go to step 1.
In this model, the "root set" for garbage collection is just whatPrint.
In a real program, we don't produce a list; we produce an IO action. But such an action is also a lazy data structure (conceptually). You can think of >>=, return, and each primitive IO operation as a constructor for IO. Think of it as
data IO :: * -> * where
Return :: a -> IO a
Bind :: IO a -> (a -> IO b) -> IO b
PrintInt :: Int -> IO ()
ReadInt :: IO Int
...
The initial value of whatShouldIDo is main, but its value evolves over time. Only what it points to directly is in the root set. There is no magical analysis necessary.
Can IO actions in IO blocks call within unsafePerformIO be reordered?
I have effectively the IO function.
assembleInsts :: ... -> IO S.ByteString
assembleInsts ... = do
tmpInputFile <- generateUniqueTmpFile
writeFile tmpInputFile str
(ec,out,err) <- readProcessWithExitCode asm_exe [tmpInputFile] ""
-- asm generates binary output in tmpOutputFile
removeFile tmpInputFile
let tmpOutputFile = replaceExtension tmpIsaFile "bits" -- assembler creates this
bs <- S.readFile tmpOutputFile -- fails due to tmpOutputFile not existing
removeFile tmpOutputFile
return bs
where S.ByteString is a strict byte string.
Sadly, I need to call this in a tree of pure code far from the IO monad,
but since I the assembler behaves as a referentially transparent
(given unique files) tool, I figured for the time being I could make
an unsafe interface for the time being.
{-# NOINLINE assembleInstsUnsafe #-}
assembleInstsUnsafe :: ... -> S.ByteString
assembleInstsUnsafe args = unsafePerformIO (assembleInsts args)
In addition I added to the top of the module the following annotation
as per the documentation's (System.IO.Unsafe's) instructions.
{-# OPTIONS -fno-cse #-}
module Gen.IsaAsm where
(I tried to also add -fnofull-laziness as well, as per a reference that
I consulted, but this was rejected by the compiler. I don't think that
case applies here though.)
Running in ghci it reports the following error.
*** Exception: C:\Users\trbauer\AppData\Local\Temp\tempfile_13516_0.dat: openBinaryFile: does not exist (No such file or directory)
But if I remove removeFile tmpOutputFile, then it magically works.
Hence, it seems like the removeFile is executing ahead of the process termination.
Is this possible? The bytestring is strict, and I even tried to force the output at one point with a:
S.length bs `seq` return ()
before the removeFile.
Is there a way to dump intermediate code to find out what's going on?
(Maybe I can trace this with Process Monitor or something to find out.)
Unfortunately, I'd like to clean up within this operation (remove the file).
I think the exe version might work, but under ghci it fails (interpreted).
I am using GHC 7.6.3 from the last Haskell Platform.
I know unsafePerformIO is a really big hammer and has other risks associated with it, but it would really limit the complexity of my software change.
This may not be applicable, since it is based on assumptions unspecified in your question. In particular, this answer is based on the following two assumptions. S, which is unspecified, is Data.ByteString.Lazy and tmpDatFile, which is undefined, is tmpOutputFile.
import qualified Data.ByteString.Lazy as S
...
let tmpDatFile = tmpOutputFile
Possible Cause
If these assumptions are true, removeFile will run too early, even without the use of unsafePerformIO. The following code
import System.Directory
import qualified Data.ByteString.Lazy as S
assembleInsts = do
-- prepare a file, like asm might have generated
let tmpOutputFile = "dataFile.txt"
writeFile tmpOutputFile "a bit of text"
-- read the prepared file
let tmpDatFile = tmpOutputFile
bs <- S.readFile tmpOutputFile
removeFile tmpDatFile
return bs
main = do
bs <- assembleInsts
print bs
Results in the error
lazyIOfail.hs: DeleteFile "dataFile.txt": permission denied (The process cannot access the file because it is being used by another process.)
Removing the line removeFile tmpDatFile will make this code execute correctly, just like you describe, but leaving behind the temporary file isn't what is desired.
Possible Solution
Changing the import S to
import qualified Data.ByteString as S
instead results in the correct output,
"a bit of text".
Explanation
The documentation for Data.ByteSting.Lazy's readFile states that it will
Read an entire file lazily into a ByteString. The Handle will be held open until EOF is encountered.
Internally, readfile accomplishes this by calling unsafeInterleaveIO. unsafeInterleaveIO defers execution of the IO code until the term it returns is evaluated.
hGetContentsN :: Int -> Handle -> IO ByteString
hGetContentsN k h = lazyRead -- TODO close on exceptions
where
lazyRead = unsafeInterleaveIO loop
loop = do
c <- S.hGetSome h k -- only blocks if there is no data available
if S.null c
then do hClose h >> return Empty
else do cs <- lazyRead
return (Chunk c cs)
Because nothing tries to look at the constructor of the bs defined in the example above until it is printed, which doesn't happen until after removeFile has been executed, no chunks are read from the file (and the file is not closed) before removeFile is executed. Therefore, when removeFile is executed, the Handle opened by readFile is still open, and the file can't be removed.
Even if you are using unsafePerformIO, IO actions should not be reordered. If you want to be sure of that, you can use the -ddump-simpl flag to see the intermediate Core language which GHC produces, or even one of the other -dump-* flags showing all the compilation intermediate steps up to assembly.
I am aware that this answers what you asked, and not what you actually need, but you can rule out GHC bugs at least. It seems unlikely there's a bug affecting this in GHC, though.
Totally my fault.... sorry everyone. GHC does not reorder IO actions in an IO block under the above stated conditions as mentioned by those above. The assembler was just failing to assemble the output and create the assumed file. I simply forgot to check the exit code or the output stream of the assembler. I assumed the input to be syntactically correct since it is generated, the assembler rejected it and simply failed to create the file. It gave a valid error code and error diagnostic too, so that was really bad on my part. I may have been using readProcess the first time around, which raises an exception on a non-zero exit, but must have eventually changed this. I think the assembler had a bug where it didn't correctly indicate a failing exit code for some cases, and I had to change from readProcessWithExitCode.
I am still not sure why the error went away when I elided the removeFile.
I thought about deleting the question, but I a hoping the suggestions above help others debug similar (more valid) problems as well. I've been burned by the lazy IO thing Cirdec mentioned, and the -ddump-simpl flag mentioned by chi is good to know as well.
I need to make sure that some actions are run on a specific OS thread. I wrote an API where this thread runs a loop listening to a TQueue and executes the given commands. From the API user side, there is an opaque value that is really just a newtype over this queue.
One problem is that what I really need is to embed arbitrary actions (type IO a), but I believe I can't directly exchange messages of that type. So I currently have something like this (pseudo code) :
makeSafe :: RubyInterpreter -> IO a -> IO (Either RubyError a)
makeSafe (RubyInterpreter q) a = do
mv <- newEmptyTMVarIO
-- embedded is of type IO (), letting me send this in my queue
let embedded = handleErrors a >>= atomically . putTMVar mv
atomically (writeTQueue q (SomeMessage embedded))
atomically (readTMVar mv)
(for more details, this is for the hruby package)
edit - clarifications :
Being able to send actions of type IO a would be nicer, but is not my main objective.
My main problem is that you can shoot yourself in the foot with this API, for example if there is a makeSafe call in the IO action that is passed as a parameter, this will hang.
My secondary problem is that this solution feels a bit contrived, and I wondered if there was a nicer/safer solution around.
I want to write a function
forkos_try :: IO (Maybe α) -> IO (Maybe α)
which Takes a command x. x is an imperative operation which first mutates state, and then checks whether that state is messed up or not. (It does not do anything external, which would require some kind of OS-level sandboxing to revert the state.)
if x evaluates to Just y, forkos_try returns Just y.
otherwise, forkos_try rolls back state, and returns Nothing.
Internally, it should fork() into threads parent and child, with x running on child.
if x succeeds, child should keep running (returning x's result) and parent should die
otherwise, parent should keep running (returning Nothing) and child should die
Question: What's the way to write something with equivalent, or more powerful semantics than forkos_try? N.B. -- the state mutated (by x) is in an external library, and cannot be passed between threads. Hence, the semantic of which thread to keep alive is important.
Formally, "keep running" means "execute some continuation rest :: Maybe α -> IO () ". But, that continuation isn't kept anywhere explicit in code.
For my case, I think it will (for the time) work to write it in different style, using forkOS (which takes the entire computation child will run), since I can write an explicit expression for rest. But, it troubles me that I can't figure out how do this with the primitive function forkOS -- one would think it would be general enough to support any specific case (which could appear as a high-level API, like forkos_try).
EDIT -- please see the example code with explicit rest if the problem's still not clear [ http://pastebin.com/nJ1NNdda ].
p.s. I haven't written concurrency code in a while; hopefully my knowledge of POSIX fork() is correct! Thanks in advance.
Things are a lot simpler to reason about if you model state explicitly.
someStateFunc :: (s -> Maybe (a, s))
-- inside some other function
case someStateFunc initialState of
Nothing -> ... -- it failed. stick with initial state
Just (a, newState) -> ... -- it suceeded. do something with
-- the result and new state
With immutable state, "rolling back" is simple: just keep using initialState. And "not rolling back" is also simple: just use newState.
So...I'm assuming from your explanation that this "external library" performs some nontrivial IO effects that are nevertheless restricted to a few knowable and reversible operations (modify a file, an IORef, etc). There is no way to reverse some things (launch the missiles, write to stdout, etc), so I see one of two choices for you here:
clone the world, and run the action in a sandbox. If it succeeds, then go ahead and run the action in the Real World.
clone the world, and run the action in the real world. If it fails, then replace the Real World with the snapshot you took earlier.
Of course, both of these are actually the same approach: fork the world. One world runs the action, one world doesn't. If the action succeeds, then that world continues; otherwise, the other world continues. You are proposing to accomplish this by building upon forkOS, which would clone the entire state of the program, but this would not be sufficient to deal with, for example, file modifications. Allow me to suggest instead an approach that is nearer to the simplicity of immutable state:
tryIO :: IO s -> (s -> IO ()) -> IO (Maybe a) -> IO (Maybe a)
tryIO save restore action = do
initialState <- save
result <- action
case result of
Nothing -> restore initialState >> return Nothing
Just x -> return (Just x)
Here you must provide some data structure s, and a way to save to and restore from said data structure. This allows you the flexibility to perform any cloning you know to be necessary. (e.g. save could copy a certain file to a temporary location, and then restore could copy it back and delete the temporary file. Or save could copy the value of certain IORefs, and then restore could put the value back.) This approach may not be the most efficient, but it's very straightforward.