module Main where
import Control.Parallel(par,pseq)
import Text.Printf
import Control.Exception
import System.CPUTime
import Data.List
import IO
import Data.Char
import Control.DeepSeq
time :: IO t -> IO t
time a = do
start <- getCPUTime
v <- a
end <- getCPUTime
let diff = (fromIntegral (end - start)) / (10^12)
printf "Computation time: %0.3f sec\n" (diff :: Double)
return v
learquivo :: FilePath -> IO ([[Int]])
learquivo s = do
content <- readFile s
return (read content)
main :: IO ()
main = do
t5 <- getCPUTime
content <- learquivo "mkList1.txt"
let !mapasort = rnf $ map sort content
t6 <- getCPUTime
let diffft6t5 = (fromIntegral (t6 - t5)) / (10^12)
printf "Computation time Mapasort: %0.3f sec\n" (diffft6t5 :: Double)
How to tell if it evaluates all elements of content?
let !mapasort = rnf $ map sort content
I used the line in winghci:
*Main> let !mapasort = rnf $ map sort content
But, returned:
*Main> mapasort ()
Thanks
I see two questions:
1) Why is mapsort evaluating to unit, ().
Because the rnf function always returns (). See the documentation.
2) Is everything evaluated
Yes. The DeepSeq instance (which is where rnf lives) for list just calls the deepseq instance for each element in the list:
rnf [] = ()
rnf (x:xs) = rnf x `seq` rnf xs
Your elements all Ints, which have a correct NFData instance.
And I'd add two more questions:
3) How should this benchmarking be done correctly?
Using Criterion. There are many Criterion advocates here on SO, you can find answers that would serve as good examples with a search.
4) How should this evaluation be forced for non-benchmarking purposes?
Using the parallel package.
import Control.Parallel.Strategies
...
let !mapsort = (map sort content) `using` (evalList rdeepseq)
or still using rnf:
let mapsort = map sort content
!_ = rnf mapsort
Related
With the following code:
(lazy_test.hs)
-- Testing lazy evaluation of monadically constructed lists, using State.
import Control.Monad.State
nMax = 5
foo :: Int -> State [Int] Bool
foo n = do
modify $ \st -> n : st
return (n `mod` 2 == 1)
main :: IO ()
main = do
let ress = for [0..nMax] $ \n -> runState (foo n) []
sts = map snd $ dropWhile (not . fst) ress
print $ head sts
for = flip map
I can set nMax to 5, or 50,000,000, and I get approximately the same run time:
nMax = 5:
$ stack ghc lazy_test.hs
[1 of 1] Compiling Main ( lazy_test.hs, lazy_test.o )
Linking lazy_test ...
$ time ./lazy_test
[1]
real 0m0.019s
user 0m0.002s
sys 0m0.006s
nMax = 50,000,000:
$ stack ghc lazy_test.hs
[1 of 1] Compiling Main ( lazy_test.hs, lazy_test.o )
Linking lazy_test ...
$ time ./lazy_test
[1]
real 0m0.020s
user 0m0.002s
sys 0m0.005s
which is as I expect, given my understanding of lazy evaluation mechanics.
However, if I switch from State to StateT:
(lazy_test2.hs)
-- Testing lazy evaluation of monadically constructed lists, using StateT.
import Control.Monad.State
nMax = 5
foo :: Int -> StateT [Int] IO Bool
foo n = do
modify $ \st -> n : st
return (n `mod` 2 == 1)
main :: IO ()
main = do
ress <- forM [0..nMax] $ \n -> runStateT (foo n) []
let sts = map snd $ dropWhile (not . fst) ress
print $ head sts
for = flip map
then I see an extreme difference between the respective run times:
nMax = 5:
$ stack ghc lazy_test2.hs
[1 of 1] Compiling Main ( lazy_test2.hs, lazy_test2.o )
Linking lazy_test2 ...
$ time ./lazy_test2
[1]
real 0m0.019s
user 0m0.002s
sys 0m0.004s
nMax = 50,000,000:
$ stack ghc lazy_test2.hs
[1 of 1] Compiling Main ( lazy_test2.hs, lazy_test2.o )
Linking lazy_test2 ...
$ time ./lazy_test2
[1]
real 0m29.758s
user 0m25.488s
sys 0m4.231s
And I'm assuming that's because I'm losing lazy evaluation of the monadically constructed list, when I switch to the StateT-based implementation.
Is that correct?
Can I recover lazy evaluation of a monadically constructed list, while keeping with the StateT-based implementation?
In your example, you're only running one foo action per runState, so your use of State and/or StateT is essentially irrelevant. You can replace the use of foo with the equivalent:
import Control.Monad
nMax = 50000000
main :: IO ()
main = do
ress <- forM [0..nMax] $ \n -> return (n `mod` 2 == 1, [n])
let sts = map snd $ dropWhile (not . fst) ress
print $ head sts
and it behaves the same way.
The issue is the strictness of the IO monad. If you ran this computation in the Identity monad instead:
import Control.Monad
import Data.Functor.Identity
nMax = 50000000
main :: IO ()
main = do
let ress = runIdentity $ forM [0..nMax] $ \n -> return (n `mod` 2 == 1, [n])
let sts = map snd $ dropWhile (not . fst) ress
print $ head sts
then it would run lazily.
If you want to run lazily in the IO monad, you need to do it explicitly with unsafeInterleaveIO, so the following would work:
import System.IO.Unsafe
import Control.Monad
nMax = 50000000
main :: IO ()
main = do
ress <- lazyForM [0..nMax] $ \n -> return (n `mod` 2 == 1, [n])
let sts = map snd $ dropWhile (not . fst) ress
print $ head sts
lazyForM :: [a] -> (a -> IO b) -> IO [b]
lazyForM (x:xs) f = do
y <- f x
ys <- unsafeInterleaveIO (lazyForM xs f)
return (y:ys)
lazyForM [] _ = return []
The other answer by K A Buhr explains why State vs StateT is not the pertinent factor (IO is), and also points out how your example is strangely constructed (in that the State(T) part isn't actually used as each number uses a new state []). But aside from those points, I'm not sure I would say "losing lazy evaluation of the monadically constructed list", because if we understand something like "lazy evaluation = evaluated only when needed", then foo does indeed need to run on every element on the input list in order to perform all the effects, so lazy evaluation is not being "lost". You are getting what you asked for. (It just so happens that foo doesn't perform any IO, and perhaps someone else can comment with if it's ever possible for a compiler/GHC to optimize it away on this basis, but you can easily see why GHC does the naive thing here.)
This is a common, well-known problem in Haskell. There are various libraries (best known of which are streaming, pipes, conduit) which solve the problem by giving you streams (basically lists) which are lazy in the effects too. If I recreate your example in a streaming style,
import Data.Function ((&))
import Control.Monad.State
import Streaming
import qualified Streaming.Prelude as S
foo :: Int -> StateT [Int] IO Bool
foo n =
(n `mod` 2 == 1) <$ modify (n:)
nMax :: Int
nMax = 5000000
main :: IO ()
main = do
mHead <- S.head_ $ S.each [0..nMax]
& S.mapM (flip runStateT [] . foo)
& S.dropWhile (not . fst)
print $ snd <$> mHead
then both versions run practically instantaneously. To make the difference more apparent, imagine that foo also called print "hi". Then the streaming version, being lazy in the effects, would print only twice, whereas your original versions would both print nMax times. As they're lazy in the effects, then the whole list doesn't need to be traversed in order to short-circuit and finish early.
I'm trying to follow this tutorial: https://wiki.haskell.org/Tutorials/Programming_Haskell/String_IO.
In the last part 7 Extension: using SMP parallelism I copy the code but it fails to compile with this error message
/home/dhilst/parallelspell.hs:13:20: error:
Variable not in scope: chunk :: Int -> [String] -> t
I searched for chunks at Hoogle and got Data.Text.Internal.Lazy, but this seems to be an internal module. And I couldn't import it anyway.
Here is the code:
import Data.Set hiding (map)
import Data.Maybe
import Data.Char
import Text.Printf
import System.IO
import System.Environment
import Control.Concurrent
import Control.Monad
main = do
(f,g,n) <- readFiles
let dict = fromList (lines f)
work = chunk n (words g)
run n dict work
run n dict work = do
chan <- newChan
errs <- getChanContents chan
mapM_ (forkIO . thread chan dict) (zip [1..n] work)
wait n errs 0
wait n xs i = when (i < n) $ case xs of
Nothing : ys -> wait n ys $! i+1
Just s : ys -> putStrLn s >> wait n ys i
thread chan dict (me,xs) = do
mapM_ spellit xs
writeChan chan Nothing
where spellit w = when (spell dict w) $
writeChan chan . Just $ printf "Thread %d: %-25s" (me::Int) w
spell d w = w `notMember` d
readFiles = do
[s,n] <- getArgs
f <- readFile "/usr/share/dict/words"
g <- readFile s
return (f,g, read n)
And here is the compilation line:
ghc -O --make -threaded parallelspell.hs
--
Update: I write my own version of chunk based on this quest:How to partition a list in Haskell?
chunk :: Int -> [a] -> [[a]]
chunk _ [] = []
chunk n xs = (take n xs) : (chunk n (drop n xs))
Still, does this means that the tutorial that I'm following is very old and out of date!? Can anyone confirm if that function already existed some day or if I'm missing something?
Regards,
Looks like the tutorial just forgot to define chunk. I encourage you to update the wiki to include a suitable definition.
I'm learning Haskell for two years now and I'm still confused, whats the best (fastest) way to read tons of numbers from a single input line.
For learning I registered into hackerearth.com trying to solve every challenge in Haskell. But now I'm stuck with a challenge because I run into timeout issues. My program is just too slow for beeing accepted by the site.
Using the profiler I found out it takes 80%+ of the time for parsing a line with lots of integers. The percentage gets even higher when the number of values in the line increases.
Now this is the way, I'm reading numbers from an input line:
import qualified Data.ByteString.Char8 as C8
main = do
scores <- fmap (map (fst . fromJust . C8.readInt) . C8.words) C8.getLine :: IO [Int]
Is there any way to get the data faster into the variable?
BTW: The biggest testcase consist of a line with 200.000 9-digits values. Parsing takes incredible long (> 60s).
It's always difficult to declare a particular approach "the fastest", since there's almost always some way to squeeze out more performance. However, an approach using Data.ByteString.Char8 and the general method you suggest should be among the fastest methods for reading numbers. If you encounter a case where performance is poor, the problem likely lies elsewhere.
To give some concrete results, I generated a 191Meg file of 20 million 9-digit numbers, space-separate on a single line. I then tried several general methods of reading a line of numbers and printing their sum (which, for the record, was 10999281565534666). The obvious approach using String:
reader :: IO [Int]
reader = map read . words <$> getLine
sum' xs = sum xs -- work around GHC ticket 10992
main = print =<< sum' <$> reader
took 52secs; a similar approach using Text:
import qualified Data.Text as T
import qualified Data.Text.IO as T
import qualified Data.Text.Read as T
readText = map parse . T.words <$> T.getLine
where parse s = let Right (n, _) = T.decimal s in n
ran in 2.4secs (but note that it would need to be modified to handle negative numbers!); and the same approach using Char8:
import qualified Data.ByteString.Char8 as C
readChar8 :: IO [Int]
readChar8 = map parse . C.words <$> C.getLine
where parse s = let Just (n, _) = C.readInt s in n
ran in 1.4secs. All examples were compiled with -O2 on GHC 8.0.2.
As a comparison benchmark, a scanf-based C implementation:
/* GCC 5.4.0 w/ -O3 */
#include <stdio.h>
int main()
{
long x, acc = 0;
while (scanf(" %ld", &x) == 1) {
acc += x;
}
printf("%ld\n", acc);
return 0;
}
ran in about 2.5secs, on par with the Text implementation.
You can squeeze a bit more performance out of the Char8 implementation. Using a hand-rolled parser:
readChar8' :: IO [Int]
readChar8' = parse <$> C.getLine
where parse = unfoldr go
go s = do (n, s1) <- C.readInt s
let s2 = C.dropWhile C.isSpace s1
return (n, s2)
runs in about 0.9secs -- I haven't tried to determine why there's a difference, but the compiler must be missing an opportunity to perform some optimization of the words-to-readInt pipeline.
Haskell Code for Reference
Make some numbers with Numbers.hs:
-- |Generate 20M 9-digit numbers:
-- ./Numbers 20000000 100000000 999999999 > data1.txt
import qualified Data.ByteString.Char8 as C
import Control.Monad
import System.Environment
import System.Random
main :: IO ()
main = do [n, a, b] <- map read <$> getArgs
nums <- replicateM n (randomRIO (a,b))
let _ = nums :: [Int]
C.putStrLn (C.unwords (map (C.pack . show) nums))
Find their sum with Sum.hs:
import Data.List
import qualified Data.Text as T
import qualified Data.Text.IO as T
import qualified Data.Text.Read as T
import qualified Data.Char8 as C
import qualified Data.ByteString.Char8 as C
import System.Environment
-- work around https://ghc.haskell.org/trac/ghc/ticket/10992
sum' xs = sum xs
readString :: IO [Int]
readString = map read . words <$> getLine
readText :: IO [Int]
readText = map parse . T.words <$> T.getLine
where parse s = let Right (n, _) = T.decimal s in n
readChar8 :: IO [Int]
readChar8 = map parse . C.words <$> C.getLine
where parse s = let Just (n, _) = C.readInt s in n
readHand :: IO [Int]
readHand = parse <$> C.getLine
where parse = unfoldr go
go s = do (n, s1) <- C.readInt s
let s2 = C.dropWhile C.isSpace s1
return (n, s2)
main = do [method] <- getArgs
let reader = case method of
"string" -> readString
"text" -> readText
"char8" -> readChar8
"hand" -> readHand
print =<< sum' <$> reader
where:
./Sum string <data1.txt # 54.3 secs
./Sum text <data1.txt # 2.29 secs
./Sum char8 <data1.txt # 1.34 secs
./Sum hand <data1.txt # 0.91 secs
I can compile the following, but when I use :t qsort I get the long, complex type signature below. However, when I add that, the program will no longer type check, without additional imports (see answer below). My best guess at defining the type is also pasted below, but doen't type check (types and monads are bewildering).
Ok, so I could add an extra import statement and it type checks, but I've never seen a code signature this complex on any code published online. So my question is:
I can drop Ord t and exhange Int for t and it checks, but how do I substitute in ST or IO for Control.Monad.Primitive.PrimMonad, and what do I do with v.
I'm trying to write a blog post that provides an example of how to use these various Monads, and by contrast for Arrays the type signature of qsort is a more manageable qsort :: (STArray s Int Int) -> Int -> Int -> ST s (). (For those that want to understand s there are lots of explanations on line, all of which passed slightly above my head - I get that it is a clever trick to get the type checker itself to prevent the author writing code where data from the Monad leaks out and thus leads to impurity.)
import Control.Monad
import Control.Monad.ST
import qualified Data.Vector as V
import qualified Data.Vector.Generic.Mutable as MV
main = do
lines <- BS.lines `fmap` BS.readFile "10.txt"
let
inputData = Prelude.map (maybe (error "can't read Int") fst . BS.readInt) lines
initialImmutableVector = V.fromList inputData
print $ runST $ do
state <- V.thaw initialImmutableVector
qsort state 0 (Prelude.length inputData - 1)
frozen <- V.freeze state
return frozen
--qsort :: (MV.MVector s Int) -> Int -> Int -> ST s ()
--qsort
-- :: (Ord t, Control.Monad.Primitive.PrimMonad m, MV.MVector v t) =>
-- v (Control.Monad.Primitive.PrimState m) t -> Int -> Int -> m ()
qsort vec min mx =
if mx - min < 1 then
return ()
else do
p <- MV.read vec min
final_i <- foldM (partitioner p) (min+1) [(min+1)..mx]
swap min (final_i - 1)
qsort vec min (final_i-2)
qsort vec final_i mx
where
swap i j = do
vec_i <- MV.read vec i
vec_j <- MV.read vec j
MV.write vec i vec_j
MV.write vec j vec_i
partitioner p i acc = do
vec_acc <- MV.read vec acc
if vec_acc > p then
return i
else do
swap i acc
return $ i+1
Importing function(-s) does not import their types. If your code explicitly refers to the type, it has to be imported. You can use imported functions without importing the types of their arguments or return values as long as the code does not explicitly refer to those types. Once you start using types or classes explicitly, you have to import them, this is intentional.
It looks like your last attempt is correct, but perhaps you didn't import everything you needed? The code you pasted as-is doesn't compile with or without type signatures. Here's a very slightly modified version of what you have that works just fine for me (with GHC 7.8.3):
import Control.Monad
import Control.Monad.ST
import qualified Data.Vector as V
import qualified Data.Vector.Generic.Mutable as MV
import qualified Data.ByteString.Char8 as BS
import Control.Monad.Primitive (PrimState, PrimMonad)
import Prelude hiding (lines, min)
main :: IO ()
main = do
lines <- BS.lines `fmap` BS.readFile "10.txt"
let
inputData = map (maybe (error "can't read Int") fst . BS.readInt) lines
initialImmutableVector = V.fromList inputData
print $ runST $ do
state <- V.thaw initialImmutableVector
qsort state 0 (Prelude.length inputData - 1)
frozen <- V.freeze state
return frozen
qsort :: (Ord t, PrimMonad m, MV.MVector v t)
=> v (PrimState m) t -> Int -> Int -> m ()
qsort vec min mx =
if mx - min < 1 then
return ()
else do
p <- MV.read vec min
final_i <- foldM (partitioner p) (min+1) [(min+1)..mx]
swap min (final_i - 1)
qsort vec min (final_i-2)
qsort vec final_i mx
where
swap i j = do
vec_i <- MV.read vec i
vec_j <- MV.read vec j
MV.write vec i vec_j
MV.write vec j vec_i
partitioner p i acc = do
vec_acc <- MV.read vec acc
if vec_acc > p then
return i
else do
swap i acc
return $ i+1
What is wrong here, is Lazy Evaluation too?
teste.hs
module Main where
import Control.Parallel(par,pseq)
import Text.Printf
import Control.Exception
import System.CPUTime
import Data.List
import IO
import Data.Char
import Control.DeepSeq
--Calcula o tempo entre o inicio e o fim de rodagem do programa
time :: IO t -> IO t
time a = do
start <- getCPUTime
v <- a
end <- getCPUTime
let diff = (fromIntegral (end - start)) / (10^12)
printf "Computation time: %0.3f sec\n" (diff :: Double)
return v
learquivo :: FilePath -> IO ([[Int]])
learquivo s = do
conteudo <- readFile s
return (read conteudo)
main :: IO ()
main = do
conteudo <- learquivo "mkList1.txt"
mapasort <- return (map sort conteudo)
time $ mapasort `seq` return ()
*Main> main
Computation time: 0.125 sec
mkList1.txt is a list of 100 lists of 100 random numbers in each, more or less like this: [[23,45,89,78,89 ...], [4783, 44, 34 ...]...]
I did a test printing mapasort:
time $ print ("Sort usando map = ", mapasort)
And the computing time increased considerably so I think something is wrong.
Computation time: 1.188 sec
Thanks
Yes, this is due to Haskell's laziness. You're trying to get around the laziness by using seq, but since seq is "shallow" (i.e. it doesn't traverse the whole structure of the expression - only the "outer" layer), it will force the evaluation of the map, but not the evaluation of the sorts.
To fix this either use deepseq instead of seq or, even better, use a library for benchmarking instead of using getCPUTime.