I'm doing two programs in Haskell, which one of them provides a tree filled with values.
The other program has to get the same tree now filled. I searched about it, but i haven't found something about how to do something like it.
For example i execute ./Generate and it saves the tree with values. Then i execute ./Work and it works with the values of the tree. Can somebody please help me?
The simplest way may be
data MyData = ... deriving (Read, Show)
producer
makeMyData :: MyData
makeMyData = ....
main = writeFile "output.data" (show makeMyData)
consumer
ioUseMyData :: MyData -> IO ()
ioUseMyData myData = ....
main = readFile "output.data" >>= ioUseMyData . read
you can use standard input/output using getContents and putStrLn.
Complete example:
-- probably as module
data Tree = Node { value :: Int
, left :: Tree
, right :: Tree
}
| Empty
deriving (Read, Show)
-- your producer program
producerProgram = do
let makeTree = Node 3 (Node 5 Empty Empty) (Node 7 Empty Empty)
writeFile "output.data" (show makeTree)
-- your consumer program
consumerProgram = do
let ioUseTree t = do
let countNodes Empty = 0
countNodes (Node _ l r) = 1 + countNodes l + countNodes r
putStrLn $ "Tree with " ++ show (countNodes t) ++ " nodes"
readFile "output.data" >>= ioUseTree . read
-- simulate call both
main = do
-- produce
producerProgram
-- consume
consumerProgram
result
Tree with 3 nodes
replacing
writeFile "output.data" (show makeTree)
by
print makeTree
and
readFile "output.data" >>= ioUseTree . read
by
getContents >>= ioUseTree . read
you can use pipes (bash, cmd.exe, ...)
$ ./producer | ./consumer
Tree with 3 nodes
The easiest way is: don't use 2 separate programs. If you really need 2 separate programs then there are 3 things you will need to choose:
A way of converting the tree into a String or ByteString
A way of sending the string or bytestring to the other program
A way of converting the String or ByteString back into a tree
For #1 and #3: it's probably easiest to use show and read, but you can also define your own functions if you want as long as they both define the same format.
For #2: if the 2 programs are meant to be run separately then you have no choice: you have to use a file. The easiest way to do so is using the writeFile and readFile functions. If the 2 programs are meant to be run simultaneously you can also use the functions in Network.Socket
Related
I'm trying to learn how to work with IO in Haskell by writing a function that, if there is a flag, will take a list of points from a file, and if there is no flag, it asks the user to enter them.
dispatch :: [String] -> IO ()
dispatch argList = do
if "file" `elem` argList
then do
let (path : otherArgs) = argList
points <- getPointsFile path
else
print "Enter a point in the format: x;y"
input <- getLine
if (input == "exit")
then do
print "The user inputted list:"
print $ reverse xs
else (inputStrings (input:xs))
if "help" `elem` argList
then help
else return ()
dispatch [] = return ()
dispatch _ = error "Error: invalid args"
getPointsFile :: String -> IO ([(Double, Double)])
getPointsFile path = do
handle <- openFile path ReadMode
contents <- hGetContents handle
let points_str = lines contents
let points = foldl (\l d -> l ++ [tuplify2 $ splitOn ";" d]) [] points_str
hClose handle
return points
I get this: do-notation in pattern Possibly caused by a missing 'do'?` after `if "file" `elem` argList.
I'm also worried about the binding issue, assuming that I have another flag that says which method will be used to process the points. Obviously it waits for points, but I don't know how to make points visible not only in if then else, constructs. In imperative languages I would write something like:
init points
if ... { points = a}
else points = b
some actions with points
How I can do something similar in Haskell?
Here's a fairly minimal example that I've done half a dozen times when I'm writing something quick and dirty, don't have a complicated argument structure, and so can't be bothered to do a proper job of setting up one of the usual command-line parsing libraries. It doesn't explain what went wrong with your approach -- there's an existing good answer there -- it's just an attempt to show what this kind of thing looks like when done idiomatically.
import System.Environment
import System.Exit
import System.IO
main :: IO ()
main = do
args <- getArgs
pts <- case args of
["--help"] -> usage stdout ExitSuccess
["--file", f] -> getPointsFile f
[] -> getPointsNoFile
_ -> usage stderr (ExitFailure 1)
print (frobnicate pts)
usage :: Handle -> ExitCode -> IO a
usage h c = do
nm <- getProgName
hPutStrLn h $ "Usage: " ++ nm ++ " [--file FILE]"
hPutStrLn h $ "Frobnicate the points in FILE, or from stdin if no file is supplied."
exitWith c
getPointsFile :: FilePath -> IO [(Double, Double)]
getPointsFile = {- ... -}
getPointsNoFile :: IO [(Double, Double)]
getPointsNoFile = {- ... -}
frobnicate :: [(Double, Double)] -> Double
frobnicate = {- ... -}
if in Haskell doesn't inherently have anything to do with control flow, it just switches between expressions. Which, in Haskell, happen to include do blocks of statements (if we want to call them that), but you still always need to make that explicit, i.e. you need to say both then do and else do if there are multiple statements in each branch.
Also, all the statements in a do block need to be indented to the same level. So in your case
if "file" `elem` argList
...
if "help" `elem` argList
Or alternatively, if the help check should only happen in the else branch, it needs to be indented to the statements in that do block.
Independent of all that, I would recommend to avoid parsing anything in an IO context. It is usually much less hassle and easier testable to first parse the strings into a pure data structure, which can then easily be processed by the part of the code that does IO. There are libraries like cmdargs and optparse-applicative that help with the parsing part.
I'm trying to get performance increases in a program I have that parses XML. The program can parse multiple XML files so I thought that I could make this run in parallel, but all my attempts have resulted in lower performance!
For XML parsing, I am using HXT.
I have a run function defined like this:
run printTasks xs = pExec xs >>= return . concat >>= doPrint printTasks 1
'pExec' is given a list of file names and is defined as:
pExec xs = do
ex <- mapM exec xs
as <- ex `usingIO` parList rdeepseq
return as
where 'exec' is defined as:
exec = runX . process
threadscope shows only one thread e ver being used (until the very end).
Can anyone explain why I have failed so miserably to parallelise this code?
In case it helps:
exec :: FilePath -> [CV_scene]
pExec :: [FilePath] -> IO [[CV_scene]]
data CV_scene = Scene [CV_layer] Time deriving (Show)
data CV_layer = Layer [DirtyRects] SourceCrop deriving (Show)
data Rect = Rect Int Int Int Int deriving (Show)-- Left Top Width Height
instance NFData CV_scene where
rnf = foldScene reduceScene
where reduceScene l t = rnf (seq t l)
instance NFData CV_layer where
rnf = foldLayer reduceLayer
where reduceLayer d s = rnf (seq s d)
instance NFData Rect where
rnf = foldRect reduceRect
where reduceRect l t w h = rnf [l,t,w,h]
type SourceCrop = Rect
type DirtyRect = Rect
type Time = Int64
Thanks in advance for your help!
First, it looks like you mislabeled the signature of exec, which should probably be:
exec :: FilePath -> IO [CV_scene]
Now for the important part. I've commented inline on what I think you think is going on.
pExec xs = do
-- A. Parse the file found at each location via exec.
ex <- mapM exec xs
-- B. Force the lazy parsing in parallel.
as <- ex `usingIO` parList rdeepseq
return as
Note that line A does not happen in paralell, which you might think is okay since it will just set up the parsing thunks which are forced in parallel in B. This is a fair assumption, and a clever use of laziness, but the results pull that into question for me.
I suspect that the implementation of exec forces most of the parsing before line B is even reached so that the deep seq doesn't do much. That fits pretty well with my experince parsing and the profiling supports that explanation.
Without the ability to test your code, I can only make the following suggestions. First try separating the parsing of the file from the IO and put the parsing in the parallel execution strategy. In that case lines A and B become something like:
ex <- mapM readFile xs
as <- ex `usingIO` parList (rdeepseq . exec')
with exec' the portion of exec after the file is read from disk.
exec' :: FilePath -> [CVScene]
Also, you may not even need rdeepSeq after this change.
As an alternative, you can do the IO and parsing in parallel using Software Transactional Memory. STM approaches are normally used for separate IO threads which act more like services, rather than pure computations. But if for some reason you cant get the strategies based approach to work, this might be worth a try.
import Control.Concurrent.STM.TChan --(from stm package)
import Control.Concurrent(forkIO)
pExec'' :: [FilePath] -> IO [[CVSene]]
pExec'' xs = do
-- A. create [(Filename,TChan [CVScene])]
tcx <- mapM (\x -> (x,) <$> newTChanIO) xs
-- B. do the reading/parsing in separate threads
mapM_ (forkIO . exec'') tcx
-- C. Collect the results
cvs <- mapM (atomically . readTChan . snd) tcx
exec'' :: [(FilePath,TChan [CVScene])] -> IO ()
exec'' (x,tch) = do
--D. The original exec function
cv <- exec x
--E. Put on the channel fifo buffer
atomically $ writeTChan tch cv
Good luck!
I have a following program in Haskell:
processDate :: String -> IO ()
processDate date = do
...
let newFlattenedPropertiesWithPrice = filter (notYetInserted date existingProperties) flattenedPropertiesWithPrice
geocodedProperties <- propertiesWithGeocoding newFlattenedPropertiesWithPrice
propertiesWithGeocoding :: [ParsedProperty] -> IO [(ParsedProperty, Maybe LatLng)]
propertiesWithGeocoding properties = do
let addresses = fmap location properties
let batchAddresses = chunksOf 100 addresses
batchGeocodedLocations <- mapM geocodeAddresses batchAddresses
let geocodedLocations = fromJust $ concat <$> sequence batchGeocodedLocations
return (zip properties geocodedLocations)
geocodeAddresses :: [String] -> IO (Maybe [Maybe LatLng])
geocodeAddresses addresses = do
mapQuestKey <- getEnv "MAP_QUEST_KEY"
geocodeResponse <- openURL $ mapQuestUrl mapQuestKey addresses
return $ geocodeResponseToResults geocodeResponse
geocodeResponseToResults :: String -> Maybe [Maybe LatLng]
geocodeResponseToResults inputResponse =
latLangs
where
decodedResponse :: Maybe GeocodingResponse
decodedResponse = decodeGeocodingResponse inputResponse
latLangs = fmap (fmap geocodingResultToLatLng . results) decodedResponse
decodeGeocodingResponse :: String -> Maybe GeocodingResponse
decodeGeocodingResponse inputResponse = Data.Aeson.decode (fromString inputResponse) :: Maybe GeocodingResponse
It reads a list of properties (homes and apartments) from html files, parses them, geocodes the addresses and saves the results into sqlite db.
Everything works fine except for a very high memory usage (around 800M).
By commenting code out I have pinpointed the problem to be the geocoding step.
I send 100 addresses at a time to MapQuest api (https://developer.mapquest.com/documentation/geocoding-api/batch/get/).
The response for 100 addresses is quite massive so it might be one of the culprits, but 800M? I feel like it holds to all of the results until the end which drives the memory usage so high.
After commenting out the geocoding part of the program memory usage is around 30M which is fine.
You can get the full version which reproduces the issue here: https://github.com/Leonti/haskell-memory-so
I'm quite a newbie in Haskell, so not sure how I can optimize it.
Any ideas?
Cheers!
It might be worth recording that this turned out to be a simple streaming problem arising from use of mapM and sequence, which with replicateM and traverse and other things that make you "extract a list from IO" always raise accumulation worries. So a little detour by a streaming library was needed. So in the repo it was necessary just to replace
processDate :: String -> IO ()
processDate date = do
allFiles <- listFiles date
allProperties <- mapM fileToProperties allFiles
let flattenedPropertiesWithPrice = filter hasPrice $ concat allProperties
geocodedProperties <- propertiesWithGeocoding flattenedPropertiesWithPrice
print geocodedProperties
propertiesWithGeocoding :: [ParsedProperty] -> IO [(ParsedProperty, Maybe LatLng)]
propertiesWithGeocoding properties = do
let batchProperties = chunksOf 100 properties
batchGeocodedLocations <- mapM geocodeAddresses batchProperties
let geocodedLocations = fromJust $ concat <$> sequence batchGeocodedLocations
return geocodedLocations
with something like this
import Streaming
import qualified Streaming.Prelude as S
processDate :: String -> IO ()
processDate date = do
allFiles <- listFiles date -- we accept an unstreamed list
S.print $ propertiesWithGeocoding -- this was the main pain point see below
$ S.filter hasPrice
$ S.concat
$ S.mapM fileToProperties -- this mapM doesn't accumulate
$ S.each allFiles -- the list is converted to a stream
propertiesWithGeocoding
:: Stream (Of ParsedProperty) IO r
-> Stream (Of (ParsedProperty, Maybe LatLng)) IO r
propertiesWithGeocoding properties =
S.concat $ S.concat
$ S.mapM geocodeAddresses -- this mapM doesn't accumulate results from mapquest
$ S.mapped S.toList -- convert segments to haskell lists
$ chunksOf 100 properties -- this is the streaming `chunksOf`
-- concat here flattens a stream of lists of as into a stream of as
-- and a stream of maybe as into a stream of as
Then the memory use looks like so, each peak corresponding to a trip to Mapquest promply followed by a little processing and a print, whereupon ghc forgets all about it and moves on:
Of course this could be done with pipes or conduit. But here we just need a little bit of simple mapM / sequence/ traverse / replicateM avoidance and streaming is perhaps simplest for this sort of quick local refactoring. Note that this list is quite short so the thought 'but short lists are cool with mapM/traverse/etc !" can be quite spectacularly false. Why not just get rid of them? Whenever you are about to write list mapM f it is a good idea to consider S.mapM f . S.each (or conduit or pipes equivalent) . You will now have a stream and can recover a list with S.toList or an equivalent, but it is likely that, as in this case, you will find you don't need a reified accumulated list but can e.g. use some streaming process like printing to file or stdout or writing things to a database, after making whatever list like manipulations are needed (here we use eg. streaming filter and also concat to flatten streamed lists and as a sort of catMaybe).
I am building some moderately large DIMACS files, however with the method used below the memory usage is rather large compared to the size of the files generated, and on some of the larger files I need to generate I run in to out of memory problems.
import Control.Monad.State.Strict
import Control.Monad.Writer.Strict
import qualified Data.ByteString.Lazy.Char8 as B
import Control.Monad
import qualified Text.Show.ByteString as BS
import Data.List
main = printDIMACS "test.cnf" test
test = do
xs <- freshs 100000
forM_ (zip xs (tail xs))
(\(x,y) -> addAll [[negate x, negate y],[x,y]])
type Var = Int
type Clause = [Var]
data DIMACSS = DS{
nextFresh :: Int,
numClauses :: Int
} deriving (Show)
type DIMACSM a = StateT DIMACSS (Writer B.ByteString) a
freshs :: Int -> DIMACSM [Var]
freshs i = do
next <- gets nextFresh
let toRet = [next..next+i-1]
modify (\s -> s{nextFresh = next+i})
return toRet
fresh :: DIMACSM Int
fresh = do
i <- gets nextFresh
modify (\s -> s{nextFresh = i+1})
return i
addAll :: [Clause] -> DIMACSM ()
addAll c = do
tell
(B.concat .
intersperse (B.pack " 0\n") .
map (B.unwords . map BS.show) $ c)
tell (B.pack " 0\n")
modify (\s -> s{numClauses = numClauses s + length c})
add h = addAll [h]
printDIMACS :: FilePath -> DIMACSM a -> IO ()
printDIMACS file f = do
writeFile file ""
appendFile file (concat ["p cnf ", show i, " ", show j, "\n"])
B.appendFile file b
where
(s,b) = runWriter (execStateT f (DS 1 0))
i = nextFresh s - 1
j = numClauses s
I would like to keep the monadic building of clauses since it is very handy, but I need to overcome the memory problem. How do I optimize the above program so that it doesn't use too much memory?
If you want good memory behavior, you need to make sure that you write out the clauses as you generate them, instead of collecting them in memory and dumping them as such, either using lazyness or a more explicit approach such as conduits, enumerators, pipes or the like.
The main obstacle to that approach is that the DIMACS format expects the number of clauses and variables in the header. This prevents the naive implementation from being sufficiently lazy. There are two possibilities:
The pragmatic one is to write the clauses first to a temporary location. After that the numbers are known, so you write them to the real file and append the contents of the temporary file.
The prettier approach is possible if the generation of clauses has no side effects (besides the effects offered by your DIMACSM monad) and is sufficiently fast: Run it twice, first throwing away the clauses and just calculating the numbers, print the header line, run the generator again; now printing the clauses.
(This is from my experience with implementing SAT-Britney, where I took the second approach, because it fitted better with other requirements in that context.)
Also, in your code, addAll is not lazy enough: The list c needs to be retained even after writing (in the MonadWriter sense) the clauses. This is another space leak. I suggest you implement add as the primitive operation and then addAll = mapM_ add.
As explained in Joachim Breitner's answer the problem was that DIMACSM was not lazy enough, both because the strict versions of the monads was used and because the number of variables and clauses are needed before the ByteString can be written to the file. The solution is to use the lazy versions of the Monads and execute them twice. It turns out that it is also necessary to have WriterT be the outer monad:
import Control.Monad.State
import Control.Monad.Writer
...
type DIMACSM a = WriterT B.ByteString (State DIMACSS) a
...
printDIMACS :: FilePath -> DIMACSM a -> IO ()
printDIMACS file f = do
writeFile file ""
appendFile file (concat ["p cnf ", show i, " ", show j, "\n"])
B.appendFile file b
where
s = execState (execWriterT f) (DS 1 0)
b = evalState (execWriterT f) (DS 1 0)
i = nextFresh s - 1
j = numClauses s
I have a problem, user inputs number m (meaning matrix has m-rows, m-columns), then the matrix is read from stdin. I want to transpose the matrix and then output the matrix back on the screen in the same format.
for example:
Main> main
3
1 2 3
4 5 6
7 8 9
1 4 7
2 5 8
3 6 9
Anyway, I got somewhere in the middle, but i dont know how to parse the [[Int]] back to IO, so it can be print on the screen.
here is my code:
import Control.Monad (replicateM)
transpose :: [[a]]->[[a]]
transpose ([]:_) = []
transpose x = (map head x) : transpose (map tail x)
readMany :: Read a => IO [a]
readMany = fmap (map read . words) getLine
parse :: IO ([[Int]])
parse = do
[m] <- readMany
xss <- replicateM m readMany
let matrix = transpose xss
return (matrix)
main :: IO ()
main = do
parse
?? --
print ??
First, you have to extract the value you parse:
main :: IO ()
main = do
matrix <- parse
...
print ???
Now, what's the type of print?
print :: (Show a) => a -> IO ()
Well, [[Int]] is an instance of Show, so we can treat print as a specialisation of its type:
print :: [[Int]] -> IO ()
So, in fact, it's as simple as filling in the list we want to print in place of the ???:
main :: IO ()
main = do
matrix <- parse
print matrix
However, this prints the data out in the same way that GHCi does. If you want to print the data out in a custom format, you want:
putStr :: String -> IO ()
Let's say you write a function to format the matrix the way you want into a string:
formatMatrix :: [[Int]] -> String
This makes the complete program:
main :: IO ()
main = do
matrix <- parse
putStr (formatMatrix matrix)
The difference between foo <- bar and let foo = bar is that the former extracts the result of an IO computation, while the latter just gives a name to a value.
So basically, the answer is that you don't need to put anything into IO to do this, since all the functions that create IO actions to do things like print strings out take pure values. This doesn't make them limited to printing the results of completely pure computations, however, since you can extract values from previous IO computations, and then feed them in to the printing functions as pure values. (In fact, this is the essence of monads, and why Haskell uses them to model IO in the first place!)
It is entirely possible to do exactly what you ask for, but sometimes it is just a little bit easier to make the string and then print it. In this case it is in fact very easy. All you need to know are the two functions words and lines, and their inverse unwords and unlines.
Here is a possible solution to your problem (note that I don't check that the inputs are integers and that I use the transpose from Data.List)
import Data.List
import Control.Monad
main = do
n <- liftM read getLine
xs <- replicateM n getLine
putStrLn (unlines . map unwords . transpose . map words $ xs)
since replicateM n getLine gives us a list of lines, what lines would have done to the entire input, we are just sandwiching transpose in between map words . lines and its inverse.