Related
I have a directory traversal function in Haskell, but I want it to ignore symlinks. I figured out how to filter out the files alone, albeit with a slightly inelegant secondary filterM. But after some diagnosis I realize that I'm failing to filter symlinked directories.
I'd like to be able to write something like this:
-- Lazily return (normal) files from rootdir
getAllFiles :: FilePath -> IO [FilePath]
getAllFiles root = do
nodes <- pathWalkLazy root
-- get file paths from each node
let files = [dir </> file | (dir, _, files) <- nodes,
file <- files,
not . pathIsSymbolicLink dir]
normalFiles <- filterM (liftM not . pathIsSymbolicLink) files
return normalFiles
However, all the variations I have tried get some version of the "Couldn't match expected type ‘Bool’ with actual type ‘IO Bool’" message (without the filter clause in the comprehension it works, but fails to filter those linked dirs).
Various hints at ways I might completely restructure the function are in partial form at online resources, but I'm pretty sure that every such variation will run into some similar issue. The list comprehension would certainly be the most straightforward way... if I could just somehow exclude those dirs that are links.
Followup: Unfortunately, the solution kindly provided by ChrisB behaves (almost?!) identically to my existing version. I defined three functions, and run them within a test program:
-- XXX: debugging
files <- getAllFilesRaw rootdir
putStrLn ("getAllFilesRaw: " ++ show (length files))
files' <- getAllFilesNoSymFiles rootdir
putStrLn ("getAllFilesNoSymFiles: " ++ show (length files'))
files'' <- getAllFilesNoSymDirs rootdir
putStrLn ("getAllFilesNoSymDirs: " ++ show (length files''))
The first is my version with the normalFiles filter removed. The second is my original version (minus the type error in the listcomp). The final one is ChrisB's suggestion.
Running that, then also the system find utility:
% find $CONDA_PREFIX -type f | wc -l
449667
% find -L $CONDA_PREFIX -type f | wc -l
501153
% haskell/find-dups $CONDA_PREFIX
getAllFilesRaw : 501153
getAllFilesNoSymFiles: 464553
getAllFilesNoSymDirs: 464420
Moreover, this question came up because—for my own self-education—I've implemented the same application in a bunch of languages: Python; Golang; Rust; Julia; TypeScript; Bash, except the glitch, Haskell; others are planned. The programs actually do something more with the files, but that's not the point of this question.
The point of this is that ALL other languages report the same number as the system find tool. Moreover, the specific issue is things like this:
% ls -l /home/dmertz/miniconda3/pkgs/ncurses-6.2-he6710b0_1/lib/terminfo
lrwxrwxrwx 1 dmertz dmertz 17 Apr 29 2020 /home/dmertz/miniconda3/pkgs/ncurses-6.2-he6710b0_1/lib/terminfo -> ../share/terminfo
There are about 16k examples here (on my system currently), but looking at some in the other version of the tool, I see specifically that all the other languages are excluding the contents of that symlink directory.
EDIT:
Instead of just fixing a Bool / IO Bool issue we now want to mach find's behavior.
After looking at the documentation,
this seems to be quite hard to implement reasonably performantly
with the PathWalk library, so i just handrolled it.
(Using do-notation, as requested in the comments.)
In my quick and dirty tests the results match those of find:
import System.FilePath
import System.Directory
getAllFiles' :: FilePath -> IO [FilePath]
getAllFiles' path = do
isSymlink <- pathIsSymbolicLink path
if isSymlink
-- if this is a symlink, return the empty list.
-- even if this was the original root. (matches find's behavior)
then return []
else do
isFile <- doesFileExist path
if isFile
then return [path] -- if this is a file, return it
else do
-- if it's not a file, we assume it to be a directory
dirContents <- listDirectory path
-- run this function recursively on all the children
-- and accumulate the results
fmap concat $ mapM (getAllFiles' . (path </>)) dirContents
Original Answer solving the IO Bool / Bool issue
getAllFiles :: FilePath -> IO [FilePath]
getAllFiles root = pathWalkLazy root
-- remove dirs that are symlinks
>>= filterM (\(dir, _, _) -> fmap not $ pathIsSymbolicLink dir)
-- flatten to list of files
>>= return . concat . map (\(dir, _, files) -> map (\f -> dir </> f) files)
-- remove files that are symlinks
>>= filterM (fmap not . pathIsSymbolicLink)
This is a bizzare behavior, even for Haskell. Look at the code segments below:
import System.Directory
import System.FilePath
-- This spins infinitely
loadCtx :: FilePath -> IO ()
loadCtx dir = do
lsfiles <- listDirectory dir
let files = mapM (dir </>) lsfiles
putStrLn $ "Files " ++ show files
-- This does what I'd expect, prepending the dir path to each file
loadCtx dir = do
lsfiles <- listDirectory dir
let files = map (dir </>) lsfiles
putStrLn $ "Files " ++ show files
Both definitions are accepted from the typechecker but give completely
different behavior. What is the output of the first mapM? It looks like an infinite loop on reading some files. Also is it possible to compose the listDirectory do-arrow line with the map (dir </>) that prepends the path, in one-line?
What is the output of the first mapM? It looks like an infinite loop on reading some files.
It is not an infinite loop -- merely a very, very long one.
You are not using mapM for IO; you are using mapM in the nondeterminism monad. Here is the type of mapM, specialized to that monad:
Traversable t => (a -> [b]) -> t a -> [t b]
Read this in the following way:
First, give me a way to turn an element of a container (type a) into a nondeterministic choice between many possible replacement elements (type [b]).
Then give me a containerful of elements (type t a).
I will give you a nondeterministic choice between containers with replacement elements in them (type [t b]). (And, this part is not in the type, but: the way I will do this is by taking all possible combinations; for each position in the container, I'll try each possible b, and give you every which way of making one choice for each position in the container.)
For example, if we were to define the function f :: Int -> [Char] for which f n chose nondeterministically between the first n letters of the alphabet, then we could see this kind of interaction:
> f 3
"abc"
> f 5
"abcde"
> f 2
"ab"
> mapM f [3,5,2]
["aaa","aab","aba","abb","aca","acb","ada","adb","aea","aeb","baa","bab","bba","bbb","bca","bcb","bda","bdb","bea","beb","caa","cab","cba","cbb","cca","ccb","cda","cdb","cea","ceb"]
In each result, the first letter is one of the first three in the alphabet (a, b, or c); the second is from the first five, and the third from the first two. What's more, we get every list which has this property.
Now let's think about what that means for your code. You have written
mapM (dir </>) lsfiles
and so what you will get back is a collection of lists. Each list in the collection will be exactly as long as lsfiles is. Let's focus on one of the lists in the collection; call it cs.
The first element of cs will be drawn from dir </> filename, where filename is the first element of lsfiles; that is, it will be one of the characters in dir, or a slash, or one of the characters in filename. The second element of cs will be similar: one of the characters of dir, or a slash, or one of the characters from the second filename in lsfiles. I guess you can see where this is going... there's an awful lot of possibilities here. =)
Also is it possible to compose the listDirectory do-arrow line with the map (dir </>) that prepends the path, in one-line?
Yes:
loadCtx dir = do
files <- map (dir </>) <$> listDirectory dir
putStrLn $ "Files " ++ show files
Well according to the documentation,
type FilePath = String
That is,
type FilePath = [Char]
So in this line,
let files = mapM (dir </>) lsfiles
you have that the argument of mapM, which is (dir </>), is of type FilePath -> FilePath. Now look at the type of mapM,
mapM :: (Traversable t, Monad m) => (a -> m b) -> t a -> m (t b)
^^^^^
So the type a -> m b is instantiated to FilePath -> FilePath, which is FilePath -> [Char]. So you're performing a monadic mapping using the list monad, which is the "nondeterminism" monad in this case for values of type Char.
To complement Jorge's answer, here's an exponential blowup, demonstrated:
> map ("XY" </>) ["a","b","c"]
["XY\\a","XY\\b","XY\\c"]
> mapM ("XY" </>) ["a","b","c"]
["XXX","XXY","XX\\","XXc","XYX","XYY","XY\\","XYc","X\\X","X\\Y","X\\\\",
"X\\c","XbX","XbY","Xb\\","Xbc","YXX","YXY","YX\\","YXc","YYX","YYY","YY\\","YYc",
"Y\\X","Y\\Y","Y\\\\","Y\\c","YbX","YbY","Yb\\","Ybc","\\XX","\\XY","\\X\\",
"\\Xc","\\YX","\\YY","\\Y\\","\\Yc","\\\\X","\\\\Y","\\\\\\","\\\\c","\\bX",
"\\bY","\\b\\","\\bc","aXX","aXY","aX\\","aXc","aYX","aYY","aY\\","aYc","a\\X",
"a\\Y","a\\\\","a\\c","abX","abY","ab\\","abc"]
Indeed, mapM = sequence . map, and sequence in the list monad performs the cartesian product of a list-of-lists, ["XY\\a","XY\\b","XY\\c"] in this case, so we get 4*4*4 combinations. (Ouch!)
I have a text file which contains two lists on each line. Each list can contain any number of alphanumeric arguments.
eg [t1,t2,...] [m1,m2,...]
I can read the file into ghc, but how can I read this into another main file and how can the main file recognise each argument separately to then process it?
I think it's best for you to figure out most of this for yourself, but I've got some pointers for you.
Firstly, try not to deal with the file access until you've got the rest of the code working, otherwise you might end up having IO all over the place. Start with some sample data:
sampleData = "[m1,m2,m3][x1,x2,x3,x4]\n[f3,f4,f5][y7,y8,y123]\n[m4,m5,m6][x5,x6,x7,x8]"
You should not mention sampleData anywhere else in your code, but you should use it in ghci for testing.
Once you have a function that does everything you want, eg processLists::String->[(String,String)], you can replcae readFile "data.txt" :: IO String with
readInLists :: FilePath -> IO [(String,String)]
readInLists filename = fmap processLists (readFile filename)
If fmap makes no sense to you, you could read a tutorial I accidentally wrote.
If they really are alphanumeric, you can split them quite easily. Here are some handy functions, with examples.
tail :: [a] -> [a]
tail "(This)" = "This)"
You can use that to throw away something you don't want at the front of your string.
break :: (Char->Bool) -> String -> (String,String)
break (== ' ') "Hello Mum" = ("Hello"," Mum")
So break uses a test to find the first character of the second string, and breaks the string just before it.
Notice that the break character is still there at the front of the next string. span is the same but uses a test for what to have in the first list, so
span :: (Char->Bool) -> String -> (String,String)
span (/= ' ') "Hello Mum" = ("Hello"," Mum")
You can use these functions with things like (==','), or isAlphaNum (you'll have to import Data.Char at the top of your file to use it).
You might want to look at the functions splitWith and splitOn that I have in this answer. They're based on the definitions of split and words from the Prelude.
How do I get a search match from a list of strings in Haskell?
module Main
where
import List
import IO
import Monad
getLines = liftM lines . readFile
main = do
putStrLn "Please enter your name: "
name <- getLine
list <- getLines "list.txt"
-- mapM_ putStrLn list -- this part is to list out the input of lists
The first thing to do, the all-important first principle, is to get as much of the thinking out of main or out of IO as possible. main should where possible contain all the IO and maybe nothing but IO decorated with pure terms you define elsewhere in the module. Your getLines is mixing them unnecessarily.
So, to get that out of the way, we should have a main that is something like
main =
do putStrLn "What is your name?"
name <- getContents
names <- readFile "names.txt"
putStrLn (frankJ name names)
-- or maybe the more austere segregation of IO from all else that we get from:
main =
do putStrLn greeting
name <- getContents
names <- readFile nameFile
putStrLn (frankJ name names)
together with the 'pure' terms:
greeting, nameFile :: String
greeting = "What is your name?"
nameFile = "names.txt"
Either way, we are now really in Haskell-land: the problem is now to figure out what the pure function:
frankJ :: String -> String -> String
should be.
We might start with a simple matching function: we get a match when the first string appears on a list of strings:
match :: String -> [String] -> Bool
match name namelist = name `elem` namelist
-- pretty clever, that!
or we might want to normalize a bit, so that white space at the beginning and end of the name we are given and the names on the list doesn't affect the match. Here's a rather shabby way to do that:
clean :: String -> String
clean = reverse . omitSpaces . reverse . omitSpaces
where omitSpaces = dropWhile (== ' ')
Then we can improve on our old match, i.e. elem:
matchClean :: String -> [String] -> Bool
matchClean name namelist = match (clean name) (map clean namelist)
Now we need to follow the types, figuring out how to fit the type of, say, matchClean:: String -> [String] -> Bool with that of frankJ :: String -> String -> String. We want to fit it inside our definition of frankJ.
Thus, to 'provide input' for matchClean, we need a function to take us from a long string with newlines to the list of stings (the names) that matchClean needs: that's the Prelude function lines.
But we also need to decide what to do with the Bool that matchClean yields as value; frankJ, as we have it, returns a String. Let us continue with simple-minded decomposition of the problem:
response :: Bool -> String
response False = "We're sorry, your name does not appear on the list, please leave."
response True = "Hey, you're on the A-list, welcome!"
Now we have materials we can compose into a reasonable candidate for the function frankJ :: String -> String -> String that we are feeding into our IO machine defined in main:
frankJ name nametext = response (matchClean name (lines nametext))
-- or maybe the fancier:
-- frankJ name = response . matchClean name . lines
-- given a name, this
-- - pipes the nametext through the lines function, splitting it,
-- - decides whether the given name matches, and then
-- - calculates the 'response' string
So here, almost everything is a matter of pure functions, and it is easy to see how to emend things for further refinement. For example, maybe the name entered and the lines of the text file should be further normalized. Internals spaces should be restricted to one space, before the comparison. Or maybe there is a comma in lines on the list since people are listed as "lastname, firstname", etc. etc. Or maybe we want the response function to use the person's name:
personalResponse :: String -> Bool -> String
personalResponse name False = name ++ " is a loser, as far as I can tell, get out!"
personalResponse name True = "Ah, our old friend " ++ name ++ "! Welcome!"
together with
frankJpersonal name = personalResponse name . matchClean name . lines
Of course there are a million ways of going about this. For example, there are regex libraries. The excellent and simple Data.List.Split from Hackage might also be of use, but I'm not sure it can be used by Hugs, which you might be using.
I note that you are using old-fashioned names for the imported modules. What I have written uses only the Prelude so imports are unnecessary, but the other modules are now called "System.IO", "Data.List" and "Control.Monad" in accordance with the hierarhical naming system. I wonder if you are using an old tutorial or manual. Maybe the pleasant 'Learn You a Haskell' site would be better? He affirms he's using ghc but I think that won't affect much.
If you wan't a list of all lines in your list.txt that contain the name,
you can simply use
filter (isInfixOf name) list
but I'm not sure if I understood your question correct.
first off sorry for doing the typical thing of 'where do I begin', but I'm totally lost.
I've been reading the 'Learn you a haskell for great good' site for what feels like an age now (pretty much half a semester. I'm just about to finish the 'Input and Output' chapter, and I still have no clue how to write a multi line program.
I've seen the do statement, and that you can only use it to concat IO actions into a single function, but I can't see how I'm gonna go about writing a realistic application.
Can someone point me in the right direction.
I'm from a C background, and basically I'm using haskell for one of my modules this semester at uni, I want to compare C++ against haskell (in many aspects). I'm looking to create a series of searching and sorting programs so that I can comment on how easy they are in the respective languages versus their speed.
However, I'm really starting to loose my faith in using Haskell as its been six weeks, and I still have no idea how to write a complete application, and the chapters in the site I'm reading seem to be getting longer and longer.
I basically need to create a basic object which will be stored in the structure (which I know how to do), more what I'm struggling with is, how do I create a program which reads data in from some text file, and populates the structure with that data in the first place, then goes on to process it. As haskell seems to split IO and other operations and it won't just let me write multiple lines in a program, I'm looking for something like this:
main = data <- getContent
let allLines = lines data
let myStructure = generateStruct allLines
sort/search/etc
print myStructure
how do I go about this? any good tutorials which will help me get going with realistic programs?
-A
You mentioned seeing do notation, now it's time to learn how to use do. Consider your example main is an IO, you should be using do syntax or binds:
main = do
dat <- getContent
let allLines = lines dat
myStructure = generateStruct allLines
sorted = mySort myStructure
searchResult = mySearch myStructure
print myStructure
print sorted
print searchResult
So now you have a main that gets stdin, turns it into [String] via lines, presumably parses it into a structure and runs sorting and searches on that structure. Notice the interesting code is all pure - mySort, mySearch, and generateStruct doesn't need to be IO (and can't be, being inside a let binding) so you are actually properly using pure and effectful code together.
I suggest you look at how bind works (>>=) and how do notation desugars into bind. This SO question should help.
See also Explaining Haskell IO without Monads by Neil Mitchell.
I'll try to start with a simplified example. Let's say this is what we want to do:
Open a file which contains a list of integers and return it.
Sort this list
Let's also reverse the list
Print the result on the screen
Let's also say that we have these functions that we can use:
getContent :: IO [Int]
sort :: [Int] -> [Int]
reverse :: [Int] -> [Int]
show :: a -> String
putStrLn :: String -> IO ()
Just so we are clear, I'll have a word about these functions:
getContent: I made up this function, but if there was such function that would be it's signature (you can use getContent = return [3,7,2,1] for testing purposes). I'm sure you've seen such signature before and at least vaguely understand that since it does IO its signature can not be just getContent :: [Int].
sort: It's a function defined in Data.List module, usage is simple: sort [3,1,2] returns [1,2,3]
reverse: Also defined in Data.List module: reverse [1,3,2] returns [2,3,1]
show: don't need to import anything, just use it: show 11 returns the string "11"; show [1,2,3] returns the string "[1,2,3]", etc.
putStrLn: takes a string, puts it on the screen and returns IO (), now again, since it does IO its signature can not be just putStrLn :: Stiring -> ().
OK, now we have all we need to create our program, the problem now is about connecting these functions together. Let's start with connecting functions:
getContent :: IO [Int] with sort :: [Int] -> [Int]
I think if you get this part, you'll easily get the rest as well. So, the problem is that since getContent returns IO [Int] and not just [Int], you can't just ignore or get rid of the IO part and shove it into sort. That is, this is what you can not do to connect these functions:
sort (getRidOfIO getContent)
Here is where the >>= :: m a -> (a -> m b) -> m b operation comes to the rescue. Now notice that m, a and b are type variables so if we substitute m for IO, a for [Int] and b for [Int], we get the signagure:
>>= :: IO [Int] -> ([Int] -> IO [Int]) -> IO [Int]
Have a look again at those getContent and sort functions and their signatures and try to think about how they'll fit into the >>=. I'm sure you'll notice that you can use getContent directly as the first argument to >>=. So far what >>= will do is take the [Int] out getContent and shoves it into the function provided as a second argument. But what will be the function in the second argument? We can't use the sort :: [Int] -> [Int] directly, the next best thing we can try is
\listOfInts -> sort listOfInts
but that still has signature [Int] -> [Int] so that did not help much. Here is where the other hero comes to the play, the
return :: a -> m a.
Again, a and m are type variables, lets substitute them and we will get
return :: [Int] -> IO [Int]
so adding \listOfInts -> sort listOfInts and return together we will get:
\listOfInts -> return $ sort listOfInts :: [Int] -> IO [Int]
Which is exactly what we want to put as a second argument to >>=. So lets finaly connect getContent and sort using our glue together:
getContent >>= (\listOfInts -> return $ sort listOfInts)
which is the same thing as (using the do notation):
do listOfInts <- getContent
return $ sort listOfInts
There, that is the end of the most terrifying part. And now comes possibly one of the aha moments, try to think about what is the result type of the connection we just made up. I'll spoil it for you,... the type of
getContent >>= (\listOfInts -> return $ sort listOfInts) is IO [Int] again.
Lets summarize: we took something of type IO [Int] and something of type [Int] -> [Int], glued those two things together and got again something of type IO [Int]!
Now go ahead and try exactly the same thing: Take the IO [Int] object we have just created and glue it together (using >>= and return) with reverse :: [Int] -> [Int].
I think I wrote way too much, but let me know if anything was not clear or if you need help with the rest.
Wha I've described so far can look something like this:
getContent :: IO [Int]
getContent = return [5,2,1,7]
main :: IO ()
main = do
listOfInts <- getContent
return $ sort listOfInts
return () -- This is only to sattisfy the signature of main
If it is a question of reading from stdin and writing a result to stdout, with no further intevening user input -- as your mention of getContents suggests -- then the ancient interact :: (String -> String) -> IO (), or the several other versions, e.g. Data.ByteString.interact :: (ByteString -> ByteString) -> IO () or Data.Text.interact :: (Text -> Text) -> IO() are all that are needed. interact is basically the 'make a little unix tool out of this function' function -- it maps pure functions of the right type to executable actions (i.e. values of the type IO().) All Haskell tutorials should mention it on the third or fourth page, with instructions on compilation.
So if you write
main = interact arthur
arthur :: String -> String
arthur = reverse
and compile with ghc --make -O2 Reverse.hs -o reverse then whatever you pipe to ./reverse will be understood as a list of characters and emerge reversed. Similarly, whatever you pipe to
main = interact (unlines . meredith . lines)
meredith :: [String] -> [String]
meredith = filter (not.null)
will emerge with the empty lines omitted. More interestingly,
main = interact ( unlines . map show . luther . map read . lines)
luther :: [Int] -> [Int]
luther = filter even
will take a stream of characters separated by newlines, read them as Ints, removing the odd ones, and yielding the suitably filtered stream.
main = interact ( unlines . map show . emma . map read . lines)
emma :: [Int] -> Int
emma = sum . map square
where square x = x * x
will print the sum of the squares of the newline-separated numerals.
In these last two cases, luther and emma the internal 'data structure' is [Int], which is pretty dull, and the function applied to it is idiot simple, of course. The main point is to let one of the forms of interact take care of all of the IO, and thus get images like 'populating a structure' and 'processing it' out of your head. To use interact you need to use composition to make the whole yield some sort of String -> String function. But even here, as in the runt first example arthur:: String -> String you are defining a genuine function in something more like the mathematical sense. Values in the types String and ByteString are just as pure as those in Bool or Int.
In more complicated cases of this basic interact type, your task is thus, first, to think how the desired pure values of the function you will be focussing on can be mapped to String values (here, it's just show for an Int or unlines . map show for a [Int]). interact knows what to "do" with the string. -- And then to figure out how to define a pure mapping from Strings or ByteString (which will contain your 'raw' data) to values in the type or types your principal function takes as arguments. Here I was just using map read . lines resulting in a [Int]. If you are working on some more complicated, say tree structure you'd need a function from [Int] to MyTree Int. A more elaborate function to put in this position would be a Parser, of course.
Then you can go to town, in this sort of case: there is really no reason to think of yourself as 'programming', 'populating' and 'processing' at all. This is where all the cool devices of LYAH kick in. Your duty is to define a mapping within the specific definitional discipline. In the last two cases, these are from [Int] to [Int] and from [Int] to Int, but here is a similar example derived from the excellent, still incomplete, tutorial on the super-excellent Vector package where the initial numerical structure one is dealing with is Vector Int
{-# LANGUAGE BangPatterns #-}
import qualified Data.ByteString.Lazy.Char8 as L
import qualified Data.Vector.Unboxed as U
import System.Environment
main = L.interact (L.pack . (++"\n") . show . roman . parse)
where
parse :: L.ByteString -> U.Vector Int
parse bytestr = U.unfoldr step bytestr
step !s = case L.readInt s of
Nothing -> Nothing
Just (!k, !t) -> Just (k, L.tail t)
-- now the IO and stringy nonsense is out of the way
-- so we can calculate properly:
roman :: U.Vector Int -> Int
roman = U.sum
Here again roman is moronic, any function from a Vector of Ints to an Int, however complex, can take its place. Writing a better roman will never be a question of "populating" "multi-line programming" "processing" etc., though of course we speak this way; it is just a question of defining a genuine function by composition of the functions in Data.Vector and elsewhere. The sky is the limit, check out that tutorial too.