Haskell Assignment - direction needed to split a String into words - haskell

Question

we started a paper on Haskell a few weeks ago and just received our first assignment. I'm aware that SO doesn't like homework questions, so I'm not going to ask how to do it. Instead, it would be very much appreciated if anyone could push me in the right direction with this. Seeing as it might not be a specific question, would it be more appropriate in a discussion / community wiki?
Question: Tokenize a String, that is: "Hello, World!" -> ["Hello", "World"]
Coming from a Java background, I have to forget everything about the usual way to go about this. The problem is that I am still very clueless with Haskell. This is what I've come up with:
module Main where
main :: IO()
main = do putStrLn "Type in a string:\n"
x <- getLine
putStrLn "The string entered was:"
putStrLn x
putStrLn "\n"
print (tokenize x)
tokenize :: String -> [String]
tokenize [] = []
tokenize l = token l ++ tokenize l
token :: String -> String
token [] = []
token l = takeWhile (isAlphaNum) l
What would be the first glaring mistake?
Thank you.

Answer 1

The first glaring mistake is
tokenize l = token l ++ tokenize l
(++) :: [a] -> [a] -> [a] appends two lists of the same type. Since token :: String -> String (and type String = [Char]), the type of tokenize that is inferred from that line is tokenize :: String -> String.
You should use (:) :: a -> [a] -> [a] here.
The next mistake in that line is that in the recursive call, you pass the same input l once again, so you have an infinite recursion, always doing the same without change. You have to remove the first token (and a bit more) from the input for the argument to the recursive call.
Another problem is that your token supposes that the input begins with alphanumeric characters.
You also need a function that ensures that condition for what you pass to token.

Answer 2

This line results in an infinite list (which is OK, since Haskell is lazy, so the list only gets constructed "on demand"), because it is recurring with no change in the arguments:
tokenize l = token l ++ tokenize l
We can visualise what is happening when tokenize is called as:
tokenize l = token l ++ tokenize l
= token l ++ (token l ++ tokenize l)
= token l ++ (token l ++ (token l ++ tokenize l))
= ...
To stop this happening, you need to change what the argument to tokenize so that it recurs sensibly:
tokenize l = token l ++ tokenize <something goes here>

Answer 3

As others already pointed out your mistake, just a little hint: While you found already the very useful takeWhile function, you should have a look at span, as this could be even more helpful here.

Answer 4

This has something in it that feels similar to a parser monad. However, as you're a newcomer to Haskell, it's unlikely that you're in a position to understand how parsing monads work (or use them in your code) quite yet. To give you the basics, consider what you want:
tokenize :: String -> [String]
This takes a String, chomps it up into more pieces, and generates a list of strings corresponding to the words in the input string. How might we represent this? What we want to do is find a function that processes a single string, and at the first sign of whitespace, adds that string on to the sequence of words. But then you have to process what's left over. (I.e., the rest of the string.) For example, let's say you want to tokenize:
The brown fox jumped
You first pull out "The" and then continue processing " brown fox jumped" (note the space at the beginning of the second string). You will do this recursively, so naturally you will need a recursive function.
The natural solution that sticks out is to take something where you accumulate a set of strings you've tokenized so far, keep munching on the current input until you hit whitespace, then also accumulate what you've seen in the current string (this leads to an implementation where you're mostly consing stuff, and then occasionally reversing stuff).

Answer 5

Your exercise seemed a bit challenging to me so I decided to solve it just for self-training. Here's what I came up with:
import Data.List
import Data.Maybe
splitByAnyOf yss xs =
foldr (\ys acc -> concat $ map (splitBy ys) acc) [xs] yss
splitBy ys xs =
case (precedingElements ys xs, succeedingElements ys xs) of
(Just "", Just s) -> splitBy ys s
(Just p, Just "") -> [p]
(Just p, Just s) -> p : splitBy ys s
otherwise -> [xs]
succeedingElements ys xs =
fromMaybe Nothing . find isJust $ map (stripPrefix ys) $ tails xs
precedingElements ys xs =
fromMaybe Nothing . find isJust $ map (stripSuffix ys) $ inits xs
where
stripSuffix ys xs =
if ys `isSuffixOf` xs then Just $ take (length xs - length ys) xs
else Nothing
main = do
print $ splitBy "!" "Hello, World!"
print $ splitBy ", " "Hello, World!"
print $ splitByAnyOf [", ", "!"] "Hello, World!"
outputs:
["Hello, World"]
["Hello","World!"]
["Hello","World"]