Related
I'm stuck on a problem with writing a parser in Haskell that I hope someone can help out with!
It is a bit more complicated than my usual parser because there are two layers of parsing. First a language definition is parsed into an AST, then that AST is transformed into another parser that parses the actual language.
I have made pretty good progress so far but I'm stuck on implementing recursion in the language definition. As the language definition is transformed from AST into a parser in a recursive function, I can't work out how it can call itself if it doesn't exist yet.
I'm finding it a bit hard to explain my problem, so maybe an example will help.
The language definition might define that a language consists of three keywords in sequence and then optional recursion in brackets.
A B C ($RECURSE)
Which would be parsed into an AST like:
[Keyword "A", Keyword "B", Keyword "C", Optional (Many [Recurse])]
The Many is not really required for this example, but in my actual project, optional blocks can have multiple syntax elements in them so an Optional would contain a Many with n elements.
I would then want it to get transformed into a parser that parses strings like:
A B C
A B C (A B C)
A B C (A B C (A B C))
I've boiled down my project into the simplest possible example. You can see my TODO comment where I'm stuck trying to implement the recursion.
{-# LANGUAGE OverloadedStrings #-}
module Example
( runExample,
)
where
import Control.Applicative hiding (many, some)
import Data.Text (Text)
import Data.Void
import System.IO as SIO
import Text.Megaparsec hiding (State)
import Text.Megaparsec.Char (space1, string')
import qualified Text.Megaparsec.Char.Lexer as L
import Text.Megaparsec.Debug
import Text.Pretty.Simple (pPrint)
-- Types
type Parser = Parsec Void Text
data SyntaxAst = Keyword Text | Recurse | Optional SyntaxAst | Many [SyntaxAst]
-- Megaparsec Base Parsers
-- Space consumer - used by other parsers to ignore whitespace
sc :: Parser ()
sc =
L.space
space1
(L.skipLineComment "--")
(L.skipBlockComment "/*" "*/")
-- Runs a parser, then consumes any left over space with sc
lexeme :: Parser a -> Parser a
lexeme = L.lexeme sc
-- Parses a string, then consumes any left over space with sc
symbol :: Text -> Parser Text
symbol = L.symbol sc
-- Parses something between parentheses
inParens :: Parser a -> Parser a
inParens =
between
(symbol "(")
(symbol ")")
-- Transforms the AST into a parser
transformSyntaxExprToParser :: SyntaxAst -> Parser [Text]
transformSyntaxExprToParser (Many exprs) = dbg "Many" (createParser exprs)
transformSyntaxExprToParser (Keyword text) = dbg "Keyword" (pure <$> lexeme (string' text))
transformSyntaxExprToParser (Optional inner) = dbg "Optional" (option [] (try (inParens (transformSyntaxExprToParser inner))))
transformSyntaxExprToParser Recurse = dbg "Recurse" (pure ["TODO"]) -- TODO: How do I recurse here?
-- transformSyntaxExprToParser s Recurse = dbg "Recurse" (createParser s) -- Seems to work in the example, but in my actual application creates an infinite loop and freezes
-- Walks over the parser AST and convert it to a parser
createParser :: [SyntaxAst] -> Parser [Text]
createParser expressions =
do
foldr1 (liftA2 (<>)) (fmap transformSyntaxExprToParser expressions)
runExample :: IO ()
runExample = do
-- To make the example simple, lets cut out the language definition parsing and just define
-- it literally.
let languageParser = createParser [Keyword "A", Keyword "B", Keyword "C", Optional (Many [Recurse])]
let run p = runParser p "" "A B C (A B C (A B C))"
let result = run languageParser
case result of
Left bundle -> SIO.putStrLn (errorBundlePretty bundle)
Right xs -> pPrint xs
A few things I've tried:
Pass the original AST up to the transformSyntaxExprToParser function and call createParser when the Recurse token is encountered. This didn't work due to infinite loops.
Using mutable references like IORef/STRef to pass in a reference that is updated to reference the final parser once the transformation is finished. I couldn't work out how to thread the IO/ST monads into the parser transform function.
State monads. I couldn't work out how to pass a reference through the state monad.
I hope that makes sense, let me know if I need to elaborate more. I can also push up my full project if it will help.
Thanks for reading!
Edit: I've made changes to my original example to demonstrate the infinite loop problem (integrating the excellent suggestions in the answer below) at https://pastebin.com/DN0JJ9BA
I believe you can use laziness here. Pass the final parser as a parameter to transformSyntaxExprToParser, and when you see a Recurse, return that parser.
transformSyntaxExprToParser :: Parser [Text] -> SyntaxAst -> Parser [Text]
transformSyntaxExprToParser self = go
where
go (Keyword text) = dbg "Keyword" (pure <$> lexeme (string' text))
go (Optional inner) = dbg "Optional" (option [] (try (inParens (go inner))))
go Recurse = dbg "Recurse" self
createParser :: [SyntaxAst] -> Parser [Text]
createParser expressions = parser
where
parser = foldr1 (liftA2 (<>))
(fmap (transformSyntaxExprToParser parser) expressions)
This ought to produce exactly the same kind of recursive parser as if you had written it directly. A Parser is ultimately just a data structure which you can construct using its instances of Monad, Applicative, Alternative, &c.
Your idea of doing this with a mutable reference such as an IORef is essentially what’s happening under the hood anyway when constructing and evaluating a thunk.
Your idea here was almost correct:
Pass the original AST up to the transformSyntaxExprToParser function and call createParser when the Recurse token is encountered. This didn't work due to infinite loops.
The problem is that you were constructing a new parser for every Recurse, from the same input, which contains a Recurse, thus constructing a new parser…and so on. What my code above does is just pass in the same parser.
If you need to perform monadic side effects while constructing the parser, such as logging, then you can use a recursive do, for example, with some hypothetical MonadLog class for illustration:
{-# Language RecursiveDo #-}
transformSyntaxExprToParser :: (MonadLog m) => Parser [Text] -> SyntaxAst -> m (Parser [Text])
transformSyntaxExprToParser self = go
where
go (Keyword text) = do
logMessage "Got ‘Keyword’"
pure $ dbg "Keyword" (pure <$> lexeme (string' text))
go (Optional inner) = do
logMessage "Got ‘Optional’"
inner' <- go inner
pure $ dbg "Optional" (option [] (try (inParens inner')))
go Recurse = do
logMessage "Got ‘Recurse’"
pure $ dbg "Recurse" self
createParser :: (MonadFix m, MonadLog m) => [SyntaxAst] -> m (Parser [Text])
createParser expressions = do
rec
parser <- fmap (foldr1 (liftA2 (<>)))
(traverse (transformSyntaxExprToParser parser) expressions)
pure parser
The rec block introduces a recursive binding which you may construct using side effects. In general, some care is required to ensure that recursive definitions like this are sufficiently lazy, that is, you don’t force the result sooner than intended, but here the recursion pattern is very simple, and you never examine the self parser, only treat it as a black box to hook up to other parsers.
This method also makes it explicit what the scope of a Recurse is, and opens the possibility of introducing local recursive parsers, with a new call to transformSyntaxExprToParser with a new local self argument.
I have a DSL, and a parser for it written in Haskell with the Parsec package. Now I want to deprecate a specific language feature of the DSL. In the next release, I want the parser to accept both the new and the old syntax, but I want the parser to spawn a deprecation message. I could not find how to do this. Is this possible, and if so, how can this be done?
Instead of emitting messages during parsing, it would be better to return extra information at the end of parsing: whether or not deprecated syntax was encountered.
The ParsecT type admits a type parameter for state set by the user during parsing:
ParsecT s u m a is a parser with stream type s, user state type u, underlying monad m and return type a. Parsec is strict in the user state.
The user state can be set with putState and modifyState. It can be obtained using getState.
Most parsec combinators are polymorphic on the user state. Most combinators for your own DSL should be, as well. But parsers for deprecated parts of the syntax should set a "flag" in your user state.
Something like this:
import Text.Parsec
import Text.Parsec.Char
import Data.Functor.Identity
type Parser = ParsecT [Char] Bool Identity -- using a Bool state
myParser :: Parser Char
myParser =
try (do char 'a'
putState True
char 'b')
<|>
try (do char 'a'
char 'c')
main :: IO ()
main = do
print $ runParser ((,) <$> myParser <*> getState) False "" "ab"
print $ runParser ((,) <$> myParser <*> getState) False "" "ac"
-- results:
-- Right ('b',True)
-- Right ('c',False)
Of course, instead of a simple boolean flag, it would be better to put more information into the state.
Notice that state set by a sub-parser is "forgotten" if the sub-parser backtracks. That is the correct behavior for our purposes: otherwise, we would get "false positives" triggered by branches that are ultimately discarded.
A common alternative to parsec is megaparsec. The latter doesn't allow for user-defined state in the parser type itself, but it can be emulated using a StateT transformer over the ParsecT type.
I'm actually using Parsec to make an Expression Parser and I have a little question (I'm french also sorry for my english ).
I have this code :
data Expression ... -- Recursive type Expression
type Store [(String, Float)] -- variable's storage
type Parser a = Parsec String () a
parseExpr :: [Char] -> Either ParseError Expression
parseExpr string = parse expr "" stream
where
stream = filter (not . isSpace) string
-- Parser's rules ...
raiseError a = Nothing
evalParser :: [Char] -> Store -> Float
evalParser expr store = fromMaybe 0 (either raiseError (eval store)(parseExpr expr))
This code works really well, but i need this function :
parseExpression :: String -> Maybe Expression
And I have no ideas of the right syntax to use.
Someone can help me please ?
I'll start it for you, and you can finish it:
eitherToMaybe :: Either a b -> Maybe b
eitherToMaybe (Left a) = ???
eitherToMaybe (Right b) = ???
A severe over-generalization looks like this (you can find this function with a less direct implementation in the monadplus package):
import Control.Applicative (Alternative (..))
import Data.Profunctor.Unsafe ((#.))
import Data.Monoid (Alt (..))
afold :: (Foldable f, Alternative g)
=> f a -> g a
afold = getAlt . foldMap (Alt #. pure)
But you don't really need to get into that business just yet.
I'm learning Haskell and as an exercise I'm trying to convert write the read_from function following code to Haskell. Taken from Peter Norvig's Scheme interpreter.
Is there a straightforward way do this?
def read(s):
"Read a Scheme expression from a string."
return read_from(tokenize(s))
parse = read
def tokenize(s):
"Convert a string into a list of tokens."
return s.replace('(',' ( ').replace(')',' ) ').split()
def read_from(tokens):
"Read an expression from a sequence of tokens."
if len(tokens) == 0:
raise SyntaxError('unexpected EOF while reading')
token = tokens.pop(0)
if '(' == token:
L = []
while tokens[0] != ')':
L.append(read_from(tokens))
tokens.pop(0) # pop off ')'
return L
elif ')' == token:
raise SyntaxError('unexpected )')
else:
return atom(token)
def atom(token):
"Numbers become numbers; every other token is a symbol."
try: return int(token)
except ValueError:
try: return float(token)
except ValueError:
return Symbol(token)
There is a straightforward way to "transliterate" Python into Haskell. This can be done by clever usage of monad transformers, which sounds scary, but it's really not. You see, due to purity, in Haskell when you want to use effects such as mutable state (e.g. the append and pop operations are performing mutation) or exceptions, you have to make it a little more explicit. Let's start at the top.
parse :: String -> SchemeExpr
parse s = readFrom (tokenize s)
The Python docstring said "Read a Scheme expression from a string", so I just took the liberty of encoding this as the type signature (String -> SchemeExpr). That docstring becomes obsolete because the type conveys the same information. Now... what is a SchemeExpr? According to your code, a scheme expression can be an int, float, symbol, or list of scheme expressions. Let's create a data type that represents these options.
data SchemeExpr
= SInt Int
| SFloat Float
| SSymbol String
| SList [SchemeExpr]
deriving (Eq, Show)
In order to tell Haskell that the Int we are dealing with should be treated as a SchemeExpr, we need to tag it with SInt. Likewise with the other possibilities. Let's move on to tokenize.
tokenize :: String -> [Token]
Again, the docstring turns into a type signature: turn a String into a list of Tokens. Well, what's a Token? If you look at the code, you'll notice that the left and right paren characters are apparently special tokens, which signal particular behaviors. Anything else is... unspecial. While we could create a data type to more clearly distinguish parens from other tokens, let's just use Strings, to stick a little closer to the original Python code.
type Token = String
Now let's try writing tokenize. First, let's write a quick little operator for making function chaining look a bit more like Python. In Haskell, you can define your own operators.
(|>) :: a -> (a -> b) -> b
x |> f = f x
tokenize s = s |> replace "(" " ( "
|> replace ")" " ) "
|> words
words is Haskell's version of split. However, Haskell has no pre-cooked version of replace that I know of. Here's one that should do the trick:
-- add imports to top of file
import Data.List.Split (splitOn)
import Data.List (intercalate)
replace :: String -> String -> String -> String
replace old new s = s |> splitOn old
|> intercalate new
If you read the docs for splitOn and intercalate, this simple algorithm should make perfect sense. Haskellers would typically write this as replace old new = intercalate new . splitOn old, but I used |> here for easier Python audience understanding.
Note that replace takes three arguments, but above I only invoked it with two. In Haskell you can partially apply any function, which is pretty neat. |> works sort of like the unix pipe, if you couldn't tell, except with more type safety.
Still with me? Let's skip over to atom. That nested logic is a bit ugly, so let's try a slightly different approach to clean it up. We'll use the Either type for a much nicer presentation.
atom :: Token -> SchemeExpr
atom s = Left s |> tryReadInto SInt
|> tryReadInto SFloat
|> orElse (SSymbol s)
Haskell doesn't have the automagical coersion functions int and float, so instead we will build tryReadInto. Here's how it works: we're going to thread Either values around. An Either value is either a Left or a Right. Conventionally, Left is used to signal error or failure, while Right signals success or completion. In Haskell, to simulate the Python-esque function call chaining, you just place the "self" argument as the last one.
tryReadInto :: Read a => (a -> b) -> Either String b -> Either String b
tryReadInto f (Right x) = Right x
tryReadInto f (Left s) = case readMay s of
Just x -> Right (f x)
Nothing -> Left s
orElse :: a -> Either err a -> a
orElse a (Left _) = a
orElse _ (Right a) = a
tryReadInto relies on type inference in order to determine which type it is trying to parse the string into. If the parse fails, it simply reproduces the same string in the Left position. If it succeeds, then it performs whatever function is desired and places the result in the Right position. orElse allows us to eliminate the Either by supplying a value in case the former computations failed. Can you see how Either acts as a replacement for exceptions here? Since the ValueExceptions in the Python code are always caught inside the function itself, we know that atom will never raise an exception. Similarly, in the Haskell code, even though we used Either on the inside of the function, the interface that we expose is pure: Token -> SchemeExpr, no outwardly-visible side effects.
OK, let's move on to read_from. First, ask yourself the question: what side effects does this function have? It mutates its argument tokens via pop, and it has internal mutation on the list named L. It also raises the SyntaxError exception. At this point, most Haskellers will be throwing up their hands saying "oh noes! side effects! gross!" But the truth is that Haskellers use side effects all the time as well. We just call them "monads" in order to scare people away and avoid success at all costs. Mutation can be accomplished with the State monad, and exceptions with the Either monad (surprise!). We will want to use both at the same time, so we'll in fact use "monad transformers", which I'll explain in a bit. It's not that scary, once you learn to see past the cruft.
First, a few utilities. These are just some simple plumbing operations. raise will let us "raise exceptions" as in Python, and whileM will let us write a while loop as in Python. For the latter, we simply have to make it explicit in what order the effects should happen: first perform the effect to compute the condition, then if it's True, perform the effects of the body and loop again.
import Control.Monad.Trans.State
import Control.Monad.Trans.Class (lift)
raise = lift . Left
whileM :: Monad m => m Bool -> m () -> m ()
whileM mb m = do
b <- mb
if b
then m >> whileM mb m
else return ()
We again want to expose a pure interface. However, there is a chance that there will be a SyntaxError, so we will indicate in the type signature that the result will be either a SchemeExpr or a SyntaxError. This is reminiscent of how in Java you can annotate which exceptions a method will raise. Note that the type signature of parse has to change as well, since it might raise the SyntaxError.
data SyntaxError = SyntaxError String
deriving (Show)
parse :: String -> Either SyntaxError SchemeExpr
readFrom :: [Token] -> Either SyntaxError SchemeExpr
readFrom = evalStateT readFrom'
We are going to perform a stateful computation on the token list that is passed in. Unlike the Python, however, we are not going to be rude to the caller and mutate the very list passed to us. Instead, we will establish our own state space and initialize it to the token list we are given. We will use do notation, which provides syntactic sugar to make it look like we're programming imperatively. The StateT monad transformer gives us the get, put, and modify state operations.
readFrom' :: StateT [Token] (Either SyntaxError) SchemeExpr
readFrom' = do
tokens <- get
case tokens of
[] -> raise (SyntaxError "unexpected EOF while reading")
(token:tokens') -> do
put tokens' -- here we overwrite the state with the "rest" of the tokens
case token of
"(" -> (SList . reverse) `fmap` execStateT readWithList []
")" -> raise (SyntaxError "unexpected close paren")
_ -> return (atom token)
I've broken out the readWithList portion into a separate chunk of code,
because I want you to see the type signature. This portion of code introduces
a new scope, so we simply layer another StateT on top of the monad stack
that we had before. Now, the get, put, and modify operations refer
to the thing called L in the Python code. If we want to perform these operations
on the tokens, then we can simply preface the operation with lift in order
to strip away one layer of the monad stack.
readWithList :: StateT [SchemeExpr] (StateT [Token] (Either SyntaxError)) ()
readWithList = do
whileM ((\toks -> toks !! 0 /= ")") `fmap` lift get) $ do
innerExpr <- lift readFrom'
modify (innerExpr:)
lift $ modify (drop 1) -- this seems to be missing from the Python
In Haskell, appending to the end of a list is inefficient, so I instead prepended, and then reversed the list afterwards. If you are interested in performance, then there are better list-like data structures you can use.
Here is the complete file: http://hpaste.org/77852
So if you're new to Haskell, then this probably looks terrifying. My advice is to just give it some time. The Monad abstraction is not nearly as scary as people make it out to be. You just have to learn that what most languages have baked in (mutation, exceptions, etc), Haskell instead provides via libraries. In Haskell, you must explicitly specify which effects you want, and controlling those effects is a little less convenient. In exchange, however, Haskell provides more safety so you don't accidentally mix up the wrong effects, and more power, because you are in complete control of how to combine and refactor effects.
In Haskell, you wouldn't use an algorithm that mutates the data it operates on. So no, there is no straightforward way to do that. However, the code can be rewritten using recursion to avoid updating variables. Solution below uses the MissingH package because Haskell annoyingly doesn't have a replace function that works on strings.
import Data.String.Utils (replace)
import Data.Tree
import System.Environment (getArgs)
data Atom = Sym String | NInt Int | NDouble Double | Para deriving (Eq, Show)
type ParserStack = (Tree Atom, Tree Atom)
tokenize = words . replace "(" " ( " . replace ")" " ) "
atom :: String -> Atom
atom tok =
case reads tok :: [(Int, String)] of
[(int, _)] -> NInt int
_ -> case reads tok :: [(Double, String)] of
[(dbl, _)] -> NDouble dbl
_ -> Sym tok
empty = Node $ Sym "dummy"
para = Node Para
parseToken (Node _ stack, Node _ out) "(" =
(empty $ stack ++ [empty out], empty [])
parseToken (Node _ stack, Node _ out) ")" =
(empty $ init stack, empty $ (subForest (last stack)) ++ [para out])
parseToken (stack, Node _ out) tok =
(stack, empty $ out ++ [Node (atom tok) []])
main = do
(file:_) <- getArgs
contents <- readFile file
let tokens = tokenize contents
parseStack = foldl parseToken (empty [], empty []) tokens
schemeTree = head $ subForest $ snd parseStack
putStrLn $ drawTree $ fmap show schemeTree
foldl is the haskeller's basic structured recursion tool and it serves the same purpose as your while loop and recursive call to read_from. I think the code can be improved a lot, but I'm not so used to Haskell. Below is an almost straight transliteration of the above to Python:
from pprint import pprint
from sys import argv
def atom(tok):
try:
return 'int', int(tok)
except ValueError:
try:
return 'float', float(tok)
except ValueError:
return 'sym', tok
def tokenize(s):
return s.replace('(',' ( ').replace(')',' ) ').split()
def handle_tok((stack, out), tok):
if tok == '(':
return stack + [out], []
if tok == ')':
return stack[:-1], stack[-1] + [out]
return stack, out + [atom(tok)]
if __name__ == '__main__':
tokens = tokenize(open(argv[1]).read())
tree = reduce(handle_tok, tokens, ([], []))[1][0]
pprint(tree)
What does the constraint (Stream s Identity t) mean in the following type declaration?
parse :: (Stream s Identity t)
=> Parsec s () a -> SourceName -> s -> Either ParseError a
What is Stream in the following class declaration, what does it mean. I'm totally lost.
class Monad m => Stream s m t | s -> t where
When I use Parsec, I get into a jam with the type-signatures (xxx :: yyy) all the time. I always skip the signatures, load the src into ghci, and then copy the type-signature back to my .hs file. It works, but I still don't understand what all these signatures are.
EDIT: more about the point of my question.
I'm still confused about the 'context' of type-signature:
(Show a) =>
means a must be a instance of class Show.
(Stream s Identity t) =>
what's the meaning of this 'context', since t never showed after the =>
I have a lot of different parser to run, so I write a warp function to run any of those parser with real files. but here comes the problem:
Here is my code, It cannot be loaded, how can I make it work?
module RunParse where
import System.IO
import Data.Functor.Identity (Identity)
import Text.Parsec.Prim (Parsec, parse, Stream)
--what should I write "runIOParse :: ..."
--runIOParse :: (Stream s Identity t, Show a) => Parsec s () a -> String -> IO ()
runIOParse pa filename =
do
inh <- openFile filename ReadMode
outh <- openFile (filename ++ ".parseout") WriteMode
instr <- hGetContents inh
let result = show $ parse pa filename instr
hPutStr outh result
hClose inh
hClose outh
the constraint: (Stream s Identity t) means what?
It means that the input s your parser works on (i.e. [Char]) must be an instance of the Stream class. In the documentation you see that [Char] is indeed an instance of Stream, since any list is.
The parameter t is the token type, which is normally Char and is determinded by s, as states the functional dependency s -> t.
But don't worry too much about this Stream typeclass. It's used only to have a unified interface for any Stream-like type, e.g. lists or ByteStrings.
what is Stream
A Stream is simply a typeclass. It has the uncons function, which returns the head of the input and the tail in a tuple wrapped in Maybe. Normally you won't need this function. As far as I can see, it's only needed in the most basic parsers like tokenPrimEx.
Edit:
what's the meaning of this 'context', since t never showed after the =>
Have a look at functional dependencies. The t never shows after the ´=>´, because it is determiend by s.
And it means that you can use uncons on whatever s is.
Here is my code, It cannot be loaded, how can I make it work?
Simple: Add an import statement for Text.Parsec.String, which defines the missing instance for Stream [tok] m tok. The documentation could be a bit clearer here, because it looks as if this instance was defined in Text.Parsec.Prim.
Alternatively import the whole Parsec library (import Text.Parsec) - this is how I always do it.
The Stream type class is an abstraction for list-like data structures. Early versions of Parsec only worked for parsing lists of tokens (for example, String is a synonym for [Char], so Char is the token type), which can be a very inefficient representation. These days, most substantial input in Haskell is handled as Text or ByteString types, which are not lists, but can act a lot like them.
So, for example, you mention
parse :: (Stream s Identity t)
=> Parsec s () a -> SourceName -> s -> Either ParseError a
Some specializations of this type would be
parse1 :: Parsec String () a -> SourceName -> String -> Either ParseError a
parse2 :: Parsec Text () a -> SourceName -> Text -> Either ParseError a
parse3 :: Parsec ByteString () a -> SourceName -> ByteString -> Either ParseError a
or even, if you have a separate lexer with a token type MyToken:
parse4 :: Parsec [MyToken] () a -> SourceName -> [MyToken] -> Either ParseError a
Of these, only the first and last use actual lists for the input, but the middle two use other Stream instances that act enough like lists for Parsec to work with them.
You can even declare your own Stream instance, so if your input is in some other type that acts sort of list-like, you can write an instance, implement the uncons function, and Parsec will work with your type, as well.