I need to parse json API responses which don't have strict structure:
{
response: { /* any object here */ }
}
How is it possible to write parseResponse which will leave ability to parse (or choose a Parser for it) for later use?
My last try is below. I don't like it because it doesn't allow to choose response type as Aeson's decode does.
data APIResponse =
APIResponse { response :: Value } deriving (Show,Generic)
instance FromJSON APIResponse
parseResponse :: BC.ByteString -> Either String Value
parseResponse resp =
case eitherDecode . BLC.fromStrict $ resp of
Right x -> Right $ response x
Left msg -> Left msg
I like to think of Aeson as happening in two distinct steps, the parsing from ByteString -> Value and the mapping from FromJSON a => Value -> a. If you don't immediately know what the correct mapping is you can simply parse to a Value as you've done. Later at runtime when you decide the proper mapping you can do that later.
import qualified Data.HashMap.Strict as Hm
decodeResponse :: ByteString -> Either String Value
decodeResponse bs = do
val <- eitherDecode bs
case val of
Object o -> case Hm.lookup "response" o of
Nothing -> fail "Invalid structure, missing 'response' key"
Just resp -> return resp
_ -> fail "Invalid structure, expected an object"
To perform a mapping without a parse, to convert a Value to a FromJSON a => a, you use the parse/parseEither/parseMaybe family and run the parser generated by parseJSON
mapValue :: FromJSON a => Value -> Either String a
mapValue = parseString parseJSON
Related
I have an application that gets JSON response from an API and I'm trying to get a specific key from that response. The working code looks as below:
{-# LANGUAGE OverloadedStrings #-}
module Main (main) where
import Data.Aeson
import Network.HTTP.Req
tinyUIDRequest :: String -> Req (JsonResponse Object)
tinyUIDRequest url = let payload = object [ "url" .= url ]
in req
POST
(https "tinyuid.com" /: "api" /: "v1" /: "shorten") (
ReqBodyJson payload)
jsonResponse
mempty
main :: IO ()
main = do
response <- runReq defaultHttpConfig (tinyUIDRequest "http://google.com")
let body = responseBody response
print body
And the function to process the response should look similar to:
tinyUIDResponseHandler :: HttpResponseBody (JsonResponse Object) -> String
tinyUIDResponseHandler r = case lookup "result_url" r of
Just url -> url
Nothing -> ""
unfortunately I'm not able to figure out how to transform HttpResponseBody (JsonResponse Object) to something like hashmap/list and find a proper key. I was checking the documentation and req source code but maybe, as a beginner in Haskell, I don't know exactly what to look for to understand the internals of the response type.
HttpResponseBody is a red herring. It's an associated type, and HttpResponseBody (JsonResponse a) is the same as just a, so HttpResponseBody (JsonResponse Object) is really just Object. As such, once you have body, req's job is done, and it's now just aeson that you need to be concerned with.
Since body is an Object and not a list of tuples, you can't use Prelude's lookup on it. If your Aeson version is older than 2, then Object is just HashMap Text Value, so do this instead:
import Data.Text
import qualified Data.HashMap.Strict as HM
tinyUIDResponseHandler :: Object -> Text
tinyUIDResponseHandler r = case HM.lookup "result_url" r of
Just (String url) -> url
_ -> ""
If your Aeson version is 2 or newer, then Object is KeyMap Value, so do this instead:
import Data.Text
import qualified Data.Aeson.KeyMap as AKM
tinyUIDResponseHandler :: Object -> Text
tinyUIDResponseHandler r = case AKM.lookup "result_url" r of
Just (String url) -> url
_ -> ""
In either case, tinyUIDResponseHandler body will then be the value you want, so for example, you could do print (tinyUIDResponseHandler body).
I'm building a RESTful endpoint that should throw an error if there is invalid data. Here's a simplified (and slightly silly) version:
getEventsR :: Handler Value
getEventsR = do
mpage <- lookupGetParam "page"
let filters = case hasPage mpage of
Right val -> val
Left val -> error $ T.unpack val -- Throw an error
-- The rest...
hasPage :: Maybe Text -> Either Text Text
hasPage =
Left "This should error out"
Is this the correct way - a function should return an Either that will later be expanded to an error?
I probably shouldn't use error, right? What instead?
As a side note, I'd be happy if my thrown error would be sent also in JSON format.
Update:
The reason I've used the term "throw exception", is because my code looks something like this (simplified):
let selectOpt = case addPager [] of
Right val -> val
Left val -> error $ T.unpack val
let filters = case addFilter [] of
Right val -> val
Left val -> error $ T.unpack val
events <- runDB $ selectList filters selectOpt :: Handler [Entity Event]
So if it errors out on selectOpt I'd like it to short-circuit, and not continue with the rest of the handler.
You can use the sendResponseStatus to directly specify the response:
getEventsR = do
mpage <- lookupGetParam "page"
case hasPage mpage of
Left message -> sendResponseStatus status400 message
Right val -> ...continue processing with val...
The second argument to sendResponseStatus can be any type which has a ToTypedContent instance. This includes Text, Value and other common types. See this page for types which have this instance defined by default.
Update
In response to your comments...
get404 is defined as:
get404 key = do
mres <- get key
case mres of
Nothing -> notFound'
Just res -> return res
notFound' :: MonadIO m => m a
notFound' = liftIO $ throwIO $ HCError NotFound
HCError is a constructor of the HandlerContents type:
http://hackage.haskell.org/package/yesod-core-1.4.0/docs/Yesod-Core-Types.html#t:HandlerContents
data HandlerContents:
HCContent Status !TypedContent
HCError ErrorResponse
HCSendFile ContentType FilePath (Maybe FilePart)
HCRedirect Status Text
HCCreated Text
HCWai Response
HCWaiApp Application
An ErrorResponse has the following constructors:
data ErrorResponse:
NotFound
InternalError Text
InvalidArgs [Text]
NotAuthenticated
PermissionDenied Text
BadMethod Method
So you can throwIO a limited number of errors, and the error messages are limited to Text.
I want to simplify this code
{-# LANGUAGE OverloadedStrings #-}
import Data.Aeson
import Network.HTTP.Types
import Data.Text
getJSON :: String -> IO (Either String Value)
getJSON url = eitherDecode <$> simpleHttp url
--------------------------------------------------------------------
maybeJson <- getJSON "abc.com"
case maybeJson of
Right jsonValue -> case jsonValue of
(Object jsonObject) ->
case (HashMap.lookup "key123" jsonObject) of
(Just (String val)) -> Data.Text.IO.putStrLn val
_ -> error "Couldn't get the key"
_ -> error "Unexpected JSON"
Left errorMsg -> error $ "Error in parsing: " ++ errorMsg
by using do syntax for Monad
maybeJson <- getJSON "abc.com/123"
let toPrint = do
Right jsonValue <- maybeJson
Object jsonObject <- jsonValue
Just (String val) <- HashMap.lookup "key123" jsonObject
return val
case toPrint of
Just a -> Data.Text.IO.putStrLn a
_ -> error "Unexpected JSON"
And it gave me 3 errors:
src/Main.hs:86:19:
Couldn't match expected type `Value'
with actual type `Either t0 (Either String Value)'
In the pattern: Right jsonValue
In a stmt of a 'do' block: Right jsonValue <- maybeJson
src/Main.hs:88:19:
Couldn't match expected type `Value' with actual type `Maybe Value'
In the pattern: Just (String val)
In a stmt of a 'do' block:
Just (String val) <- HashMap.lookup "key123" jsonObject
src/Main.hs:88:40:
Couldn't match type `Maybe' with `Either String'
Expected type: Either String Value
Actual type: Maybe Value
Even when I replace
Just (String val) <- HashMap.lookup "key123" jsonObject
with
String val <- HashMap.lookup "key123" jsonObject
I'm getting another similar error about Either:
Couldn't match type `Maybe' with `Either String'
Expected type: Either String Value
Actual type: Maybe Value
In the return type of a call of `HashMap.lookup'
In a stmt of a 'do' block:
String val <- HashMap.lookup "key123" jsonObject
How do I fix those errors?
You can't easily simplify that into a single block of do-notation, because each case is matching over a different type. The first is unpacking an either, the second a Value and the third a Maybe. Do notation works by threading everything together through a single type, so it's not directly applicable here.
You could convert all the cases to use the same monad and then write it all out in a do-block. For example, you could have helper functions that do the second and third pattern match and produce an appropriate Either. However, this wouldn't be very different to what you have now!
In fact, if I was going for this approach, I'd just be content to extract the two inner matches into their own where variables and leave it at that. Trying to put the whole thing together into one monad just confuses the issue; it's just not the right abstraction here.
Instead, you can reach for a different sort of abstraction. In particular, consider using the lens library which has prisms for working with nested pattern matches like this. It even supports aeson nateively! Your desired function would look something like this:
decode :: String -> Maybe Value
decode json = json ^? key "key123"
You could also combine this with more specific prisms, like if you're expecting a string value:
decode :: String -> Maybe String
decode json = json ^? key "key123" . _String
This takes care of parsing the json, making sure that it's an object and getting whatever's at the specified key. The only problem is that it doesn't give you a useful error message about why it failed; unfortunately, I'm not good enough with lens to understand how to fix that (if it's possible at all).
So every line in a do expression for a Monad must return a value in that Monadic type. Monad is a typeclass here, not a type by itself. So putting everything in a do Monad is not really a sensible statement.
You can try your code with everything wrapped in a Maybe monad.
Assuming you've fetched your JSON value:
{-# LANGUAGE OverloadedStrings #-}
import Data.Aeson
import Network.HTTP
import qualified Data.Map as M
import Control.Applicative
import qualified Data.HashMap.Strict as HM
--------------------------------------------------------------------
main = do
maybeJson <- return $ toJSON (M.fromList [("key123","value")] :: M.Map String String)
ioVal <- return $ do -- The Maybe monad do expression starts here
maybeJson <- Just maybeJson
jsonObject <- case maybeJson of
Object x -> Just x
_ -> Nothing
val <- HM.lookup "key123" jsonObject
return val
putStrLn $ show ioVal
Once we start working in the Maybe monad, every expression must return a Maybe Something value. The way the Maybe monad works is that anything that is a Just something comes out as a pure something that you can work with, but if you get a Nothing, the rest of the code will be skipped and you'll get a Nothing.
This property of falling through is unique to the Maybe monad. Different monads behave differently.
You should read up more about Monads and the IO monad here: http://www.haskell.org/haskellwiki/Introduction_to_IO
You should read more about monads and what they really help you do:
http://learnyouahaskell.com/a-fistful-of-monads
(You should work through the previous chapters and then get to this chapter. Once you do, you'll have a pretty solid understanding of what is happening).
I also think your HTTP request is screwed up. Here's an example of a POST request that you can use.
import qualified Network.HTTP as H
main = do
postData <- return $ H.urlEncodeVars [("someVariable","someValue")]
request <- return $ H.postRequestWithBody "http://www.google.com/recaptcha/api/verify" "application/x-www-form-urlencoded" postData
putStrLn $ show request
-- Make the request
(Right response) <- H.simpleHTTP request
-- Print status code
putStrLn $ show $ H.rspCode response
-- Print response
putSrLn $ show $ H.rspBody response
UPDATED:
Use the following to help you get a JSON value:
import qualified Data.ByteString.Lazy as LC
import qualified Data.ByteString.Char8 as C
import qualified Data.Aeson as DA
responseBody <- return $ H.rspBody response
responseJSON <- return (DA.decode (LC.fromChunks [C.pack responseBody]) :: Maybe DA.Value)
You'll have to make a request object to make a request. There are quite a few helpers. I meant the post request as the most generic case:
http://hackage.haskell.org/package/HTTP-4000.0.5/docs/Network-HTTP.html
Since you are in the IO monad, all the <- are going to assume that you are dealing with IO operations. When you write
do
Right jsonValue <- maybeJson
Object jsonObject <- jsonValue
you are saying that jsonValue must be an IO action just like maybeJson. But this is not the case! jsonValue is but a regular Either value. The silution here would ge to use a do-block let instead of a <-:
do
Right jsonValue <- maybeJson
let Object jsonObject = jsonValue
However, its important to note that in both versions of your code you are using an irrecoverable error to abort your program if the JSON parsing fails. If you want to be able to collect errors, the basic idea would be to convert your values to Either (and then use the monad instance for Either to avoid having lots of nested case expressions)
I'm a complete newbie to Haskell and having some trouble parsing JSON from a url. I have managed to code the parsing sides of things and have tested with Strings and they work fine. The trouble I'm getting is when I start working with IO.
I have the following code :
data Movie = Movie
{ adult :: Bool,
backdrop_path :: Maybe String,
id :: Int,
original_title :: String,
release_date :: String,
poster_path :: Maybe String,
popularity :: Int,
title :: String,
vote_average :: Double,
vote_count :: Int
} deriving (Show)
data MovieList = MovieList {getMovies ::[Movie]} deriving (Show)
instance FromJSON Movie where
parseJSON (Object v) = Movie <$>
(v .: "adult") <*>
(v .:? "backdrop_path") <*> -- Optional
(v .: "id") <*>
(v .: "original_title") <*>
(v .: "release_date") <*>
(v .:? "poster_path") <*> -- Optional
(v .: "popularity") <*>
(v .: "title") <*>
(v .: "vote_average") <*>
(v .: "vote_count")
instance FromJSON MovieList where
parseJSON (Object o) = MovieList <$> o .: "results"
parseJSON _ = mzero
movieAPIRequest :: String -> IO String
movieAPIRequest movieURI =
do resp <- simpleHTTP request
case resp of
Left x -> return $ "Error connecting: " ++ show x
Right r ->
case rspCode r of
(2,_,_) -> return $ rspBody r -- Request Fulfilled
_ -> return $ show r -- Request Failed
where request = Request {rqURI = uri, rqMethod = GET, rqHeaders = [], rqBody = ""}
uri = fromJust $ parseURI movieURI
convertToByteString s = BS.pack s
main = do
response <- movieAPIRequest url
decoded <- decode (convertToByteString response):: Maybe MovieList
return $ decoded
I can't get the main to work. I want to automatically retrieve JSON from a url. The movieAPIRequest gives me the body of the request (JSON) as IO String. convertToByteString takes a String and converts to Data.ByteString.Lazy.Char8.ByteString as the decode function in Aeson takes a bytestring as an argument. With the above code I'm getting the following error:
[1 of 1] Compiling MovieDataType ( MovieDataType.hs, interpreted )
MovieDataType.hs:62:20:
Couldn't match type `Maybe' with `IO'
Expected type: IO MovieList
Actual type: Maybe MovieList
In a stmt of a 'do' block:
decoded <- decode (convertToByteString response) :: Maybe MovieList
In the expression:
do { response <- movieAPIRequest url;
decoded <- decode (convertToByteString response) ::
Maybe MovieList;
return $ decoded }
In an equation for `main':
main
= do { response <- movieAPIRequest url;
decoded <- decode (convertToByteString response) ::
Maybe MovieList;
return $ decoded }
Failed, modules loaded: none.
I've tried fixing it but keep getting different that I can't get my head around. For example this one is telling its expecting IO MovieList but decode should return a Maybe MovieList.
I see a couple of errors in your main function
main = do
response <- movieAPIRequest url
decoded <- decode (convertToByteString response):: Maybe MovieList
return $ decoded
First of all, if you look up "decode" at http://hackage.haskell.org/package/aeson-0.6.1.0/docs/Data-Aeson.html, you will see that it is of type
decode :: FromJSON a => ByteString -> Maybe a
Note that no IO is involved in this definition, so you can not extract any data from it using "<-". Instead, use "let decoded = ....".
You can read up on the theory behind this distinction as you learn more Haskell, but for now, just note that if a function returns type IO a, you extract data from it using "value <- function x y" (this is why the movieAPIRequest is fine), whereas if the function has no side effects, you extract the value using "let value = function x y". (it is more complicated than this, but this is the 90% answer that will keep you going until you learn about monads).
Second, you are returning a value from main, which is of type IO(). This is the hardcoded type of main in any Haskell program, so you can't return any value except "()". Instead, you have to do something with the value "decoded" (like print it) before the program ends.
I have a server that receives JSON that looks like:
{ "foo": "bar", "bono": "bobo",
"result": { "some": ["complex", "JSON", "structure",...
}
where all the stuff is for the Server except for "result", which is to be forwarded to the client (Worker --JSON--> Server --value of "result"--> Client). Therefore, when parsing this thing with aeson, I want to keep the value of "result" as a string (or Text or whatnot), just so I can forward it to the client without caring what's inside. Problem is that "result" can be anything (array, object, etc.).
So If I do
data RPCResult = RPCResult { foo :: Text, result :: Text }
the decode function of aeson is going to return Nothing, since "result" is not necessarily a JSON string...
How do I tell aeson to keep parts of the JSON object as-is and just give them to me so I can do what I want with them?
I seem to have somewhat of a solution by making the type of result be Data.Aeson.Value and then after doing a decode on the incoming JSON, I extract the result and run encode on it before forwarding it. I'm not sure if this is the best solution (because I'm not "keeping it as a string" as the question states, but decoding it then encoding it again...) but it works:
import Data.Aeson
import Data.Maybe
import Data.ByteString.Lazy (fromStrict, ByteString)
import Data.ByteString.UTF8 (fromString)
import Control.Applicative
data RPCResult = RPCResult { foo :: Text, result :: Value }
instance FromJSON RPCResult where
parseJSON (Object v) = RPCResult <$> v .: "foo" <*> v .: "result"
-- example just so you get the idea:
toRPCResult :: String -> RPCResult
toRPCResult = fromJust . decode . fromStrict . fromString
getResult :: String -> ByteString
getResult = encode . result . toRPCResult