I receive a JSON data using httpLbs and read it
import qualified Data.ByteString.Lazy.UTF8 as LB
sendSimpleRequest :: Credentials -> IO LB.ByteString
sendSimpleRequest creds = do
<...>
let request = applyBasicAuth user pass $ fromJust $ parseUrl url
manager <- newManager tlsManagerSettings
response <- httpLbs request manager
return $ responseBody response
After that, I can print the result with putStr . LB.toString and get "summary":"Обсуждение рабочих вопросов".
However, when I try to use aeson's decode to put this value into data and print it
data Fields = Fields
{ fi_summary :: String
} deriving (Show, Generic)
instance FromJSON Fields where parseJSON = genericParseJSON parseOptions
instance ToJSON Fields where toJSON = genericToJSON parseOptions
parseOptions :: Options
parseOptions = defaultOptions { fieldLabelModifier = drop 3 }
parseAndShow = putStr . show . fromJust . decode
I get escaped characters: Fields {fi_summary = "\1054\1073\1089\1091\1078\1076\1077\1085\1080\1077 \1088\1072\1073\1086\1095\1080\1093 \1074\1086\1087\1088\1086\1089\1086\1074"}
Seems like I need to configure aeson to correctly put ByteString into String, but I don't want to implement the FromJSON instance myself because I have a dozen more structures like data Fields. Changing the fi_summary type is also a possibility, but I had no luck with any so far.
If you're seeing escaped characters, then the data is in the string just fine. The default string Show instance prints all non-ASCII characters like this. So you're got the data, it's just a matter of trying to output it again appropriately.
You might try using putStrLn to print the string, or maybe write it to a text file. (I know sometimes putStrLn does strange things if the locale is set wrong...)
Related
I am trying to write a FromJSON implementation which would parse a list of objects while at the same time skipping some of them - those which contain a certain json property.
I have the code like this, but without proper handling of mzero it returns an error once it encounters a value with "exclude: true".
newtype Response = Response [Foo]
newtype Foo = Foo Text
instance FromJSON Response where
parseJSON = withArray "Foos" $ \arr -> do
-- can I filter out here the ones which return `mzero`?
foos <- mapM parseJSON arr
pure $ Response (toList foos)
instance FromJSON Foo where
parseJSON = withObject "Foo" $ \foo -> do
isExcluded <- foo .: "exclude"
if isExcluded
then mzero
else do
pure $ Foo "bar"
I've found a few questions which hint at using parseMaybe, but I can't figure out how I can use it from within the FromJSON definition, it seems to be more suited to running the parser from "outside". Is it possible to do skipping "inside"? Or am I going the wrong route here?
What you're looking for is the optional function. It can be a bit tricky to find because it's very general-purpose and not simply a helper function in aeson. For your purposes, it will have the type Parser a -> Parser (Maybe a) and used in conjunction with catMaybes should do what you want.
I have written this ManagerSettings to log all requests and responses for my http-conduit application. (By the way, I am importing ClassyPrelude).
tracingManagerSettings :: ManagerSettings
tracingManagerSettings =
tlsManagerSettings { managerModifyRequest = \req -> do
putStr "TRACE: "
print req
putStrLn ""
pure req
, managerModifyResponse = \r -> do
responseChunks <- brConsume $ responseBody r
let fullResponse = mconcat responseChunks
putStr "TRACE: RESPONSE: "
putStrLn $ decodeUtf8 fullResponse
pure $ r { responseBody = pure fullResponse }
}
However, it's not working - when I use it, the application is hanging and trying to consume all the RAM in the machine after printing the first request and first response, which suggests some kind of infinite loop.
Also, the request is printed twice.
I made a previous attempt that was similar, but didn't modify r. That failed because after I had already read the response completely, there was no more response data to read.
If I replace this with tlsManagerSettings, http-conduit works again.
My application is using libstackexchange, which I have modified to allow the ManagerSettings to be customised. I am using http-conduit version 2.2.4.
How can I diagnose the issue? How can I fix it?
managerModifyResponse doesn't work with a Response ByteString, it works with a Response BodyReader, where type BodyReader = IO ByteString along with the contract that if it produces a non-empty ByteString there is more input that can be read.
The problem you're running into is that pure fullResponse never returns an empty ByteString unless it always does. You need to provide a somewhat more complex IO action to capture the intended behavior. Maybe something along these lines (untested):
returnOnce :: Monoid a => a -> IO (IO a)
returnOnce x = do
ref <- newIORef x
pure $ readIORef ref <* writeIORef ref mempty
As for how to debug this? Not sure about generic methods. I was just suspicious that you probably needed a solution along these lines, and the docs for BodyReader confirmed it.
I want to simplify this code
{-# LANGUAGE OverloadedStrings #-}
import Data.Aeson
import Network.HTTP.Types
import Data.Text
getJSON :: String -> IO (Either String Value)
getJSON url = eitherDecode <$> simpleHttp url
--------------------------------------------------------------------
maybeJson <- getJSON "abc.com"
case maybeJson of
Right jsonValue -> case jsonValue of
(Object jsonObject) ->
case (HashMap.lookup "key123" jsonObject) of
(Just (String val)) -> Data.Text.IO.putStrLn val
_ -> error "Couldn't get the key"
_ -> error "Unexpected JSON"
Left errorMsg -> error $ "Error in parsing: " ++ errorMsg
by using do syntax for Monad
maybeJson <- getJSON "abc.com/123"
let toPrint = do
Right jsonValue <- maybeJson
Object jsonObject <- jsonValue
Just (String val) <- HashMap.lookup "key123" jsonObject
return val
case toPrint of
Just a -> Data.Text.IO.putStrLn a
_ -> error "Unexpected JSON"
And it gave me 3 errors:
src/Main.hs:86:19:
Couldn't match expected type `Value'
with actual type `Either t0 (Either String Value)'
In the pattern: Right jsonValue
In a stmt of a 'do' block: Right jsonValue <- maybeJson
src/Main.hs:88:19:
Couldn't match expected type `Value' with actual type `Maybe Value'
In the pattern: Just (String val)
In a stmt of a 'do' block:
Just (String val) <- HashMap.lookup "key123" jsonObject
src/Main.hs:88:40:
Couldn't match type `Maybe' with `Either String'
Expected type: Either String Value
Actual type: Maybe Value
Even when I replace
Just (String val) <- HashMap.lookup "key123" jsonObject
with
String val <- HashMap.lookup "key123" jsonObject
I'm getting another similar error about Either:
Couldn't match type `Maybe' with `Either String'
Expected type: Either String Value
Actual type: Maybe Value
In the return type of a call of `HashMap.lookup'
In a stmt of a 'do' block:
String val <- HashMap.lookup "key123" jsonObject
How do I fix those errors?
You can't easily simplify that into a single block of do-notation, because each case is matching over a different type. The first is unpacking an either, the second a Value and the third a Maybe. Do notation works by threading everything together through a single type, so it's not directly applicable here.
You could convert all the cases to use the same monad and then write it all out in a do-block. For example, you could have helper functions that do the second and third pattern match and produce an appropriate Either. However, this wouldn't be very different to what you have now!
In fact, if I was going for this approach, I'd just be content to extract the two inner matches into their own where variables and leave it at that. Trying to put the whole thing together into one monad just confuses the issue; it's just not the right abstraction here.
Instead, you can reach for a different sort of abstraction. In particular, consider using the lens library which has prisms for working with nested pattern matches like this. It even supports aeson nateively! Your desired function would look something like this:
decode :: String -> Maybe Value
decode json = json ^? key "key123"
You could also combine this with more specific prisms, like if you're expecting a string value:
decode :: String -> Maybe String
decode json = json ^? key "key123" . _String
This takes care of parsing the json, making sure that it's an object and getting whatever's at the specified key. The only problem is that it doesn't give you a useful error message about why it failed; unfortunately, I'm not good enough with lens to understand how to fix that (if it's possible at all).
So every line in a do expression for a Monad must return a value in that Monadic type. Monad is a typeclass here, not a type by itself. So putting everything in a do Monad is not really a sensible statement.
You can try your code with everything wrapped in a Maybe monad.
Assuming you've fetched your JSON value:
{-# LANGUAGE OverloadedStrings #-}
import Data.Aeson
import Network.HTTP
import qualified Data.Map as M
import Control.Applicative
import qualified Data.HashMap.Strict as HM
--------------------------------------------------------------------
main = do
maybeJson <- return $ toJSON (M.fromList [("key123","value")] :: M.Map String String)
ioVal <- return $ do -- The Maybe monad do expression starts here
maybeJson <- Just maybeJson
jsonObject <- case maybeJson of
Object x -> Just x
_ -> Nothing
val <- HM.lookup "key123" jsonObject
return val
putStrLn $ show ioVal
Once we start working in the Maybe monad, every expression must return a Maybe Something value. The way the Maybe monad works is that anything that is a Just something comes out as a pure something that you can work with, but if you get a Nothing, the rest of the code will be skipped and you'll get a Nothing.
This property of falling through is unique to the Maybe monad. Different monads behave differently.
You should read up more about Monads and the IO monad here: http://www.haskell.org/haskellwiki/Introduction_to_IO
You should read more about monads and what they really help you do:
http://learnyouahaskell.com/a-fistful-of-monads
(You should work through the previous chapters and then get to this chapter. Once you do, you'll have a pretty solid understanding of what is happening).
I also think your HTTP request is screwed up. Here's an example of a POST request that you can use.
import qualified Network.HTTP as H
main = do
postData <- return $ H.urlEncodeVars [("someVariable","someValue")]
request <- return $ H.postRequestWithBody "http://www.google.com/recaptcha/api/verify" "application/x-www-form-urlencoded" postData
putStrLn $ show request
-- Make the request
(Right response) <- H.simpleHTTP request
-- Print status code
putStrLn $ show $ H.rspCode response
-- Print response
putSrLn $ show $ H.rspBody response
UPDATED:
Use the following to help you get a JSON value:
import qualified Data.ByteString.Lazy as LC
import qualified Data.ByteString.Char8 as C
import qualified Data.Aeson as DA
responseBody <- return $ H.rspBody response
responseJSON <- return (DA.decode (LC.fromChunks [C.pack responseBody]) :: Maybe DA.Value)
You'll have to make a request object to make a request. There are quite a few helpers. I meant the post request as the most generic case:
http://hackage.haskell.org/package/HTTP-4000.0.5/docs/Network-HTTP.html
Since you are in the IO monad, all the <- are going to assume that you are dealing with IO operations. When you write
do
Right jsonValue <- maybeJson
Object jsonObject <- jsonValue
you are saying that jsonValue must be an IO action just like maybeJson. But this is not the case! jsonValue is but a regular Either value. The silution here would ge to use a do-block let instead of a <-:
do
Right jsonValue <- maybeJson
let Object jsonObject = jsonValue
However, its important to note that in both versions of your code you are using an irrecoverable error to abort your program if the JSON parsing fails. If you want to be able to collect errors, the basic idea would be to convert your values to Either (and then use the monad instance for Either to avoid having lots of nested case expressions)
I need to parse json API responses which don't have strict structure:
{
response: { /* any object here */ }
}
How is it possible to write parseResponse which will leave ability to parse (or choose a Parser for it) for later use?
My last try is below. I don't like it because it doesn't allow to choose response type as Aeson's decode does.
data APIResponse =
APIResponse { response :: Value } deriving (Show,Generic)
instance FromJSON APIResponse
parseResponse :: BC.ByteString -> Either String Value
parseResponse resp =
case eitherDecode . BLC.fromStrict $ resp of
Right x -> Right $ response x
Left msg -> Left msg
I like to think of Aeson as happening in two distinct steps, the parsing from ByteString -> Value and the mapping from FromJSON a => Value -> a. If you don't immediately know what the correct mapping is you can simply parse to a Value as you've done. Later at runtime when you decide the proper mapping you can do that later.
import qualified Data.HashMap.Strict as Hm
decodeResponse :: ByteString -> Either String Value
decodeResponse bs = do
val <- eitherDecode bs
case val of
Object o -> case Hm.lookup "response" o of
Nothing -> fail "Invalid structure, missing 'response' key"
Just resp -> return resp
_ -> fail "Invalid structure, expected an object"
To perform a mapping without a parse, to convert a Value to a FromJSON a => a, you use the parse/parseEither/parseMaybe family and run the parser generated by parseJSON
mapValue :: FromJSON a => Value -> Either String a
mapValue = parseString parseJSON
I'm a complete newbie to Haskell and having some trouble parsing JSON from a url. I have managed to code the parsing sides of things and have tested with Strings and they work fine. The trouble I'm getting is when I start working with IO.
I have the following code :
data Movie = Movie
{ adult :: Bool,
backdrop_path :: Maybe String,
id :: Int,
original_title :: String,
release_date :: String,
poster_path :: Maybe String,
popularity :: Int,
title :: String,
vote_average :: Double,
vote_count :: Int
} deriving (Show)
data MovieList = MovieList {getMovies ::[Movie]} deriving (Show)
instance FromJSON Movie where
parseJSON (Object v) = Movie <$>
(v .: "adult") <*>
(v .:? "backdrop_path") <*> -- Optional
(v .: "id") <*>
(v .: "original_title") <*>
(v .: "release_date") <*>
(v .:? "poster_path") <*> -- Optional
(v .: "popularity") <*>
(v .: "title") <*>
(v .: "vote_average") <*>
(v .: "vote_count")
instance FromJSON MovieList where
parseJSON (Object o) = MovieList <$> o .: "results"
parseJSON _ = mzero
movieAPIRequest :: String -> IO String
movieAPIRequest movieURI =
do resp <- simpleHTTP request
case resp of
Left x -> return $ "Error connecting: " ++ show x
Right r ->
case rspCode r of
(2,_,_) -> return $ rspBody r -- Request Fulfilled
_ -> return $ show r -- Request Failed
where request = Request {rqURI = uri, rqMethod = GET, rqHeaders = [], rqBody = ""}
uri = fromJust $ parseURI movieURI
convertToByteString s = BS.pack s
main = do
response <- movieAPIRequest url
decoded <- decode (convertToByteString response):: Maybe MovieList
return $ decoded
I can't get the main to work. I want to automatically retrieve JSON from a url. The movieAPIRequest gives me the body of the request (JSON) as IO String. convertToByteString takes a String and converts to Data.ByteString.Lazy.Char8.ByteString as the decode function in Aeson takes a bytestring as an argument. With the above code I'm getting the following error:
[1 of 1] Compiling MovieDataType ( MovieDataType.hs, interpreted )
MovieDataType.hs:62:20:
Couldn't match type `Maybe' with `IO'
Expected type: IO MovieList
Actual type: Maybe MovieList
In a stmt of a 'do' block:
decoded <- decode (convertToByteString response) :: Maybe MovieList
In the expression:
do { response <- movieAPIRequest url;
decoded <- decode (convertToByteString response) ::
Maybe MovieList;
return $ decoded }
In an equation for `main':
main
= do { response <- movieAPIRequest url;
decoded <- decode (convertToByteString response) ::
Maybe MovieList;
return $ decoded }
Failed, modules loaded: none.
I've tried fixing it but keep getting different that I can't get my head around. For example this one is telling its expecting IO MovieList but decode should return a Maybe MovieList.
I see a couple of errors in your main function
main = do
response <- movieAPIRequest url
decoded <- decode (convertToByteString response):: Maybe MovieList
return $ decoded
First of all, if you look up "decode" at http://hackage.haskell.org/package/aeson-0.6.1.0/docs/Data-Aeson.html, you will see that it is of type
decode :: FromJSON a => ByteString -> Maybe a
Note that no IO is involved in this definition, so you can not extract any data from it using "<-". Instead, use "let decoded = ....".
You can read up on the theory behind this distinction as you learn more Haskell, but for now, just note that if a function returns type IO a, you extract data from it using "value <- function x y" (this is why the movieAPIRequest is fine), whereas if the function has no side effects, you extract the value using "let value = function x y". (it is more complicated than this, but this is the 90% answer that will keep you going until you learn about monads).
Second, you are returning a value from main, which is of type IO(). This is the hardcoded type of main in any Haskell program, so you can't return any value except "()". Instead, you have to do something with the value "decoded" (like print it) before the program ends.