Continuing my exploration of conduit and aeson, how would I go about using my own data type in stead of Value in this (slightly modified) code snippet from the Yesod book.
{-# LANGUAGE OverloadedStrings, TemplateHaskell #-}
import Network.Wai (Response, responseLBS, Application, requestBody)
import Network.HTTP.Types (status200, status400)
import Network.Wai.Handler.Warp (run)
import Data.Aeson.Parser (json)
import Data.Conduit.Attoparsec (sinkParser)
import Control.Monad.IO.Class (liftIO)
import Data.Aeson (Value(..), encode, object, (.=))
import Control.Exception (SomeException)
import Data.ByteString (ByteString)
import Data.Conduit (ResourceT, ($$))
import Control.Exception.Lifted (handle)
import qualified Data.HashMap.Strict as M
import Data.Aeson.TH (deriveJSON)
-- I ADDED THIS
data JSONRequest = JSONRequest {
command :: ByteString,
params :: M.HashMap ByteString ByteString
}
deriveJSON id ''JSONRequest
-- END OF WHAT I ADDED
main :: IO ()
main = run 3000 app
app :: Application
app req = handle invalidJson $ do
value <- requestBody req $$ sinkParser json
newValue <- liftIO $ dispatch value
return $ responseLBS
status200
[("Content-Type", "application/json")]
$ encode newValue
invalidJson :: SomeException -> ResourceT IO Response
invalidJson ex = return $ responseLBS
status400
[("Content-Type", "application/json")]
$ encode $ object
[ ("message" .= show ex)
]
-- Application-specific logic would go here.
dispatch :: Value -> IO Value
dispatch = return
Basically, I want to change the type of dispatch to JSONRequest -> IO JSONRequest. How do I tell the parser to use my own derived instance of fromJSON?
I tried just adding a type declaration, praying for polymorphic return type on json, but I realised it is strictly for Value.
Just looking at the types, don't you just need to fmap your fromJSON over the result coming from json? With a suitable signature for dispatch we just need:
-- import Data.Aeson
app :: Application
app req = handle invalidJson $ do
result <- requestBody req $$ sinkParser (fmap fromJSON json)
next_result <- liftIO $ dispatch result
return $ responseLBS status200 [("Content-Type", "application/json")]
$ encode next_result
dispatch :: Result JSONRequest -> IO JSONRequest
dispatch (Error str) = undefined
dispatch (Success jsonreq) = return jsonreq
But maybe it's a little clearer written thus:
-- import Data.Aeson
-- import qualified Data.Attoparsec as Atto
toRequest :: Value -> Result JSONRequest
toRequest = fromJSON -- specialized now to your fromJSON
jsonRequestParser :: Atto.Parser (Result JSONRequest)
jsonRequestParser = fmap toRequest json
app :: Application
app req = handle invalidJson $ do
result <- requestBody req $$ sinkParser jsonRequestParser
next_result <- liftIO $ dispatch result
return $ responseLBS status200 [("Content-Type", "application/json")]
$ encode next_result
dispatch :: Result JSONRequest -> IO JSONRequest
dispatch (Error str) = undefined
dispatch (Success jsonreq) = return jsonreq
I left the parser returning a Result JSONRequest so dispatch is handling Error cases too, which might mean you need your exception handling somehow?
Related
I'm trying to create some middleware that will send 500 errors to a remote server. The error information is in the response body.
How can I get the response body from a Response as any kind of string? I see responseToStream but I can't figure out how to use it.
import Network.Wai
import Data.ByteString.Lazy (ByteString)
responseBody :: Response -> IO ByteString
responseBody res = _
An implementation of the comment by #user2407038:
import Data.IORef (newIORef,modifyIORef',readIORef)
import Data.Monoid ((<>))
import Data.ByteString.Lazy (ByteString)
import Data.ByteString.Builder (toLazyByteString)
import Network.Wai
responseBody :: Response -> IO ByteString
responseBody res =
let (status,headers,body) = responseToStream res in
body $ \f -> do
content <- newIORef mempty
f (\chunk -> modifyIORef' content (<> chunk)) (return ())
toLazyByteString <$> readIORef content
I am trying to create a conduit that will stream data from HTTP through a conduit source. Here is what I have so far:
import qualified Network.HTTP.Client.Conduit as CC
getStream :: String -> IO (ConduitM () BS.ByteString IO ())
getStream url = do
req <- parseUrl url
return $ CC.withResponse req $ \res -> do
responseBody res $= (awaitForever $ \bytes -> liftIO $ do
putStrLn $ "Got " ++ show (BS.length bytes) ++ " but will ignore them")
But I am getting
No instance for (Control.Monad.Reader.Class.MonadReader env0 IO) …
arising from a use of ‘CC.withResponse’
In the expression: CC.withResponse req
In the second argument of ‘($)’, namely
‘CC.withResponse req
$ \ res
-> do { responseBody res $= (awaitForever $ \ bytes -> ...) }’
In a stmt of a 'do' block:
return
$ CC.withResponse req
$ \ res
-> do { responseBody res $= (awaitForever $ \ bytes -> ...) }
How come a MonadReader is expected? It doesn't make any sense to me.
How about this variation of the example in the Network.HTTP.Conduit docs:
{-# LANGUAGE OverloadedStrings #-}
module Lib2 () where
import Data.Conduit (($$+-), awaitForever)
import qualified Network.HTTP.Client.Conduit as CC
import Network.HTTP.Conduit (http, tlsManagerSettings, newManager)
import Control.Monad.IO.Class (liftIO)
import Control.Monad.Trans.Resource (runResourceT)
import Data.Conduit.Binary (sinkFile) -- Exported from the package conduit-extra
main2 :: IO ()
main2 = do
request <- CC.parseUrl "http://google.com/"
manager <- newManager tlsManagerSettings
runResourceT $ do
response <- http request manager
CC.responseBody response $$+- (awaitForever $ \x -> liftIO $ putStrLn "Chunk")
Original answer
The return type for getStream is wrong. Try removing the type signature and use FlexibleContexts, e.g.:
{-# LANGUAGE OverloadedStrings, FlexibleContexts #-}
module Lib () where
import Data.Conduit
import qualified Data.ByteString as BS
import qualified Network.HTTP.Client.Conduit as CC
import Control.Monad.IO.Class
getStream url = do
req <- CC.parseUrl url
CC.withResponse req $ \res -> do
CC.responseBody res $= (awaitForever $ \x -> liftIO $ putStrLn "Got a chunk")
And then :t getStream reports:
getStream
:: (monad-control-1.0.0.4:Control.Monad.Trans.Control.MonadBaseControl
IO (ConduitM a c m),
mtl-2.2.1:Control.Monad.Reader.Class.MonadReader env m, MonadIO m,
CC.HasHttpManager env,
exceptions-0.8.0.2:Control.Monad.Catch.MonadThrow m) =>
String -> ConduitM a c m ()
which shows that the return type has the form ConduitM ..., not IO ....
This also shows how MonadReader gets into the picture... The monad m must have access to an HTTP manager through a reader environment as expressed by the following constraints:
CC.HasHttpManager env
MonadReader env m
All this is saying is that m has a reader environment of some type env which itself has a way of accessing an HTTP manager.
In particular, m cannot be just the plain IO monad, which is what the error message is complaining about.
Answer to question in the comments
Here is an example of how to create a Producer from a HTTP response:
{-# LANGUAGE OverloadedStrings #-}
module Lib3 () where
import qualified Data.ByteString as BS
import qualified Network.HTTP.Client.Conduit as CC
import Network.HTTP.Conduit (http, tlsManagerSettings, newManager)
import qualified Network.HTTP.Client as Client (httpLbs, responseOpen, responseClose)
import Data.Conduit (Producer, addCleanup)
import Data.Conduit (awaitForever, await, ($$))
import qualified Network.HTTP.Client.Conduit as HCC
import Control.Monad.IO.Class (liftIO, MonadIO)
getStream url = do
request <- CC.parseUrl url
manager <- newManager tlsManagerSettings
response <- Client.responseOpen request manager
let producer :: Producer IO BS.ByteString
producer = HCC.bodyReaderSource $ CC.responseBody response
cleanup _ = do liftIO $ putStrLn "(cleaning up)"; Client.responseClose response
producerWithCleanup = addCleanup cleanup producer
return $ response { CC.responseBody = producerWithCleanup }
test = do
res <- getStream "http://google.com"
let producer = CC.responseBody res
consumer = awaitForever $ \_ -> liftIO $ putStrLn "Got a chunk"
producer $$ consumer
I am trying to get back a value that is in a json feed (via Aeson) directly inside a StateT stacked on IO:
{-# LANGUAGE DeriveGeneric #-}
module MyFeed where
import Data.Aeson
import Network.URI (parseURI, URI(..))
import Data.Maybe (fromJust)
import Data.Text (Text, unpack)
import Control.Monad.State
import Network.HTTP
import GHC.Generics
import Control.Applicative
import Network.HTTP.Conduit (simpleHttp)
import qualified Data.ByteString.Lazy as B
type Feed a = StateT MyIndex IO a
data MyIndex = MyIndex {
index :: Int
}
data FooBar = Foo | Bar
data MyFeed = MyFeed {
idx :: !Text,
key :: !Text
} deriving (Show,Generic)
instance FromJSON MyFeed
instance ToJSON MyFeed
getJSON :: String -> IO B.ByteString
getJSON url = simpleHttp url
getFeed :: String -> IO (Maybe MyFeed)
getFeed url = (decode <$> getJSON url) :: IO (Maybe MyFeed)
getIndex :: FooBar -> Feed MyIndex
getIndex fb = do
cursor <- get
let newCursor = case fb of
Foo -> do myFeed <- liftIO $ getFeed "http://echo.jsontest.com/key/value/idx/1"
let i = read $ unpack $ idx $ fromJust myFeed
return $ cursor { index = i }
Bar -> return cursor
put newCursor
return newCursor
In the Foo case I fetch the feed as expected but when the required value is returned I get:
src/MyFeed.hs:47:10:
Couldn't match expected type ‘MyIndex’
with actual type ‘m0 MyIndex’
Relevant bindings include
newCursor :: m0 MyIndex (bound at src/MyFeed.hs:40:7)
In the first argument of ‘return’, namely ‘newCursor’
In a stmt of a 'do' block: return newCursor
The Actual Type looks still in a Monad context (do {...}). Is there a way to take it out or I am using a wrong approach?
The error was due to the fact I used:
let newCursor = case fb of
instead of
newCursor <- case fb of
For this reason the final value never get "unwrapped" from its monad context.
I'm trying to create a JSON REST api using Happstack. It should allow POSTS with a JSON body. How can I do this? All the functions in happstack's API seem to look things up based on parameter name. It thinks the body is always url-encoded.
If it isn't possible with Happstack, which framework should I use?
Alright, here's what I figured out.
{-# LANGUAGE OverloadedStrings, DeriveDataTypeable #-}
import qualified Data.ByteString.Lazy.Char8 as L
import Happstack.Server
import Happstack.Server.Types
import Control.Monad.IO.Class (liftIO)
import Data.Data (Data, Typeable)
-- easiest to serialize/deserialize objects
data Unit = Unit { x :: Int, y :: Int } deriving (Show, Eq, Data, Typeable)
-- put this function in a library somewhere
getBody :: ServerPart L.ByteString
getBody = do
req <- askRq
body <- liftIO $ takeRequestBody req
case body of
Just rqbody -> return . unBody $ rqbody
Nothing -> return ""
myRoute :: ServerPart Response
myRoute = do
body <- getBody -- it's a ByteString
let unit = fromJust $ A.decode body :: Unit -- how to parse json
ok $ toResponse $ A.encode unit -- how to send json back.
Good day, i have mongodb database filled with some data, i ensured that data stored in correct charset, to fetch data i use following snippet:
{-# LANGUAGE OverloadedStrings #-}
import Network.Wai
import Network.Wai.Handler.Warp (run)
import Data.Enumerator (Iteratee (..))
import Data.Either (either)
import Control.Monad (join)
import Data.Maybe (fromMaybe)
import Network.HTTP.Types (statusOK, status404)
import qualified Data.ByteString as B
import qualified Data.ByteString.Lazy as L
import Data.ByteString.Char8 (unpack)
import Data.ByteString.Lazy.Char8 (pack)
import qualified Data.Text.Lazy as T
import Data.Text (Text(..))
import Control.Monad.IO.Class (liftIO, MonadIO)
import Data.Aeson (encode)
import qualified Data.Map as Map
import qualified Database.MongoDB as DB
application dbpipe req = do
case unpack $ rawPathInfo req of
"/items" -> itemsJSON dbpipe req
_ -> return $ responseLBS status404 [("Content-Type", "text/plain")] "404"
indexPage :: Iteratee B.ByteString IO Response
indexPage = do
page <- liftIO $ processTemplate "templates/index.html" []
return $ responseLBS statusOK [("Content-Type", "text/html; charset=utf-8")] page
processTemplate f attrs = do
page <- L.readFile f
return page
itemsJSON :: DB.Pipe -> Request -> Iteratee B.ByteString IO Response
itemsJSON dbpipe req = do
dbresult <- liftIO $ rundb dbpipe $ DB.find (DB.select [] $ tu "table") >>= DB.rest
let docs = either (const []) id dbresult
-- liftIO $ L.putStrLn $ encode $ show $ map docToMap docs
return $ responseLBS statusOK [("Content-Type", "text/plain; charset=utf-8")]
(encode $ map docToMap docs)
docToMap doc = Map.fromList $ map (\f -> (T.dropAround (== '"') $ T.pack $ show $ DB.label f, T.dropAround (== '"') $ T.pack $ show $ DB.value f)) doc
main = do
pipe <- DB.runIOE $ DB.connect $ DB.host "127.0.0.1"
run 3000 $ application pipe
rundb pipe act = DB.access pipe DB.master database act
tu :: B.ByteString -> UString
tu = DB.u . C8.unpack
Then the result is suprprising, DB.label works well, but DB.value giving me native characters as some escape codes, so the result is look like:
curl http://localhost:3000/items gives:
[{"Марка": "\1058\1080\1087 \1087\1086\1076",
"Model": "BD-W LG BP06LU10 Slim \1058\1080\1087 \1087\1086\1076\1082\1083\1102\1095\1077\1085\1080\1103"},
...
]
This happens in case i trying to print data and also in case i return data encoded as JSON
Any idea how correctly extract values from MongoDB driver ?
The following line confirms that aeson's encoding works properly (using the utf8-string library to read utf8 data off the lazy bytestring back to a haskell string:
> putStrLn $ Data.ByteString.Lazy.UTF8.toString $ encode $ ("\1058\1080\1087 \1087\1086\1076",12)
["Тип под",12]
Looking at your code more closely I see the real problem. You're calling T.pack $ show $ DB.value -- this will render out as literal codepoints, and then pack those into a text object. The fix is to switch from show to something smarter. Look at this (untested)
smartShow :: DB.Value -> Text
smartShow (String s) = Data.Text.Encoding.decodeUtf8 $ Data.CompactString.UTF8.toByteString s
smartShow x = T.pack $ show x
Obviously to handle the recursive cases, etc. you need to be smarter than that, but that's the general notion...
In fact, the "best" thing to do is to write a function of BSON -> JSON directly, rather than go through any intermediate structures at all.
Everything is working as expected -- only your expectations are wrong. =)
What you're seeing there are not raw Strings; they are String's which have been escaped to exist purely in the printable ASCII range by the show function, called by print:
print = putStrLn . show
Never fear: in memory, the string that prints as "\1058" is in fact a single Unicode codepoint long. You can observe this by printing the length of one of the Strings you're interested in and comparing that to the number of Unicode codepoints you expect.