convert dynamic key fields in JSON - haskell

here is my JSON structure, there are N records that has a name as ID to represent a children
{"Kids":
{"Jack":{"age":10}
,"Jane":{"age":9}
, .......
}
}
in the data type in Haskell
data Kid = Kid { name::String, age::Int}
instance FromJSON Kid where
parseJSON (Object v) =
....
question is ,how to make the key ( name ) as part of the constructor ? the expected output signature is like:
decode "input json string" -> [Kid]
when the expect decode function was called, it will return a list of type Kid. Thanks for reading this & appreciate any help .

By using the withObject function, you get access to an Object which is actually a KeyMap which you can manipulate much like the usual Map from e.g. containers. If you're on an older aeson version, Object will instead be a HashMap, so you can use that as well.
EDIT: I remember that Map itself also has a FromJSON, so you can probably use that instead for a shorter "solution":
{-# LANGUAGE OverloadedStrings #-}
import Data.Aeson (FromJSON(..), withObject, (.:), fromJSON)
import Data.Map (Map)
import qualified Data.Map as Map
data Kid = MkKid {name :: String, age :: Int}
newtype Kids = MkKids {unKids :: [Kid]}
instance FromJSON Kids where
parseJSON = withObject "Kids" $ \o -> do
kvmap <- o .: "Kids"
pure $ MkKids $ map (uncurry MkKid) $ Map.toList kvmap
Old "solution", which manipulates they KeyMap
{-# LANGUAGE NamedFieldPuns #-}
{-# LANGUAGE OverloadedStrings #-}
import Data.Aeson.Key (Key)
import qualified Data.Aeson.Key as Key
import qualified Data.Aeson.KeyMap as KeyMap
import Data.Aeson.Types (Parser, Object, Value)
import Data.Aeson (FromJSON(..), withObject, (.:), fromJSON)
data Kid = MkKid {name :: String, age :: Int}
newtype Kids = MkKids {unKids :: [Kid]}
instance FromJSON Kids where
parseJSON = withObject "Kids" $ \o -> do
inner <- o .: "Kids"
withObject "inner" parseKids inner
where
parseKids :: Object -> Parser Kids
parseKids obj =
fmap MkKids $ traverse toKid $ KeyMap.toList obj
toKid :: (Key, Value) -> Parser Kid
toKid (k, v) = do
age <- parseJSON v
let name = Key.toString k
pure $ MkKid {name, age}

Related

Better ways to collect all unused field of an Object in aeson's Parser?

Suppose I want to implement FromJSON for a data type. Below are the complete source code:
{-# LANGUAGE
NamedFieldPuns
, OverloadedStrings
, TupleSections
, ViewPatterns
#-}
module Main
( main
) where
import Data.Aeson
import Control.Monad
import qualified Data.HashMap.Strict as HM
import qualified Data.Map.Strict as M
import qualified Data.Text as T
data Foo
= Foo
{ aaa :: Int
, bbb :: T.Text
, ccc :: Maybe (Int, Int)
, extra :: M.Map T.Text T.Text
}
instance FromJSON Foo where
parseJSON = withObject "Foo" $ \obj -> do
aaa <- obj .: "aaa"
bbb <- obj .: "bbb"
ccc <- obj .:? "ccc"
let existingFields = T.words "aaa bbb ccc"
obj' =
-- for sake of simplicity, I'm not using the most efficient approach.
filter ((`notElem` existingFields) . fst)
. HM.toList
$ obj
(M.fromList -> extra) <- forM obj' $ \(k,v) ->
withText "ExtraText" (pure . (k,)) v
pure Foo {aaa,bbb,ccc,extra}
main :: IO ()
main = pure ()
This data type Foo has a bunch of fields of potentially different types and in the end there is extra to collect all remaining fields.
Obviously no one would enjoy updating existingFields every time some fields get add/remove/update-ed, any recommended approach on collecting unused fields?
An alternative that I can think of is to stack a StateT on top with obj (converted to Map) as the initial state, and use something like Data.Map.splitLookup to "discharge" used fields. But I'm reluctant to do so as it will involve some lifting around monad stacks and it doesn't sound very good performance-wise removing elements one at a time from Map in comparison to filtering through HashMap in one pass in the end.
no one would enjoy updating existingFields every time some fields get
add/remove/update-ed
Consider this function
import Data.Aeson.Types (Parser)
import Data.Text (Text)
import Control.Monad.Trans.Writer
import Data.Functor.Compose
keepName :: (Object -> Text -> Parser x)
-> Object -> Text -> Compose (Writer [Text]) Parser x
keepName f obj fieldName = Compose $ do
tell [fieldName]
pure (f obj fieldName)
It takes as input an operator like .: or .:? and "enriches" its result value so that, instead of returning a Parser, it returns a Parser nested inside a Writer that serves to accumulate the supplied field names. The composition is wrapped in the Compose newtype, which automatically gives us an Applicative instance because, as mentioned in the docs:
(Applicative f, Applicative g) => Applicative (Compose f g)
(The composition is not a Monad though. Also take note that we are using Writer and not WriterT. We are nesting Applicatives, not applying monad transformers).
The rest of the code doesn't change that much:
{-# LANGUAGE ApplicativeDo #-}
instance FromJSON Foo where
parseJSON = withObject "Foo" $ \obj -> do
let Compose (runWriter -> (parser,existingFields)) =
do aaa <- keepName (.:) obj "aaa"
bbb <- keepName (.:) obj "bbb"
ccc <- keepName (.:?) obj "ccc"
pure Foo {aaa,bbb,ccc,extra = mempty}
obj' =
filter ((`notElem` existingFields) . fst)
. HM.toList
$ obj
(M.fromList -> extra) <- forM obj' $ \(k,v) ->
withText "ExtraText" (pure . (k,)) v
r <- parser
pure $ r { extra }

Aeson: parsing dynamic keys as type field

Let's say there is a JSON like:
{
"bob_id" : {
"name": "bob",
"age" : 20
},
"jack_id" : {
"name": "jack",
"age" : 25
}
}
Is it possible to parse it to [Person] with Person defined like below?
data Person = Person {
id :: Text
,name :: Text
,age :: Int
}
You cannot define an instance for [Person] literally, because aeson already includes an instance for [a], however you can create a newtype, and provide an instance for that.
Aeson also includes the instance FromJSON a => FromJSON (Map Text a), which means if aeson knows how to parse something, it knows how to parse a dict of that something.
You can define a temporary datatype resembling a value in the dict, then use the Map instance to define FromJSON PersonList, where newtype PersonList = PersonList [Person]:
data PersonInfo = PersonInfo { infoName :: Text, infoAge :: Int }
instance FromJSON PersonInfo where
parseJSON (Object v) = PersonInfo <$> v .: "name" <*> v .: "age"
parseJSON _ = mzero
data Person = Person { id :: Text, name :: Text, age :: Int }
newtype PersonList = PersonList [Person]
instance FromJSON PersonList where
parseJSON v = fmap (PersonList . map (\(id, PersonInfo name age) -> Person id name age) . M.toList) $ parseJSON v
If you enable FlexibleInstances, you can make instance for [Person]. You can parse your object to Map Text Value and then parse each element in map:
{-# LANGUAGE UnicodeSyntax, OverloadedStrings, FlexibleInstances #-}
module Person (
) where
import Data.Aeson
import Data.Aeson.Types
import Data.Text.Lazy
import Data.Text.Lazy.Encoding
import Data.Map (Map)
import qualified Data.Map as M
data Person = Person {
id ∷ Text,
name ∷ Text,
age ∷ Int }
deriving (Eq, Ord, Read, Show)
instance FromJSON [Person] where
parseJSON v = do
objs ← parseJSON v ∷ Parser (Map Text Value)
sequence [withObject "person"
(\v' → Person i <$> v' .: "name" <*> v' .: "age") obj |
(i, obj) ← M.toList objs]
test ∷ Text
test = "{\"bob_id\":{\"name\":\"bob\",\"age\":20},\"jack_id\":{\"name\":\"jack\",\"age\":25}}"
res ∷ Maybe [Person]
res = decode (encodeUtf8 test)
mniip's answer converts the JSON Object to a Map, which leads to a result list sorted by ID. If you don't need the results sorted in that fashion, it's probably better to use a more direct approach to speed things up. In particular, an Object is really just a HashMap Text Value, so we can use HashMap operations to work with it.
Note that I renamed the id field to ident, because most Haskell programmers will assume that id refers to the identity function in Prelude or to the more general identity arrow in Control.Category.
module Aes where
import Control.Applicative
import Data.Aeson
import Data.Text (Text)
import qualified Data.HashMap.Strict as HMS
data PersonInfo = PersonInfo { infoName :: Text, infoAge :: Int }
instance FromJSON PersonInfo where
-- Use mniip's definition here
data Person = Person { ident :: Text, name :: Text, age :: Int }
newtype PersonList = PersonList [Person]
instance FromJSON PersonList where
parseJSON (Object v) = PersonList <$> HMS.foldrWithKey go (pure []) v
where
go i x r = (\(PersonInfo nm ag) rest -> Person i nm ag : rest) <$>
parseJSON x <*> r
parseJSON _ = empty
Note that, like Alexander VoidEx Ruchkin's answer, this sequences the conversion from PersonInfo to Person explicitly within the Parser monad. It would therefore be easy to modify it to produce a parse error if the Person fails some sort of high-level validation. Alexander's answer also demonstrates the utility of the withObject combinator, which I'd have used if I'd known it existed.

Parsing a homogenous polymorphic json array

I have a data type where one of the fields is a list of one of n other data types (n is small and the types are known in advance). I would like to make a JSON parser but I can't quite figure it out. I've tried creating a Pet type class and making them both instances of it, but it seemed to be a dead end. Any help would be appreciated!
As a simplified example - I have a Person data type who can have a list of pets, either dogs or cats - but not a mix of both.
Here's the example:
{-# LANGUAGE OverloadedStrings #-}
import Control.Applicative
import Data.Aeson
import Data.ByteString.Lazy as L
import Data.Aeson.Types (Parser)
import Control.Monad (mzero)
data Person = Person {
name :: String,
species :: String,
pets :: [?] -- a list of dogs OR cats
} deriving Show
instance FromJSON (Person a) where
parseJSON (Object v) = ???
data Dog = Dog {
dogField :: String
} deriving Show
instance FromJSON Dog where
parseJSON (Object v) = Dog <$>
v .: "dogField"
data Cat = Cat {
catField :: String
} deriving Show
instance FromJSON Cat where
parseJSON (Object v) = Cat <$>
v .: "catField"
A standard way of representing either one type or another is to use the Either type, e.g.:
data Person { ..., pets :: Either [Dog] [Cat] }
Also, you might be interested in use GHC Generics to auto-derive the To/FromJSON instances.
An example with a data structure that uses Either:
{-# LANGUAGE DeriveGeneric #-}
import Data.Aeson
import GHC.Generics
data Person = Person {
name :: String,
species :: String,
pets :: Either [String] [Int]
} deriving (Show,Generic)
instance ToJSON Person -- instances are auto-derived
instance FromJSON Person
doit = do
let me = Person "user5402" "Human" (Right [42])
print $ encode me
If you have more than two alternatives you can easily create your own sum type like this:
-- assume the possible pet types are: Dog, Cat, Rodent, Reptile
data Pets = Dogs [Dog] | Cats [Cat] | Rodents [Rodent] | Reptiles [Reptile]
deriving (Show, Generic)
data Person { ..., pets :: Pets }
deriving (Show, Generic)
doit = do
let me = Person "user5402" "Human" (Rodents [agerbil, amouse])
print $ encode me
where agerbil and amouse are Rodent values.
I'm modifying #user5402's answer because I don't like the "tag" and "contents" fields that Generics added. Also accepting his answer since he gave me the key insight of how to structure the sum type
instance FromJSON Pets where
parseJSON (Object o) = (parsePets o "pets")
parseJSON _ = mzero
parsePets :: Object -> T.Text -> Parser Pets
parsePets o key = case H.lookup key o of
Nothing -> fail $ "key " ++ show key ++ " not present"
Just v -> parseToCatsOrDogs (o .: "species") v
{-# INLINE parsePets #-}
parseToCatsOrDogs :: Parser String -> Value -> Parser Pets
parseToCatsOrDogs speciesParser (Array v) = speciesParser >>= \species -> case species of
"dog" -> (V.mapM (\x -> parseJSON $ x) v) >>= \ dogVector -> return $ Dogs (V.toList dogVector)
"cat" -> (V.mapM (\x -> parseJSON $ x) v) >>= \ catVector -> return $ Cats (V.toList catVector)
_ -> mzero
parseToCatsOrDogs _ _ = mzero

Parsing data types with all nullary constructors using generic decode

I have the following code:
{-# LANGUAGE DeriveGeneric, OverloadedStrings #-}
import Data.Aeson
import GHC.Generics
data CharClass = Fighter | Rogue | Wizard deriving (Generic, Show)
instance FromJSON CharClass
instance ToJSON CharClass
I can encode values of this type:
*Main> encode Fighter
"\"Fighter\""
But round-tripping doesn't work:
*Main> eitherDecode $ encode Fighter
Left "Failed reading: satisfy"
*Main> :t eitherDecode $ encode Fighter
eitherDecode $ encode Fighter :: FromJSON a => Either String a
It looks a lot like this answered question, but adding the expected type doesn't work:
*Main> eitherDecode $ encode Fighter :: Either String CharClass
Left "Failed reading: satisfy"
Interestingly, it does work for fromJSON/toJSON:
*Main> fromJSON $ toJSON Fighter :: Result CharClass
Success Fighter
Making at least one of the constructors non-nullary also makes things work, like if I do this:
data CharClass = Fighter Int | Rogue | Wizard deriving (Generic, Show)
And then try to round-trip again:
*Main> decode $ encode (Fighter 0) :: Maybe CharClass
Just (Fighter 0)
I'm sure I'm missing something simple, but attempting to trace this through the generic code made my head spin.
JSON is fundamentally a collection of key-value pairs, where values can be a few primitive types or another collection of key-value pairs. Nullary types just don't fit in very well with the whole idea of being JSON entities by themselves. However, they work fine when placed within other types that mesh well with the JSON concept.
data F = F { a :: CharClass, b :: CharClass }
deriving (Generic, Show)
instance FromJSON F
instance ToJSON F
This looks more like the sort of key-value collection JSON was designed for.
*Main> let x = F Fighter Rogue
*Main> x
F {a = Fighter, b = Rogue}
*Main> decode $ encode x :: Maybe F
Just (F {a = Fighter, b = Rogue})
The version of aeson that stack installed on my machine was from the 0.8 series, and in aeson 0.8 or earlier, only objects and arrays were parsed at the root level.
In aeson 0.9, decode uses value parser. So nullable object (represented as string) at top-level will work.
For 0.8 the below example works. I don't know why decodeWith isn't exposed.
{-# LANGUAGE DeriveGeneric #-}
import Control.Applicative
import GHC.Generics
import Data.Aeson
import Data.Aeson.Parser
import Data.ByteString.Lazy as L
import Data.Attoparsec.ByteString.Char8 (endOfInput, skipSpace)
import qualified Data.Attoparsec.Lazy as L
data CharClass = Fighter | Rogue | Wizard deriving (Generic, Show)
instance ToJSON CharClass
instance FromJSON CharClass
decodeWith p to s =
case L.parse p s of
L.Done _ v -> case to v of
Success a -> Just a
_ -> Nothing
_ -> Nothing
{-# INLINE decodeWith #-}
valueEOF = value <* skipSpace <* endOfInput
decodeValue :: (FromJSON a) => L.ByteString -> Maybe a
decodeValue = decodeWith valueEOF fromJSON
main :: IO ()
main = print (decodeValue (encode Fighter) :: Maybe CharClass)

'Nothing' Result from Decode in Aeson

I'm using this code:
{-# LANGUAGE OverloadedStrings #-}
import Data.Aeson
import Data.Maybe
import Data.ByteString.Lazy
import Control.Applicative
import Debug.Trace
import Control.Monad
import qualified Data.Aeson.Types as T
main = do
res <- liftA show (liftA decodeOriginal (Data.ByteString.Lazy.readFile "./a.json"))
Prelude.putStrLn res
interpretResult :: Maybe String -> String
interpretResult Nothing = "Error."
interpretResult x = fromJust x
data TotalLine1 = TotalLine1 {
timestamp :: Integer,
value :: Integer
} deriving (Eq, Show)
data Original = Original {
totals :: [TotalLine1]
} deriving (Eq, Show)
instance FromJSON Original where
parseJSON (Object v) = traceStack "Original" (Original <$> (parseJSON =<< (v .: "visitors.total")))
parseJSON _ = mzero
instance FromJSON TotalLine1 where
parseJSON (Object v) = TotalLine1 <$>
v .: "timestamp" <*>
v .: "value"
decodeOriginal :: ByteString -> Maybe Original
decodeOriginal b = traceStack "decoding" (do
a <- decode b :: Maybe Original
return a)
to try and parse JSON like this:
{
visitors.total: [
{
timestamp: 1365548400,
value: 1
},
{
timestamp: 1365548700,
value: 2
},
{
timestamp: 1365549000,
value: 5
},
]
}
But main just returns Nothing every time. What have I done wrong? It seems that even parseJSON isn't being called for Original.
Your JSON file is not valid.
On the one hand, the names of the fields have to be quoted,
"timestamp"
etc. and on the other, you have a trailing comma in the list of TotalLine1s, which causes the decoding of the ByteString to a Value to fail. Quote the field names and remove the trailing comma, and it works.

Resources