I'm trying to parse XML that looks like this:
<h1>Collection A</h2>
<table>
<tr>Property 1</tr>
<tr>Property 2</tr>
</table>
<h2>Collection 2</h2>
<table>
<tr>Property 1</tr>
<tr>Property 88</tr>
</table>
I would like to parse that info as such:
MyClass "Collection 1" "Property 1"
MyClass "Collection 1" "Property 2"
MyClass "Collection 2" "Property 1"
MyClass "Collection 2" "Property 88"
I'm not sure how to go about doing this. My first thought was doing something like element "h1" $| followingSibling &// element "tr" &/ content, but that doesn't work, since it will capture all of the tr's, even the ones that don't "belong" to the table that I'm trying to read from, and I won't be able to know what properties belong to which collection.
How do I go about solving this?
You have to define your own XML Axis of a "immediate sibling" as followingSibling returns every node after the context. It's possible since Axis in Text.XML.Cursor are type synonym of Cursor -> [Cursor]:
immediateSibling = take 1 . (anyElement <=< followingSibling)
And combining information from different level is just nested list comprehension:
selected = root $/ selector
selector = element "h2" >=> toTuple
-- replace tuple with your constructor
toTuple c = [ (coll, prop)
| coll <- c $/ content
, prop <- c $| (immediateSibling >=> element "table" &/ element "tr" &/ content) ]
Related
I have problem using scalpel to capture block of tags.
Given following HTML snippet store in testS :: String
<body>
<h2>Apple</h2>
<p>I Like Apple</p>
<p>Do you like Apple?</p>
<h2>Banana</h2>
<p>I Like Banana</p>
<p>Do you like Banana?</p>
<h2>Carrot</h2>
<p>I Like Carrot</p>
<p>Do you like Carrot?</p>
</body>
I want to parse block of h2 and two p as a single record Block.
{-#LANGUAGE OverloadedStrings #-}
import Control.Monad
import Text.HTML.Scalpel
data Block = B String String String
deriving Show
block :: Scraper String Block
block = do
h <- text $ "h2"
pa <- text $ "p"
pb <- text $ "p"
return $ B h pa pb
blocks :: Scraper String [Block]
blocks = chroot "body" $ replicateM 3 block
But the result of scraping is not what I want, look like it keep repeat capturing the first block and never consume it.
λ> traverse (mapM_ print) $ scrapeStringLike testS blocks
B "Apple" "I Like Apple" "I Like Apple"
B "Apple" "I Like Apple" "I Like Apple"
B "Apple" "I Like Apple" "I Like Apple"
Expected output:
B "Apple" "I Like Apple" "Do you like Apple?"
B "Banana" "I Like Banana" "Do you like Banana?"
B "Carrot" "I Like Carrot" "Do you like Carrot?"
How to make it work?
First, I apologize for proposing a solution without testing or knowing anything about scalpel (such arrogance). Let me make it up to you; here's my totally rewritten attempt.
First, this monstrosity works.
blocks :: Scraper String [Block]
blocks = chroot "body" $ do
hs <- texts "h2"
ps <- texts "p"
return $ combine hs ps
where
combine (h:hs) (p:p':ps) = B h p p' : combine hs ps
combine _ _ = []
I call it a monstrosity because it erases the structure of the document with the two texts calls and then recreates it in the assumed order via combine. This probably isn't such a big deal in practice though, since most pages will be structured by combining tags via <div>.
So, if we were to have a different page:
testS' :: String
testS'= unlines [ "<body>",
"<div>",
" <h2>Apple</h2>",
" <p>I Like Apple</p>",
" <p>Do you like Apple?</p>",
"</div>",
"",
"<div>",
" <h2>Banana</h2>",
" <p>I Like Banana</p>",
" <p>Do you like Banana?</p>",
"",
"</div>",
"<div>",
" <h2>Carrot</h2>",
" <p>I Like Carrot</p>",
" <p>Do you like Carrot?</p>",
"</div>",
"</body>"
]
Then we can parse via:
block' :: Scraper String Block
block' = do
h <- text $ "h2"
[pa,pb] <- texts $ "p"
return $ B h pa pb
blocks' :: Scraper String [Block]
blocks' = chroots ("body" // "div") $ block'
Yielding,
B "Apple" "I Like Apple" "Do you like Apple?"
B "Banana" "I Like Banana" "Do you like Banana?"
B "Carrot" "I Like Carrot" "Do you like Carrot?"
Edit: re >>= and combine
My combine, above, is a local where definition. What you see there is what you get. Its unrelated to the function used in >>=, which incidentally is also a locally defined function with a slightly different name—combined. Even if they had the same name, however, it wouldn’t matter since each is only in scope within their respective functions.
As for the >>=, and just going by the observed behavior, each scrape starts from the beginning of the currently selected tags. So in your block definition, chroot “body” returns all tags in the body, text “h2” matches the first <h2>, and the next two text “p” both match the first <p>. So the bind is acting like an “and”: given the scalpel context of a bunch of tags match an <h2> and a <p> and (redundantly) a <p>. Notice that in my <div> based parse i could use texts (note the “s”) to get the two <p> i was expecting.
Finally, this behavior clicked for me when i saw it was based on tag soup. (Simultaneously with why they named it tag soup). Each of these scrapes are like dipping a spoon into an unordered soup of tags. The selector makes the soup, the scraper is your spoon. Hope that helps.
This is now supported in version 0.6.0 of scalpel through the use of SerialScrapers. SerialScrapers allow you to focus on one child of the current root at a time and expose APIs to move the focus and execute Scrapers on the currently focused node.
Adapting the example code in the documentation to your HTML gives:
-- Copyright 2019 Google LLC.
-- SPDX-License-Identifier: Apache-2.0
-- Chroot to the body tag and start a SerialScraper context with inSerial.
-- This will allow for focusing each child of body.
--
-- Many applies the subsequent logic repeatedly until it no longer matches
-- and returns the results as a list.
chroot "body" $ inSerial $ many $ do
-- Move the focus forward until text can be extracted from an h2 tag.
title <- seekNext $ text "h2"
-- Create a new SerialScraper context that contains just the tags between
-- the current focus and the next h2 tag. Then until the end of this new
-- context, move the focus forward to the next p tag and extract its text.
ps <- untilNext (matches "h2") (many $ seekNext $ text "p")
return (title, ps)
Which would return:
[
("Apple", ["I like Apple", "Do you like Apple?"]),
("Banana", ["I like Banana", "Do you like Banana?"]),
("Carrot", ["I like Carrot", "Do you like Carrot?"])
]
I would like to extract the text contents from the below Html page. All the paragraphs from the <div>.
I use the xml-conduit package for html parsing and came up with the following code:
getWebPageContents :: Url -> IO [T.Text]
getWebPageContents u = do
cursor <- cursorFor u
return $ cursor $// filter &/ content
filter = element "div" >=> attributeIs "id" "article-body-blocks" &// element "p"
This will return most of the text but not the ones from the links("front page of today's Daily Mirror")
Could anyone help?
You need to filter to all the descendants of the p tags, not just the children. You probably just need to replace &/ content with &// content.
I have been following the yesod tutorial and I am stuck on how to build a unit test involving parameters in a view that also hit a database. Backtracking a little, I followed the Echo.hs example:
getEchoR :: Text -> Handler Html
getEchoR theText = do
defaultLayout $ do
$(widgetFile "echo")
The corresponding test, note I have to cast the parameter into Text using Data.Text.pack
yit "Echo some text" $ do
get $ EchoR $ pack "Hello"
statusIs 200
Now I have the model defined like so:
Tag
name Text
type Text
With a handler that can render that that obviously take a TagId as the parameter
getTagR :: TagId -> Handler Html
getTagR tagId = do
tag <- runDB $ get404 tagId
defaultLayout $ do
setTitle $ toHtml $ tagName tag
$(widgetFile "tag")
This is where the test fails.
yit "Get a tag" $ do
-- tagId is undefined
get $ TagR tagId
statusIs 200
I am not sure how to define the tagId. It wouldn't work with a String or Text or Num, and I can't seem to figure out how to generate one as I can't find any example code in various Data.Persist tutorials. Or better yet, some other way to call the get method.
You want to use the Key data constructor to construct an ID value, which takes a PersistValue as a parameter. A simple example of creating one is:
Key $ PersistInt64 5
Another option is to call get with a textual URL, e.g. get ("/tag/5" :: Text).
Since times have changed, I'll leave this note here to say that these days one would use something like:
fromBackendKey 5
See the docs for fromBackendKey.
In Yesod, I've got a form that populates the type
data Field = Field Text Text text
deriving Show
When I write the hamlet html to display it, I'm running into the problem that Field is wrapped up in Maybe Maybe Field. So in hamlet I'm trying to do the following as shown here
(Snippet in postHomeR function)
let fieldData = case result of
FormSuccess res -> Just res
_ -> Nothing
(In the hamlet file)
<ul>
$maybe (Field one two three) <- fieldData
<li>#{show one}
However, when compiling there is Not in scope: one error. Why is the variable one not created/populated correctly?
You need to indent the <li> so that it's inside the $maybe block. As it sits now, it's a sibling to the $maybe, and thus the variables bound by $maybe are not in scope.
I've "inherited" some Tcl code, and while I worked through some tutorials and can make sense out of the language, my own Tcl constructs lack a certain... finesse.
For example, I have this code:
puts "Column 'name': [ $queryRs getString name ]"
$queryRs is the result set of a SQL query. The [ $queryRs getString name ] construct retrieves the content of the table column "name" from the current row in the result set. If the database field is NULL, the puts does not print anything.
I would like to print a "default" string instead, i.e. if [ $queryRs getString name ] results in nothing, I'd like to replace it with "--".
Now, I could do something like this:
set nameVar "[ $queryRs getString name ]"
if { [ string length $nameVar ] == 0 } {
set nameVar "--"
}
puts "Column 'name': $nameVar"
But there has to be a more compact solution, something that can be done inline instead of adding four lines and a temporary variable. Help, please?
Two things.
First, Tcl doesn't have a notion of NULL (nil, undefined or whatever) value, and when you want to simulate such a value you have to either use a variable and test for its existence or use an entry in an array of dictionary and test for its existence, too. If such a variable/entry exists, then the value is defined, otherwise it's not.
Unfortunately, the creator of your inherited code apparently did not care about the case where a variable can be NULL so NULLs are indistinguishable from variables having default values (an empty string).
Next, you can use a helper procedure to do what you need:
proc ValueOrDef {val {def --}} {
expr {$val ne "" ? $val : $def}
}
and then go like this:
puts [ValueOrDef [$queryRs getString name]]
puts [ValueOrDef [$queryRs getString name] "some other default"]
You could use the x?y:z construct of the expr command. x is the condition, y is the alternative if the condition is met and z is the alternative if x is not met.
E.g. (but still with a temporary variable):
set nameVar [ $queryRs getString name ]
puts "Column 'name': [expr {[string length $nameVar]>0 ? $nameVar : "--"}]"