I'm trying to find a way to check if a webpage exists in Haskell. The server is HTTP2 / HTTPS only and I'm trying to check if the page exists in a servant application.
Is there any Haskell packages with good documentation to just check if the status code is 200 or 404? And working with strong HTTPS and HTTP2 servers?
Here what I currently have with http-conduit but I'm receiving weird exceptions (TlsExceptionHostPort (HandshakeFailed (Error_Protocol ("expecting server hello, got alert : [(AlertLevel_Fatal,HandshakeFailure)]",True,HandshakeFailure))) "thibaud.dauce.fr" 443 and StatusCodeException).
... other imports
import qualified Network.HTTP.Conduit as HTTP
... other types
type AppM = ReaderT Config (EitherT ServantErr IO)
newComment :: String -> OneComment -> AppM Int64
newComment baseUrl oneComment = do
time <- liftIO getCurrentTime
response <- HTTP.withManager $ \manager -> do
request <- HTTP.parseUrl $ url oneComment
HTTP.httpLbs request manager
case (statusIsSuccessful $ HTTP.responseStatus response, startswith baseUrl (url oneComment)) of
(_, False) -> return 0
(True, True) -> do
theNewComment <- runDb $ insert $ Comment (url oneComment) (content oneComment) time
return $ fromSqlKey theNewComment
_ -> return 0
Some examples using wreq
{-# LANGUAGE OverloadedStrings #-}
import Network.Wreq
import Control.Lens
import Control.Exception as E
import Network.HTTP.Client (HttpException)
test1 = do
r <- get "https://httpbin.org/get"
print $ r ^. responseStatus . statusCode
-- throws an exception
test2 = do
r <- get "https://www.google123123.com"
print $ r ^. responseStatus . statusCode
testUrl url = do
r <- get url
return $ r ^. responseStatus . statusCode
-- catching the exception
test3 = do
st <- testUrl "https://www.google123123123.com" `E.catch` handler
print st
where
handler :: HttpException -> IO Int
handler _ = return 999
Related
I am writing an API server using Servant. The server includes persistent state. I would like to use QuickCheck to write tests for the server.
The implementation of various endpoints that make up the Servant Application require a database value. Unsurprisingly, creation of the database value is in the IO monad.
I don't understand how to combine the pieces from Hspec, Wai, QuickCheck, and Servant in a way that satisfies them all.
I see that I can perform an IO as part of creating the Hspec Spec itself and I see that I can specify that an IO be performed before each item in the Hspec Spec. Neither of these capabilities seems helpful in this case. The IO needs to be performed for each QuickCheck iteration of the property. Without this, the database accumulates state from each iteration which invalidates the definition of the property (or at least makes it greatly more complicated).
Below is my attempt to create a minimal, self-contained example of this scenario.
{-# LANGUAGE DataKinds #-}
{-# LANGUAGE TypeOperators #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE FlexibleContexts #-}
module Main where
import Data.IORef
import Test.QuickCheck
import Test.QuickCheck.Monadic
import qualified Test.Hspec.Wai.QuickCheck as QuickWai
import Test.Hspec
import Test.Hspec.Wai
import Text.Printf
import Servant
import Servant.API
import Data.Aeson
import Data.Text.Encoding
import Data.ByteString.UTF8
( fromString
)
data Backend = Backend (IORef Integer)
openBackend :: Integer -> IO Backend
openBackend n = Backend <$> newIORef n
data Acknowledgement = Ok Integer
instance ToJSON Acknowledgement where
toJSON (Ok n) = object [ "value" .= n ]
serveSomeNumber :: Backend -> Integer -> IO Acknowledgement
serveSomeNumber (Backend a) b = do
a' <- readIORef a
modifyIORef a (\n -> n + 1)
return $ Ok (a' + b)
type TheAPI = Capture "SomeNumber" Integer :> Post '[JSON] Acknowledgement
theServer :: Backend -> Server TheAPI
theServer backend = liftIO . serveSomeNumber backend
theAPI :: Proxy TheAPI
theAPI = Proxy
app :: Backend -> Application
app backend = serve theAPI (theServer backend)
post' n =
let
url = printf "/%d" (n :: Integer)
encoded = fromString url
in
post encoded ""
spec_g :: Backend -> Spec
spec_g (Backend expectedResult) =
describe "foo" $
it "bar" $ property $ \genN -> monadicIO $ do
n <- run genN
m <- run $ readIORef expectedResult
post' n `shouldRespondWith` ResponseMatcher { matchStatus = fromInteger (n + m) }
main :: IO ()
main = do
spec_g' <- spec_g `fmap` openBackend 16
hspec spec_g'
This doesn't type check:
/home/exarkun/Scratch/QuickCheckIOApplication/test/Spec.hs:119:3: error:
* Couldn't match type `WaiSession' with `PropertyM IO'
Expected type: PropertyM IO ()
Actual type: WaiExpectation
* In a stmt of a 'do' block:
post' n
`shouldRespondWith`
ResponseMatcher {matchStatus = fromInteger (n + m)}
In the second argument of `($)', namely
`do n <- run genN
m <- run $ readIORef expectedResult
post' n
`shouldRespondWith`
ResponseMatcher {matchStatus = fromInteger (n + m)}'
In the expression:
monadicIO
$ do n <- run genN
m <- run $ readIORef expectedResult
post' n
`shouldRespondWith`
ResponseMatcher {matchStatus = fromInteger (n + m)}
|
119 | post' n `shouldRespondWith` ResponseMatcher { matchStatus = fromInteger (n + m) }
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
I don't know if there is a way to fit a WaiExpectation into a PropertyM IO () at all. I don't even know if monadicIO is helpful here at all.
How can I fit these pieces together?
Define spec_g :: Background -> Spec, then take advantage of IO's Functor and Monad instances.
main = do
spec <- fmap spec_g (openBackend 16) -- fmap spec_g :: IO Background -> IO Spec
hspec spec
or more concisely,
main = spec_g <$> openBackend 16 >>= hspec
IIRC, you're supposed to run each spec or property with the with function. Here's a few properties I wrote some time ago:
with app $ describe "/reservations/" $ do
it "responds with 404 when no reservation exists" $ WQC.property $ \rid ->
get ("/reservations/" <> toASCIIBytes rid) `shouldRespondWith` 404
it "responds with 200 after reservation is added" $ WQC.property $ \
(ValidReservation r) -> do
_ <- postJSON "/reservations" $ encode r
let actual = get $ "/reservations/" <> toASCIIBytes (reservationId r)
actual `shouldRespondWith` 200
The app value serves the service, and as far as I recall, it runs the IO action for each test. I did it with an in-memory database using an IORef, and that seems to be working just fine:
app :: IO Application
app = do
ref <- newIORef Map.empty
return $
serve api $
hoistServer api (Handler . runInFakeDBAndIn2019 ref) $
server 150 []
The WQC.property function is from a qualified import:
import qualified Test.Hspec.Wai.QuickCheck as WQC
I wasn't too happy, however, with the way I had to structure my tests and properties with HSpec, so I ultimately rewrote all the tests to be driven by HUnit. I've an upcoming blog post that describes this, but I haven't published it yet.
I am trying to make client bindings for the News api provided by newsapi.org using the Haskell servant library. I have created the following endpoint:
type NewsAPI = "top-headlines" :> QueryParam "country" String :> QueryParam "apiKey" String :> Get '[JSON] TopHeadlines
And attempted to call it as follows:
topheadlines :: Maybe String -> Maybe String -> ClientM TopHeadlines
api :: Proxy NewsAPI
api = Proxy
topheadlines = client api
query = topheadlines (Just "us") (Just "<api key>")
run3 :: IO ()
run3 = do
manager' <- newManager defaultManagerSettings
users <- runClientM query (mkClientEnv manager' (BaseUrl Https "newsapi.org/v2" 443 ""))
print users
I keep getting a connection error that I don't entirely understand how to reason about:
Left (ConnectionError "HttpExceptionRequest Request {\n host = \"newsapi.org/v2\"\n port = 443\n secure = True\n requestHeaders = [(\"Accept\",\"application/json;charset=utf-8,application/json\")]\n path = \"/top-headlines\"\n queryString = \"?country=us&api_key=90a38fab85c440fa88521e0789248f83\"\n method = \"GET\"\n proxy = Nothing\n rawBody = False\n redirectCount = 10\n responseTimeout = ResponseTimeoutDefault\n requestVersion = HTTP/1.1\n}\n TlsNotSupported")
Not sure why there is no connection. Another set of client bindings I have is working fine.
It is a combination of two things:
api_key should be apiKey
Rely on parseBaseUrl
burl <- parseBaseUrl "http://newsapi.org/v2"
This worked for me in the sample project I set up.
{-# LANGUAGE DataKinds #-}
{-# LANGUAGE TypeOperators #-}
module Main where
import Control.Monad.Free
import Servant.Client.Free
import qualified Network.HTTP.Client as HTTP
import qualified Servant.Client.Internal.HttpClient as I
import Network.Wai.Handler.Warp (run)
import Servant
import System.Environment (getArgs)
type NewsAPI = "top-headlines" :> QueryParam "country" String :> QueryParam "apiKey" String :> Get '[JSON] String
topheadlines :: Maybe String -> Maybe String -> Free ClientF String
topheadlines = client api
api :: Proxy NewsAPI
api = Proxy
main :: IO ()
main = do
test
test :: IO ()
test = case topheadlines (Just "us") (Just "API_KEY") of
Pure n ->
putStrLn $ "ERROR: got pure result: " ++ show n
Free (Throw err) ->
putStrLn $ "ERROR: got error right away: " ++ show err
Free (StreamingRequest _req _k) ->
putStrLn $ "ERROR: need to do streaming request"
Free (RunRequest req k) -> do
burl <- parseBaseUrl "http://newsapi.org/v2"
mgr <- HTTP.newManager HTTP.defaultManagerSettings
let req' = I.requestToClientRequest burl req
putStrLn $ "Making request: " ++ show req'
res' <- HTTP.httpLbs req' mgr
putStrLn $ "Got response: " ++ show res'
I am making some http calls using wreq and would like to catch any exception and return an Either type. I tried something like this but could not figure out how to manipulate the calls so it will type check.
-- exhaustive pattern match omitted here
safeGetUrl :: URL -> Maybe Login -> Maybe Password -> IO (Either String (Response LBS.ByteString))
safeGetUrl url (Just login) (Just pass) = do
let def = defaults
opts = def & auth ?~ basicAuth (BS.pack login) (BS.pack pass)
r <- getWith opts url `E.catch` handler
return $ Right r
where
handler :: HttpException -> Either String (Response LBS.ByteString)
handler (StatusCodeException s _ _) = do
return $ Left $ LBS.unpack (s ^. statusMessage)
I am pasting the type error below but I know the above code will not compile. The issue is r <- getWith opts url E.catch handler. The first part returns IO (Res... but the exception handler returns Either... I tried adding lifting the getWith.. into Either but that did not type check either.
Couldn't match type ‘Either String (Response LBS.ByteString)’
with ‘IO (Response LBS.ByteString)’
Expected type: HttpException -> IO (Response LBS.ByteString)
Actual type: HttpException
-> Either String (Response LBS.ByteString)
In the second argument of ‘catch’, namely ‘handler’
In a stmt of a 'do' block: r <- getWith opts url `catch` handler
Is there a way to catch this exception and return an IO Either type?
Since #jozefg answer, the API has changed a little bit and the answer doesn't compile anymore.
Here is an updated version that compiles:
import qualified Control.Exception as E
import Control.Lens
import qualified Data.ByteString.Char8 as BSC
import qualified Data.ByteString.Lazy as LBS
import Network.HTTP.Client
import Network.Wreq as NW
type URL = String
type Login = String
type Password = String
safeGetUrl ::
URL
-> Maybe Login
-> Maybe Password
-> IO (Either String (Response LBS.ByteString))
safeGetUrl url (Just login) (Just pass) = do
let def = defaults
opts = def & auth ?~ basicAuth (BSC.pack login) (BSC.pack pass)
(Right <$> getWith opts url) `E.catch` handler
where
handler :: HttpException -> IO (Either String (Response LBS.ByteString))
handler (HttpExceptionRequest _ (StatusCodeException r _)) =
return $ Left $ BSC.unpack (r ^. NW.responseStatus . statusMessage)
Your issue is that one side of the handle returns an unwrapped response (no Either) and the other side returns an Either-wrapped exception. You then attempt to wrap the response in an Either, which you do need to do, but it's just at the wrong place. You can fix this merely by switching where you do the wrapping
safeGetUrl :: URL -> Maybe Login -> Maybe Password -> IO (Either String (Response LBS.ByteString))
safeGetUrl url (Just login) (Just pass) = do
let def = defaults
opts = def & auth ?~ basicAuth (BS.pack login) (BS.pack pass)
(Right <$> getWith opts url) `E.catch` handler
where
handler :: HttpException -> IO (Either String (Response LBS.ByteString))
handler (StatusCodeException s _ _) = do
return $ Left $ LBS.unpack (s ^. statusMessage)
However there are some other problems with your functions, remember that unpack gives back Word8s not Char. You may want to import Data.ByteString.Char as the version of unpack defined in there should work better than LBS.unpack. Without your imports though I cannot confirm this definitively. The final (working) code for me is
import Control.Lens
import Network.Wreq
import Network.HTTP.Client
import qualified Control.Exception as E
import qualified Data.ByteString.Char8 as BSC
import qualified Data.ByteString.Lazy as LBS
type URL = String
type Login = String
type Password = String
safeGetUrl :: URL
-> Maybe Login
-> Maybe Password
-> IO (Either String (Response LBS.ByteString))
safeGetUrl url (Just login) (Just pass) = do
let def = defaults
opts = def & auth ?~ basicAuth (BSC.pack login) (BSC.pack pass)
(Right <$> getWith opts url) `E.catch` handler
where
handler :: HttpException -> IO (Either String (Response LBS.ByteString))
handler (StatusCodeException s _ _) = do
return $ Left $ BSC.unpack (s ^. statusMessage)
I am new to Haskell and am trying to a sample code of http-conduit version 2.0.0.4, but it dose not work
Here is the sample code
{-# LANGUAGE OverloadedStrings #-}
import Network.HTTP.Conduit
import Network
import Data.Time.Clock
import Data.Time.Calendar
import qualified Control.Exception as E
past :: UTCTime
past = UTCTime (ModifiedJulianDay 56200) (secondsToDiffTime 0)
future :: UTCTime
future = UTCTime (ModifiedJulianDay 562000) (secondsToDiffTime 0)
cookie :: Cookie
cookie = Cookie { cookie_name = "password_hash"
, cookie_value = "abf472c35f8297fbcabf2911230001234fd2"
, cookie_expiry_time = future
, cookie_domain = "example.com"
, cookie_path = "/"
, cookie_creation_time = past
, cookie_last_access_time = past
, cookie_persistent = False
, cookie_host_only = False
, cookie_secure_only = False
, cookie_http_only = False
}
main = withSocketsDo $ do
request' <- parseUrl "http://example.com/secret-page"
let request = request' { cookieJar = Just $ createCookieJar [cookie] }
E.catch (withManager $ httpLbs request)
(\(StatusCodeException statusCode _ _) ->
if statusCode==403 then putStrLn "login failed" else return ())
ref: http://hackage.haskell.org/package/http-conduit-2.0.0.4/docs/Network-HTTP-Conduit.html
the error message when I load it
samplecode.hs:33:39:
Couldn't match type `()'
with `Response Data.ByteString.Lazy.Internal.ByteString'
Expected type: IO
(Response Data.ByteString.Lazy.Internal.ByteString)
Actual type: IO ()
In the return type of a call of `putStrLn'
In the expression: putStrLn "login failed"
In the expression:
if statusCode == 403 then putStrLn "login failed" else return ()
samplecode.hs:33:75:
Couldn't match expected type `Response
Data.ByteString.Lazy.Internal.ByteString'
with actual type `()'
In the first argument of `return', namely `()'
In the expression: return ()
In the expression:
if statusCode == 403 then putStrLn "login failed" else return ()
Failed, modules loaded: none.
How can I fix it?
Many Thanks
Update
Following Abrahamson's advice, I have changed my code little bit to the following and now have proper StatusCodeException handling.
main = withSocketsDo $ do
request' <- parseUrl "http://example.com/secret-page"
let request = request' { cookieJar = Just $ createCookieJar [cookie] }
eitherResp <- E.try (withManager $ httpLbs request)
case eitherResp of
Left (StatusCodeException s _ _)
| statusCode s == 403 -> putStrLn "login failed"
| otherwise -> return ()
Right resp -> print (L.length (responseBody resp))
You're not using E.catch as it was intended. If you take a look at the type:
E.catch :: Exception e => IO a -> (e -> IO a) -> IO a
it's clear that the return type of the first and second arguments must match. In your case you have
withManager $ httpLbs request :: IO (Response ByteString)
in the first branch and either
putStrLn "login failed" -- or
return ()
in the second. These types do not match and thus you're getting the error you see.
In higher level terms, the problem is that you're not handling the success case. For instance, we could rewrite this using E.try to make that more clear
eitherResp <- E.try (withManager $ httpLbs request)
case eitherResp of
Left (StatusCodeException statusCode _ _)
| statusCode == 403 -> putStrLn "login failed"
| otherwise -> return ()
Right resp -> print (ByteString.length (responseBody resp))
Here since I explicitly pattern match on the Either StatusCodeException (Response ByteString) it's clear that I needed to provide both the failing and the succeeding branche and give them the same return types. To do so I introduced an action to perform on the successful case.
Generally, I find E.try easier to use. E.catch is primarily useful when you want to provide a default under failure.
I'm trying to download all png files contained in an html file.
I have trouble catching 404 status exceptions though, instead my program just crashes.
Here is some sample to demonstrate:
import Network.HTTP.Conduit
import qualified Data.ByteString.Lazy as L
main = do
let badUrl = "http://www.google.com/intl/en_com/images/srpr/WRONG.png"
imgData <- (simpleHttp badUrl) `catch` statusExceptionHandler
L.writeFile "my.png" imgData
statusExceptionHandler :: t -> IO L.ByteString
statusExceptionHandler e = (putStrLn "oops") >> (return L.empty)
My "oops" message never prints, instead app crashes with:
StatusCodeException (Status {statusCode = 404, statusMessage = "Not Found"}) [("Content-Type","text/html; charset=UTF-8"),("X-Content-Type-Options","nosniff"),("Date","Fri, 27 Jan 2012 03:10:34 GMT"),("Server","sffe"),("Content-Length","964"),("X-XSS-Protection","1; mode=block")]
What am I doing wrong?
Update:
Following Thoma's advice, I changed my code to the following snippet and now have proper exception handling in place.
main = do
let badUrl = "http://www.google.com/intl/en_com/images/srpr/WRONG.png"
imgData <- (simpleHttp badUrl) `X.catch` statusExceptionHandler
case imgData of x | x == L.empty -> return ()
| otherwise -> L.writeFile "my.png" imgData
statusExceptionHandler :: HttpException -> IO L.ByteString
statusExceptionHandler (StatusCodeException status headers) =
putStr "An error occured during download: "
>> (putStrLn $ show status)
>> (return L.empty)
In addition to Thomas's answer, you could tell http-conduit not to throw an exception by overriding the checkStatus record of your Request type.
You should probably read the Marlow paper on extensible exceptions. The original catch, exported by Prelude and used in your code snipt, only works for IOError's. The http-conduit code is throwing exceptions of a different type, HttpException to be exact. (there is some dynamic typing going on via the Typeable class, see the paper).
The solution? Use catch from Control.Exception and only catch the error types you want to handle (or SomeException for all of them).
import Network.HTTP.Conduit
import qualified Data.ByteString.Lazy as L
import Control.Exception as X
main = do
let badUrl = "http://www.google.com/intl/en_com/images/srpr/WRONG.png"
imgData <- (simpleHttp badUrl) `X.catch` statusExceptionHandler
L.writeFile "my.png" imgData
statusExceptionHandler :: SomeException -> IO L.ByteString
statusExceptionHandler e = (putStrLn "oops") >> (return L.empty)