I have following code working with proxy for a GET Request:
import Control.Applicative ((<$>))
import Data.Maybe (fromJust)
import Network.Browser
import Network.HTTP
import Network.HTTP.Proxy (parseProxy)
main = do
rsp <- browse $ do
setProxy . fromJust $ parseProxy "127.0.0.1:8118"
request $ getRequest "http://www.google.com"
print $ rspBody <$> rsp
And this one for Post, but without proxy:
main = do
r <- post "http://www.geocodeip.com" ["IP" := Data.ByteString.Lazy.Char8.pack "79.212.82.103"]
html <- r ^. responseBody
print html
But how to make a post request with proxy? I dont get it. please help me!
It's pretty simple if you keep track of what you're doing.
We need to use request but feed it a POST request rather than a GET request. To make these we use postRequestWithBody which Hackage tells us has the parameters
postRequestWithBody :: String | URL to POST to
-> String | Content-Type of body
-> String | The body of the request
-> Request_String | The constructed request
So replace request $ getRequest "http://www.google.com" with:
request $ postRequestWithBody "http://www.geocodeip.com/" "application/x-www-form-urlencoded" "IP=79.212.82.103"
...and you'll be good.
Related
Network.HTTP.Proxy has a nice function called fetchProxy:
fetchProxy :: Bool -> IO Proxy
fetchProxy flg gets the local proxy settings and parse the string into
a Proxy value. If you want to be informed of ill-formed proxy
configuration strings, supply True for flg. Proxy settings are sourced
from the HTTP_PROXY environment variable [...]
I want to use the Proxy obtained this way with Wreq library, which has it's own Proxy defined like this, by importing it from HTTP:
import Network.HTTP.Client.Internal (Proxy(..), Response)
There appears to be a type mismatch between Network.HTTP.Proxy.Proxy and Network.Wreq.Proxy, where I presume they must be identical.
I import both like this:
import Network.Wreq
import Network.HTTP.Proxy (fetchProxy)
How can I use HTTP.Proxy.Proxy with Wreq and why does GHC see them as different types?
It's likely that the Wreq authors were just unaware of the other Proxy as they seem to be storing equivalent information. It'll be tricky to get them to talk to each other, however, since fetchProxy stores the host:port as a string and Wreq's Proxy wants the individual host and port. You'll have to do some URI parsing:
import Control.Lens
import Data.Text.Strict.Lens
import Network.HTTP.Proxy
import Network.Wreq
import URI.ByteString
main :: IO ()
main = do
Network.HTTP.Proxy.Proxy host _ <- fetchProxy True
case parseURI strictURIParserOptions (host ^. packed . re utf8) of
Left e -> do
putStrLn "uh oh"
print e
Right uri ->
case ( uri ^? uriAuthorityL . _Just . authorityHostL . hostBSL
, uri ^? uriAuthorityL . _Just . authorityPortL . _Just . portNumberL) of
(Just host_, Just port_) -> do
let opts = defaults & proxy ?~ httpProxy host_ port_
response <- getWith opts "http://example.com"
print response
_ ->
putStrLn "uh oh"
I'm using lens here to do the boring bits and pieces of packing/unpacking strings, encoding UTF8, and talking to the uri-bytestring package to get URI parsing. But the general idea is that datatypes in Haskell can be sliced and diced simply by pattern matching on the constructor; once extracted, the host:string here is funneled down into the httpProxy call, which returns Wreq's Proxy type. By qualifying the name of the constructor (Newtork.HTTP.Proxy.Proxy) I've let the compiler know which module I want that name from.
It would also not be too difficult, and probably less code to boot, to manually parse proxy information from the environment variables yourself. You could even have a separate environment variable for host and port, which would obviate the need for URI parsing. URIs are have such massively low entropy that they're an awful format for storing configuration information.
I have two http-servers working with a json api using the snap framework
my first prototype contains a handler similar to this example handler
import Data.ByteString (ByteString)
import Data.ByteString.Char8 as B (unwords, putStrLn)
import Data.ByteString.Lazy.Char8 as L (putStrLn)
import Control.Monad.IO.Class (liftIO)
import Data.Monoid ((<>))
import Snap.Core (getParam, modifyResponse, setHeader, writeLBS)
import Network.HTTP.Conduit
import Network.HTTP.Client (defaultManagerSettings)
exampleHandler :: AppHandler ()
exampleHandler = do resp <- liftIO
$ do L.putStrLn "Begin request ..."
initReq <- parseUrl "http://localhost:8001/api"
manager <- newManager defaultManagerSettings
let req = initReq { method = "GET"
, proxy = Nothing}
r <- httpLbs req manager
L.putStrLn "... finished request."
return $ responseBody r
liftIO . L.putStrLn $ "resp: " <> resp
modifyResponse $ setHeader "Content-Type" "application/json"
writeLBS $ "{ \"data\": \""<> resp <>"\" }"
If I issue an ajax-request, the response is sent and received - i see this when the server writes resp: testdata on the console, but the response sent to the browser with writeLBS is not. Now if I change the last line to
writeLBS $ "{ \"data\": \""<> "something fixed" <>"\" }"
everything works like a charm. I think I am meeting one of the pitfalls of lazy IO, but I don't know how to remedy this.
I also tried a few variations with no singe liftIO-block but putting liftIO where necessary.
EDIT
based on the comment by #MichaelSnoyman I did some research regarding writeLBS and tried to
modifyResponse $ setBufferingMode False
. setHeader "Content-Type" "application/json"
writeLBS resp
as I thought maybe buffering could be the problem - no it is not
Furthermore I tried to write explicitly a setResponseBody
let bb = enumBuilder . fromLazyByteString $ "{ \"data\": \""<> resp <>"\" }"
modifyResponse $ setBufferingMode False
. setHeader "Content-Type" "application/json"
. setResponseBody bb
Which showed also no success.
I have solved this issue - it actually was a problem with the javascript getting the handwritten json (note to self: never do that again). There was a non-breaking space at the end of the input data that was not encoded correctly, and I as I am a newbie at JS I didn't get that from the error message.
The intermediate solution is to add urlEncode and make a strict ByteString
let respB = urlEncode . L.toStrict $ C.responseBody resp
modifyResponse $ setBufferingMode False
. setHeader "Content-Type" "application/json"
writeBS $ "{ \"data\": \"" <> respB <> "\" }"
of course you have to change imports accordingly.
The long term solution is: write a proper from/toJSON instance and let the library deal with this.
I'm having an issue with the Scotty web server right now - rescue isn't working for unfound parameters - I'm still getting a 404 with the following code:
post "/newsletter/create" ( do
(param "subscriber[email]") `rescue` (\msg -> text msg)
formContent <- param "subscriber[email]"
text $ "found! " ++ show formContent )
I can see that when I just use params instead, my data is there, and indexed with "subscriber[email]". Is there something going on with [ escaping? Any help with this would be tremendous.
With some cleanup I got it to work:
{-# LANGUAGE OverloadedStrings #-}
import Web.Scotty
import qualified Data.Text.Lazy as TL
main = scotty 3000 $ do
post "/newsletter/create" $ do
formContent <- (param "subscriber[email]") `rescue` (\msg -> return msg)
text $ "found! " `TL.append` formContent
I made a bunch of modifications, but the key point was that rescue is used as a wrapper around param, not to change any internal state, hence you shouldn't call it twice. The square brackets didn't cause me any trouble.
I'm trying to upload files from a web form directly to to Amazon S3 asynchronously. In order to do this I must authenticate the client request to upload files on the server.
By digitally signing an upload request w/ my AWS Secret key I can create a temporary authenticated URL that the client can use to upload files to a S3 bucket.
The amazon S3 docs specify that the signature must be generated by the following
Signature = URL-Encode( Base64( HMAC-SHA1( YourSecretAccessKeyID,
UTF-8-Encoding-Of( StringToSign ) ) ) );
I'm using Haskell on the server so my implementation looks like:
{-# LANGUAGE OverloadedStrings #-}
import qualified Data.ByteString.Base64.Lazy as B64
import qualified Data.Digest.Pure.SHA as SHA
import qualified Data.ByteString.Lazy.Char8 as BL8
sign :: BL8.ByteString -> BL8.ByteString
sign = B64.encode . SHA.bytestringDigest . SHA.hmacSha1 secret
where secret = "aws-secret-key"
The format of the amazon docs requires that StringToSign look like:
StringToSign = HTTP-VERB + "\n" +
Content-MD5 + "\n" +
Content-Type + "\n" +
Expires + "\n" +
CanonicalizedAmzHeaders +
CanonicalizedResource;
Another example from Amazon:
GET\n
\n
\n
1175139620\n
/johnsmith/photos/puppy.jpg
So my string looks like:
"PUT\n\n\n1384330538\n/bucketname/objname"
I sign the string above (w/ the sign function) and craft a url that looks like:
https://s3.amazonaws.com/bucketname/objname?AWSAccessKeyId=accessskey&Signature=signature=&Expires=1384330979
This is then sent to the client via an AJAX request before an upload. I have updated the CORS policy on the bucket as well to allow for PUT requests.
The problem is that every time I try to upload something with the above signed url I get this message (in an XML doc).
The request signature we calculated does not match the signature you
provided. Check your key and signing method.
So I'm not sure where I went wrong. Note: I can upload if I use the public url (https://s3.amazonaws.com/bucketname/objname) (but this shouldn't be, I only want users to upload blobs, not read nor delete, etc.)
As someone who's done this dance a lot, it's very difficult to build software that correctly signs an HTTP-digest authenticated request like this. In particular, if you rely only on the server response to guide you it will take a long time. For security purposes the servers are deliberately cryptic when rejecting you.
My best tip is to (a) get an alternative implementation that you know works and (b) build your Haskell interface to be pure so that it's easy to make it exactly replicate a request from that other framework and (c) make sure you can get both the exact request text and exact String-To-Sign from both the alternative framework and your own code. In particular, you'll often have to impute exact timestamps and nonces and pay close attention to percent encodings.
With these two tools just create a variety of successful requests from the alternative implementation and see if you can replicate the exact String-To-Sign and exact request text using your own framework.
Most often my own errors involved improper encoding, missing quotes, not including all of the proper parameters (or the wrong ones), or using the hmac function incorrectly.
here is my upload url code, i might have missed a couple of imports since i pulled it out of the deep.
{-# LANGUAGE OverloadedStrings, FlexibleContexts, TypeFamilies, DeriveDataTypeable, TemplateHaskell, QuasiQuotes #-}
import qualified Aws
import qualified Aws.Core as Aws
import qualified Aws.S3 as S3
import qualified Data.Text as T
import qualified Codec.Binary.Base64 as B64
import qualified Data.ByteString as BS
import Text.Shakespeare.Text(st)
import qualified Codec.Binary.Url as Url
import System.Posix.Time(epochTime)
import Crypto.MAC.HMAC(hmac)
import Crypto.Hash.SHA1(hash)
data Cfg = Cfg { baseCfg :: Aws.Configuration
, s3Cfg :: S3.S3Configuration Aws.NormalQuery
, s3Bucket :: S3.Bucket
}
uploadUrl :: Cfg -> T.Text -> T.Text -> IO T.Text
uploadUrl cfg mime filename = do
time <- epochTime
let expires = show $ time + 600
msg = E.encodeUtf8 $ [st|PUT
#{mime}
#{expires}
x-amz-acl:public-read
/#{s3Bucket cfg}/#{filename}|] --the gap is necessary
key = Aws.secretAccessKey $ Aws.credentials $ baseCfg cfg
accessid = T.pack $ Url.encode $ BS.unpack $ Aws.accessKeyID $ Aws.credentials $ baseCfg cfg
signature = encode . T.pack $ B64.encode $ BS.unpack $ hmac hash 64 key msg
encode = T.pack . Url.encode . BS.unpack . E.encodeUtf8
return $ [st|http://#{s3Bucket cfg}.s3.amazonaws.com/#{filename}?AWSAccessKeyId=#{accessid}&Expires=#{expires}&Signature=#{signature}|]
I'm struggling converting a Response from http-conduit to an XML document via xml-conduit.
The doPost function takes an XML Document and posts it to the server. The server responds with an XML Document.
doPost queryDoc = do
runResourceT $ do
manager <- liftIO $ newManager def
req <- liftIO $ parseUrl hostname
let req2 = req
{ method = H.methodPost
, requestHeaders = [(CI.mk $ fromString "Content-Type", fromString "text/xml" :: Ascii) :: Header]
, redirectCount = 0
, checkStatus = \_ _ -> Nothing
, requestBody = RequestBodyLBS $ (renderLBS def queryDoc)
}
res <- http req2 manager
return $ res
The following works and returns '200':
let pingdoc = Document (Prologue [] Nothing []) (Element "SYSTEM" [] []) []
Response status headers body <- doPost pingdoc
return (H.statusCode status)
However, when I try and parse the Response body using xml-conduit, I run into problems:
Response status headers body <- doPost xmldoc
let xmlRes' = parseLBS def body
The resulting compilation error is:
Couldn't match expected type `L.ByteString'
with actual type `Source m0 ByteString'
In the second argument of `parseLBS', namely `body'
In the expression: parseLBS def body
In an equation for `xmlRes'': xmlRes' = parseLBS def body
I've tried connecting the Source from http-conduit to the xml-conduit using $= and $$, but I'm not having any success.
Does anyone have any hints to point me in the right direction? Thanks in advance.
Neil
You could use httpLbs rather than http, so that it returns a lazy ByteString rather than a Source — the parseLBS function is named because that's what it takes: a Lazy ByteString. However, it's probably best to use the conduit interface that the two are based on directly, as you mentioned. To do this, you should remove the runResourceT line from doPost, and use the following to get an XML document:
xmlRes' <- runResourceT $ do
Response status headers body <- doPost xmldoc
body $$ sinkDoc def
This uses xml-conduit's sinkDoc function, connecting the Source from http-conduit to the Sink from xml-conduit.
Once they're connected, the complete pipeline has to be run using runResourceT, which ensures all allocated resources are released in a timely fashion. The problem with your original code is that it runs the ResourceT too early, from inside doPost; you should generally use runResourceT right at the point that you want an actual result out, because a pipeline has to run entirely within the scope of a single ResourceT.
By the way, res <- http req2 manager; return $ res can be simplified to just http req2 manager.