Reading file contents once at haskell application startup - haskell

I have an API written using the Servant library which connects to a postgres db. The connection string for my db is stored in a configuration file. Everytime I make a request to any endpoint that interacts with the db I have to read the file to get the connection string, this is what im trying to avoid.
Step by step example of what im trying to achieve:
Application starts up.
Contents of the config file are read and bound to some type/object.
I make a request to my endpoint to create an entry in the db.
I read the connection string from the type/object that I bound it to and NOT the config file.
Every subsequent request for the lifetime of the application does not have to read the config file everytime it wants to interact with the database.
Doing this in something like java/c# you would just bind the contents of a file to some POCO which would be added to your DI container as a singleton so it can be referenced anywhere in your application and persist between each api request. If I have 100 requests that ineract with the db, none of those 100 requests would need to read config file to get the connection string as it was already loaded into memory when the app started.
I have thought about using the cache package, but is there an easier way to do something like this without a third party package?

Let's begin with this trivial Servant server:
import Servant
import Servant.Server
type FooAPI = Get '[JSON] Int
fooServer :: Server FooAPI
fooServer = pure 1
Suppose we don't want to hardcode that 1. We could turn fooServer into a function like
makeFooServer :: Int -> Server FooAPI
makeFooServer n = pure n
Then, in main, we could read the n from a file then call makeFooServer to construct the server. Something similar could be done for your database connection.
There's another approach that might be sometimes preferrable. Servant lets you define servers whose handlers live in a monad different from Handler, and then transform them into regular servers (tutorial).
We can write a server in which the handler monad is a ReaderT holding the configuration:
import Control.Monad.Trans.Reader
type RHandler env = ReaderT env Handler
fooServer' :: ServerT FooAPI (RHandler Int)
fooServer' = do
n <- ask
pure n
Where ServerT is a more general form of Server that lets you specify the handler monad in an extra type argument.
Then, we use the hoistServer function to supply the initial environment and go back to a regular server:
-- "Server FooAPI" is the same as "ServerT FooAPI Handler"
-- so the transformation we need is simply to run the `ReaderT`
-- by supplying an environment.
backToNormalServer :: Int -> Server FooAPI
backToNormalServer n = hoistServer (Proxy #FooAPI) (flip runReaderT n) fooServer'
The ServerT FooAPI (RHandler Int) approach has the advantage that you still have a server value that you can directly manipulate and pass around, instead of it being the result of a function.
Also, for some advanced use cases, the environment might reflect information derived from the structure of each endpoint.

Related

handle one connection for all the server request

Here is the following code :
start :: Settings -> IO ()
start settings # Settings {healthCheckLoggerId} = do
waitTillHealthy
healthCheckLoggerId
settings
Server.getDependencies
Server.healthCheck
Server.getDependencies
settings
(runServerOnWarp)
where
runServerOnWarp :: Server.Dependencies -> IO()
runServerOnWarp dependencies # Server.Dependencies {logger,port} = do
run port $ application
(proxy :: Proxy GSDMonitoringStreamingApi)
monitoringServer
dependencies
monitoringServer :: ServantServer GSDMonitoringStreamingApi Server.Dependencies
monitoringServer dependencies = streamCommand dependencies
where
streamCommandResponse :: Server.Dependencies ->
WorkspaceId ->
Handler (PipeStream (Either StreamIssue (Persisted CommandResponse)))
streamCommandResponse Server.Dependencies {eventStoreClientDependencies} =
return . toPipes . GsdMonitoring.streamCommandResponse eventStoreClientDependencies
Explanation of the issue :
the function Server.getDependencies is wrapping a bracket pattern that get a connection to a database (the client of that database recommend using one connection for all not one for each request).
First I'm running a healthCheck which is blocking till everything is fine. Each time I'm testing the health check of a dependency each time I'm acquiring a new connection. Once the environnement is healthy, I'm getting again the Server.Dependencies (db connection), and I'm running the server. Everything running into runServerOnWarp is executed in the -- ^ computation to run in-between section from
bracket
:: IO a -- ^ computation to run first (\"acquire resource\")
-> (a -> IO b) -- ^ computation to run last (\"release resource\")
-> (a -> IO c) -- ^ computation to run in-between
-> IO c -- returns the value from the in-between computation
so all the requests are executed into that section and get the same connection, so far so good... Because that section :
run port $ application
(proxy :: Proxy GSDMonitoringStreamingApi)
monitoringServer dependencies
is never ending, we are always staying in the computation to run in-between (except when I kill the app...)
Now the issue I have is when the connection get closed.... all my request will return Left StreamIssue which is fine but I have to get a new connection from Server.getDependencies. I don't know how to do it properly :-( Hacks that I have in mind but I don't like are :
1) Having one connection for one request
2) Raising an exception when the request stream a Left streamIssue, that somehow restart the server to get new dependencies... The issue is that the client of the server never get a Left value and the communication between the 2 is violently closed because the server is restarted...
Do you have a cleaner way in mind ?
P.S : I hope the details are enough to understand the issue, don't hesitate for more information if necessary...
If losing the database connection is rare, letting the server crash, and some supervisory process restart it, works quite well.
If losing the DB connection is common enough that you can't tolerate client timeouts from restarting the server, you can add reconnect logic to whatever code uses the DB connection. I usually use resource-pool for this. I believe you can create the "pool" with only one resource, if that fits your database.
Usually getting a DB connection is fast, so I'm guessing it's not worth extra effort to route incoming requests elsewhere. withResource in resource-pool blocks until a resource is ready, so active requests can easily wait while you reconnect.

Haskell, Yesod and Keter - How can I run a routine periodically (every 5 minutes)?

There are some database queries I want to run periodically, and according to its state, send notifications to users email and change the state of their accounts. Can I do it within Yesod itself?
I moved from Yesod's issue.
Run Handler code at a specific time within yesod · Issue #1529 · yesodweb/yesod
I do not know your complete code.
So, this is proposal.
makeApplication :: App -> IO Application
makeApplication foundation = do
unsafeHandler foundation $
forkHandler (\_ -> catchError) $ forever $ do -- catchError do not exist
waitUntil10AM -- waitUntil10AM do not exist
getCheckupR
logWare <- makeLogWare foundation
-- Create the WAI application and apply middlewares
appPlain <- toWaiAppPlain foundation
return $ logWare $ (acceptOverride . autohead . gzip def) appPlain
This code point that use unsafeHandler and forkHandler.
waitUntil10AM
I do not know your timezone, environment, database structure etc, so I want you to write the details yourself.
For example, if you put threadDelay in forever and check it once every ten minutes, put the date on which you sent the mail already in the database and call it if you do not send it and it exceeds 10AM.
catchError
Please decide what kind of processing should be done at the time of error.
I would like to handle errors in a way that it would never stop
You can name the function to be passed inside forkHandler and call it again on error.

Haskell Servant and streaming

I am trying to add a functionality to my servant server that would get a file from Amazon S3 and stream it back to the user.
Because files can be big I don't want to download them locally and then serve them to clients, I'd rather prefer to stream them directly from S3 to clients.
I use Amazonka for what I do with S3 and I can get a stream for an S3 file as a Conduit sink.
But now I don't know how to get from Sink to EitherT ServantErr IO a.
Can anyone explain me how to do it or show me some example of how it can be done?
The is nothing in Servant to do this out of the box, however all the needed parts are available.
Before we begin, I guess that if you can stream to a Sink, that means you have a source (the gorsBody of GetObjectResponse is a RsBody, which is a Source)
First of all, Servant provides us with the possibility to add support for new return types, by creating a new instance of HasServer, so we could serve a EitherT ServantErr IO (Source ...) and have it stream.
To create that instance, we must implement route :: Proxy layout -> Server layout -> RoutingApplication. Server layout, in this case, just means EitherT ServantErr IO layout, layout being the source we want to server, so it's the function that returns a source (and may fail with an HTTP error).
We have to return a RoutingApplication, which is (in continuation style) a function that takes a Request and returns a RouteResult Response, which means either an unmatched route error, or a response. Both Request and Response are standard wai, not Servant, so we can now look at the rest of the ecosystem to find how to implement it.
Fortunately, we don't have to go far: Network.Wai.Conduit contains just what we need to implement the route function: responseSource takes a status value, some response headers and your source and gives you a Response.
So, that is quite a lot of work to do, but everything we need is there. Looking a the source of the instance HasServer * (Get ...) might help.

How can I limit size of request body and headers in WAI?

I am developing an application using Scotty and of course WAI. I would like to be able to limit the size of requests, both for body length and for headers. How can I do that? Is it possible to do it using a plain WAI middleware ?
I don't know details of Scotty, but it's certainly possible to set up a WAI middleware that will look at the requestBodyLength and, if it's too large, return an appropriate 413 status code page. One thing you'd need to deal with is if the upload body is sent with chunked encoding, in which case no content-length is present. but that's uncommon. You have the option of either rejecting those requests, or adding code to wrap the request body and return an error if it turns out to be too large (that's what Yesod does).
The marked solution points in the correct direction, but if you're like me you might still struggle to explicitely derive the full code needed. Here is an implementation (thanks to the help of an experienced Haskell friend):
import qualified Network.HTTP.Types as Http
import qualified Network.Wai as Wai
limitRequestSize :: Wai.Middleware
limitRequestSize app req respond = do
case Wai.requestBodyLength req of
Wai.KnownLength len -> do
if len > maxLen
then respond $ Wai.responseBuilder Http.status413 [] mempty
else app req respond
Wai.ChunkedBody ->
respond $ Wai.responseBuilder Http.status411 [] mempty
where
maxLen = 50*1000 -- 50kB
The middleware then just runs in scotty's do block like this
import Network.Wai.Middleware.RequestLogger (logStdout)
main :: IO ()
main = do
scotty 3000 $ do
middleware logStdout
middleware limitRequestSize
get "/alive" $ do
status Http.status200
-- ...
If you're curious as to how to derive it (or why I found this not overly trivial), consider that Middleware is an alias for
Application -> Application
where Application itself is an alias for
Request -> (Response -> IO ResponseReceived) -> IO ResponseReceived
Hence there are quite a bunch of arguments to (mentally) unpack, even if the solution is pretty terse.
As of wai-extra-3.1.1 the code described above has been added to the Network.Wai.Middleware.RequestSizeLimit module, so it can just be pulled in as a dependency.

Haskell: HDBC, connection state and possibly pool

Say, I have simple app which uses HDBC to interact with a database. It could be a snap app or a command line app. I'd like a generic view on this - not snap or anything specific.
A good example would be the the chapter 22 from "Real World Haskell".
So I would hide all my SQL related functions like connect, saveArticle, getArticle etc. in a separate DB.hs file. All the DB functions would take a connection handle as an argument.
Now, how would I initiate the connection once and put it in a state so that all those DB functions just use it without initiating connection individually?
Is this possible? Maybe I am thinking wrong or with OO concepts in mind but ... I would like to "initialize" my connection (like you would initialize an object in OO).
I don't want every DB function to create and close new connection.
Do I have to create a pool and pass the pool to the functions as argument instead of connection handle? What is the simplest example of doing it?
Whether it is a pool or connection handle, how do I put it in a state so my functions just grab it when needed and don't repetitively "open" and "close" all the time? If yes, how is it done correctly?
Do I need to create a pool and put it in a state so functions just query the pool for connections from global state? Again, example would be really appreciated.
Thanks.
There is no such thing as module state or application state, so you have to find an other strategy. I much recommend a reader monad, with the environment holding the db connection. That will cost you a few returns in some functions, but will be very solid.
Below is a simple pseudo-haskell example.
type AppState = AppState { dbcon :: DatabaseConnection }
type App = ReaderT AppState IO
main = do
db <- makeNewDbConnection
runReaderT getDbTableCount $ AppState db
getDbTableCount :: App Integer
getDbTableCount = do
(count:_) <- runDb "select count(*) from table;"
return $ read count
runDb :: String -> App [String]
runDb req = do
con <- asks dbcon
return $ dbQuery con req
You may want to take a look at bos' resource-pool library.

Resources