How do I shut down `runTCPServer`? - haskell

I'm writing a socket server with runTCPServer from conduit-extra (formerly known as network-conduit). My goal is to interact with my editor using this server --- activate the server from the editor (most likely just by calling external command), use it, and terminate the server when the work is done.
For simplicity, I start with a simple echo server, and let's say I'd like to shut down the whole process when the connection is closed.
So I tried:
{-# LANGUAGE OverloadedStrings #-}
module Main where
import Data.Conduit
import Data.Conduit.Network
import Data.ByteString (ByteString)
import Control.Monad.IO.Class (liftIO)
import System.Exit (exitSuccess)
import Control.Exception
defaultPort :: Int
defaultPort = 4567
main :: IO ()
main = runTCPServer (serverSettings defaultPort "*") $ \ appData ->
appSource appData $$ conduit =$= appSink appData
conduit :: ConduitM ByteString ByteString IO ()
conduit = do
msg <- await
case msg of
Nothing -> liftIO $ do
putStrLn "Nothing left"
exitSuccess
-- I'd like the server to shut down here
(Just s) -> do
yield s
conduit
But this doesn't work -- the program continues to accept new connections. If I am not mistaken, this is because the thread listening to the connection we're dealing with exits with exitSuccess, but the entire process doesn't. So this is totally understandable, but I haven't been able to find a way to exit the whole process.
How do I terminate a server run by runTCPServer? Is runTCPServer something that's supposed to serve forever?

Here's a simple implementation of the idea described in comments:
main = do
mv <- newEmptyMVar
tid <- forkTCPServer (serverSettings defaultPort "*") $ \ appData ->
appSource appData $$ conduit mv =$= appSink appData
() <- takeMVar mv -- < -- wait for done signal
return ()
conduit :: MVar () -> ConduitM ByteString ByteString IO ()
conduit mv = do
msg <- await
case msg of
Nothing -> liftIO $ do
putStrLn "Nothing left"
putMVar mv () -- < -- signal that we're done
(Just s) -> do
yield s
conduit mv

Related

Capture the output while terminating a process

I need to run a process, do something while it is running, and finally terminate it. The
process in question writes things to standard output that I would like to retain. Unfortunately,
it seems that the process dies before I can connect and extract its last words. Having scarce
experience with asynchronous programming, I am having a difficulty finding a nice solution. It
would be fortunate if I can accomplish this task within the framework of RIO.Process, although I
am prepared to step outside of it if it cannot be avoided. (Note that RIO employs an unusual
way of invoking external processes via a callback system.)
Below is a highly simplified runnable example of what I am trying to achieve.
Here is an emulation of the program to be run:
(Put it in a file called x.sh and say chmod +x x.sh to make it executable.)
#!/bin/sh
trap 'echo "Terminating..."; exit 0' TERM
echo "Initialization complete."
while true; do sleep 1; done
Here is my code:
(Put it in a file called X.hs and compile with ghc -package rio X.hs.)
{-# language NoImplicitPrelude #-}
{-# language BlockArguments #-}
{-# language OverloadedStrings #-}
module Main where
import RIO
import RIO.Process
import Data.Text.IO (hGetContents, hGetLine)
main :: IO ()
main = runSimpleApp do
proc "./x.sh" [ ]
\processConfig -> withProcessWait_ (setStdout createPipe processConfig)
\processHandle -> bracket_
(initialize processHandle)
(terminate processHandle)
(return ())
initialize :: (HasProcessContext env, HasLogFunc env) => Process () Handle () -> RIO env ()
initialize processHandle = do
x <- liftIO $ hGetLine (getStdout processHandle)
if x == "Initialization complete." then return () else error "This should not happen."
terminate :: HasLogFunc env => Process () Handle () -> RIO env ()
terminate processHandle = do
log' <- async $ liftIO $ hGetContents (getStdout processHandle)
stopProcess processHandle
log <- wait log'
logInfo $ display log
Here is what happens:
% ./X
X: fd:3: hGetBuffering: illegal operation (handle is closed)
— x.sh is saying something, but I cannot hear.
What is the right way to manage this?
From the documentation for stopProcess:
Close a process and release any resources acquired. This will ensure terminateProcess is called, wait for the process to actually exit, and then close out resources allocated for the streams. In the event of any cleanup exceptions being thrown this will throw an exception.
(emphasis mine) You don't want stopProcess to do that before you read the output. You just want terminateProcess. withProcessWait_ will take care of the rest of it. Unfortuntately, you do have to step outside of RIO to do that, with import System.Process (terminateProcess) and then liftIO $ terminateProcess (unsafeProcessHandle processHandle).
Side notes: You're kind of misusing bracket_. Since the "middle" of your bracket_ is a no-op, and especially now that the beginning and end aren't actually acquiring or releasing any resources, it's kind of pointless. Also, instead of using async at all, you can just read the output normally after terminating the process, since the output that a process already produced doesn't just disappear when it's terminated.
Here's your code with all of the above fixed:
{-# language NoImplicitPrelude #-}
{-# language BlockArguments #-}
{-# language OverloadedStrings #-}
module Main where
import RIO
import RIO.Process
import Data.Text.IO (hGetContents, hGetLine)
import System.Process (terminateProcess)
main :: IO ()
main = runSimpleApp do
proc "./x.sh" [ ]
\processConfig -> withProcessWait_ (setStdout createPipe processConfig)
\processHandle -> do
initialize processHandle
terminate processHandle
initialize :: (HasProcessContext env, HasLogFunc env) => Process () Handle () -> RIO env ()
initialize processHandle = do
x <- liftIO $ hGetLine (getStdout processHandle)
if x == "Initialization complete." then return () else error "This should not happen."
terminate :: HasLogFunc env => Process () Handle () -> RIO env ()
terminate processHandle = do
liftIO $ terminateProcess (unsafeProcessHandle processHandle)
log <- liftIO $ hGetContents (getStdout processHandle)
logInfo $ display log

Can't pass data via stdin to process spawned with conduit-extra

In my program I am starting external process and communicate with it via stdin and stdout. I'm feeding the input through conduit (producer) started from STMs TQueue. It worked like a charm until I've decided to bump lts version. It worked great with lts <= 8.24.
Here is the minimized program that reproduces my problem:
#!/usr/bin/env stack
-- stack --resolver lts-10.4 --install-ghc runghc --package conduit-extra --package stm-conduit
{-# LANGUAGE OverloadedStrings #-}
import Control.Concurrent
import Control.Monad.STM
import Control.Concurrent.STM.TQueue
import Data.Conduit
import qualified Data.Conduit.Binary as CB
import qualified Data.Conduit.List as CL
import Data.Conduit.Process (CreateProcess (..),
proc, sourceProcessWithStreams)
import qualified Data.Conduit.TQueue as CTQ
import qualified Data.ByteString.Char8 as BS
import Data.Monoid ((<>))
main :: IO ()
main = do
putStrLn "Enter \"exit\" to exit."
q <- open
putStrLn "connection opened"
loop q
where loop q = do
s <- BS.getLine
case s of
"exit" -> return ()
req -> do
atomically $ writeTQueue q req
loop q
open :: IO (TQueue BS.ByteString)
open = do
req <- atomically newTQueue
let chat :: CreateProcess
chat = proc "cat" []
input :: Producer IO BS.ByteString
input = toProducer
$ CTQ.sourceTQueue req
-- .| CL.mapM_ (\bs -> BS.putStrLn (("queue: " :: BS.ByteString) <> bs))
output :: Consumer BS.ByteString IO ()
output = toConsumer
$ CL.mapM_ BS.putStrLn
_ <- forkIO (sourceProcessWithStreams chat input output output >> pure ())
pure req
With newer lts it seems like the problem is not with communication via TQueue, as uncommenting the line which prints content from input conduit gives shows data from the queue. It looks like the spawned process never receives anything on it's stdin.
Furthermore writing to spawned cat stdin from console, like so:
echo "test" > /proc/<pid of spawned cat>/fd/0
produces output in my program.
Am I missing something that changed between versions?
So the issue was that default behaviour of sinkHandle was changed to not flush after every chunk of data.
I've fixed the issue by first porting to Data.Conduit.Process.Typed and then rolling my own variant of createSink that is using sinkHandleFlush instead of sinkHandle.

Haskell: start a long running process, silently, capturing stdout

I have a long running process which I need to start.
It takes a few seconds to start, and outputs logs to stdout, with one that indicates it is ready.
I would like to:
start the process silently, so that the stdout from the process is not displayed in my session.
capture the output as it streams so that I can determine that it is ready.
have some handle on the process so that I can stop the process at a later point.
I have come close using Shelly, Turtle and System.Process, but fail to capture the stdout.
Using System.Process I had:
import Control.Concurrent (threadDelay)
import Control.Concurrent.Async (race)
import System.IO
import System.Process
startService :: IO ProcessHandle
startService = do
let cmd = "./my-service"
args = [ "-p 1234" ]
(_, Just hout, _, p) <- createProcess $ (proc cmd args) { std_out = CreatePipe }
started <- either id id <$> race (checkStarted hout) timeOut
unless started $ fail "Service not started"
pure p
where
checkStarted :: Handle -> IO Bool
checkStarted h = do
str <- hGetLine h
-- check str for started log, else loop
timeOut :: IO Bool
timeOut = do
threadDelay 10000000
pure False
But The handler hout was never in a ready state.
Using Shelly I had:
import Control.Concurrent (threadDelay)
import Control.Concurrent.Async (race)
import Control.Concurrent.MVar
import Shelly
import System.IO
startService :: IO (Async ())
startService = do
let cmd = "./my-service"
args = [ "-p 1234" ]
startedMVar <- newEmptyMVar
async <- shelly $ asyncSh $ runHandle cmd args $ recordWhenStarted startedMVar
started <- either id id <$> race (readMVar startedMVar) timeOut
unless started $ fail "Service not started"
pure async
where
recordWhenStarted :: MVar Bool -> Text -> IO ()
recordWhenStarted mvar txt =
when (isStartedLog txt) $
modifyMVar_ mvar (const $ pure True)
timeOut :: IO Bool
timeOut = do
threadDelay 10000000
pure False
But the recordWhenStarted is never called.
The following is example of starting process and reading stdout in a program of mine:
runMystem :: [T.Text] -> IO T.Text
runMystem stemWords = do
(i, o, _, ph) <- createProcess (proc mystemExecutabe mystemParams) { std_in = CreatePipe, std_out = CreatePipe }
res <- flip (maybe (return T.empty)) i $ \hIn ->
flip (maybe (return T.empty)) o $ \hOut -> do
hSetEncoding hIn utf8
hSetEncoding hOut utf8
forM_ stemWords $ TIO.hPutStrLn hIn
TIO.hGetContents hOut
void $ waitForProcess ph
return res
This answer uses the process-streaming library (written by the author of this answer) which is a set of helpers over process.
{-# language OverloadedStrings #-}
{-# language NumDecimals #-}
import System.Process.Streaming (execute,piped,shell,foldOut,transduce1)
import qualified System.Process.Streaming.Text as PT
import Data.Text.Lazy (isInfixOf)
import Control.Applicative
import Control.Monad
import Control.Concurrent (threadDelay)
import Control.Concurrent.Async
import Control.Concurrent.MVar
main :: IO ()
main = do
started <- newEmptyMVar
let execution =
execute (piped (shell "{ sleep 3 ; echo fooo ; sleep 3 ; }")) $
foldOut . transduce1 PT.utf8x . PT.eachLine $ lookline
lookline line = do
when (isInfixOf "foo" line) (putMVar started ())
return (Right ())
stopOrNot =
do abort <- race (threadDelay 4e6) (readMVar started)
case abort of
Left () -> return () -- stop immediately
Right () -> runConcurrently empty -- sleep forever
result <- race stopOrNot execution
print result
execute installs exception handlers that terminate the external process when an asynchronous exceptions arrives, to it is safe to use race with it.
execute also takes care to drain any standard stream that is not read explicitly (like stderr in this case) to avoid a common source of deadlocks.

How to exit a conduit when using mergeSources

I have a simple forked conduit setup, with two inputs feeding one single output....
{-# LANGUAGE OverloadedStrings #-}
import Control.Concurrent (threadDelay)
import Control.Monad.IO.Class
import Control.Monad.Trans.Resource
import qualified Data.ByteString as B
import Data.Conduit
import Data.Conduit.TMChan
import Data.Conduit.Network
main::IO ()
main = do
runTCPClient (clientSettings 3000 "127.0.0.1") $ \server -> do
runResourceT $ do
input <- mergeSources [
transPipe liftIO (appSource server),
infiniteSource
] 2
input $$ transPipe liftIO (appSink server)
infiniteSource::MonadIO m=>Source m B.ByteString
infiniteSource = do
liftIO $ threadDelay 10000000
yield "infinite source"
infiniteSource
(here I connect to a tcp socket, then combine the socket input with a timed infinite source, then respond back to the socket)
This works great, until the connection is dropped.... Because the second input still exists, the conduit keeps running. (In this case, the program does end when the timed input fires and there is no socket to write to, but this isn't always the case in my real example).
What is the proper way to shut down the full conduit when one of the inputs is closed?
I tried to brute force a crash by adding the following
crashOnEndOfStream::MonadIO m=>Conduit B.ByteString m B.ByteString
crashOnEndOfStream = do
awaitForever $ yield
error "the peer connection has disconnected" --tried with error
liftIO $ exitWith ExitSuccess --also tried with exitWith
but because the input conduit runs in a thread, the executable was immune to runtime exceptions shutting it down (plus, there is probably a smoother way to shut stuff down than halting the program).
the Source created by mergeSources keeps a count of unclosed sources. It's only closed when the count reaches 0 i.e. every upstream source is closed. This mechanism and the underlying TBMChannel is hidden from user code so you have no way to change its behavior.
One possible solution is to create the channel and the source manually with some medium-level functions exported by Data.Conduit.TMChan, so you can finalize the source by closing the TBMChannel. I haven't tested the code below since your program is not runnable on my machine.
{-# LANGUAGE OverloadedStrings #-}
import Control.Concurrent (threadDelay)
import Control.Monad.IO.Class
import Control.Monad.Trans.Resource
import qualified Data.ByteString as B
import Data.Conduit
import Data.Conduit.Network
import Data.Conduit.TMChan
main::IO ()
main = do
runTCPClient (clientSettings 3000 "127.0.0.1") $ \server -> do
runResourceT $ do
-- create the TBMChannel
chan <- liftIO $ newTBMChanIO 2
let
-- everything piped to the sink will appear at the source
chanSink = sinkTBMChan chan True
chanSource = sourceTBMChan chan
tid1 <- resourceForkIO $ appSource server $$ chanSink
tid2 <- resourceForkIO $ infiniteSource $$ chanSink
chanSource $$ transPipe liftIO (appSink server)
-- and call 'closeTBMChan chan' when you want to exit.
-- 'chanSource' will be closed when the underlying TBMChannel is closed.
infiniteSource :: MonadIO m => Source m B.ByteString
infiniteSource = do
liftIO $ threadDelay 10000000
yield "infinite source"
infiniteSource

Using a monad inside the IO monad

Is there something that is like the opposite of liftIO? I'm using websockets, and I want to be able to listen for messages from the server in a separate thread. Here's what I'm doing:
import Network.WebSockets
import qualified Data.Text as T
import Control.Monad.IO.Class
import Control.Monad
import Control.Concurrent
import Control.Applicative
printMessages :: WebSockets Hybi00 ()
printMessages = forever $ do
resp <- receiveDataMessage
liftIO $ print resp
run :: WebSockets Hybi00 ()
run = do
liftIO . forkIO $ printMessages
forever $ do
line <- liftIO getLine
sendTextData . T.pack $ line
main = connect "0.0.0.0" 8080 "/" run
So printMessages listens for messages from the server and keeps printing them out. The problem is, forkIO expects a function that returns IO (). Is there any way for me to run printMessages in the IO monad?
If I'm understanding this right, the reason you want to receive messages in another thread is because the main thread will be waiting for user input to send.
From a look at the documentation, it seems like you'll have an easier time if you reverse the roles of the threads: receive in the main thread, and send asynchronously from the other.
Then you can use getSink :: Protocol p => WebSockets p (Sink p) to grab a sink before forking, which you can then use with sendSink :: Sink p -> Message p -> IO () which lives in IO, avoiding the whole problem of mixing monads.
In other words, restructure your code to something like this:
sendMessages :: Sink Hybi00 -> IO ()
sendMessages sink = forever $ do
line <- getLine
let msg = textData . T.pack $ line
sendSink sink msg
run :: WebSockets Hybi00 ()
run = do
sink <- getSink
liftIO . forkIO $ sendMessages sink
forever $ do
resp <- receiveDataMessage
liftIO $ print resp
main = connect "0.0.0.0" 8080 "/" run

Resources