From what I understand (please correct me if I am wrong), in tomcat incoming websocket messages are processed sequentially. Meaning that if you have 100 incoming messages in one websocket, they will be processed using only one thread one-by-one from message 1 to message 100.
But this does not work for me. I need to concurrently process incoming messages in a websocket in order to increase my websocket throughput. The messages coming in do not depend on each other hence do not need to be processed sequentially.
The question is how to configure tomcat such that it would assign multiple worker threads per websocket to process incoming messages concurrently?
Any hint is appreciated.
This is where in tomcat code that I think it is blocking per websocket connection (which makes sense):
/**
* Called when there is data in the ServletInputStream to process.
*
* #throws IOException if an I/O error occurs while processing the available
* data
*/
public void onDataAvailable() throws IOException {
synchronized (connectionReadLock) {
while (isOpen() && sis.isReady()) {
// Fill up the input buffer with as much data as we can
int read = sis.read(
inputBuffer, writePos, inputBuffer.length - writePos);
if (read == 0) {
return;
}
if (read == -1) {
throw new EOFException();
}
writePos += read;
processInputBuffer();
}
}
}
You can't configure Tomcat to do what you want. You need to write a message handler that consumes the message, passes it to an Executor (or similar for processing) and then returns.
Related
I am using boost ASIO for a TCP client. for the most part the ASIO is a glorified event loop for read and write. There is actually only one client managed by the ASIO.
The architecture is like this -
The TCP server streams continuous messages. The Client will read the messages, process it and ack back with proper code.
My code runs in client side. There is one thread running io_service. The io_service thread reads messages and distributes it to N number of worker threads using a boost lockfree SPSC queue. The workers after processing posts the replies to the io_service thread.
most important concern for me is the rate of read and write. So I am using synchronous reads and writes.
Read Code:
void read ()
{
if (_connected && !_readInProgress) {
_socket.async_read_some(boost::asio::null_buffers(),
make_boost_alloc_handler(_readAllocator,
[self = shared_from_this(), this] (ErrorType err, unsigned a)
{
connection()->handleRead(err);
_readInProgress = false;
if (err) disconnect();
else asyncRead();
});
_readInProgress = true;
}
}
Basically I use read_some with nullbuffer() and then directly use Unix system calls to read the messages. The read give N number of messages which are enqueued to threads in a loop.
I want use the boost SPSC queue in the reverse direction for writes to the socket from workers.
Write:
// Get the queue to post writes
auto getWriteQ ()
{
static thread_local auto q =
std::make_shared< LFQType >(_epoch);
return q;
}
So each thread gets a thread-local Q using getWriteQ. The writes to the queue looks like this:
void write (Buf& buf) override
{
auto q = getWriteQ();
while (!q->enqueue(buf) && _connected);
if (!_connected) return;
_ioService.post( [self = shared_from_this(), this, q]()
{
writeHelper(q); });
}
}
Now this is inefficent, as we do a ioservice post for each write. The write handler at a time actually writes upto 32 messages in a single system-call using sendmmsg()
So I am looking for help with 2 things:
Is the design any good?
Any fool proof way to minimize the no. of posts. I was thinking of keep an atomic enqueue count. The worker thread will do this -
the writing thread does this - (Pseudo code)
bool post = false;
if(enqueue_count == 0) post = true
// enqueue the message
++enqueue_count
if(post)
// post the queue event
The io-service thread does this -
enqueue_count -= num_processed;
if (enqueue_count)
// repost the queue for further processing
Would this work if the enqueue_count is atomic ?
If i have following code to send data repeatedly to client every 10ms:
setInterval(function() {
res.write(somedata);
}, 10ms);
What would happen if the client is very slow to receive the data?
Will server get out-of-memory error?
Edit:
actually the connection is kept alive, sever send jpeg data endlessly (HTTP multipart/x-mixed-replace header + body + header + body.....)
Because node.js response.write is asynchronous,
so some users guess it may store data in internal buffer and wait until low layer tells it can send,
so the internal buffer will grow, am i right?
If i am right, then how to resolve this?
the problem is node.js does not notify me when data is send for a single write call.
In other word, i can not tell user this way is theoretically no risk of "out of memory" and how to fix it.
Update:
By the keyword "drain" event given by user568109, i studied the source of node.js, and got conclusion:
it really will cause "out-of-memory" error. I should check return value of response.write(...)===false and then handle "drain" event of the response.
http.js:
OutgoingMessage.prototype._buffer = function(data, encoding) {
this.output.push(data); //-------------No check here, will cause "out-of-memory"
this.outputEncodings.push(encoding);
return false;
};
OutgoingMessage.prototype._writeRaw = function(data, encoding) { //this will be called by resonse.write
if (data.length === 0) {
return true;
}
if (this.connection &&
this.connection._httpMessage === this &&
this.connection.writable &&
!this.connection.destroyed) {
// There might be pending data in the this.output buffer.
while (this.output.length) {
if (!this.connection.writable) { //when not ready to send
this._buffer(data, encoding); //----------> save data into internal buffer
return false;
}
var c = this.output.shift();
var e = this.outputEncodings.shift();
this.connection.write(c, e);
}
// Directly write to socket.
return this.connection.write(data, encoding);
} else if (this.connection && this.connection.destroyed) {
// The socket was destroyed. If we're still trying to write to it,
// then we haven't gotten the 'close' event yet.
return false;
} else {
// buffer, as long as we're not destroyed.
this._buffer(data, encoding);
return false;
}
};
Some gotchas:
If sending over http it is not be a good idea. The browser may consider the request as timeout if it is not finished within specified amount of time. Server too will close connection which is idle for too long. If client cannot keep up, the timeout is almost certain.
setInterval for 10ms is also subject to some restrictions. It doesn't mean it will repeat after every 10ms, 10ms is the minimum it will wait before repeating. It will be slower than what you set the interval.
Let's say you chance to overload the response with data, then at some point the server will end connection and respond by 413 Request Entity Too Large depending on what the limit is set.
Node.js has single threaded architecture with a max memory limitation of around 1.7 GB. If you set your above server limits to too high and have many incoming connections you will get process out of memory error.
So with appropriate limits it will either give timeout or be request too large. (And there are no other errors in your program.)
Update
You need to use drain event. The http response is a writable stream. It has its own internal buffer. When the buffer is emptied the drain event is triggered. You should learn more about streams as you would go in deeper. This will help you not just in http. You can find several resources about streams on web.
I have one thread in the thread-pool servicing blocking request.
def sync = Action {
import Contexts.blockingPool
Future {
Thread.sleep(100)
}
Ok("Done")
}
In Contexts.blockingPool is configured as:
custom-pool {
fork-join-executor {
parallelism-min = 1
parallelism-max = 1
}
}
In theory, if above request receives 100 simultaneous requests, the expected behaviour should be: 1 request should sleep(100) and rest of 99 requests should be rejected (or queued until timeout?). However I observed that extra worker threads are created to service rest of requests. I also observed that latency increases as (gets slower to service request) as number of threads in the pool gets smaller than the requests received.
What is expected behavior if a request larger than configured thread-pool size is received?
Your test is not correctly structured to test your hypothesis.
If you go over this section in the docs you will see that Play has a few thread pools/execution contexts. The one that is important with regards to your question is the default thread pool and how that relates to the HTTP requests served by your action.
As the doc describes, the default thread pool is where all application code is run by default. I.e. all action code, including all Future's (not explicitly defining their own execution context), will run in this execution context/thread pool. So using your example:
def sync = Action {
// *** import Contexts.blockingPool
// *** Future {
// *** Thread.sleep(100)
// ***}
Ok("Done")
}
All the code in your action not commented by // *** will run in the default thread pool.
I.e. When a request gets routed to your action:
the Future with the Thread.sleep will be dispatched to your custom execution context
then without waiting for that Future to complete (because it's running in it's own thread pool [Context.blockingPool] and therefore not blocking any threads on the default thread pool)
your Ok("Done") statement is evaluated and the client receives the response
Approx. 100 milliseconds after the response has been received, your Future completes
So to explain you observation, when you send 100 simultaneous requests, Play will gladly accept those requests, route to your controller action (executing on the default thread pool), dispatch to your Future and then respond to the client.
The default size of the default pool is
play {
akka {
...
actor {
default-dispatcher = {
fork-join-executor {
parallelism-factor = 1.0
parallelism-max = 24
}
}
}
}
}
to use 1 thread per core up to a max of 24.
Given that your action does very little (excl. the Future), you will be able to handle into the 1000's of requests/sec without a sweat. Your Future will however take much longer to work through the backlog because you are blocking the only thread in your custom pool (blockingPool).
If you use my slightly adjusted version of your action, you will see what confirms the above explanation in the log output:
object Threading {
def sync = Action {
val defaultThreadPool = Thread.currentThread().getName;
import Contexts.blockingPool
Future {
val blockingPool = Thread.currentThread().getName;
Logger.debug(s"""\t>>> Done on thread: $blockingPool""")
Thread.sleep(100)
}
Logger.debug(s"""Done on thread: $defaultThreadPool""")
Results.Ok
}
}
object Contexts {
implicit val blockingPool: ExecutionContext = Akka.system.dispatchers.lookup("blocking-pool-context")
}
All your requests are swiftly dealt with first and then your Future's complete one by one afterwards.
So in conclusion, if you really want to test how Play will handle many simultaneous requests with only one thread handling requests, then you can use the following config:
play {
akka {
akka.loggers = ["akka.event.Logging$DefaultLogger", "akka.event.slf4j.Slf4jLogger"]
loglevel = WARNING
actor {
default-dispatcher = {
fork-join-executor {
parallelism-min = 1
parallelism-max = 1
}
}
}
}
}
you might also want to add a Thread.sleep to your action like this (to slow the default thread pools lonesome thread down a bit)
...
Thread.sleep(100)
Logger.debug(s"""<<< Done on thread: $defaultThreadPool""")
Results.Ok
}
Now you will have 1 thread for requests and 1 thread for your Future's.
If you run this with high concurrent connections you will notice that the client blocks while Play handles the requests one by one. Which is what you expected to see...
Play uses AkkaForkJoinPool which extends scala.concurrent.forkjoin.ForkJoinPool.
It has internal queue of tasks.
You may also find this description interesting in respect to handling blocking code by fork-join-pool: Scala: the global ExecutionContext makes your life easier
In scala, how can I tell a thread: sleep t seconds, or until you receive a message? i.e. sleep at most t seconds, but wake up in case t is not over and you receive a certain message.
The answer depends greatly on what the message is. If you're using Actors (either the old variety or the Akka variety) then you can simply state a timeout value on receive. (React isn't really running until it gets a message, so you can't place a timeout on it.)
// Old style
receiveWithin(1000) {
case msg: Message => // whatever
case TIMEOUT => // Handle timeout
}
// Akka style
context.setTimeoutReceive(1 second)
def receive = {
case msg: Message => // whatever
case ReceiveTimeout => // handle timeout
}
Otherwise, what exactly do you mean by "message"?
One easy way to send a message is to use the Java concurrent classes made for exactly this kind of thing. For example, you can use a java.util.concurrent.SynchronousQueue to hold the message, and the receiver can call the poll method which takes a timeout:
// Common variable
val q = new java.util.concurrent.SynchronousQueue[String]
// Waiting thread
val msg = q.poll(1000)
// Sending thread will also block until receiver is ready to take it
q.offer("salmon", 1000)
An ArrayBlockingQueue is also useful in these situations (if you want the senders to be able to pack messages in a buffer).
Alternatively, you can use condition variables.
val monitor = new AnyRef
var messageReceived: Boolean = false
// The waiting thread...
def waitUntilMessageReceived(timeout: Int): Boolean = {
monitor synchronized {
// The time-out handling here is simplified for the purpose
// of exhibition. The "wait" may wake up spuriously for no
// apparent reason. So in practice, this would be more complicated,
// actually.
while (!messageReceived) monitor.wait(timeout * 1000L)
messageReceived
}
}
// The thread, which sends the message...
def sendMessage: Unit = monitor synchronized {
messageReceived = true
monitor.notifyAll
}
Check out Await. If you have some Awaitable objects then that's what you need.
Instead of making it sleep for a given time, make it only wake up on a Timeout() msg and then you can send this message prematurely if you want it to "wake up".
Here's my situation:
I'm writing a chat client to connect to a chat server. I create the connection using a TcpClient and get a NetworkStream object from it. I use a StreamReader and StreamWriter to read and write data back and forth.
Here's what my read looks like:
public string Read()
{
StringBuilder sb = new StringBuilder();
try
{
int tmp;
while (true)
{
tmp = StreamReader.Read();
if (tmp == 0)
break;
else
sb.Append((char)tmp);
Thread.Sleep(1);
}
}
catch (Exception ex)
{
// log exception
}
return sb.ToString();
}
That works fine and dandy. In my main program I create a thread that continually calls this Read method to see if there is data. An example is below.
private void Listen()
{
try
{
while (IsShuttingDown == false)
{
string data = Read();
if (!string.IsNullOrEmpty(data))
{
// do stuff
}
}
}
catch (ThreadInterruptedException ex)
{
// log it
}
}
...
Thread listenThread = new Thread(new ThreadStart(Listen));
listenThread.Start();
This works just fine. The problem comes when I want to shut down the application. I receive a shut down command from the UI, and tell the listening thread to stop listening (that is, stop calling this read function). I call Join and wait for this child thread to stop running. Like so:
// tell the thread to stop listening and wait for a sec
IsShuttingDown = true;
Thread.Sleep(TimeSpan.FromSeconds(1.00));
// if we've reach here and the thread is still alive
// interrupt it and tell it to quit
if (listenThread.IsAlive)
listenThread.Interrupt();
// wait until thread is done
listenThread.Join();
The problem is it never stops running! I stepped into the code and the listening thread is blocking because the Read() method is blocking. Read() just sits there and doesn't return. Hence, the thread never gets a chance to sleep that 1 millisecond and then get interrupted.
I'm sure if I let it sit long enough I'd get another packet and get a chance for the thread to sleep (if it's an active chatroom or a get a ping from the server). But I don't want to depend on that. If the user says shut down I want to shut it down!!
One alternative I found is to use the DataAvailable method of NetworkStream so that I could check it before I called StreamReader.Read(). This didn't work because it was undependable and I lost data when reading from packets from the server. (Because of that I wasn't able to login correctly, etc, etc)
Any ideas on how to shutdown this thread gracefully? I'd hate to call Abort() on the listening thread...
Really the only answer is to stop using Read and switch to using asynchronous operations (i.e. BeginRead). This is a harder model to work with, but means no thread is blocked (and you don't need to dedicate a thread—a very expensive resource—to each client even if the client is not sending any data).
By the way, using Thread.Sleep in concurrent code is a bad smell (in the Refactoring sense), it usually indicates deeper problems (in this case, should be doing asynchronous, non-blocking, operations).
Are you actually using System.IO.StreamReader and System.IO.StreamWriter to send and receive data from the socket? I wasn't aware this was possible. I've only ever used the Read() and Write() methods on the NetworkStream object returned by the TcpClient's GetStream() method.
Assuming this is possible, StreamReader returns -1 when the end of the stream is reached, not 0. So it looks to me like your Read() method is in an infinite loop.