Let's assume I've got two Finagle services, as per below
val service1: Service[http.Request, http.Response] = ???
val service2: Service[http.Request, http.Response] = ???
and I run them both against different ports
val server1 = Http.serve(":9090", service1)
val server2 = Http.serve(":8080", service2)
would requests coming to these 2 servers be handled by different Netty worker/thread pools?
if not, is there a way to configure Finagle so they would be handled by different Netty worker/thread pools?
If you are using netty 3, you can set up execution handlers in a pipeline.
http://netty.io/3.5/api/org/jboss/netty/handler/execution/ExecutionHandler.html
Then transform the channel pipeline into a Finagle Codec, which offers builder methods to create a stack client or stack server, and just set the netty3 threadpool size to 0, finagle has been moving towards, hiding netty from it's users which can be a pain when you're trying to get fine grained control.
Hopefully that works, since I see no reason to do that when you can just run the servers in separate jvm instances.
Related
I was going through multiple docs and examples about the correct configuration of EventLoopGroup but couldn't decide what's best for my use case.
I am using Netty 4.1.86.Final on linux.
Use case:
We have a web server that accepts 50-100k connections (both http :80 and https :443) and handles 20-30k requests per second. It's server-to-server communication, where multiple servers maintain connections for a short duration (keep-alive few minutes to an hour) and then recycle (close and open a new connection) the connection. Every request typically takes 50-100ms to process. The processing involves parsing JSON payload and running some complex business logic and some async network call to internal servers as well (cache, db etc.).
Question:
What I understood so far is that there are a couple of options that I can try -
use same EventLoopGroup for boss and worker and use separate EventExecutorGroup in pipeline for my business logic like: pipeline.addLast(handlerExecutorGroup, "handler", new MyHttpEndServer());
use different EventLoopGroup for boss and worker (.group(ioThreadExecutors, workerThreadExecutors)) and use a very high number of threads for the worker EventLoopGroup (cpu * 8)
use 3 different EventLoopGroup for boss, worker and handler (cpu * 8 thread).
use same EventLoopGroup for boss and worker and use SO_REUSEPORT and bind multiple threads to the same port so that instead of 1 I/O thread per port, now we will have more I/O thread per port to accept the connection along with using separate handlerExecutorGroup in pipeline for my business logic same as option 1.
.group(ioThreadExecutors)
.option(UnixChannelOption.SO_REUSEPORT, true)
for (int i = 0; i < channelThreadCount; i++) {
ChannelFuture f = httpServer.bind(port).sync();
channelFutureSet.add(f);
}
which among these seems a better option for my use case? Or if there is some other better approach that I should try.
Note: I did the load test with all 4 options, but didn't notice much difference.
Using Ktor and Kotlin 1.5 to implement a REST service backed by Netty. A couple of things about this service:
"Work" takes non-trivial amount of time to complete.
A unique client endpoint sends multiple requests in parallel to this service.
There are only a handful of unique client endpoints.
The service is not scaling as expected. We ran a load test with parallel requests coming from a single client and we noticed that we only have two threads on the server actually processing the requests. It's not a resource starvation problem - there is plenty of network, memory, CPU, etc. and it doesn't matter how many requests we fire up in parallel - it's always two threads keeping busy, while the others are sitting idle.
Is there a parameter we can configure to increase the number of threads available to process requests for specific endpoints?
Netty use what is called Non-blocking IO model (http://tutorials.jenkov.com/java-concurrency/single-threaded-concurrency.html).
In this case you have only a single thread and it can handle a lot of sub-processes in parallel, as long as you follow best practices (not blocking the main thread event loop).
You might need to check the following configuration options for Netty https://ktor.io/docs/engines.html#configure-engine
connectionGroupSize = x
workerGroupSize = y
callGroupSize = z
Default values usually are set rather low and tweaking them could be useful for the time-consuming 'work'. The exact values might vary depending on the available resources.
What is the different use cases of Java NIO transport connector vs PoolConnectionFactory in ActiveMQ. Both serves the pool of connections.I want to use thousand of clients connect to the broker and maintain a seperate queue for each client. Where is is use case for both of this in the scenario?
The NIO Transport connector is a server side incoming connection API that utilizes a selector based event loop to share the load of multiple active connections where normally on the normal transport connector a single thread is created per connection to process IO leading to higher thread counts when large numbers of connections are active.
The PooledConnectionFactory is a client side device that provides a pool of one or more open connections that can be used by application code to reduce the number of connection create / destroy events thereby leading to faster client side code in some cases and lower overhead on the remote broker as it would not need to process connection create / destroy events from an application whose model causes this sort of behavior. Depending on how you've coded your application or what API layering you have such as Camel or Spring etc a pool may or may not be of benefit.
The two things are not related and should not be equated with one another.
NIO transport uses on low level the selector which is much more performant then Pool connectionfactory.
It means it get notification if any new data is ready while Pool wait for each Connection. For your use case i would strongly suggest NIO Connector
I have a scenario where i have server which is sending tasks for processing to different workers. I tried doing it with ZeroMQ in scala by setting up the server as a DEALER and multiple workers as DEALERS too connected to the server. But it wasn't efficient. As this was using round robin scheduling for sending the tasks. I want to do event driven scheduling of tasks without blocking any thread and i have to be sure to send tasks to those workers which are free. So how can i do that using ZeroMQ?
Your scenario requires more than a .connect() of the ZeroMQ basic elements & put these into a more abstract communication pattern, that meets both your application needs and also serves well for the load-balancing and failure-resilience aspects of the real-world use.
You need to also propagate <state> of the [worker]-client back to the work-units' [dispatcher]-server
This may sound complicated for the first few things one tries to code with ZeroMQ, but if you at least jump to the page 265 of the [Code Connected, Volume 1][asPdf->]1, if it were not the case of reading step-by-step there
The fastest-ever learning-curve would be to have first an un-exposed view on the Fig.60 Republishing Updates and Fig.62 HA Clone Server pair for a possible High-availability approach and then go back to the roots, elements and details.
My understanding is that in Tomcat, each request will take up one Java/(and thus OS) thread.
Imagine I have an app with lots of long-running requests (eg a poker game with multiple players,) that involves in-game chat, and AJAX long-polling etc.
Is there a way to change the tomcat configuration/architecture for my webapp so that I'm not using a thread for each request but 'intercept' the request and response so they can be processed as part of a queue?
I think you're right about tomcat likes to handle each request in its own thread. This could be problematic for several concurrent threads. I have the following suggestions:
Configure maxThreads and acceptCount attributes of the Connector elements in server.xml. In this way you limit the number of threads that can get spawned to a threshold. Once that limit is reached, requests get queued. The acceptCount attribute is to set this queue size. Simplest to implement but not a good long term solution
Configure multiple Connector elements in server.xml and make them share a threadpool by adding an Executor element in server.xml. You probably want to point tomcat to your own implementation of Executor interface.
If you want finer grain control no how requests are serviced, consider implementing your own connector. The 'protocol' attribute of the Connector element in server.xml should point to your new connector. I have done this to add a custom SSL connector and this works great.
Would you reduce this problem to a general requirement to make tomcat more scalable in terms of the number of requests/connections? The generic solution to that would be configuring a loadbalancer to handle multiple instances of tomcat.