I have a Multi-Client Single-Server application where client and server gets connected through sockets. Client and Server are in different machine.
In client Application, client socket gets connected to server and sends data periodically to server.
In server application server socket listens for client to connect. When a client is connected, new thread is created for client to receive data.
for example: 1 client = 1 thread created by server for receiving data. If its 10000 client, server creates 10000 threads. This seems not good and scalable too.
My Application is in Java.
Is there an alternate method for this problem?
Thanks in advance
This is a typical C10K problem. There are patterns to solve this, one examples is Reactor pattern
Java NIO is another way where the incoming request can be processed in non blocking way. See a reference implementation here
Yes, you should not need a separate thread for each client. There's a good tutorial here that explains how to use await to handle asynchronous socket communication. Once you receive data over the socket you can use a fixed number of threads. The tutorial also covers techniques to handle thousands of simultaneous communications.
Unfortunately given the complexity it's not possible to post the code here, so although link-only answers are frowned upon ...
I would say it's a perfect candidate for an Erlang/Elixir application. Whatsapp, RabbitMQ...
Erlang processes are cheap and fast to start, Erlang manages the scheduling for you so you don't have to think about the number of threads, CPUs or even machines, Erlang manages garbage collection for each process after you don't need it anymore.
Haskell is slow, Erlang is fast enough for most applications that are not doing heavy calculations and even then you can use it and hand off the heavy lifting to a C process.
What are you writing in?
Yes, you can use the Actor model, with e.g. Akka or Akka.net. This allows you to create millions of actors that run on e.g. 4 threads. Erlang is a programming language that implements the actor model natively.
However, actors and non-blocking code won't do you much good if you are relying on blocking library calls for backend services that you rely on, such as (the most prominent example in the JVM world) JDBC calls.
There is also a rather interesting approach that Haskell uses, called green threads. It means that the runtime threads are very lightweight and are dynamically mapped to OS threads. It also means that you get a certain amount of scalability "for free", with no need to write non-blocking IO code. It does however require a good IO manager in the runtime to schedule the IO operations efficiently, and GHC Haskell has had a substantial amount of work put into that in recent years.
Related
As I've done some more research into web server software, I've begun to question if Apache's thread/process based method is the way to go vs. the the asynchronous request handling provided by servers like Nginx a Lighttpd, which tend to scale better with heavier loads.
I understand there are many other differences between these latter two and Apache. My question is under what circumstances would I pick a thread/process based method over the asynchronous handling.
Are there any features/technologies that I can't use with an asynchronous method (or would function poorly/not as well)?
What situations would cause the performance of an asynchronous method to perform worse than a thread/process based approach? Are these common or rare cases, and how big is the difference?
Are there any other factors I should take into consideration when comparing the two? Keep in mind I'm focusing mainly on the thread/process based method vs. asynchronous, not any particular server software which happens to utilize one of these methods. These concerns might be difficulty of managing/debugging, security issues, etc.
This is old, but worth answering. Let's first start by saying how each model works.
In threaded, you have a request come in to a handler, the handler spawns a new OS thread to handle that request, and all work for that request happens in that thread until a response is sent and the thread is ended. This model supports as many concurrent requests as threads that your server can spawn (but threads can be somewhat heavyweight).
When doing async a request comes in to a handler but instead of creating a thread to deal with it, it adds the connection to what's known as an event loop. The event loop listens for data/state changes on the connection and fires callbacks each time "something" happens. Once the connection is added to the event loop, the handler immediately listens for new connections to add. This allows you to have many (sometimes 100K) concurrent connections at the same time.
Are there any features/technologies that I can't use with an asynchronous method (or would function poorly/not as well)?
Yes, when you're doing number crunching. The architecture of an async (or "evented") system is such that it is great at passing data around but not processing data. It can handle thousands of concurrent operations, but because it only runs on one OS thread, the callbacks it fires need to do as little as possible to get the most throughput. This is because if one of your callbacks does some number crunching that takes 5 seconds, your entire server is frozen for 5 seconds until that operation completes. The idea is to get data, send it to where it's going (database, API, etc) and send a response all with minimal processing.
Async is good for network I/O: passing data between multiple sources/destinations (and also user interfaces, but that's beyond this post).
What situations would cause the performance of an asynchronous method to perform worse than a thread/process based approach? Are these common or rare cases, and how big is the difference?
See above, but any time you're doing more CPU work than network I/O, you should switch to a threaded model. However, there are architecture workarounds...for instance, you could have an async app, and anytime it needs to do real work, it sends a job to a worker queue. However, if every request requires CPU processing then that architecture is overkill and you might as well just use a threaded server.
Are there any other factors I should take into consideration when comparing the two? Keep in mind I'm focusing mainly on the thread/process based method vs. asynchronous, not any particular server software which happens to utilize one of these methods. These concerns might be difficulty of managing/debugging, security issues, etc.
Programming in async is generally more complicated than threaded. That said, if you're not doing the programming yourself (ie you're choosing between nginx and apache) then I usually recommend you go async (nginx) because you'll generally be able to squeeze more juice out of your server that way. I'm always in favor of using as much async in the stack as possible.
That said, if you're programming an app and trying to decide whether to use a threaded or async model, you will have to take developer time into account. Unless you're using a language that has green threads over an event loop (like scheme), expect to tear your hair out quite a bit over rogue exceptions crashing your entire app and in general wrapping your head around CPS/using callbacks for everything. Futures/promises are your friend, but are only a bandaid to make async nicer.
TL;DR
Async, when used in a server, can squeeze (a lot) more concurrent operations than threading if you're doing network IO and nothing else.
If you're doing any kind of number crunching, either use a threaded app server or use an async app with a background queuing system.
Async is a lot harder to program in unless your language supports "fake" threading over it (ie green threads). Once you get past the initial hump you're fine, generally. If you don't have green threads, use promises.
If you have the choice between threaded and async as a component in your stack (apache vs nginx), and they provide the exact same features, slightly favor async. Don't just pick it because you think it will make everything 20x faster though.
Processes have several advantages compared to threads and async models related to security and reliability. Most websites don't need these particular advantages, but sometimes they're indispensable.
Security: you can run your worker processes in a sandbox, as a low privileged user, and handle only one request per worker process. This mitigates against some kinds of security vulnerabilities: even if an attacker takes over your entire worker process, as long as you sandboxed it tightly based on request metadata (i.e. it doesn't have write access to all your data), then it can't harm system stability or affect the responses made to requests.
Security #2: sometimes you need to sandbox untrusted code, or to enforce segregation between different code or different requests, and the only way to do this is with a separate one-shot process. (Think running user-provided code.)
Reliability: memory leaks and memory corruption are much less severe if you teardown and replace worker processes regularly (or for each request).
It's easy to enforce hard limits on CPU time, disk and network quota, etc. spent on handling a user request in a separate process. Even if the request-handling code goes into an infinite loop, the master process (or the OS) can enforce a timeout.
Im currently trying to decide on a design for a TCP server where the services that the server will provide consist of performing synchronous I/O (tons of DB queries - existing code!)
The system that this server will be part of has a couple of hundred clients that are typically all connected simultaniously and stay connected for several hours.
Basically all client requests are a result of human interaction, so the frequency is low but the response time should be as fast possible.
As I said, the service implementation must perform synchronous I/O, so a fully event based server is obviously out of the question.
Threads seem like a natural choice to serialize blocking IO, but you see advice to not use more threads than CPU cores.
Currently I'm leaning towards using a thread pool with a number of threads that is actually higher than the core count, since the threads will mostly be blocking anyway.
Would such a design be reasonable? What alternatives exist for a server with these requirements?
It seems, (yes again, from ohter posts, not my direct experience), that 200 .NET managed threads 'causes all kinds of problems'. 200 unmanaged threads is not a big problem on Windows/Linux.
200 persistent database connections, however, seems like a lot! You may wish to pool them, or use a threadpool for DB access with suitable inter-thread comms.
I tend to use the following as my standard threading model, but maybe it isn't such a great model. What other suggestions do people have or do they think this is set up well? This is not for a high performance internet server, though performance is sometimes pretty critical and in those cases I use asynchronous networking methods and reuse buffers, but it is the same model.
There is a gui thread to run the gui.
There is a backend thread that handles anything that is computationally intensive (basically anything the gui can hand off that isn't pretty quick to run) and also is in charge of parsing and acting on incoming messages or gui actions.
There is one or more networking threads that take care of breaking an outgoing send into peices if necessary, recieving packets from various sockets and reassembling them into messages.
There is an intermediary static class which serves as an intermediary between the networking and backend threads. It acts as a post office. Messages that need to go out are posted to it by backend threads and networking threads check the "outbox" of this class to find messages to send and post any incoming messages in a static "inbox" this class has (regardless of the socket they arrive from, though that information is posted with the incoming message) which the backend thread checks to find messages from other machines it should act on.
The gui / backend threading interface tends to be more ad hoc and should probably have its own post office like class or some alternative intermediary?
Any comments/suggestions on this threading setup?
My primary concern is that you don't really want to lock yourself into the idea that there can only be one back-end thread. My normal model is to use the MVC at first, make sure all the data structures I use aren't inherently unsafe for a threaded environment, avoid singletons, and then profile like crazy, splitting things out as I go while trying to minimize the number of condition variables I'm leveraging. For long asynchronous tasks, I prefer to spawn a new process, particularly if it's something that might want to let the OS give it a differing priority.
This architecture sounds like the classic Model-View-Controller architecture which is usually considered as good.
We've all read the benchmarks and know the facts - event-based asynchronous network servers are faster than their threaded counterparts. Think lighttpd or Zeus vs. Apache or IIS. Why is that?
I think event based vs thread based is not the question - it is a nonblocking Multiplexed I/O, Selectable sockets, solution vs thread pool solution.
In the first case you are handling all input that comes in regardless of what is using it- so there is no blocking on the reads- a single 'listener'. The single listener thread passes data to what can be worker threads of different types- rather than one for each connection. Again, no blocking on writing any of the data- so the data handler can just run with it separately. Because this solution is mostly IO reads/writes it doesn't occupy much CPU time- thus your application can take that to do whatever it wants.
In a thread pool solution you have individual threads handling each connection, so they have to share time to context switch in and out- each one 'listening'. In this solution the CPU + IO ops are in the same thread- which gets a time slice- so you end up waiting on IO ops to complete per thread (blocking) which could traditionally be done without using CPU time.
Google for non-blocking IO for more detail- and you can prob find some comparisons vs. thread pools too.
(if anyone can clarify these points, feel free)
Event-driven applications are not inherently faster.
From Why Events Are a Bad Idea (for High-Concurrency Servers):
We examine the claimed strengths of events over threads and show that the
weaknesses of threads are artifacts of specific threading implementations
and not inherent to the threading paradigm. As evidence, we present a
user-level thread package that scales to 100,000 threads and achieves
excellent performance in a web server.
This was in 2003. Surely the state of threading on modern OSs has improved since then.
Writing the core of an event-based server means re-inventing cooperative multitasking (Windows 3.1 style) in your code, most likely on an OS that already supports proper pre-emptive multitasking, and without the benefit of transparent context switching. This means that you have to manage state on the heap that would normally be implied by the instruction pointer or stored in a stack variable. (If your language has them, closures ease this pain significantly. Trying to do this in C is a lot less fun.)
This also means you gain all of the caveats cooperative multitasking implies. If one of your event handlers takes a while to run for any reason, it stalls that event thread. Totally unrelated requests lag. Even lengthy CPU-invensive operations have to be sent somewhere else to avoid this. When you're talking about the core of a high-concurrency server, 'lengthy operation' is a relative term, on the order of microseconds for a server expected to handle 100,000 requests per second. I hope the virtual memory system never has to pull pages from disk for you!
Getting good performance from an event-based architecture can be tricky, especially when you consider latency and not just throughput. (Of course, there are plenty of mistakes you can make with threads as well. Concurrency is still hard.)
A couple important questions for the author of a new server application:
How do threads perform on the platforms you intend to support today? Are they going to be your bottleneck?
If you're still stuck with a bad thread implementation: why is nobody fixing this?
It really depends what you're doing; event-based programming is certainly tricky for nontrivial applications. Being a web server is really a very trivial well understood problem and both event-driven and threaded models work pretty well on modern OSs.
Correctly developing more complex server applications in an event model is generally pretty tricky - threaded applications are much easier to write. This may be the deciding factor rather than performance.
It isn't about the threads really. It is about the way the threads are used to service requests. For something like lighttpd you have a single thread that services multiple connections via events. For older versions of apache you had a process per connection and the process woke up on incoming data so you ended up with a very large number when there were lots of requests. Now however with MPM apache is event based as well see apache MPM event.
I am looking for any strategies people use when implementing server applications that service client TCP (or UDP) requests: design patterns, implementation techniques, best practices, etc.
Let's assume for the purposes of this question that the requests are relatively long-lived (several minutes) and that the traffic is time sensitive, so no delays are acceptable in responding to messages. Also, we are both servicing requests from clients and making our own connections to other servers.
My platform is .NET, but since the underlying technology is the same regardless of platform, I'm interested to see answers for any language.
The modern approach is to make use of the operating system to multiplex many network sockets for you, freeing your application to only processing active connections with traffic.
Whenever you open a socket it's associated it with a selector. You use a single thread to poll that selector. Whenever data arrives, the selector will indicate the socket which is active, you hand off that operation to a child thread and continue polling.
This way you only need a thread for each concurrent operation. Sockets which are open but idle will not tie up a thread.
Using the select() and poll() methods
Building Highly Scalable Servers with Java NIO
A more sophosticated aproach would be to use IO Completion ports. (Windows)
With IO Completion ports you leave to the operating system to manage polling, which lets it potentially use very high level of optimization with NIC driver support.
Basically, you have a queue of network operations which is OS managed, and provide a callback function which is called when the operation completes. A bit like (Hard-drive) DMA but for network.
Len Holgate wrote an eccelent series on IO completion ports a few years ago on Codeproject:
http://www.codeproject.com/KB/IP/jbsocketserver2.aspx
And
I found an article on IO completion ports for .net (haven't read it though)
http://www.codeproject.com/KB/cs/managediocp.aspx
I would also say that it is easy to use completion ports compared to try and write a scaleable alternative. The problem is that they are only available on NT (2000, XP, Vista)
If you were using C++ and the Win32 directly then I'd suggest that you read up about overlapped I/O and I/O Completion ports. I have a free C++, IOCP, client/server framework with complete source code, see here for more details.
Since you're using .Net you should be looking at using the asynchronous socket methods so that you don't need have a thread for every connection; there are several links from this blog posting of mine that may be useful starting points: http://www.lenholgate.com/blog/2005/07/disappointing-net-sockets-article-in-msdn-magazine-this-month.html (some of the best links are in the comments to the original posting!)
G'day,
I'd start by looking at the metaphor you want to use for your thread framework.
Maybe "leader follower" where a thread is listening for incoming requests and when a new request comes in it does the work and the next thread in the pool starts listening for incoming requests.
Or thread pool where the same thread is always listening for incoming requests and then passing the requests over to the next available thread in the thread pool.
You might like to visit the Reactor section of the Ace Components to get some ideas.
HTH.
cheers,
Rob