I know that Vibe.D implementation is based on Fibers.
But I don't know how high load scenarios are handled by Vibe.D. Is it the scheduler in Vibe.D allocating fibers on multiple threads or just one thread for all fibers?
This consideration is very important because even with the high efficiency of Fibers a lot of CPU time is wasted is no more than one thread is used to attend all incoming requests.
Their front page says yes:
http://vibed.org/
this page has the details
http://vibed.org/features#multi-threading
Distributed processing of incoming connections
The HTTP server (as well as any other TCP based server) can be instructed to process incoming connections across the worker threads of the thread pool instead of in the main thread. For applications that don't need to share state across different connections in the process, this can increase the maximum number of requests per second linearly with the number of cores in the system. This feature is enabled using the HTTPServerOption.distribute or TCPListenOptions.distribute settings.
Related
I have an express backend application, which listens for http requests from a web application. This backend application is running on AWS ECS Fargate.
So my question is, does it makes sense use multithreading, worker-threads in Node.js, in this backend application? There are both CPU intensive and non-intensive functions in the endpoints. For instance should I just distribute any incoming request right away ,regardless of the intensity of the call, to other threads so that Main thread is never blocked? Or should I only use multithreading on intensive jobs etc.
Any suggestion, pros and cons on this topic is very much appreciated.
Questions like this ultimately only really get answered by profiling the system under representative load. This is because our guestimates about what takes CPU and how much are just rarely very good - this stuff has to be measured to really know.
That said, there are some general design guidelines you could consider:
If you have specific requests that you believe in advance are likely to be CPU heavy, then you may want to consider putting those into a queue that is served by a pool of threads.
If you have specific requests that you believe in advance are really just doing non-blocking I/O, then those don't need to get complicated at all by threading. The main thread can likely handle those just fine.
If only a relatively small fraction of your requests are CPU heavy, the simplest design may be to use the nodejs clustering module and just let it spread your requests out among CPUs figuring that this by itself (without any other major design changes) will get any CPU-bound requests off the main loop. If a high percentage of your requests are CPU-bound and you want to prioritize the non-CPU requests so they are always quick, then you'd probably be better with the thread pool that CPU-heavy requests are passed off to so the non-CPU heavy requests aren't held up by them.
So my question is, does it makes sense use multithreading, worker-threads in Node.js, in this backend application?
Yes, in some circumstances. But, when to add this extra layer of complication really depends upon the specific metrics of CPU usage in your requests.
For instance should I just distribute any incoming request right away ,regardless of the intensity of the call, to other threads so that Main thread is never blocked? Or should I only use multithreading on intensive jobs etc.
This depends upon how you want to prioritize different requests. If you want requests prioritized in a FIFO order, where each one gets to start in order based on when it arrived, regardless of what type of request, then yes you can just distribute all requests to threads. In fact, it's probably easier to just use the clustering module for this because that's what it does. If, on the other hand, you want non-CPU-heavy requests to always run quick and not have to wait behind CPU-heavy requests, then you may want to push only the CPU-heavy requests into a queue that is processed by a thread pool. This, then allows the main thread to work on the non-CPU-heavy requests immediately regardless of how many CPU-heavy requests are currently crunching away in the thread pool.
Which one of these are better to implement in a server? I was wondering about the tradeoffs between spawning a thread per request versus using a worker pool consisting of a fixed number of threads, and how the performance will vary as the number of users increases. These are some questions that came up to my mind what I started to think about the way I want to implement my server.
I have looked up online and all I seem to find are answers related to the question of "how does Node benefit from running in a multi core cpu?"
But. If you have a machine with just one core, you can only be running one process at any given time. (I am considering task scheduling here). And node uses a single threaded model.
My question: is there any scenario in which it makes sense to run multiple node processes in one core? And if the process is a web server that listens on a port, how can this ever work given that only one process can listen?
My question: is there any scenario in which it makes sense to run
multiple node processes in one core?
Yes, there are some scenarios. See details below.
And if the process is a web server that listens on a port, how can
this ever work given that only one process can listen?
The node.js clustering module creates a scenario where there is one listener on the desired port, but incoming requests are shared among all the clustered processes. More details to follow...
You can use the clustering module to run more than one process that are all configured to handle incoming requests on the same port. If you want to know how incoming requests are shared among the different clustered processes, you can read the explanation in this blog post. In a nutshell, there ends up being only one listener on the desired port and the incoming requests are shared among the various clustered processes.
As to whether you could benefit from more processes than you have cores, the answer is that it depends on what type of benefit you are looking for. If you have a properly written server with all async I/O, then adding more processes than cores will likely not improve your overall throughput (as measured by requests per second that your server could process).
But, if you have any CPU-heavy processing in your requests, having a few more processes may provide a bit fairer scheduling between simultaneous requests because the OS will "share" the CPU among each of your processes. This will likely slightly decrease overall throughput (because of the added overhead of task switching the CPU between processes), but it may make request processing more even when there are multiple requests to be processed together.
If your requests don't have much CPU-heavy processing and are really just waiting for I/O most of the time, then there's probably no benefit to adding more processes than cores.
So, it really depends what you want to optimize for and what your situation is.
I see from the server side, the benefit of NIO is the capability to manage multiple network connections with fewer thread comparing to the comparing to one thread per connection blocking IO.
However, if I have a IO client which connects to thousand of servers at the same time, can I just have similar approach to manage these connections IO using fewer threads. I tried the approach in Netty 4 multiple client and found it spawn a "Reader" thread for each channel it created.
So, my questions are:
1) what are the benefits using netty/NIO in the client side?
2) is it possible to manage multiple connections with fewer threads in the client side?
Thanks!
I have uploaded the code samples in github: https://github.com/hippoz/ogop-lseb
The sample server/client class is moc.ogop.ahsp.demo.nio.MultipleConnectionNioMain and moc.ogop.ahsp.demo.nio.NettyNioServerMain
Having lots of threads creates a context-switch problem in the kernel where lots more memory is being loaded and unloaded from each core as the kernel tries to reschedule the threads across the cores.
The benefit of NIO anywhere is performance. Thats pretty much the only reason we use it. Using Blocking IO is MUCH more simple. Using the worker model and NIO you can limit the number of threads (and potential computational time) the process uses. So if you have two workers and they go bonkers using 100% cpu time the whole system won't go to a crawl because you have 2-4 more cores available.
Have fun!
https://en.wikipedia.org/wiki/Context_switch
Why should I use non-blocking or blocking sockets?
I heard there is some limitation for a single thread to use network bandwidth? if this is true, is this the reason to use multithread programming to achieve the maximum bandwidth?
The reason to use multithreading for network tasks is that one thread might be waiting for a response from the remote server. Creating multiple threads can help you having at least one thread downloading from different requests at one time.
The usual reason for issuing more than one network request at a time, (either implicitly with user threads, or implicitly with kernel threads and asynchronous callbacks), is that the effects of network latency can be be minimised. Latency can have a large effect. A web connection, for example, needs a DNS lookup first, then a TCP 3-way connect, then some data transfer and finally a 4-way close. If the page size is small and the bandwidth large compared with the latency, most time is spent waiting for protocol exchanges.
So, if you are crawling multiple servers, a multithreaded design is hugely faster even on a single-core machine. If you are downloading a single video file from one server, not so much..