According to https://nodejs.org/api/cluster.html#cluster_cluster, one should run the same number of Node.js processes in parallel as the number of cores on the machine.
The supposed reasoning behind this is that Node.js is single threaded.
However, is this really true? Sure the JavaScript code and the event loop run on one thread but Node also has a worker thread pool. The default number of thread in this pool is 4. So why does it make sense to run one Node process per core?
This article has an extension review on the threading mechanism of node.js, worth a read.
In short, the main point is in plain node.js only a few function calls uses thread pool (DNS and FS calls). Your call mostly runs on the event loop only. So for example if you wrote a web app that each request takes 100ms synchronously, you are bound to 10req/s. Thread pool won't be involved. And to increase throughput on a multicore system is to use other cores.
Then it comes asynchronous or callback functions. While it does give you a sense of parallelization, what really happens is it waits for the async code to finish in background so that event loop can work on another function call. Afterwards, the callback codes still has to run in event loop, therefore all your written code are still ran in the one and only one event loop, thus won't be able to harness multi-core systems' power.
The said document clearly states that Node is single-threaded:
A single instance of Node.js runs in a single thread. To take advantage of multi-core systems, the user will sometimes want to launch a cluster of Node.js processes to handle the load.
This way Node process has a single thread, unless new threads are created with respective APIs like child_process, cluster, native add-ons or several built-in modules that use libuv treadpool:
Asynchronous system APIs are used by Node.js whenever possible, but where they do not exist, libuv's threadpool is used to create asynchronous node APIs based on synchronous system APIs. Node.js APIs that use the threadpool are:
all fs APIs, other than the file watcher APIs and those that are
explicitly synchronous
crypto.pbkdf2()
crypto.randomBytes(), unless it is used without a callback
crypto.randomFill()
dns.lookup()
all zlib APIs, other than those that are explicitly synchronous
A single thread uses 1 CPU core, in order to use available resources to the fullest extent and utilize multicore CPU, there should be several threads, the number of cores is used as a rule of thumb.
If cluster processes occupy 100% CPU and it's known there are other threads or external processes (database service) that would fight over CPU cores with cluster processes, the number of cluster processes can be decreased.
Related
I would like to know how to take full advantage of the Worker class in nodejs' worker_threads, specifically, on a 1 or 2 cpu system, do tasks get scheduled better than if I had just blocked in a for-loop in a regular nodejs program (without making use of any worker api)? Are they just delegated to the OS?
Also, can I block inside a Worker? I assumed that's what they are for.
How does Node.js schedule Workers on a limited resource system
Nodejs worker threads use underlying OS threads so worker threads are scheduled by the OS, not by nodejs. If you have more active threads than you have CPU cores, then the underlying OS will time slice (e.g. share) the cores among the active threads. In general, you shouldn't write a blocked for loop in the main nodejs event loop thread, but for more specifics on that part of the question, we would need to see the actual code you're talking about, what the precise context is and what the alternatives are.
Also, can I block inside a Worker? I assumed that's what they are for.
Yes, you can. It will not have any adverse effect on the main event loop thread. You will, of course, not be able to do anything else in the worker thread while it is blocked. Also, you may want to know that worker threads in nodejs are not lightweight things (in terms of memory usage). Each one comes with a separate V8 interpreter environment. So, in a low resource system, you will have to very carefully plan out your memory usage as nodejs + multiple worker threads do not make for low memory usage.
Keep in mind that each V8 interpreter instance also creates its own thread pool for the libuv engine to use for things like crypto operations and file operations to allow blocking OS system calls to present an asynchronous interface to the JS engine. So, in addition to your Javascript threads, there are also these libuv threads involved in some nodejs APIs.
Let's assume i have a nodejs serverProgram with one api and it does some manipulations on the video file, sent via the http request.
const saveVideoFile=(req,res)=>{
processAndSaveVideoFile(); // can run for minimum of 10 minutes
res.send({status: "video is being processed"})
}
i decided to to make use of a workerThread to do this processing as my machine has 3 cores (core1,core2,core3) and there is no hyperthreading enabled here
Assume that my nodejs program is running on core1. When i fire up a single workerThread, will the workerThread run on core2/core3 or core1?
i read that workerThread is not the same as childProcess. ChildProcess will fork a new process which will facilitate the childProcess to choose from available free cores (core2 or core3).
i read that workerThread shares memory with the mainThread. Let's assume that i create 2 workerThreads (wt1,wt2). Will my nodejs program, wt1, wt2 run on the same core i.e core1 ?
Also, in nodejs we have eventloop (mainthread) and otherThreads doing the background operations i.e I/O. is it correct to assume that all of these are utilizing the resources available in a single core (core1). if this is the case, is creating and using additional workerThread's an overkill on the nodejs server?
Below is an excerpt from this blog
We can run things in parallel in Node.js. However, we need not to
create threads. The operating system and the virtual machine
collectively run the I/O in parallel and the JS code then runs in a
single thread when it is time to send the data back to the JavaScript
code.
i keep reading this same information about nodejs in many articles and video presentations. But what i do not understand is this,
The operating system and the virtual machine collectively run the I/O in parallel
How can the operating system run the I/O requests from nodejs program in parallel without using any of the childProcess or threads spawned from nodejs? if those I/O requests from nodejs program is running in parallel, does it mean that all 3 cores (core1,core2,core3) will be utilized?
There are lot of contents on nodejs, but it doesn't clear doubts related to my above questions. if you have idea on how these things actually work, please share the detail.
A worker thread in node.js is an actual OS thread running in a different instance of V8. As such, it's totally up to the operating system to decide how to allocate it among available CPU cores. If there are cores with available time, then it will not generally be run on the same core as the main nodejs thread when that thread is busy because the OS will allocate busy threads across the various cores.
But, again this is entirely up to the OS and is not something that nodejs controls and the exact strategy for which cores are used will vary by OS. But, in all modern operating systems, the design goal is that available cores are used for threads that are currently executing. Now, if there are more threads active at once than there are cores, the threads will be time-sliced and all the cores will be active.
Also, in nodejs we have eventloop (mainthread) and otherThreads doing the background operations i.e I/O. is it correct to assume that all of these are utilizing the resources available in a single core (core1). if this is the case, is creating and using additional workerThread's an overkill on the nodejs server?
No, it is not correct to assume those threads all use the same core.
A workerThread in nodejs has its own event loop. For the most part, it does not share memory. In fact, if you want to share memory, you have to very specifically allocated SharedMemory and pass that to the workerThread.
Is it overkill? Well, it depends upon what you're doing. There are very useful things to do with workerThreads and there are things that they would not be necessary for.
The operating system and the virtual machine collectively run the I/O in parallel
I/O in node.js is either asynchronous at the OS level (such as networking) or run in separate threads (such as disk I/O). That means it runs separately from the main thread in node.js that runs your Javascript and can run in parallel with it, synchronizing only at the completion of an event. "Parallel" in this case means that both make progress at the same time. If there are multiple cores, then they can truly be running at exactly the same time. If there was only one core, then the OS will timeslice between the various threads and they will be both make progress (in an interleaved fashion that will seem to be parallel, but really they are taking turns).
How can the operating system run the I/O requests from nodejs program in parallel without using any of the childProcess or threads spawned from nodejs? if those I/O requests from nodejs program is running in parallel, does it mean that all 3 cores (core1,core2,core3) will be utilized?
The OS has its own threads for managing things like a network interface or a disk interface. The job of those threads is to interface with the hardware and bring data to an appropriate application or take data from the application and send it to the hardware. These are OS-level threads that exists independent of node.js. Yes, other cores can be used by those OS-level threads. It is important to realize that many operations such as networking are inherently non-blocking. Thus, if you're waiting for some data to arrive on a network interface, you don't need to have a thread doing something the whole time.
I want to add that it appears in your questions that you've combined questions about a several different things. Mentioned in your questions are:
Worker Threads
Internal node.js threads
Operating system threads
These are all different things.
A worker thread is a new thread you can start to run specific pieces of Javascript in another thread so you can have more than one Javascript thread running at the same time. In node.js, this is done by creating a whole new instance of V8, setting up a whole new global environment and loaded modules environment and using almost entirely separate memory.
Internal node.js threads are used by node.js as part of implementing its event loop and its standard library. Specifically, disk I/O and some crypto operations are run in internal native threads and they communicate with your Javascript via events/callbacks through the event loop.
Operating system threads are threads that the OS uses to implement it's own system APIs. Since the OS is responsible for lots of things, these threads ca have many different uses. Depending upon native implementations, they may be used to facilitate things like disk I/O or networking I/O. These threads are the responsibility of the OS to create and use and are not directly controlled by node.js.
Some additional questions asked in comments:
what is the difference b/w workerThread & childProcess concept in nodejs? is childProcess = workerThread without sharedMemory ?
A child process can be any type of program - it does not have to be a node.js program. A worker thread is node.js code.
A worker thread can share memory if sharedMemory is specifically allocated and shared with the worker thread and if it is carefully managed for concurrency issues.
It is more efficient to copy memory back and forth between worker thread and main thread than with child process.
If main program exits, worker threads will exit. If main program exits, child process can be configured to exit or to continue.
If worker thread calls process.exit(), the main thread will exit too. If child program exits, it cannot cause main program to exit without main program's cooperation.
how nodejs is able to magically interact with the os level thread without nodejs itself creating any threads?, i need additional details on this, your explanation is the common one present in most places including the blog i shared?
nodejs just calls an OS API. It's the OS API that manages communicating with its own threads (if threads are needed for that specific OS API). How it does that communication internally is implementation dependent and will vary by OS. It will even vary by OS which OS APIs use threads and which don't.
If I use async functions, or functions with callbacks like the native fs module, http etc, will they run by default across all cpu cores?
Or the entire thing will just use 1 core?
Some asynchronous operations in node.js (such as file I/O in the fs module) will use additional threads within the node.js process via a thread pool in libuv. It would depend upon the size of your thread pool and what types of operations and upon your host OS for how many additional CPUs will be engaged. It does not necessarily help overall throughput to engage many CPUs on file I/O that is all going through the same disk since reading/writing is often bottlenecked by the position of the read/write head on the disk anyway.
Some asynchronous operations such as networking (like the http module) are non-blocking and asynchronous by nature and do not do their networking with threads or trigger any meaningful use of additional CPUs.
None of this will run your own Javascript in multiple threads since Javascript itself all executes in one thread.
To fully engage multiple CPUs, you can:
Put some of your own Javascript into the new nodejs Worker Threads and communicate back to the main node.js thread via messaging.
Fire up your own node.js child processes to do work in those child processes and communicate back results using one of the many interprocess communications options.
Use node.js clustering so that incoming requests can be split among available queues. This requires making sure any server state is shareable among all the clustered processes (typically stored in some database that all processes can access). This will allow separate requests to use separate CPUs - it won't help a single request use more CPUs. You would need to use #1 and/or #2 for that.
Node.js maintains an event loop but then it also has by default four threads for the complicated requests. How this is single threaded when there are more threads available in the thread pool?
Also, the threads assigned by the event loop for the complicated task are the dedicated threads then how it's different from other multithreading concepts?
In the context to which you're referring, "single threaded" means that your Javascript runs as a single thread. No two pieces of Javascript are ever running at the same time either literally or time sliced (note: as of 2020 node.js does now have WorkerThreads, but those are something different from this original discussion). This massively simplifies Javascript development because there is no need to do thread synchronization for Javascript variables which are shared between different pieces of Javascript because only one piece of Javascript can ever be running at the same time.
All that said, node.js does use threads internal to its implementation. The default four threads you mention are used in a thread pool for disk I/O. Because disk I/O is normally a synchronous operation at the OS level that blocks the calling thread and node.js has a design where all I/O operations should be offered as asynchronous operations, the node.js designers decided to fulfill the asynchronous interface by using a pool of threads in order to implement (in native code), the fs module disk I/O interface (yes there are non-blocking disk I/O operations in some operating systems, but the node.js designers decided not to use them). This all happens under the covers in native code and does not affect the fact that your Javascript runs only in a single thread.
Here's a summary of how a disk I/O call works in node.js. Let's assume there's already an open file handle.
Javascript code calls fs.write() on an existing file handle.
fs module packages the arguments to the function and then calls native code.
Native code gets a thread from the thread pool and initiates the OS call to write data to that file
Native code returns from the function
fs module returns from the fs.write() call
Javascript continues to execute (whatever statements came after the fs.write() call
Some time later the native code fs.write() call on a thread finishes. It obtains a mutex protecting the event loop and inserts an event in the event queue.
When the Javascript engine is done executing whatever stream of Javascript it was running, it checks the event queue to see if there are any other events to run.
When it finds an event in the event queue, it removes it from the event queue and executes the callback associated with that event, starting a new stream of running Javascript.
Because a new event is never acted upon until the current stream of Javascript is done executing, this is where Javascript gets is event-driven, single threaded nature even though native code threads may be used to implement some library functions. Those threads are used to make a blocking operation into a non-blocking operation, but do not affect the single threaded-ness of Javascript execution itself.
The key here is that node.js is event driven. Every new operation that triggers some Javascript to run is serialized through the event queue and the next event is not serviced until the current stream of Javascript has finished executing.
In the node.js architecture the only way to get two pieces of Javascript to run independently and at the same time is to use a separate node.js process for each. Then, they will run as two completely separate operations and the OS will manage them separately. If your computer has at least two cores, then they can literally run at the same time, each on their own core. If your computer has only one core, they will essentially be in their own process thread and the OS will time slice them (sharing the one CPU between them).
I will tell it in a clear and simple way and clear the confusion :
Node Event Loop is SINGLE-THREADED But THE Other processes are not.
The confusion came from c++, which Node uses underline ( NodeJs is about 30% js + 70% c++ ).So, By default, The JS part of NodeJs is single-threaded BUT it uses a thread pool of c++. So, We have a single thread JS which is the event loop of NodeJs + 4 threads of c++ if needed for asynchronous I/O operations.
It is also important to know that The event loop is like a traffic organizer, Every request go through the loop ( which is single-thread ) then the loop organizes them to the pool threads if I/O processes are needed, so if you have a high computational app that does like heavy lifting image-processing, video-editing, audio-processing or 3d-graphics ..etc, which is not needed for most apps,So NodeJs will be a bottleneck for that high load computational app and the traffic organizer will be unhappy.
While NodeJS shine for I/O bound apps ( most apps ) Like apps dealing with databases and filesystem.
Again: By default, NodeJs uses a 4 thread pool (PLUS one thread for the event loop itself ). so by default (total of 5) because of the underlying c++ system.
As a general idea, The CPU could contain one or more cores, it depends on your server(money).
Each core could have threads. Watch your activity Monitor discover how many threads you are using.
Each process has multiple threads.
The multi-threading of Node is due to that node depends on V8 and libuv ( C Library ).
So Long story short:-
Node is single-threaded for the event loop itself but there are many operations that are done outside the event Loop, Like crypto and file system (fs ). if you have two calls for crypto then each of them will reach each THREAD ( imagine 3 calls to crypto and 1 for fs, These calls will be distributed one for each thread from the 4 thread pool )
Finally: It is very easy to increase the default number of threads of the C-Library libuv thread pool which is 4 by default by changing the value of process.env.uv_threadpool_size. and also you could use clustering ( PM2 recommended ) to like clone the event-loop, like have multiple event-loops in case the single-threaded one is not enough for your high load app.
So nobody illustrates that thread pool is a c++ thing that’s nodeJs control mostly not the developer, which still asking How it’s single-thread while having a thread-pool !!
Hope that simplifies that advanced topic.
By default, the execution of your JavaScript code runs on a single thread.
However, node.js tries to make most long-running calls async. For some that just involves doing async OS calls, but for some others node.js will execute the call itself on a secondary thread, while continuing to run other JS code. Once the async call terminated, the Js callback or Promise handler will run.
For async processing, Node.js was created explicitly as an experiment. It is believed that more performance and scalability can be achieved by doing async processing on a single thread under typical web loads than the typical thread based implementation.
As far as I know, all IO requests and other asynchronous tasks are done by libuv in nodejs.
I want to know if libuv is using threading. If it is, is it using all available core or not?
First of all, what is libuv. As mentioned in the documentation, it's a multi-platform support library with a focus on asynchronous I/O.
libuv doesn't use thread for asynchronous tasks, but for those that aren't asynchronous by nature.
As an example, it doesn't use threads to deal with sockets, it uses threads to make synchronous fs calls asynchronous.
When threads are involved, libuv uses a thread pool the size of which you can change at compile-time using UV_THREADPOOL_SIZE.
node.js is provided with a precompiled version of libuv and thus a fixed UV_THREADPOOL_SIZE parameter.
It goes without saying that it has nothing to do with the number of cores of your chip.
I'm tempted to affirm that you can safely ignore the topic, for libuv and thus node.js don't use threads intensively for their purposes (unless you are using them in a really perverse way or if you are running an high number of libuv work requests).
Feel free to run an instance of node.js per core if you need as most of the users do.
The design overview section of libuv is also clear enough about this point:
The I/O (or event) loop is the central part of libuv. It establishes the content for all I/O operations, and it’s meant to be tied to a single thread. One can run multiple event loops as long as each runs in a different thread.
The libuv module has a responsibility that is relevant for some particular functions in the standard library. for SOME standard library function calls, the node C++ side and libuv decide to do expensive calculations outside of the event loop entirely.They make something called a thread pool that thread pool is a series of four threads that can be used for running computationally intensive tasks such as hashing functions.
By default libuv creates four threads in this thread pool. Thread Pool in the picture is organized by the Libuv So that means that in addition to that thread used for the event loop there are four other threads that can be used to offload expensive calculations that need to occur inside of our application. Many of the functions include in the node standard library will automatically make use of this thread pool.
Network (Network IO) is responsible for api requests, File system (File IO) is fs module. so node.js single thread delegates those heavy work to the libuv
If you have too many function calls, It will use all of the cores. CPU cores do not actually speed up the processing function calls, they just allow for some amount of concurrency inside of the work that you are doing.
From here:
A single instance of Node.js runs in a single thread. To take
advantage of multi-core systems the user will sometimes want to launch
a cluster of Node.js processes to handle the load.
The cluster module allows easy creation of child processes that all
share server ports.
Multiple processes could be better than multithreading in some cases. Some people even think theads are evil. Maybe node.js is designed in such a way to take advantage of processes better than threads.