I understand the differences between sync and async server however I am wondering if have these 2 cases, which one would be more performant async or sync?
Sync: Write call will be blocking till message is ready to be sent on the wire from the internal completion queue.
Async : write call immediately returns and we need to wait on completion queue.
In Sync server what if we add queue which basically populated for evry Write call and other thread draining it and doing stream.write then performance will be same?
Sync: gRPC internally creats threadpool with threads equal to number of CPUs
Async: threading is upto implementation. So if for each client if we create separate thread and completion queue, would the performance wil be same for sync and async?
It's difficult to compare performance theoretically. As a rule of thumb, if your choices are between letting gRPC handle concurrent calls internally in a way it's designed to handle, vs managing gRPC call concurrency yourself with the sync API, chances are gRPC internals will be better tuned for performance than you can manage yourself. There may be exceptions to that advice, depending on many variables ... for example, if the server is doing something very fast and inexpensive and the messages are small, the sync API might be fine.
In the end, benchmarks are your friends.
gRPC performance advice: https://grpc.io/docs/guides/performance/#c
The official gRPC benchmarks (under development): https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5180705743044608. The tests that underly those benchmarks may be informative in your design choices.
Related
I had asked in an interview, are there any cases that may force you to use blocking code in a node.js server?
my answer was: I didn't ever need that in any project but I think it may be useful in some tasks that need much CPU processing like Some Image Processing or video generation.
so experts, can you correct that for me, is there any case that a blocking code would be a must?
First off, you have to distinguish between the different types of programs. A server that you expect to be responsive to many different incoming requests has very different needs than a single user program you write to do some file management or fetch some content and insert it in a database.
So, if you're not a multi-user server, you may be able to use synchronous I/O everywhere it's offered (most specifically for file access). For example, I have several scripts that do file management on my hard disk. These scripts don't have any server component and are run automatically in the middle of the night to trim backups, trim log files, etc... These scripts are perfectly OK to use synchronous I/O for pretty much anything.
If, on the other hand, you are a mutli-user server and you need to be responsive to incoming requests that can arrive at any time, then the only two times you can/should use blocking I/O or blocking crypto are at startup time or in some sort of shut-down scenario. For all other code in service of incoming requests, you have to use non-blocking, asynchronous I/O to avoid locking up your server during a request and making it non-responsive to new incoming requests.
If you have time consuming, CPU-intensive operations such as image processing or video generation, then you will want to offload that processing to another thread or process so that your main server thread is not blocked doing that processing. A typical way of handling that would be to create a worker pool of N processes/threads that can be sent jobs to crunch on. Then, you keep your most CPU-intensive work out of the main nodejs thread, allowing it to stay responsive to incoming requests.
so experts, can you correct that for me, is there any case that a blocking code would be a must?
Synchronous (blocking) I/O vastly simplifies server startup as you can do things like read configurations synchronously. You could write that code asynchronously, but then your module interface often end up having to return promises that indicate when it's actually ready and done with its initialization which complicates using the module.
For example, require() is synchronous and this really, really helps make initialization a lot simpler.
The only place I know of in a server where blocking code might be required is if you're trying to write something to disk right before your program exits when it's already in the process of exiting. You get notified of an exit event and if you try to use asynchronous file I/O, then your program will exit before the I/O finishes. In that case, you may need to use synchronous file I/O (which is not a problem in that circumstance).
I am trying to understanding basics of Redis.
One that that keep coming everywhere is, Redis is single threaded that makes things atomic.But I am unable to imagine how this is working internally.I have below doubt.
Don't we design a server Single thread if it is IO bound application(like Node.js),where thread got free for another request after initiating IO operation and return data to client once IO operation is finished(providing concurrency). But in case of redis all data are available in Main Memory,We are not going to do IO operation at all.So then why Redis is single threaded?What will happen if first request is taking to much time,remaining request will have to keep waiting?
TL;DR: Single thread makes redis simpler, and redis is still IO bound.
Memory is I/O. Redis is still I/O bound. When redis is under heavy load and reaches maximum requests per second it is usually starved for network bandwidth or memory bandwidth, and is usually not using much of the CPU. There are certain commands for which this won't be true, but for most use cases redis will be severely I/O bound by network or memory.
Unless memory and network speeds suddenly get orders of magnitude faster, being single threaded is usually not an issue. If you need to scale beyond one or a few threads (ie: master<->slave<->slave setup) you are already looking at Redis Cluster. In that case you can set up a cluster instance per CPU core if you are somehow CPU starved and want to maximize the number of threads.
I am not very familiar with redis source or internals, but I can see how using a single thread makes it easy to implement lockless atomic actions. Threads would make this more complex and doesn't appear to offer large advantages since redis is not CPU bound. Implementing concurrency at a level above a redis instance seems like a good solution, and is what Redis Sentinel and Redis Cluster help with.
What happens to other requests when redis takes a long time?
Those other requests will block while redis completes the long request. If needed, you can test this using the client-pause command.
The correct answer is Carl's, of course. However.
In Redis v4 we're seeing the beginning of a shift from being mostly single threaded to selectively and carefully multi threaded. Modules and thread-safe contexts are one example of that. Another two are the new UNLINK command and ASYNC mode for FLUSHDB/FLUSHALL. Future plans are to offload more work that's currently being done by the main event loop (e.g. IO-bound tasks) to worker threads.
From redis website
Redis uses a mostly single threaded design. This means that a single
process serves all the client requests, using a technique called
multiplexing. This means that Redis can serve a single request in
every given moment, so all the requests are served sequentially. This
is very similar to how Node.js works as well. However, both products
are not often perceived as being slow. This is caused in part by the
small amount of time to complete a single request, but primarily
because these products are designed to not block on system calls, such
as reading data from or writing data to a socket.
I said that Redis is mostly single threaded since actually from Redis 2.4 we use threads in Redis in order to perform some slow I/O operations in the background, mainly related to disk I/O, but this does not change the fact that Redis serves all the requests using a single thread.
Memory is no I/O operation
So, we're designing a new micro-service architecture. One of the biggest challenge is internal communication. For communication, in which response is required, we're using REST APIs. But for the services, which just wants to relay the information, this API processing is unnecessary overhead.
One way is to use Queue. The service1 will push the information into a queue, and service2 can consume from there. Therefore service1 don't have to wait (unlike an API call). (If there is any error in processing the information, service2 can either inform via a callback URL to service1, or any other way; this is not a concern at this point [1])
Now with Queue, there are two options, one is RabbitMQ. And another is AWS SQS. With RabbitMQ I've to worry about server-setup and everything (which can be done, but wants to avoid it). So after a POC of SQS, it seems like a good option, but the thing is SQS internally uses Rest APIs to communicate with AWS servers, at both point (service1 when pushing, service2 when consuming), there will be overhead. So now I'm thinking why not do it in NodeJS, service1 will hit the service2 with information. Service2 will respond immediately, acknowledging that it has received the information, if there is any error then [1].
Now Pros/Cons I could summarise is -
RabbitMQ
Easy to implement
In case of unavailability of receiver, sender won't have to worry about retrying.
Server Setup Cost + Maintenance (+ Tuning)
SQS
Easiest to implement
Pricing
Constant Polling for Messages
Overhead at push/receive
Non-blocking APIs
No 3rd medium required for communication
Service1 has to manage retry mechanism
Relative to SQS, less overhead
Information will be in-memory until processed
So to some up, my question is, is it a good idea to go with Non-blocking APIs? Or which one will be better approach, in terms of making system scalable.
Edit -
Can a PubSub provider like PubNub or Pusher can be used instead of Queue?
SQS uses XML over http, RabbitMQ uses AMQP, all protocols have overhead. Serializing/deserializing has a cost. Both the amazon SQS and AMQP are very efficient. I would exclude these "overheads" from your calculations, and instead focus on your other requirements.
One of the big advantages of using a queue is the handling of surge activity. If you get 100K hits, and need to send 100K messages, and you try to implement this as inter-service calls (non-blocking or otherwise), you will hit real limits on the scalability of your system (from a port count if nothing else). If you instead put 100K messages on a queue, those messages can be processed basically at the remote server's "leisure".
Additionally, as you have mentioned above, queues have a persistence that is much more difficult to implement on your own. If you data is not critical, this is not a big concern, but if this data is of higher importance, you really want something that pushes to a persistent store (Like SQS, or Rabbit persistent queues)...
I am late here but off late I have started working with NON Blocking I/O and see a great benefit of NIO especially when you are calling external services which cannot be given access to a message queue. Using a fixed connection pool will ensure that 100K problem is handled with non blocking I/O and too many connections are not created.
While calling internal services a message queue is prefered, but lets say you do not have that option, you can leverage NIO with a retry mechanism and connection pooling to given you the same scalability message queues would give. This is assuming that receivers are able to handle the load of NIO calls.
I've gone through some introductory articles on Node.js and Event Loop and one thing is not clear - if there are multiple concurrent requests then are the responses always sequential in the order the request was made? Say if 20 requests did complete simultaneously then will the 20th response have to wait for the other 19 to be cleared (responded back to the client) ?
Update: What I was wondering is whether this is similar to how multiple setTimeouts get queued?
node.js runs Javascript as single threaded. Thus, only one piece of Javascript is running at any given time.
But, almost all I/O (e.g. networking, file access, etc...) is asynchronous and non-blocking. So, if 20 requests are made of your server in a very short period of time, the first request to reach the server will start executing it's request handler and the other requests will be queued. But, as soon as the first request hits an asynchronous operation (such as reading from the local file system), that request will be suspended while the non-blocking asynchronous I/O is taking place and the next request in line will start to run.
This second request will then run until it either finishes or until it also hits a piece of asynchronous I/O. When that second request is waiting on the async I/O, then another request will get to run. The system scheduler will determine if the next operation is the completion of the async I/O request from the first request or if it will start the third request that was waiting in the queue.
The various requests will continue this way until all are done. Multiple requests may be "in-flight" at the same time (meaning they've been started, but have not completed yet), but only one is ever actually executing code at any given moment.
This is sometimes referred to as cooperative tasking. There is no pre-emptive multi-tasking among the different requests where each automatically gets a time slice of the host CPU. But, any time a request hits an asynchronous I/O operation, then that tells the scheduler that other requests waiting to run can run.
This is all managed from an event queue in node.js. A piece of Javascript runs until it completes. If it makes an asynchronous I/O request and then completes, then another piece of Javascript that is also waiting to run can start to run. When it is done, the JS engine pulls the next item out of the event queue and runs it. That might be a new incoming request or it might be the completion of some asynchronous I/O operation on some other request.
The advantages of this type of system are:
It scales really well, particularly for I/O bound server operations, because you can have many requests "in-flight" at the same time with only a single Javascript thread. The cooperative tasking is very lightweight and fast.
Programming a system like this has far fewer "race conditions" to watch out for because no two pieces of Javascript are ever running at the actual same time. This means you can often share state between requests without ever having to use mutexes (like you would in a multi-thread environment). Since thread-safe bugs are often very difficult to avoid and to test for, it's a major advantage to eliminate these types of bugs.
The cooperative model is conceptually simple and easier to learn and to program safely.
The disadvantages of this type of system are:
It does not share the CPU among tasks that are CPU-bound. A node.js programmer with lots of heavy CPU-bound computations often has to use clustering or child processes to handle the heave CPU computations so as to not over-burden the main request processing Javascript thread with that work and make it too non-responsive.
Clustering of processes is required to maximize the use of multiple processors and then any shared data must be shared across those processes. People often use an in-memory database like Redis to share data between processes.
You can't just willy nilly fire up another Javascript thread to go off and do something.
I don't understand several things about nodejs. Every information source says that node.js is more scalable than standard threaded web servers due to the lack of threads locking and context switching, but I wonder, if node.js doesn't use threads how does it handle concurrent requests in parallel? What does event I/O model means?
Your help is much appreciated.
Thanks
Node is completely event-driven. Basically the server consists of one thread processing one event after another.
A new request coming in is one kind of event. The server starts processing it and when there is a blocking IO operation, it does not wait until it completes and instead registers a callback function. The server then immediately starts to process another event (maybe another request). When the IO operation is finished, that is another kind of event, and the server will process it (i.e. continue working on the request) by executing the callback as soon as it has time.
So the server never needs to create additional threads or switch between threads, which means it has very little overhead. If you want to make full use of multiple hardware cores, you just start multiple instances of node.js
Update
At the lowest level (C++ code, not Javascript), there actually are multiple threads in node.js: there is a pool of IO workers whose job it is to receive the IO interrupts and put the corresponding events into the queue to be processed by the main thread. This prevents the main thread from being interrupted.
Although Question is already explained before a long time, I'm putting my thoughts on the same.
Node.js is single threaded JavaScript runtime environment. Basically it's creator Ryan Dahl concern was that parallel processing using multiple threads is not the right way or too complicated.
if Node.js doesn't use threads how does it handle concurrent requests in parallel
Ans: It's completely wrong sentence when you say it doesn't use threads, Node.js use threads but in a smart way. It uses single thread to serve all the HTTP requests & multiple threads in thread pool(in libuv) for handling any blocking operation
Libuv: A library to handle asynchronous I/O.
What does event I/O model means?
Ans: The right term is non-blocking I/O. It almost never blocks as Node.js official site says. When any request goes to node server it never queues the request. It take request and start executing if it's blocking operation then it's been sent to working threads area and registered a callback for the same as soon as code execution get finished, it trigger the same callback and goes to event queue and processed by event loop again after that create response and send to the respective client.
Useful link:
click here
Node JS is a JavaScript runtime environment. Both browser and Node JS run on V8 JavaScript engine. Node JS uses an event-driven, non-blocking I/O model that makes it lightweight and efficient. Node JS applications uses single threaded event loop architecture to handle concurrent clients. Actually its' main event loop is single threaded but most of the I/O works on separate threads, because the I/O APIs in Node JS are asynchronous/non-blocking by design, in order to accommodate the main event loop. Consider a scenario where we request a backend database for the details of user1 and user2 and then print them on the screen/console. The response to this request takes time, but both of the user data requests can be carried out independently and at the same time. When 100 people connect at once, rather than having different threads, Node will loop over those connections and fire off any events your code should know about. If a connection is new it will tell you .If a connection has sent you data, it will tell you .If the connection isn’t doing anything ,it will skip over it rather than taking up precision CPU time on it. Everything in Node is based on responding to these events. So we can see the result, the CPU stay focused on that one process and doesn’t have a bunch of threads for attention.There is no buffering in Node.JS application it simply output the data in chunks.
Though its been answered , i would like to just share my understandings in simple terms
Nodejs uses a library called Libuv , so this Libuv is written in C
language which uses the concept of threads . These threads are called
as workers and these workers take care of the multiple requests from client.
Parallel processing in nodejs is achieved with the help of 2 concepts
Asynchronous
Non blocking IO