We chose Node.js for our web project, but there are many computational tasks for which we would prefer Scala. We are highly concerned about speed, what is the best way to call a Scala "worker" from Node.js in an asynchronous non-blocking way?
When queuing jobs its best to have some kind of Broker like a message queue or a job queue. Redis is a popular choice, as it can also be used for caching, and storing data in memory. RabbitMQ is another common choice. The nice thing about having a Broker is it can hold the job until a worker pulls it out of queue when ever it has available resources. A broker also acts as a load balancer in a sense, where it holds jobs and you can have multiple worker nodes grabbing jobs allowing for high availability, scalability, and parallel processing.
You probably should not be so concerned about speed; in my experience concerns like readability and maintainability are more important in almost all projects.
For short-lived "remote procedure calls" of at most a few seconds, I would tend to use Apache Thrift, which has libraries for Javascript and the JVM (Scrooge is an alternative Scala implementation, oriented towards writing async backends using Twitter's Finagle futures library), allowing nonblocking calls; by using Thrift you get strongly typed interface definitions that are engineered for forward compatibility, and you know exactly what changes you can make to the interface without breaking compatibility.
Alternatively one could use an ordinary HTTP ("REST") interface; node is oriented towards making async HTTP calls, and libraries like Spray make it easy to offer a high-performance, async HTTP interface in Scala.
For longer-running "batch" tasks where you're less concerned about latency and more about reliability, it's probably better to use a dedicated task queue as #tsutrzl suggests.
Related
Can you please explain what is the I/O Scaling Problem in Node.js Performance?
I am reading the book "Basarat Ali Syed - Beginning Node.js-Apress" but the explanation is not enough to I/O Scaling.
A server typically has a mix of computational things to do and I/O things to do (getting data from somewhere like a database or a disk or another server). In today's modern servers with pretty fast multi-core processors, it is more common for a given server request to be limited by I/O than by CPU.
So, if you're going to scale a server to be able to handle lots of requests and to handle them with good performance, you have to find a way to be able to most efficiently handle lots of I/O requests because that's probably what your server is limited by. This would be the "I/O scaling problem". How to scale your server and code architecture to handle lots of I/O requests very efficiently.
It so happens that the node.js single-threaded architecture with asynchronous I/O is very efficient at doing lots of I/O and can be more efficient than other server architectures that use multiple threads and blocking I/O calls.
If you go to your Table of Contents in that book, you will see the following:
Understanding Node.js Performance
The I/O Scaling Problem
Traditional Web Servers Using a Process Per Request
Traditional Web Servers Using a Thread Pool
The Nginx way
Node.js Performance Secret
I don't have the book myself, but I would presume that "The I/O Scaling Problem" section of the book describes it for you. And, then you can read about the node.js performance secret for how it handles this. The servers that use a process or a thread per request take more system resources to have lots of requests in flight at the same time (which is one key to handle lots of requests). The node.js non-blocking I/O model, on the other hand, is very efficient at handling lots and lots of in-flight requests.
I have a Multi-Client Single-Server application where client and server gets connected through sockets. Client and Server are in different machine.
In client Application, client socket gets connected to server and sends data periodically to server.
In server application server socket listens for client to connect. When a client is connected, new thread is created for client to receive data.
for example: 1 client = 1 thread created by server for receiving data. If its 10000 client, server creates 10000 threads. This seems not good and scalable too.
My Application is in Java.
Is there an alternate method for this problem?
Thanks in advance
This is a typical C10K problem. There are patterns to solve this, one examples is Reactor pattern
Java NIO is another way where the incoming request can be processed in non blocking way. See a reference implementation here
Yes, you should not need a separate thread for each client. There's a good tutorial here that explains how to use await to handle asynchronous socket communication. Once you receive data over the socket you can use a fixed number of threads. The tutorial also covers techniques to handle thousands of simultaneous communications.
Unfortunately given the complexity it's not possible to post the code here, so although link-only answers are frowned upon ...
I would say it's a perfect candidate for an Erlang/Elixir application. Whatsapp, RabbitMQ...
Erlang processes are cheap and fast to start, Erlang manages the scheduling for you so you don't have to think about the number of threads, CPUs or even machines, Erlang manages garbage collection for each process after you don't need it anymore.
Haskell is slow, Erlang is fast enough for most applications that are not doing heavy calculations and even then you can use it and hand off the heavy lifting to a C process.
What are you writing in?
Yes, you can use the Actor model, with e.g. Akka or Akka.net. This allows you to create millions of actors that run on e.g. 4 threads. Erlang is a programming language that implements the actor model natively.
However, actors and non-blocking code won't do you much good if you are relying on blocking library calls for backend services that you rely on, such as (the most prominent example in the JVM world) JDBC calls.
There is also a rather interesting approach that Haskell uses, called green threads. It means that the runtime threads are very lightweight and are dynamically mapped to OS threads. It also means that you get a certain amount of scalability "for free", with no need to write non-blocking IO code. It does however require a good IO manager in the runtime to schedule the IO operations efficiently, and GHC Haskell has had a substantial amount of work put into that in recent years.
Trying to grasp some basics of Redis I came across an interesting blog post .
The author states:
Redis is single-threaded with epoll/kqueue and scale indefinitely in terms of I/O concurrency.
I surely misunderstand the whole threading thing, because I find this statement puzzling. If a program is single-threaded, how does it do anything concurrently? Why it is so great that Redis operations are atomic, if the server is single-threaded anyway?
Could anybody please shed some light on the issue?
Well it depends on how you define concurrency.
In server-side software, concurrency and parallelism are often considered as different concepts. In a server, supporting concurrent I/Os means the server is able to serve several clients by executing several flows corresponding to those clients with only one computation unit. In this context, parallelism would mean the server is able to perform several things at the same time (with multiple computation units), which is different.
For instance a bartender is able to look after several customers while he can only prepare one beverage at a time. So he can provide concurrency without parallelism.
This question has been debated here:
What is the difference between concurrency and parallelism?
See also this presentation from Rob Pike.
A single-threaded program can definitely provide concurrency at the I/O level by using an I/O (de)multiplexing mechanism and an event loop (which is what Redis does).
Parallelism has a cost: with the multiple sockets/multiple cores you can find on modern hardware, synchronization between threads is extremely expensive. On the other hand, the bottleneck of an efficient storage engine like Redis is very often the network, well before the CPU. Isolated event loops (which require no synchronization) are therefore seen as a good design to build efficient, scalable, servers.
The fact that Redis operations are atomic is simply a consequence of the single-threaded event loop. The interesting point is atomicity is provided at no extra cost (it does not require synchronization). It can be exploited by the user to implement optimistic locking and other patterns without paying for the synchronization overhead.
OK, Redis is single-threaded at user-level, OTOH, all asynchronous I/O is supported by kernel thread pools and/or split-level drivers.
'Concurrent', to some, includes distributing network events to socket state-machines. It's single-threaded, runs on one core, (at user level), so I would not refer to this as concurrent. Others differ..
'scale indefinitely in terms of I/O concurrency' is just being economical with the truth. They may get more belief if they said 'can scale better than one-thread-per-client, providing the clients don't ask for much', though they may then feel obliged to add 'blown away on heavy loading by other async solutions that use all cores at user level'.
what are the disadvantages of RPC with respect to message passing?
Are you talking about RPC vs Messaging? As in (typically) asynchronous messaging? If that's what you're talking about, then Messaging tends to be more robust at the cost of complexity and extra infrastructure.
The simplest example is if you have a scenario where you RPC->RPC->RPC, you end up having a call stack that's 3 processes/machines deep. Any one of those processes/machine could fail during processing, and the entire stack unwinds.
If you were messaging, the actual connectivity between the processes is much less. You hand the message off, and you're on your way. Now if one of the processes fail, there's a good chance of it being restarted where it left off, since, typically, the message is still sitting on a queue somewhere waiting for a new process to fetch it. The overall time may be longer, but it's a much more robust system.
Mind it's no panacea, there are a lot of pitfalls with an asynchronous architecture, but this robustness is a prime distinction between RPC and Messaging systems.
As a general rule, RPC provides a higher level of abstraction than some other means of interprocess communication. This makes it, perhaps, easier to use than lower level primitives. For this abstraction you may pay some penalty in performance due to marshaling/unmarshaling and may have to deal with added complexity in configuration for simple scenarios.
You might be interested in this thesis (pdf) by Jackie Silcock which discusses differences between message passing, RPC, and distributed shared memory with respect to several different measures of performance and implementation. You can also read one of the papers based on the thesis: Message Passing, Remote Procedure Calls and
Distributed Shared Memory as Communication
Paradigms for Distributed Systems (pdf)
We've all read the benchmarks and know the facts - event-based asynchronous network servers are faster than their threaded counterparts. Think lighttpd or Zeus vs. Apache or IIS. Why is that?
I think event based vs thread based is not the question - it is a nonblocking Multiplexed I/O, Selectable sockets, solution vs thread pool solution.
In the first case you are handling all input that comes in regardless of what is using it- so there is no blocking on the reads- a single 'listener'. The single listener thread passes data to what can be worker threads of different types- rather than one for each connection. Again, no blocking on writing any of the data- so the data handler can just run with it separately. Because this solution is mostly IO reads/writes it doesn't occupy much CPU time- thus your application can take that to do whatever it wants.
In a thread pool solution you have individual threads handling each connection, so they have to share time to context switch in and out- each one 'listening'. In this solution the CPU + IO ops are in the same thread- which gets a time slice- so you end up waiting on IO ops to complete per thread (blocking) which could traditionally be done without using CPU time.
Google for non-blocking IO for more detail- and you can prob find some comparisons vs. thread pools too.
(if anyone can clarify these points, feel free)
Event-driven applications are not inherently faster.
From Why Events Are a Bad Idea (for High-Concurrency Servers):
We examine the claimed strengths of events over threads and show that the
weaknesses of threads are artifacts of specific threading implementations
and not inherent to the threading paradigm. As evidence, we present a
user-level thread package that scales to 100,000 threads and achieves
excellent performance in a web server.
This was in 2003. Surely the state of threading on modern OSs has improved since then.
Writing the core of an event-based server means re-inventing cooperative multitasking (Windows 3.1 style) in your code, most likely on an OS that already supports proper pre-emptive multitasking, and without the benefit of transparent context switching. This means that you have to manage state on the heap that would normally be implied by the instruction pointer or stored in a stack variable. (If your language has them, closures ease this pain significantly. Trying to do this in C is a lot less fun.)
This also means you gain all of the caveats cooperative multitasking implies. If one of your event handlers takes a while to run for any reason, it stalls that event thread. Totally unrelated requests lag. Even lengthy CPU-invensive operations have to be sent somewhere else to avoid this. When you're talking about the core of a high-concurrency server, 'lengthy operation' is a relative term, on the order of microseconds for a server expected to handle 100,000 requests per second. I hope the virtual memory system never has to pull pages from disk for you!
Getting good performance from an event-based architecture can be tricky, especially when you consider latency and not just throughput. (Of course, there are plenty of mistakes you can make with threads as well. Concurrency is still hard.)
A couple important questions for the author of a new server application:
How do threads perform on the platforms you intend to support today? Are they going to be your bottleneck?
If you're still stuck with a bad thread implementation: why is nobody fixing this?
It really depends what you're doing; event-based programming is certainly tricky for nontrivial applications. Being a web server is really a very trivial well understood problem and both event-driven and threaded models work pretty well on modern OSs.
Correctly developing more complex server applications in an event model is generally pretty tricky - threaded applications are much easier to write. This may be the deciding factor rather than performance.
It isn't about the threads really. It is about the way the threads are used to service requests. For something like lighttpd you have a single thread that services multiple connections via events. For older versions of apache you had a process per connection and the process woke up on incoming data so you ended up with a very large number when there were lots of requests. Now however with MPM apache is event based as well see apache MPM event.