I am designing a service that would open ~30-50 websocket connections getting financial tick data and writing it to MongoDB; it's possible that hundreds of messages will be received every second. I am using Autobahn's Python Twisted Websocket for it, but I was wondering whether it is scalable on running one async thread. My instinct is that both single threaded Twisted and MongoDB can handle that throughput, but I want to confirm that hypothesis.
I see 3 potential options:
Using 1 thread, connect to all websocket endpoints and call reactor.run().
Each websocket connection belongs to its own thread, which has its own reactor.run(). Run it as one main unix daemon that spawns all the threads.
Each websocket connection belong to its own thread, run in different scripts as unix daemons.
What would be the most robust option? Thanks a lot!
As far as you describe your problem your application is IO bound not CPU bound. If you're going for an async solution you should try to avoid threads. It's not that Twisted doesn't support threads it does but threads are just overhead when you're programming in an async way. Twisted and Autobahn can handle lots of IO before they start using up CPU. So as long as your application code isn't high cpu either you'll be fine.
Related
I am using Node.js express for building REST api with postgres database using node-postgres package.
My question is whether I should use Client or Pool? I found this answer:
How can I choose between Client or Pool for node-postgres
but I don't understand what would be the use of Pool connection, since Nodejs is single-threaded and there won't be an attempt to use a single connection at the same time even if there are concurrent requests occurring.
Also by using a single connection, I can benefit from the prepared statements much more efficiently. I can prepare them at the initialization phase of my app, and then execute it whenever a request arrives.
Yes since Postgresql is still multithreaded.
When making a database request your process spends 0% CPU time executing code. Yes, you've read that right, zero.
The computer does not execute code in order to wait. Instead it sets up interrupt handlers and tells the hardware (ethernet card or wifi module) to send it an interrupt when there is data. Regardless of the number of requests you make to your database you still only have ONE ethernet card in your PC (well, some servers can have multiple and have increased bandwidth by trunking but I think you can see that the number of PCI cards you have does not have any relationship with the number of threads you are running - rather it is more related with the amount of $money you are willing to spend). Your hardware still basically sends all the requests out one bit at a time.
A traditional multi-threaded server therefore spends exactly the same amount of CPU time as node.js waiting for responses from the database: zero. Which means node.js improves efficiency by not needing to malloc a lot of RAM for each thread since node only has one thread.
Even when you are running your database on the same computer as your node process, communication with the database is not overly parallel. And the TCP/IP stack itself sort of serializes the communication. And while it does not go through the networking hardware the OS still schedules the responses using OS level events (instead of hardware interrupts).
So yes, it makes sense for your node.js process to make multiple parallel connections to the database even when node is singlethreaded - it is to allow the database to process requests in multiple database threads. You are making use of your database's multithreading instead of forcing your database to also use only one thread to process node's single connection.
I was trying to understand how nodejs can achieve higher concurrency compared to thread-based approaches such as Servlet servers.
I already know that in nodejs "everything runs in parallel except your code", and also there is a backend thread pool in libuv to handle File IO or database calls which are usually the bottlenecks.
So here is my question: if nodejs uses thread pool to handle database calls, how it can service higher concurrent request than Servlet servers such as Tomcat given that Tomcat can also use NIO backed by epoll/kqueue to achieve high concurrency ?
For example, if there's a 100k concurrent request coming in and each requires database operations, if these 100k request are to be serviced concurrently, with nodejs we still end up creating 100k threads which might cause memory exhaustion as Tomcat does. Yes, the 100k threads is just an imagination because (I know) that nodejs has a fixed thread pool and different operations are queued in the event loop, but with Tomcat it handles things in the same way--we also can configure the thread pool size in Tomcat and it also queues request.
Or, am I wrong to say that "nodejs uses backend thread pool in libuv to handle File IO or database calls"? Does nodejs use epoll/kqueue to handle database io without a separate thread?
I was reading this similar question but still didn't get the answer.
if nodejs uses thread pool to handle database calls
That's a wrong assumption. nodejs will typically use networking to talk to a local database running in a different process or on a different host. Networking in node.js does not use threads of any kind - it uses event driven I/O. What the database does for threads is up to the database and independent of node.js since it would be the same no matter which server environment you were using.
node.js does use a thread pool for local disk access, but high scale applications are usually using a database for the crux of their disk access which run in a separate process and have their own I/O optimizations to handle lots of requests. How a given database does it is up to that implementation, but it will not be using a nodejs thread per request.
I was trying to understand how nodejs can achieve higher concurrency compared to thread-based approaches such as Servlet servers.
The general concept is that a properly written server app in node.js uses async I/O for all I/O (except perhaps startup code that only runs during server startup). This means that it can have a lot of requests in-flight at the same time with only a single Javascript thread while most of them are waiting on some type of I/O. If you're going to have a lot of requests in-flight at the same time, it can be a lot more efficient for the system to do it the node.js way of a single thread where all the requests are cooperatively switched vs. using OS threads where every thread has OS overhead associated with it and every pre-emptive thread switch has OS and CPU overhead associated with it.
In node-js, there is no pre-emptive switching between the active requests. Only one runs at a time and it runs until it either finishes or hits an asychronous operation and has nothing else to do until that async I/O operation completes. At that point, the JS engine goes back to the event queue and picks out an event (probably for one of the other requests). This type of cooperate switching can be significantly faster and more efficient than OS-level threads. There is sometimes a programming cost in that a node.js developer has to code with async I/O in order to take advantage of this which has a learning curve in order to get proficient at writing good, clean code with proper error handling and has a learning curve for debugging it too.
For example, if there's a 100k concurrent request coming in and each requires database operations, if these 100k request are to be serviced concurrently, with nodejs we still end up creating 100k threads which might cause memory exhaustion as Tomcat does.
No, you will not be creating 100k threads. A node.js database interface layer that interfaces between node.js and the actual database code in another process or on another host may be written entirely in node.js (using TCP networking to talk to the database) and introduce no new threads at all or it may have some native code and use a small number of threads for its own native code operations, but it will likely be a small number of threads and nothing even close to one per request.
Or, am I wrong to say that "nodejs uses backend thread pool in libuv to handle File IO or database calls"? Does nodejs use epoll/kqueue to handle database io without a separate thread?
For file I/O, yes it uses a thread pool in libuv. For database calls, no - While the details depend entirely upon the database implementation, usually there is not a thread per database call. The database is typically in another process and the nodejs interface library for the DB either directly uses nodejs TCP to talk to the database (which uses no threads) or it has its own native code add-on that talks to the database which probably uses a small number of threads for its work, but typically not a thread per request.
I see from the server side, the benefit of NIO is the capability to manage multiple network connections with fewer thread comparing to the comparing to one thread per connection blocking IO.
However, if I have a IO client which connects to thousand of servers at the same time, can I just have similar approach to manage these connections IO using fewer threads. I tried the approach in Netty 4 multiple client and found it spawn a "Reader" thread for each channel it created.
So, my questions are:
1) what are the benefits using netty/NIO in the client side?
2) is it possible to manage multiple connections with fewer threads in the client side?
Thanks!
I have uploaded the code samples in github: https://github.com/hippoz/ogop-lseb
The sample server/client class is moc.ogop.ahsp.demo.nio.MultipleConnectionNioMain and moc.ogop.ahsp.demo.nio.NettyNioServerMain
Having lots of threads creates a context-switch problem in the kernel where lots more memory is being loaded and unloaded from each core as the kernel tries to reschedule the threads across the cores.
The benefit of NIO anywhere is performance. Thats pretty much the only reason we use it. Using Blocking IO is MUCH more simple. Using the worker model and NIO you can limit the number of threads (and potential computational time) the process uses. So if you have two workers and they go bonkers using 100% cpu time the whole system won't go to a crawl because you have 2-4 more cores available.
Have fun!
https://en.wikipedia.org/wiki/Context_switch
Why should I use non-blocking or blocking sockets?
I understand how Node.js works with single thread. Mostly it is using asynchronous methods/modules in order to keep the main runtime thread free as much as possible.
However, some of the asynchronous modules internally are using threads to do their job. Example for this is reading file or other high intensive CPU task. This is done in background and it is abstracted for the Node developer.
My question is , how internally Socket.IO works, does it use threads like the above examples ? Does it use separate thread per connection ? If so , does it mean that we will have 1000 threads, if we have 1000 connected clients ?
Node does not use the thread pool (or separate threads) for sockets, instead it uses whatever platform-specific mechanism for polling sockets for data (e.g. epoll on Linux, kqueue on OS X (IIRC), I/O completion ports on Windows, etc.) on the main thread.
Socket.io works on the event loop like most node applications. No tricky thread business AFAIK. You can check out the source yourself here: https://github.com/Automattic/socket.io
The following two diagrams are my understanding on how threads work in a event-driven web server (like Node.js + JavaScript) compared to a non-event driven web server (like IIS + C#)
From the diagram is easy to tell that on a traditional web server the number of threads used to perform 3 long running operations is larger than on a event-driven web server (3 vs 1.)
I think I got the "traditional web server" counts correct (3) but I wonder about the event-driven one (1). Here are my questions:
Is it correct to assume that only one thread was used in the event-driven scenario? That can't be correct, something must have been created to handle the I/O tasks. Right?
How did the evented server handled the I/O? Let's say that the I/O was to read from a database. I suspect that the web server had to create a thread to hand off the job of connecting to the database? Right?
If the event-driven web server indeed created threads to handle the I/O where is the gain?
A possible explanation for my confusion could be that on both scenarios, traditional and event-driven, three separate threads were indeed created to handle the I/O (not shown in the pictures) but the difference is really on the number of threads on the web server per-se, not on the I/O threads. Is that accurate?
Node may use threads for IO. The JS code runs in a single thread, but all the IO requests are running in parallel threads. If you want some JS code to run in parallel threads, use thread-a-gogo or some other packages out there which mitigate that behaviour.
Same as 1., threads are created by Node for IO operations.
You don't have to handle threading, unless you want to. Easier to develop. At least that's my point of view.
A node application can be coded to run like another web server. Typically, JS code runs in a single thread, but there are ways to make it behave differently.
Personally, I recommend threads-a-gogo (the package name isn't that revealing, but it is easy to use) if you want to experiment with threads. It's faster.
Node also supports multiple processes, you may run a completely separate process if you also want to try that out.
The best way to picture NodeJS is like a furious squirrel (i.e. your thread) running in a wheel with an infinite number of pigeons (your I/O) available to pass messages around.
I/O in node is "free". Your squirrel works to set up the connection and send the pigeon off, then can go on to do other things while the pigeon retrieves the data, only dealing with the data when the pigeon returns.
If you write bad code, you can end up having the squirrel waiting for each pigeon.
So always write non-blocking i/o code.
If you can encourage your Pigeons to promise to come back ;)
Promises and generators are probably the best approach you can take to this.
HOWEVER you can always use Node cluster to establish a master squirrel that will procreate child squirrels based on the number of CPUs the master squirrel can find to dole out the work.
Hope this helps and note the complete lack of a car analogy.