As a followup on this question discussing the use of the ThreadPool vs a dedicated thread:
When would you use a dedicated thread (with lowered priority) for applicationwide logging and when would you use the ThreadPool?
What I would do is completely dependent on the requirements of my app and its logging component.
If logging is mission-critical (maybe you need the ability to replay recent traffic based on the log, for example) then a dedicated thread is more likely the right approach.
If logging is 'best effort', then ThreadPool would be fine subject to other constraints on your app's required performance and latency. Async I/O for the logger would be fine here. Since you suggest lower priority for your putative logger thread, this may match your app's profile.
If more critical work is happening on the ThreadPool then I would not overload it to do logging, esp. if logging itself is important - you could well be doing synchronous, flushed I/O to write out the logs and that's a possible bottleneck depending on the volume of stuff that you wish to log.
If logging is not critical and you want to do it asynchronously then I would recommend using a single background thread for logging and a producer/consumer queue to send log messages. This can achieve improved performance over threadpool since you have a single thread performing I/O on less-critical logs which would have less of a likelyhood to block higher-priority I/O on other threads.
You can also use this mechanism to make sure critical logs are committed before logging. Add them to the queue and then have a mechanism to wait until that particular message is committed.
Related
I have an express backend application, which listens for http requests from a web application. This backend application is running on AWS ECS Fargate.
So my question is, does it makes sense use multithreading, worker-threads in Node.js, in this backend application? There are both CPU intensive and non-intensive functions in the endpoints. For instance should I just distribute any incoming request right away ,regardless of the intensity of the call, to other threads so that Main thread is never blocked? Or should I only use multithreading on intensive jobs etc.
Any suggestion, pros and cons on this topic is very much appreciated.
Questions like this ultimately only really get answered by profiling the system under representative load. This is because our guestimates about what takes CPU and how much are just rarely very good - this stuff has to be measured to really know.
That said, there are some general design guidelines you could consider:
If you have specific requests that you believe in advance are likely to be CPU heavy, then you may want to consider putting those into a queue that is served by a pool of threads.
If you have specific requests that you believe in advance are really just doing non-blocking I/O, then those don't need to get complicated at all by threading. The main thread can likely handle those just fine.
If only a relatively small fraction of your requests are CPU heavy, the simplest design may be to use the nodejs clustering module and just let it spread your requests out among CPUs figuring that this by itself (without any other major design changes) will get any CPU-bound requests off the main loop. If a high percentage of your requests are CPU-bound and you want to prioritize the non-CPU requests so they are always quick, then you'd probably be better with the thread pool that CPU-heavy requests are passed off to so the non-CPU heavy requests aren't held up by them.
So my question is, does it makes sense use multithreading, worker-threads in Node.js, in this backend application?
Yes, in some circumstances. But, when to add this extra layer of complication really depends upon the specific metrics of CPU usage in your requests.
For instance should I just distribute any incoming request right away ,regardless of the intensity of the call, to other threads so that Main thread is never blocked? Or should I only use multithreading on intensive jobs etc.
This depends upon how you want to prioritize different requests. If you want requests prioritized in a FIFO order, where each one gets to start in order based on when it arrived, regardless of what type of request, then yes you can just distribute all requests to threads. In fact, it's probably easier to just use the clustering module for this because that's what it does. If, on the other hand, you want non-CPU-heavy requests to always run quick and not have to wait behind CPU-heavy requests, then you may want to push only the CPU-heavy requests into a queue that is processed by a thread pool. This, then allows the main thread to work on the non-CPU-heavy requests immediately regardless of how many CPU-heavy requests are currently crunching away in the thread pool.
I'm just wondering if there would be any reason I might want to lock a queue. I am working on an application that has several threads that reads and writes to a database. In order to reduce traffic, I want to reduce the amount of calls to that database at any given point (I know many databases can handle some traffic already). Would it make any sense to make a queue for the read/write requests and only the request at the top executes and then protect the queue's push and pop commands with a lock? Is having a lock on each read/write call enough? Isn't a lock implemented as a "queue" by the OS anyways? Could size of this "queue" be an issue or would there be any other reason I wouldn't use a lock by itself?
Thanks!
You could limit the number of threads that are engaged in database requests or if that's not feasible due to the nature of your app, you could use a more granular approach to limit access to the shared resource. In python, you can use the built-in semaphore objects for inter-thread synchronization. For inter-process synchronization (or inter-thread), you'd use posix_ipc. It depends what your service's execution model is.
Most database clients wouldn't require any application-level throttling. In a typical system, the database connections would be pooled and the connection manager would be responsible for acquiring an available connection. Internally this usually involves a queue of some sort with timeouts to prevent waiting indefinitely. The database itself would then handle the scheduling of individual operations made by each connection.
However, a semaphore is a signalling primitive that can be used to limit the number of concurrent operations: https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Semaphore.html
Tasks can also be modeled as a producer-consumer problem which involves a shared queue, however you'll have to deal with the added complexity of managing the consumer threads in addition to the producers.
Let's say I have an XPC service and two user-visible apps (menu bar item and main app). Both connect to the XPC service.
If both of these app components send a "create Foo" message, how are they dispatched? It's a single process after all, so do they arrive in a sequence or will multiple queues be used?
The XPC service uses Core Data, so I worry about whether I have to create a global queue or ensure thread confinement in any other way.
Any time you use CoreData, XPC or not, in a non-single-threaded executable you need to address threading, either through confinement, main-thread only, or private queue. Since XPC services don't have as well-defined a concept of a main thread (obviously, there is the first thread which will forever be the main thread, but practically speaking...) NSMainQueueConcurrencyType is probably not useful. I've not seen any indication that any promises are made by XPC about the thread affinity of requests, so I generally proceed under the assumption that the XPC listener's threading management is an implementation detail over which I have no control (until execution transitions into my code). With that in mind, NSConfinementConcurrencyType looks like a bunch of work. So, if it were me, I'd run with NSPrivateQueueConcurrencyType.
Even if you empirically observed only one request executing at once, you would be better off assuming that's not guaranteed.
I want to provide a way to upload plugins (assemblies) to a site by users for a scripting propose. Through mono.cecil I can analyse those assemblies and limit access only to a predefined list of functions, but I also need to limit memory usage, execution time and kill the thread if it goes to overdraft this resources.
I think I can monitor the memory usage by the profiler api, but as I know there are no tools to abort thread with guarantee. Is there any way to abort thread with guarantee? Maybe I should run code using embedding mono and control the execution of thread in native part of an application, is it possible?
You could use Thread.Abort() as long as you don't allow the plugin code to ResetAbort().
Thread level control was not practical IMHO (anyone did that in the past). Typically you should consider process level control of memory usage or application domain level.
I know non-blocking receive is not used as much in message passing, but still some intuition tells me, it is needed. Take for example GUI event driven applications, you need some way to wait for a message in a non-blocking way, so your program can execute some computations. One of the ways to solve this is to have a special thread with message queue. Is there some use case, where you would really need non-blocking receive even if you have threads?
Threads work differently than non-blocking asynchronous operations, although you can usually achieve the same effect by having threads that does synchronous operations. However, in the end, it boils down on how to handle doing things more efficiently.
Threads are limited resources, and should be used to process long running, active operations. If you have something that is not really active doing things, but need to wait idly for some time for the result (think some I/O operation over the network like calling web services or database servers), then it is better to use the provided asynchronous alternative for it instead of wasting threads unnecessarily by putting the synchronous call on another thread.
You can have a good read on this issue here for more understanding.
One thread per connection is often not a good idea (wasted memory, not all OS are very good with huge thread counts, etc)
How do you interrupt the blocking receive call? On Linux, for example (and probably on some other POSIX OS) pthreads + signals = disaster. With a non-blocking receive you can multiplex your wait on the receiving socket and some kind of IPC socket used to communicate between your threads. Also maps to the Windows world relatively easily.
If you need to replace your regular socket with something more complex (e.g. OpenSSL) relying on the blocking behavior can get you in trouble. OpenSSL, for example, can get deadlocked on a blocking socket, because SSL protocol has sender/receive inversion scenarios where receive can not proceed before some sending is done.
My experience has been -- "when in doubt use non-blocking sockets".
With blocking IO, it's challenging on many platforms to get your application to do a best effort orderly shutdown in the face of slow, hung, or disconnected clients/services.
With non-blocking IO, you can kill the in-flight operation as soon as the system call returns, which is immediately. If your code is written with premature termination in mind - which is comparatively simple with non-blocking IO - this can allow you to clean up your saved state gracefully.
I can't think of any, but sometimes the non-blocking APIs are designed in a way that makes them easier/more intuitive to use than an explicitly multi-threaded implementation.
Here goes a real situation I have faced recently. Formerly I had a script that would run every hour, managed by crontab, but sometimes users would log to the machine and run the script manually. This had some problems, for example concurrent execution by crontab and user could cause problems, and sometimes users would log in as root - I know, bad pattern, not under my control - and run script with wrong permissions. So we decided to have the routine running as daemon, with proper permissions, and the command users were used to run would now just trigger the daemon.
So, this user executed command would basically do two things: trigger the daemon and wait for it to finish the task. But it also needed a timeout and to keep dumping daemon logs to user while waiting.
If I understand the situation you proposed, I had the case you want: I needed to keep listening from daemon while still interacting with user independently. The solution was asynchronous read.
Lucky for me, I didn't think about using threads. I probably would have thought so if I were coding in Java, but this was Python code.
My point is, that when we consider threads and messaging being perfect, the real trade-off is about writing scheduler for planning the non-blocking receive operations and writing synchronizations codefor threads with shared state (locks etc.). I would say, that both can be sometime easy and sometime hard. So an use case would be when there are many messages asynchronous messages to be received and when there is much data to be operated on based on the messages. This would be quite easy in one thread using non-blocking receive and would ask for much synchronization with many threads and shared state.... I am also thinking about some real life example, I will include it probably later.