Sorry for the novel...
I'm working on a Nodejs project where I need to decrypt millions of envelopes in multiple files. Any APIs of my application have to run on localhost.
The main API handles client requests to decrypt a batch of files. Each file contains thousands to millions of envelopes that need to be decrypted. Each file is considered a job and these jobs are queued up by the Main API and then run concurrently by forking a new process for each job. (I only allow 5 concurrent jobs/forks at one time) In each process, a script runs that goes through and decrypts the file.
This runs relatively quickly but instead of doing the decryption in the code of each process/script forked by the Main API, I want to dish this work out to another API (call it Decrypt API) that basically takes the envelope in the request and sends back the decrypted result in the response.
So I created this api and then used 'forky' to cluster it. Then from my processes, instead of doing the decryption in those, I makes multiple parallel requests to the Decrypt API and once I get the responses back just place the decrypted results in a file.
At first my problem was that I made requests right as I got each envelope without waiting for a request to return before sending the next one. I would basically send "parallel" requests if you will, and then just handle the vote in the callback of each request. This led to what I think is too many outstanding reqs at one time because I was getting an ECONNRESET error. Some requests were dropped. So my solution was to have a maximum of x outstanding reqs(I used 10) at any one time to avoid too many concurrent reqs. This seemed ok but then I realized since I was forking 5 processes from the MainAPI, and although each one had this new 'outstanding reqs' limiting code, since they were running concurrently I was still running into the problem of too many reqs at once to the Decrypt API. Also, this method of using two different microservices/APIs is slower than just having the MainAPI's forked processses just do the decryption. In the Decrypt API I'm also using the node 'crypto' library and some of those functions that I use are synchronous so I suspect that with high traffic that's a problem, but I can't avoid those sync methods.
So finally, my question is, what can I do to increase the speed of the Decrypt API with high traffic like I described, and what can I do to avoid these dropped requests?
Forgive me if I sound like a noob, but since these APIs are all running on the same machine and localhost, could this be why this method is slower than just doing decryption in each process?
Thanks!
Related
I took over a project where the developers were not fully aware of how Node.js works, so they created code accessing MongoDB with Mongoose which would leave inconsistent data in the database whenever you had any concurrent request reaching the same endpoint / modifying the same data. The project uses the Express web framework.
I already instructed them to implement a fix for this (basically, to use Mongoose transaction support with automatically managed retriable transactions), but due to the size of the project they will take a lot of time to fix it.
I need to put this in production ASAP, so I thought I could try to do it if I'm able to guarantee sequential processing of the incoming requests. I'm completely aware that this is a bad thing to do, but it would be just a temporary solution (with a low count of concurrent users) until a proper fix is in place.
So is there any way to make Node.js to process incoming requests in a sequential manner? I just basically don't want code from different requests to run interleaved, or putting it another way, I don't want non-blocking operations (.then()/await) to yield to another task and instead block until the asynchronous operation ends, so every request is processed entirely before attending another request.
I have an NPM package that can do this: https://www.npmjs.com/package/async-await-queue
Create a queue limited to 1 concurrent user and enclose the code that calls Mongo in wait()/end()
Or you can also use an async mutex, there are a few NPM packages as well.
DISCLAIMER: If this post is off-topic to this site, please recommend a site where this post would be appropriate.
On Ubuntu 18.04, in bash, I am writing a network-based, threaded application that requires multiple servers. It receives files through the network and processes them, ultimately making an API call that finishes the processing and logs the results to a database for later retrieval and reporting.
So far I have written the application using non-threaded programming models and concepts. That means the files are processed one at a time in real-time. This works great if there is no sudden burst of files and/or a backlog of files to process. The main bottle neck has been the way I sequentially send files to the API one after another, waiting until the entire operation has taken place for one file and the API returns the results. The API has a rate limit of 8 calls per second. But since each call takes from .75 to 1 second, my program waits until the operation is done and only processes about 1 file per second through the API. In short, I did not have to worry about scheduling API calls because I could barely do one call per second.
Since the capacity is there to process 8 files per second, and I need more speed, I have been converting my single-threaded, sequential application into a parallel, scalable, multi-threaded application. This new version can spawn enough threads to send 8 files per second to the REST API and much more. So now I have the opposite problem. I am sending too many requests per second to the REST API and am in danger of triggering penalties, etc. Ultimately, when my traffic is higher, I will upgrade my subscription to the API and get more calls per second, but this current dilemma has got me thinking about how to schedule the API calls with different threads.
The purpose of this post is to discuss an idea about how to schedule these REST API calls across various threads. Specifically, I want to discuss how to coordinate timing and usage of the API while maintaining efficiency and yet not overloading the API. In short, I want to coordinate a group of threads so that the API is properly used. Not too fast and not too slow.
Independent of my application, this idea could be useful in a number of generically similar scenarios.
My idea is to create an "air traffic controller" ("ATC") so that the threads of the application have a centralized timing authority to check when they are ready to submit files to the REST API. The ATC would know how many time slots/calls per time period (in this case, calls per second) the API can schedule. The ATC would be listening for the threads to request a time slot ("launch code") which would give them a time slot in the future to perform their API call. The ATC would decide based on the schedule of other launch codes that it has already handed out.
In my case, from the start of the upload of the file to the API, it could take 0.75 to 1 second to complete the processing and receive a response from the API. This does not affect the count of new API calls that can be performed. It is just a consideration of how long the threads will be waiting once they call the API. It may not be relevant to this overall discussion.
Each thread would obviously have to do some error handling. If the API timed out or threw an error, then the thread would have to handle it and get back in line with the ATC -if appropriate- and ask for a new launch code. Maybe it should report the error to the ATC for centralized logging?
In situations where the file processing needs burst above 8 files per second, there would be a scheduling backlog where the threads should wait their turn as assigned by the ATC.
Here are some other considerations:
Function
The ATC would be a lightweight daemon that does the following:
- listens on some TCP port
- receives a request
security token (?), thread id, priority
- authenticates the request (?)
- examines schedule
- reserves the next available time slot
- returns the launch code
security token (?), current time, launch timing offset to current time, URL and auth token for the API
- expunged expired launch codes
The ATC would need the following:
- to know what port it is supposed to run on
- to know how many slots per time period it was set to schedule
(e.g. 8 per second)
- to have a super fast read/write access to the schedule (associative array?)
- to know the URL and corresponding auth token for the thread to use
- maybe to know multiple URLs and auth tokens for load balancing
Here are more things to consider:
Security
How could we keep the ATC secure while ensuring high performance?
Network-level security (e.g. firewalls allowing only the IP addresses of the file-processing servers?)
Auth tokens or logins and passwords?
Performance
What would the requirements be for this ATC server? Would this be taxing to a CPU and memory?
Timing
How often would an NTP call be needed? By the ATC server? By the servers which call the API?
Scalability
Being able to provide different URLs and auth tokens would allow the ATC to load balance with different API providers.
Threading of the ATC itself
Would the ATC need to spawn threads to be able to handle each new request?
How does a web server handle requests?
How would the various threads share a common schedule?
In a non-threaded environment, the ATC would possibly keep an associative array in memory to keep performance as high as possible. How would the various threads of the ATC have access to the same schedule?
So here is my question. Does this exist? If not, what are some best practices in trying to build the above?
It seems like a beanstalkd kind of network service except it only provides permission/scheduling and is extremely dependant on timing.
I'm developing an API for sending SMS with an Http request. I use node js and mongoose. So I have a problem like the one with multi thread application.
The fact is that when a user send a sms, I verify the number of sms he has already sent in database (using mongoose) and if the number doesn't exceed a limit his sms is sent and the number of sms he has sent is increment in the database (there is a value for the number of sms he has sent in the hour,day,week and month in the schema). But the fact is that I use a callbacks for the process of read value and increment value and many other operation in my code.
So the problem (I think) is that when user send requests very quickly the server different callbacks read the same count of the sms sent, authorize user to sent sms, increment and save the same value so that the count of sms is false.
In a multi thread application that access to a variable the solution would be to prevent other threads to read a variable before the actual thread has done all of it works.
With Node js event system and access to data in mongoDB I just don't know how to solve my problem.
Thank you in advance for the answers.
PS: I don't know the solution but it will be good if it works also with clusters that allow node js to use multi core.
I think you should try some cache approach.
now I meet same situation with you.
I will try to use cache to store the record_id that is in process.
When new request come, the coming process need check cache. If the record_id is in cache that means that record is using by other thread. So that thread need wait or do something else until finish. And when the process finish that will remove the record_id in cache in callback function
Thanks Cristy, I have solved the main part of my problem using async queue.
My application works well when I run it the default way of node js.
But there is an other problem. I intend to run my code on a server that has 4 cores so I want to use the node cluster module. But when I used this... because it runs code like 4 differents process (I used a server with 4 cores) they use differents queues and the error I mention earlier always occured, they read and write to the database without waiting for other thread to finish processing verifications + update.
So I would like to know what should I do to have an optimal and fast application.
Should I stop to use the cluster module and don't take benefit of multi core server (I don't think it is the best answer)?
Should I store it in my mongodb (maybe try to not persist the queue but store it in the memory in other to make it faster) ?
Is there a way to share the queue in the code when I use cluster?
What is my best choice?
How many simultaneous requests can I make with the request package?
I am expecting data back from every request confirming the request was received and processed successfully. Is this hardware or OS dependent? Where do I start looking?
One of the more recent versions of node.js does not enforce a limit on outgoing requests (older versions did). If you were literally trying to make millions of outgoing connections at the same time, then you would probably hit a limit on your own node.js server that would be OS specific. But, the practical limit is more likely going to be determined by the target host.
Since all your requests are being sent to the same host, the more likely limit will be determined by the server you are making the requests to. It will have some sort of limit for how many simultaneous requests it can have "in-flight" at the same time before it starts refusing new connections. What that number is depends entirely upon how the server is configured and built. For http://www.google.com, the number is probably hundreds of thousands or millions of requests because they have a huge server farm and requests are balanced across all of them. For some simple single CPU server, the limit would obviously be much smaller than that.
In addition, there will little use in sending zillions of requests to a single CPU server anyway because it won't be able to work on all of them at once anyway.
So, if you want to know what would work best for a given target host, you would have to set up an adjustable test harness so you could test scenarios where you send from 1, 2, 5, 10, 50, 100, 200, 500, 1000 at a time and see what the average response time is and where you start to get errors (if any).
If you don't want to do any of that type of testing, then a reasonably safe choice that doesn't attempt to fully optimize things is to put no more than 5 requests in flight at the same time.
You can either build something yourself to manage to N requests in flight at a time or you can use one of the existing libraries that will do that for you. The Bluebird promise library has a concurrency option on some of it's functions such as Promise.map() which will automatically do that for you for whatever concurrency value you set. The async library also has something similar.
If you want more specific help crafting the code to manage how many requests are in flight at a time or to build a test harness for it, please show us some of your code for the source of all the requests so we have some idea how that works (if it's a giant array of requests or what the source of the URLs is).
In my application, I have a multiple file upload AJAX client. I noticed (using a stub file processing class) that Spring usually opens 6 threads at once, and the rest of the file upload requests are blocked until any of those 6 threads finishes its job. It is then assigned a new request, as in a thread pool.
I haven't done anything specific to reach this behavior. Is this something that Spring does by default behind the scenes?
While uploading, I haven't had any problems browsing the other parts of the application, with pretty much no significant overhead in performance.
I noticed however that one of my "behind the scenes" calls to the server (I poll for new notifications every 20 secs) gets blocked as well. On the server side, my app calls a Redis-based key-value store which should always return even if there are no new notifications. The requests to it start getting normally processed only after the uploads get finished. Any explanation for this kind of blocking?
Edit: I think it has to deal with a maximum of concurrent requests per session
I believe this type of treading belongs to the Servlet Container but not to Spring.