Best way to split up large array in a Node.js environment - node.js

Question

I’m pulling data from the Cloudflare API, getting all web request logs for a very high traffic website in a certain time frame (less than 7 days of data).
The Cloudflare API takes start and end parameters for the dates that you want to pull logs from. Start can be no later than 7 days and the difference between start and end cannot be greater than 1 hour. For this reason, in order to pull the data I need (usually 3-4 days worth), I wrote some custom code to generate a range of dates separated by one hour from the start to end I need.
With this range, I query the API with a loop and concat the array response to a single large array as I need to do analysis on all the data. This array typically has ~1 million entries (objects). I’m sure you can realize the problem here.
I’m using Deno.js (Node.js alternative) and, at first, the program wouldn’t even run as it would run out of memory. However, I figured out a workaround for this by passing the v8 engine flag to the run command: —-max-old-space-size=8000. It runs now with my massive array but it’s very slow and my computer essentially becomes a brick while it’s running.
My question is, how can I better deal with data of this size, specifically in a Node.js style environment?
Proposed Idea (please tell me if it’s stupid)
Deno gives a nice interface for creating temp directories and files so I was thinking of saving the data from each request to the API in a temporary .json file and then reading the file(s) where I need them as the next step for this data is to filter it down.
Would this approach improve speed?

Answer 1

To elaborate on my comment, the following awk script counts the number of log entries by IP. I'd start there, and then grep that IP to list the visited resources.
./ip-histogram mylogfile.log
# Output
1 127.0.0.1
3 127.0.2.2
mylogfile.log
127.0.2.2 - - [28/Jul/2006:10:27:10 -0300] "GET /foo" 200 3395
127.0.2.2 - - [28/Jul/2006:10:27:10 -0300] "GET /bar" 200 3395
127.0.2.2 - - [28/Jul/2006:10:27:10 -0300] "GET /baz" 200 3395
127.0.0.1 - - [28/Jul/2006:10:22:04 -0300] "GET /foo" 200 2216
ip-histogram
#!/usr/bin/awk -f
# Counts the frequency of each IP in a log file.
# Expects the IP to be in the first ($1) column.
#
# Sample Output:
# ./ip-histogram mylogfile.log
# 12 1.1.1.1
# 18 2.2.2.2
{
histogram[$1]++
}
END {
for (ip in histogram)
print histogram[ip], ip | "sort -n"
}
If you save the 1hr responses as follows: mylog0001.log, mylog0002.log, you can aggregate them by:
./ip-histogram mylog*.log