Foxx arangodb runs into memory limit - arangodb

Question

I have an simple query runs inside foxx
For u in collection
Filter u.someIndexedSparseFiler !=null
Return {_id:u._id}
This will return millions+ results. In the logs, arango have a message of limited memory heap reached and terminate the process.
reached heap-size limit of #3 interrupting V8 execution (heap size limit 3232954528, used 3060226424) during V8 internal collection
Even though I add the flag --javascript.v8-max-heap 3000 to the start-up. It still runs in the same error. What should I do? Is there a better approach than this

Answer 1

I'm not sure why you're getting out-of-memory errors, but it looks like the data you're returning is overflowing the V8 heap size. Another possibility is that something is causing the engine to miss/ignore the index, causing the engine to load every document before evaluating the someIndexedSparseFiler attribute.
Evaluating millions of documents (or lots of large documents) would not only cost a lot of disk/memory I/O, but could also require a lot of RAM. Try using the explain feature to return a query analysis - it should tell you what is going wrong.
For comparison, my query...
FOR u IN myCollection
FILTER u.someIndexedSparseFiler != null
RETURN u._id
...returns this when I click "explain":
Query String (82 chars, cacheable: true):
FOR u IN myCollection
FILTER u.someIndexedSparseFiler != null
RETURN u._id
Execution plan:
Id NodeType Est. Comment
1 SingletonNode 1 * ROOT
7 IndexNode 5 - FOR u IN myCollection /* persistent index scan, projections: `_id` */
5 CalculationNode 5 - LET #3 = u.`_id` /* attribute expression */ /* collections used: u : myCollection */
6 ReturnNode 5 - RETURN #3
Indexes used:
By Name Type Collection Unique Sparse Selectivity Fields Ranges
7 idx_1667363882689101824 persistent myCollection false true 100.00 % [ `someIndexedSparseFiler` ] *
Optimization rules applied:
Id RuleName
1 move-calculations-up
2 move-filters-up
3 move-calculations-up-2
4 move-filters-up-2
5 use-indexes
6 remove-filter-covered-by-index
7 remove-unnecessary-calculations-2
8 reduce-extraction-to-projection
Note that it listss my sparse index under Indexes used:. Also, try changing the != to == and you will see that now it ignores the index! This is because the optimizer knows a sparse index will never have a null value, so it skips it.
If you aren't familiar with it, the "explain" functionality is extremely useful (indispensable, really) when tuning queries and creating indexes. Also, remember that indexes should match your query; in this case, the index should only have one attribute or the "selectivity" quotient may be too low and the engine will ignore it.