How to limit node application memory size, and how to verify current memory usage and capacity? - node.js

I want to customize node application memory size, also I want to add some middleware to record realtime application memory usage and capacity.
I know memory limit can be raised by setting --max-old-space-size to a maximum of ~1gb (32-bit) and ~1.7gb (64-bit)
Example:
node --max-old-space-size=1024 app.js
But when I use v8, how to verify memory capacity and usage?
Update:
I have used another approach to raise memory for app with v8.
const v8 = require('v8');
v8.setFlagsFromString('--max_old_space_size=2048');
This is log for HeapSpaceStatistics
########## v8.getHeapSpaceStatistics() ################
[
{
"space_name": "new_space",
"space_size": 1048576,
"space_used_size": 781704,
"space_available_size": 249464,
"physical_space_size": 798416
},
{
"space_name": "old_space",
"space_size": 18874368,
"space_used_size": 16844024,
"space_available_size": 270840,
"physical_space_size": 18772464
},
{
"space_name": "code_space",
"space_size": 4718592,
"space_used_size": 4344800,
"space_available_size": 95424,
"physical_space_size": 4586176
},
{
"space_name": "map_space",
"space_size": 1572864,
"space_used_size": 962104,
"space_available_size": 584648,
"physical_space_size": 1187888
},
{
"space_name": "large_object_space",
"space_size": 1626112,
"space_used_size": 1603664,
"space_available_size": 1471614464,
"physical_space_size": 1626112
}
]
I have set memory limit for 2GB, base on log, how to verify that?

The --trace_gc V8 option and process.memoryUsage() should give you good overall numbers
node --trace_gc --max_old_space_size=4096 -e '
let a = []
for( let i=1; i>0; i++){
a.push({ a: "testing" })
if ( i % 1000000 === 0 ) console.log("%j", process.memoryUsage())
if ( i % 10000000 === 0 ) console.log(v8.getHeapSpaceStatistics())
}
'
Note that the node process won't start at the set max_old_space_size size, but that will allow node to grow to the value. I don't believe V8 has the concept of setting a minimum old space size like the Java VM does.
Also note that the other memory spaces, like "new" and possibly "external" memory, are used on top of the old space size, so what you set for max_old_space_size is not the actual OS process memory limit. The OS value will likely be higher again.

Related

Alfresco Activiti Memory Leak

Thanks in advance for the help.
I'm having this issue with Alfresco Process Service where the memory is reaching the JVM limit, CPU is usually below 10%. Usually happening quicker when we make requests to the endpoint (/activiti-app/api/enterprise/tasks/query). Just wondering has anyone seen this issue before?
Config:
-Xms8g -Xmx16g
Heap dump:
1 instance org.hibernate.engine.internal.StatefulPersistenceContext Retained Size 10.660 MB (97%)
1 instance of org.hibernate.engine.internal.EntityEntryContext 10.656 MB (100%)
67231 instances of org.hibernate.engine.internal.EntityEntryContext$ManagedEntityImpl 10.655 MB (99.9%)
67231 instances of org.hibernate.engine.spi.EntityEntry 10.653 MB (99.9%)
67230 instances of java.lang.Object[] 10.648 MB (99.9%)
67230 instances of byte[] 10.609 MB (99.5%)
Endpoint: /activiti-app/api/enterprise/tasks/query
Request Body:
{
"appDefinitionId": null,
"sort": "created-desc",
"size": 50,
"start": 0,
"text": "",
"state": "completed"
}
References:
hibernate-commons-annotations-4.0.2.Final
hibernate-core-4.2.16.Final
hibernate-entitymanager-4.2.16.Final
hibernate-jpa-2.0-api-1.0.1.Final
hibernate-jpa-2.0-api-1.0.1.Final
hibernate-validator-5.3.6.Final

Jest is not generating the report table in my terminal

The issue I'm facing is the lack of report tables in my terminal once I run my npm test.
I know for a fact that the reports are being generated, since I can see the files in the coverage directory.
However, it's a bit annoying and despite my debugging, I can't seem to find out what the issue is.
Here is my jest.config.js:
/*
* For a detailed explanation regarding each configuration property and type check, visit:
* https://jestjs.io/docs/en/configuration.html
*/
module.exports = {
// Automatically clear mock calls and instances between every test
clearMocks: true,
// Indicates whether the coverage information should be collected while executing the test
collectCoverage: true,
// The directory where Jest should output its coverage files
coverageDirectory: "coverage",
// Indicates which provider should be used to instrument code for coverage
coverageProvider: "v8",
reporters: [
"default",
[
"jest-junit",
{
outputDirectory: "./coverage",
outputName: "unit_tests_coverage.xml",
},
],
],
// A list of reporter names that Jest uses when writing coverage reports
coverageReporters: ["cobertura", "lcov"],
// The maximum amount of workers used to run your tests. Can be specified as % or a number. E.g. maxWorkers: 10% will use 10% of your CPU amount + 1 as the maximum worker number. maxWorkers: 2 will use a maximum of 2 workers.
maxWorkers: "50%",
// A list of paths to directories that Jest should use to search for files in
roots: ["test"],
testEnvironment: "node",
// Options that will be passed to the testEnvironment
// testEnvironmentOptions: {},
testRegex: ["/test/.*\\.(test|spec)?\\.(ts|tsx)$"],
transform: {
"^.+\\.ts?$": ["babel-jest"],
},
}
At the end of every test execution, I get like a summary like this:
Test Suites: 9 passed, 9 total
Tests: 155 passed, 155 total
Snapshots: 0 total
Time: 10.248 s
But no table showing line coverage, branch coverage... etc.
Is my jest.config.js incorrect or am I missing something?
Thanks in advance for your help!
Thanks to #jonrsharpe, I managed to find out what the issue was.
Since I was using reporters, the default one (text) was overridden. So in order to see it again, I had to specify it manually (check docs)
...
coverageReporters: ["cobertura", "lcov", "text"],
...

"[circuit_breaking_exception] [parent]" Data too large, data for "[<http_request>]" would be error

After smoothly working for more than 10 months, I start getting this error on production suddenly while doing simple search queries.
{
"error" : {
"root_cause" : [
{
"type" : "circuit_breaking_exception",
"reason" : "[parent] Data too large, data for [<http_request>] would be [745522124/710.9mb], which is larger than the limit of [745517875/710.9mb]",
"bytes_wanted" : 745522124,
"bytes_limit" : 745517875
}
],
"type" : "circuit_breaking_exception",
"reason" : "[parent] Data too large, data for [<http_request>] would be [745522124/710.9mb], which is larger than the limit of [745517875/710.9mb]",
"bytes_wanted" : 745522124,
"bytes_limit" : 745517875
},
"status" : 503
}
Initially, I was getting this error while doing simple term queries when I got this circuit_breaking_exception error, To debug this I tried _cat/health query on elasticsearch cluster, but still, the same error, even the simplest query localhost:9200 is giving the same error Not sure what happens to the cluster suddenly.
Her is my circuit breaker status:
"breakers" : {
"request" : {
"limit_size_in_bytes" : 639015321,
"limit_size" : "609.4mb",
"estimated_size_in_bytes" : 0,
"estimated_size" : "0b",
"overhead" : 1.0,
"tripped" : 0
},
"fielddata" : {
"limit_size_in_bytes" : 639015321,
"limit_size" : "609.4mb",
"estimated_size_in_bytes" : 406826332,
"estimated_size" : "387.9mb",
"overhead" : 1.03,
"tripped" : 0
},
"in_flight_requests" : {
"limit_size_in_bytes" : 1065025536,
"limit_size" : "1015.6mb",
"estimated_size_in_bytes" : 560,
"estimated_size" : "560b",
"overhead" : 1.0,
"tripped" : 0
},
"accounting" : {
"limit_size_in_bytes" : 1065025536,
"limit_size" : "1015.6mb",
"estimated_size_in_bytes" : 146387859,
"estimated_size" : "139.6mb",
"overhead" : 1.0,
"tripped" : 0
},
"parent" : {
"limit_size_in_bytes" : 745517875,
"limit_size" : "710.9mb",
"estimated_size_in_bytes" : 553214751,
"estimated_size" : "527.5mb",
"overhead" : 1.0,
"tripped" : 0
}
}
I found a similar issue hereGithub Issue that suggests increasing circuit breaker memory or disabling the same. But I am not sure what to choose. Please help!
Elasticsearch Version 6.3
After some more research finally, I found a solution for this i.e
We should not disable circuit breaker as it might result in OOM error and eventually might crash elasticsearch.
dynamically increasing circuit breaker memory percentage is good but it is also a temporary solution because at the end after solution increased percentage might also fill up.
Finally, we have a third option i.e increase overall JVM heap size which is 1GB by default but as recommended it should be around 30-32 GB on production, also it should be less than 50% of available total memory.
For more info check this for good JVM memory configurations of elasticsearch on production, Heap: Sizing and Swapping
In my case I have an index with large documents, each document has ~30 KB and more than 130 fields (nested objects, arrays, dates and ids).
and I was searching all fields using this DSL query:
query_string: {
query: term,
analyze_wildcard: true,
fields: ['*'], // search all fields
fuzziness: 'AUTO'
}
Since full-text searches are expensive. Searching through multiple fields at once is even more expensive. Expensive in terms of computing power, not storage.
Therefore:
The more fields a query_string or multi_match query targets, the
slower it is. A common technique to improve search speed over multiple
fields is to copy their values into a single field at index time, and
then use this field at search time.
please refer to ELK docs that recommends searching as few fields as possible with the help of copy-to directive.
After I changed my query to search one field:
query_string: {
query: term,
analyze_wildcard: true,
fields: ['search_field'] // search in one field
}
everything worked like a charm.
I got this error with my docker container so I increase the java_opts to 1GB and now it works without any error.
Here are the docker-compose.yml
version: '1'
services:
elasticsearch-cont:
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
container_name: elasticsearch
environment:
- "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
- discovery.type=single-node
ulimits:
memlock:
soft: -1
hard: -1
ports:
- 9200:9200
- 9300:9300
networks:
- elastic
networks:
elastic:
driver: bridge
In my case, I also have an index with large documents which store system running logs and I searched the index with all fields. I use the Java Client API, like this:
TermQueryBuilder termQueryBuilder = QueryBuilders.termQuery("uid", uid);
searchSourceBuilder.query(termQueryBuilder);
When I changed my code like this:
TermQueryBuilder termQueryBuilder = QueryBuilders.termQuery("uid", uid);
searchSourceBuilder.fetchField("uid");
searchSourceBuilder.fetchSource(false);
searchSourceBuilder.query(termQueryBuilder);
the error disappeared.

huge performance gap with nodejs and postgresql benchmark

I am creating an application with nodejs, it is using Hapi for web framework and knex for sql builder, The primary code is following:
server.route({
method: 'POST',
path: '/location',
config: {
tags: ['api'],
validate: {
payload: {
longitude: Joi.string().regex(/^\d+\.\d+$/).required(),
latitude: Joi.string().regex(/^\d+\.\d+$/).required(),
date: Joi.string().regex(/^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}.+$/).required(),
phone: Joi.string().required(/^1\d{10}$/)
}
}
},
handler: createLocation
})
async function createLocation(request, reply){
try{
const data = await knex('locations').insert(request.payload)
reply(data)
}catch(error){
reply(error)
}
}
It simply insert some date to postgresql. I am using Wrk to benchmark it Concurrent throughput in Google Compute Engine(cheapest machine), Result:
$ wrk -c 100 -t 12 http://localhost/api/location -s wrk.lua
Running 10s test # http://panpan.tuols.com/api/location
12 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 542.22ms 102.93ms 1.02s 88.32%
Req/Sec 21.95 18.74 70.00 78.71%
1730 requests in 10.02s, 0.94MB read
Requests/sec: 172.65
Transfer/sec: 96.44KB
Then I using pgbench to test postgresql insert performance:
$ pgbench location -U postgres -h localhost -r -t 1000 -f index.sql -c 10
transaction type: index.sql
scaling factor: 1
query mode: simple
number of clients: 10
number of threads: 1
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
latency average = 1.663 ms
tps = 6014.610692 (including connections establishing)
tps = 6029.973067 (excluding connections establishing)
script statistics:
- statement latencies in milliseconds:
1.595 INSERT INTO "public"."locations"("phone", "longitude", "latitude", "date", "createdAt", "updatedAt") VALUES('18382383428', '123,33', '123.33', 'now()', 'now()', 'now()') RETURNING "id", "phone", "longitude", "latitude", "date", "createdAt", "updatedAt";
The nodejs is 172.65 req/s, and the postgresql internal native is 6000 req/s, The are actually do some thing, if ignore the http overhead, the difference should not so much big, Why the performance is so much hug different? It is nodejs or node-pg package problem?

A kind of memory leak on nodeJs?

I have some problems with a "memory leak issue" in several versions of NodeJs.
EDIT :
I've tried several version of nodeJs : v0.8.10, v0.10.22, v0.10.23, v0.10.24 ...
I'm under MacOsX 10.9.1.
Here is my test.js :
function a() {
console.log( process.memoryUsage() );
global.gc();
}
var b = setInterval(a,5000);
I launch it with :
node --expose-gc test.js
Here is the output :
{ rss: 12312576, heapTotal: 5115392, heapUsed: 2429656 }
{ rss: 13406208, heapTotal: 6163968, heapUsed: 2139848 }
{ rss: 13463552, heapTotal: 6163968, heapUsed: 1911320 }
{ rss: 14295040, heapTotal: 6163968, heapUsed: 1919632 }
{ rss: 14434304, heapTotal: 6163968, heapUsed: 1898944 }
... etc ...
As you can notice, RSS doesn't stop growing, heapTotal is stable, heapUsed moves but stays balanced.
Does somebody understand why RSS (Resident Set Size) is still growing ? Is it normal ? Is it a native node memory leak ?
What's happen on your machines ?
EDIT :
I've tried on a linux server and the headTotal and RSS are stable ! Looks like a issue on nodeJs for MacOs ?! Somebody knows why ?
Thanks !

Resources