Thanks in advance for the help.
I'm having this issue with Alfresco Process Service where the memory is reaching the JVM limit, CPU is usually below 10%. Usually happening quicker when we make requests to the endpoint (/activiti-app/api/enterprise/tasks/query). Just wondering has anyone seen this issue before?
Config:
-Xms8g -Xmx16g
Heap dump:
1 instance org.hibernate.engine.internal.StatefulPersistenceContext Retained Size 10.660 MB (97%)
1 instance of org.hibernate.engine.internal.EntityEntryContext 10.656 MB (100%)
67231 instances of org.hibernate.engine.internal.EntityEntryContext$ManagedEntityImpl 10.655 MB (99.9%)
67231 instances of org.hibernate.engine.spi.EntityEntry 10.653 MB (99.9%)
67230 instances of java.lang.Object[] 10.648 MB (99.9%)
67230 instances of byte[] 10.609 MB (99.5%)
Endpoint: /activiti-app/api/enterprise/tasks/query
Request Body:
{
"appDefinitionId": null,
"sort": "created-desc",
"size": 50,
"start": 0,
"text": "",
"state": "completed"
}
References:
hibernate-commons-annotations-4.0.2.Final
hibernate-core-4.2.16.Final
hibernate-entitymanager-4.2.16.Final
hibernate-jpa-2.0-api-1.0.1.Final
hibernate-jpa-2.0-api-1.0.1.Final
hibernate-validator-5.3.6.Final
Related
Hello experts
I have a mongodb server running on my 2GB ec2 instance using following command
mongod --wiredTigerCacheSizeGB=0.5
Below is my memory usage:
------------------------------------------------
MALLOC: 509658616 ( 486.0 MiB) Bytes in use by application
MALLOC: + 100265984 ( 95.6 MiB) Bytes in page heap freelist
MALLOC: + 10784344 ( 10.3 MiB) Bytes in central cache freelist
MALLOC: + 8235328 ( 7.9 MiB) Bytes in transfer cache freelist
MALLOC: + 3289712 ( 3.1 MiB) Bytes in thread cache freelists
MALLOC: + 4063232 ( 3.9 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 636297216 ( 606.8 MiB) Actual memory used (physical + swap)
MALLOC: + 385024 ( 0.4 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 636682240 ( 607.2 MiB) Virtual address space used
MALLOC:
MALLOC: 23555 Spans in use
MALLOC: 26 Thread heaps in use
MALLOC: 4096 Tcmalloc page size
------------------------------------------------
As per my understanding Virtual address space used is Total memory occupied by Mongodb.
It would great help if anyone can tell me why it is increasing above the Limit set which is 0.5(500MB)
----------------More Info--------------------
{
"name" : "wiredTiger",
"supportsCommittedReads" : true,
"oldestRequiredTimestampForCrashRecovery" : Timestamp(0, 0),
"supportsPendingDrops" : true,
"dropPendingIdents" : NumberLong(0),
"supportsSnapshotReadConcern" : true,
"readOnly" : false,
"persistent" : true,
"backupCursorOpen" : false
}
{
"db" : "Database",
"collections" : 43,
"views" : 0,
"objects" : 2780441,
"avgObjSize" : 2756.87204116182,
"dataSize" : 7665320055.0,
"storageSize" : 2320449536.0,
"numExtents" : 0,
"indexes" : 58,
"indexSize" : 156573696.0,
"scaleFactor" : 1.0,
"fsUsedSize" : 77780537344.0,
"fsTotalSize" : 214746263552.0,
"ok" : 1.0
}
MongDB memory consumption goes higher than set value in wiredTigercacheSizeGB
Wired tiger is a component of MongoDB. There are many other components. It is expected that memory usage for the entire server would be greater than the limit imposed on one aspect of one component of the server.
Last two lines of code below are the issue. I have line of sight to the csv file in the bucket as can be seen in the printout below, the file in the bucket is an object that is returned with key/value conventions. The problem is the .read(). It ALWAYS times out. Per the pointers when I first posted this question I've changed my settings in AWS to 3 minutes before a function times out and I also try to download it but that returns None. I guess the central questions are why does the .read() function take so long and what is missing in my download_file command? The file is small: 1KB. Any help appreciated thanks
import boto3
import csv
s3 = boto3.resource('s3')
bucket = s3.Bucket('polly-partner')
obj = bucket.Object(key='CyclingLog.csv')
def lambda_handler(event, context):
response = obj.get()
print(response)
key = obj.key
filepath = '/tmp/' + key
print(bucket.download_file(key, filepath))
lines = response['Body'].read()
print(lines)
Printout is:
Response:
{
"errorType": "Runtime.ExitError",
"errorMessage": "RequestId: 541f6cc6-2195-409a-88d3-e98c57fbd539 Error: Runtime exited with error: signal: killed"
}
Request ID:
"541f6cc6-2195-409a-88d3-e98c57fbd539"
Function Logs:
START RequestId: 541f6cc6-2195-409a-88d3-e98c57fbd539 Version: $LATEST
{'ResponseMetadata': {'RequestId': '0860AE16F7A96522', 'HostId': 'D6k1kFcCv9Qz70ANXjEnPQEFsKpAntqJND9FRf5diae3WWmDbVDJENkPCd1oOOOfFt8BJ8b8OOY=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'D6k1kFcCv9Qz70ANXjEnPQEFsKpAntqJND9FRf5diae3WWmDbVDJENkPCd1oOOOfFt8BJ8b8OOY=', 'x-amz-request-id': '0860AE16F7A96522', 'date': 'Wed, 01 Apr 2020 17:51:49 GMT', 'last-modified': 'Thu, 19 Mar 2020 17:17:37 GMT', 'etag': '"b56479c4073a90943b3d862d5d4ff38d-6"', 'accept-ranges': 'bytes', 'content-type': 'text/csv', 'content-length': '50000056', 'server': 'AmazonS3'}, 'RetryAttempts': 1}, 'AcceptRanges': 'bytes', 'LastModified': datetime.datetime(2020, 3, 19, 17, 17, 37, tzinfo=tzutc()), 'ContentLength': 50000056, 'ETag': '"b56479c4073a90943b3d862d5d4ff38d-6"', 'ContentType': 'text/csv', 'Metadata': {}, 'Body': <botocore.response.StreamingBody object at 0x7f536df1ddc0>}
None
END RequestId: 541f6cc6-2195-409a-88d3-e98c57fbd539
REPORT RequestId: 541f6cc6-2195-409a-88d3-e98c57fbd539 Duration: 12923.11 ms Billed Duration: 13000 ms Memory Size: 128 MB Max Memory Used: 129 MB Init Duration: 362.26 ms
RequestId: 541f6cc6-2195-409a-88d3-e98c57fbd539 Error: Runtime exited with error: signal: killed
Runtime.ExitError
The error message says: Task timed out after 3.00 seconds
You can increase the Timeout on a Lambda function by opening the function in the console, going to the Basic settings section and clicking Edit.
While you say that you increased this timeout setting, the fact that it is timing-out after exactly 3 seconds suggests that the setting has not been changed.
I know this is an old post, (and hopefully solved long ago!), but I ended up here so I'll share my findings.
These generic Runtime error messages:
"Error: Runtime exited with error: signal: killed Runtime.ExitError"
...when accompanied by something like this on the REPORT line:
Memory Size: 128 MB Max Memory Used: 129 MB Init Duration: 362.26 ms
...Looks like a low memory issue. Especially when "Max Memory Used" is >= "Memory Size"
From what I've seen, Lambda can and often will utilize up to 100% memory without issue (Discussed in this post). But when you attempt to load data into memory, or perform memory intensive processing (copying large data sets stored in variables?), the Python runtime can hit a memory error and exit.
Unfortunately, it isn't very well documented, or logged, or captured with CloudWatch metrics.
I believe the same error in NodeJS runtime looks like:
"Error: Runtime exited with error: signal: aborted (core dumped)"
I converted all my logs in my Lambda to JSON using Winston to look like this:
{
"#timestamp": "2018-10-04T12:24:48.930Z",
"message": "Logging some data about my app",
"level": "INFO"
}
How can I convert the AWS Default logging to my custom format?
Is this possible? Is there a way to convert that stream?
By "Default Logging" I mean this stuff:
START RequestId: 7c759123-a7d0-11e8-b4a4-318095fdonutz Version: $LATEST
END RequestId: 7c759123-a7d0-11e8-b4a4-318095fdonutz
REPORT RequestId: 7c759123-a7d0-11e8-b4a4-318095fdonutz Duration: 109.21 ms Billed Duration: 200 ms Memory Size: 128 MB Max Memory Used: 46 MB
I want to customize node application memory size, also I want to add some middleware to record realtime application memory usage and capacity.
I know memory limit can be raised by setting --max-old-space-size to a maximum of ~1gb (32-bit) and ~1.7gb (64-bit)
Example:
node --max-old-space-size=1024 app.js
But when I use v8, how to verify memory capacity and usage?
Update:
I have used another approach to raise memory for app with v8.
const v8 = require('v8');
v8.setFlagsFromString('--max_old_space_size=2048');
This is log for HeapSpaceStatistics
########## v8.getHeapSpaceStatistics() ################
[
{
"space_name": "new_space",
"space_size": 1048576,
"space_used_size": 781704,
"space_available_size": 249464,
"physical_space_size": 798416
},
{
"space_name": "old_space",
"space_size": 18874368,
"space_used_size": 16844024,
"space_available_size": 270840,
"physical_space_size": 18772464
},
{
"space_name": "code_space",
"space_size": 4718592,
"space_used_size": 4344800,
"space_available_size": 95424,
"physical_space_size": 4586176
},
{
"space_name": "map_space",
"space_size": 1572864,
"space_used_size": 962104,
"space_available_size": 584648,
"physical_space_size": 1187888
},
{
"space_name": "large_object_space",
"space_size": 1626112,
"space_used_size": 1603664,
"space_available_size": 1471614464,
"physical_space_size": 1626112
}
]
I have set memory limit for 2GB, base on log, how to verify that?
The --trace_gc V8 option and process.memoryUsage() should give you good overall numbers
node --trace_gc --max_old_space_size=4096 -e '
let a = []
for( let i=1; i>0; i++){
a.push({ a: "testing" })
if ( i % 1000000 === 0 ) console.log("%j", process.memoryUsage())
if ( i % 10000000 === 0 ) console.log(v8.getHeapSpaceStatistics())
}
'
Note that the node process won't start at the set max_old_space_size size, but that will allow node to grow to the value. I don't believe V8 has the concept of setting a minimum old space size like the Java VM does.
Also note that the other memory spaces, like "new" and possibly "external" memory, are used on top of the old space size, so what you set for max_old_space_size is not the actual OS process memory limit. The OS value will likely be higher again.
I am creating an application with nodejs, it is using Hapi for web framework and knex for sql builder, The primary code is following:
server.route({
method: 'POST',
path: '/location',
config: {
tags: ['api'],
validate: {
payload: {
longitude: Joi.string().regex(/^\d+\.\d+$/).required(),
latitude: Joi.string().regex(/^\d+\.\d+$/).required(),
date: Joi.string().regex(/^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}.+$/).required(),
phone: Joi.string().required(/^1\d{10}$/)
}
}
},
handler: createLocation
})
async function createLocation(request, reply){
try{
const data = await knex('locations').insert(request.payload)
reply(data)
}catch(error){
reply(error)
}
}
It simply insert some date to postgresql. I am using Wrk to benchmark it Concurrent throughput in Google Compute Engine(cheapest machine), Result:
$ wrk -c 100 -t 12 http://localhost/api/location -s wrk.lua
Running 10s test # http://panpan.tuols.com/api/location
12 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 542.22ms 102.93ms 1.02s 88.32%
Req/Sec 21.95 18.74 70.00 78.71%
1730 requests in 10.02s, 0.94MB read
Requests/sec: 172.65
Transfer/sec: 96.44KB
Then I using pgbench to test postgresql insert performance:
$ pgbench location -U postgres -h localhost -r -t 1000 -f index.sql -c 10
transaction type: index.sql
scaling factor: 1
query mode: simple
number of clients: 10
number of threads: 1
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
latency average = 1.663 ms
tps = 6014.610692 (including connections establishing)
tps = 6029.973067 (excluding connections establishing)
script statistics:
- statement latencies in milliseconds:
1.595 INSERT INTO "public"."locations"("phone", "longitude", "latitude", "date", "createdAt", "updatedAt") VALUES('18382383428', '123,33', '123.33', 'now()', 'now()', 'now()') RETURNING "id", "phone", "longitude", "latitude", "date", "createdAt", "updatedAt";
The nodejs is 172.65 req/s, and the postgresql internal native is 6000 req/s, The are actually do some thing, if ignore the http overhead, the difference should not so much big, Why the performance is so much hug different? It is nodejs or node-pg package problem?