Hello experts
I have a mongodb server running on my 2GB ec2 instance using following command
mongod --wiredTigerCacheSizeGB=0.5
Below is my memory usage:
------------------------------------------------
MALLOC: 509658616 ( 486.0 MiB) Bytes in use by application
MALLOC: + 100265984 ( 95.6 MiB) Bytes in page heap freelist
MALLOC: + 10784344 ( 10.3 MiB) Bytes in central cache freelist
MALLOC: + 8235328 ( 7.9 MiB) Bytes in transfer cache freelist
MALLOC: + 3289712 ( 3.1 MiB) Bytes in thread cache freelists
MALLOC: + 4063232 ( 3.9 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 636297216 ( 606.8 MiB) Actual memory used (physical + swap)
MALLOC: + 385024 ( 0.4 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 636682240 ( 607.2 MiB) Virtual address space used
MALLOC:
MALLOC: 23555 Spans in use
MALLOC: 26 Thread heaps in use
MALLOC: 4096 Tcmalloc page size
------------------------------------------------
As per my understanding Virtual address space used is Total memory occupied by Mongodb.
It would great help if anyone can tell me why it is increasing above the Limit set which is 0.5(500MB)
----------------More Info--------------------
{
"name" : "wiredTiger",
"supportsCommittedReads" : true,
"oldestRequiredTimestampForCrashRecovery" : Timestamp(0, 0),
"supportsPendingDrops" : true,
"dropPendingIdents" : NumberLong(0),
"supportsSnapshotReadConcern" : true,
"readOnly" : false,
"persistent" : true,
"backupCursorOpen" : false
}
{
"db" : "Database",
"collections" : 43,
"views" : 0,
"objects" : 2780441,
"avgObjSize" : 2756.87204116182,
"dataSize" : 7665320055.0,
"storageSize" : 2320449536.0,
"numExtents" : 0,
"indexes" : 58,
"indexSize" : 156573696.0,
"scaleFactor" : 1.0,
"fsUsedSize" : 77780537344.0,
"fsTotalSize" : 214746263552.0,
"ok" : 1.0
}
MongDB memory consumption goes higher than set value in wiredTigercacheSizeGB
Wired tiger is a component of MongoDB. There are many other components. It is expected that memory usage for the entire server would be greater than the limit imposed on one aspect of one component of the server.
Related
Thanks in advance for the help.
I'm having this issue with Alfresco Process Service where the memory is reaching the JVM limit, CPU is usually below 10%. Usually happening quicker when we make requests to the endpoint (/activiti-app/api/enterprise/tasks/query). Just wondering has anyone seen this issue before?
Config:
-Xms8g -Xmx16g
Heap dump:
1 instance org.hibernate.engine.internal.StatefulPersistenceContext Retained Size 10.660 MB (97%)
1 instance of org.hibernate.engine.internal.EntityEntryContext 10.656 MB (100%)
67231 instances of org.hibernate.engine.internal.EntityEntryContext$ManagedEntityImpl 10.655 MB (99.9%)
67231 instances of org.hibernate.engine.spi.EntityEntry 10.653 MB (99.9%)
67230 instances of java.lang.Object[] 10.648 MB (99.9%)
67230 instances of byte[] 10.609 MB (99.5%)
Endpoint: /activiti-app/api/enterprise/tasks/query
Request Body:
{
"appDefinitionId": null,
"sort": "created-desc",
"size": 50,
"start": 0,
"text": "",
"state": "completed"
}
References:
hibernate-commons-annotations-4.0.2.Final
hibernate-core-4.2.16.Final
hibernate-entitymanager-4.2.16.Final
hibernate-jpa-2.0-api-1.0.1.Final
hibernate-jpa-2.0-api-1.0.1.Final
hibernate-validator-5.3.6.Final
I encountered a problem when using the swupdate image built by yocto.
Software Update started !
[network_initializer] : Software update started
[extract_file_to_tmp] : Found file
[extract_file_to_tmp] : filename sw-description
[extract_file_to_tmp] : size 303
[get_common_fields] : Version 0.1.0
[get_common_fields] : Description Firmware update for XXXXX Project
[parse_hw_compatibility] : Accepted Hw Revision : 1.0
[parse_hw_compatibility] : Accepted Hw Revision : 1.2
[parse_hw_compatibility] : Accepted Hw Revision : 1.3
[_parse_images] : Found Image: rootfs.ext4.gz in device : /dev/mmcblk2p4 for handler raw
[check_hw_compatibility] : Hardware myir Revision: 1.0
[check_hw_compatibility] : Hardware compatibility verified
[extract_files] : Found file
[extract_files] : filename rootfs.ext4.gz
[extract_files] : size 373258053 required
ERROR : Not enough free space to extract rootfs.ext4.gz (needed 373258053, got 223219712)
Image invalid or corrupted. Not installing ...
[network_initializer] : Main thread sleep again !
Waiting for requests...
ERROR : Writing to IPC fails due to Broken pipe
As shown in the figure, it indicates that there is not enough space, and then I use resize2fs /dev/mmcblk2p4 to expand the space. Now it has 1g of space. But still the same hint. Please let me know what you think.
Try using the installed-directly flag in the sw-description file.
files: (
{
filename = "rootfs.ext4.gz";
sha256 = "bc57b9c737033d0d6826db51618d596da7ecf3fdc0cb48dc9986a6094f529413";
type = "archive";
path = "/path/to/extract";
preserve-attributes = true;
installed-directly = true; <---------- this option
properties: {
create-destination = "true";
}
}
After smoothly working for more than 10 months, I start getting this error on production suddenly while doing simple search queries.
{
"error" : {
"root_cause" : [
{
"type" : "circuit_breaking_exception",
"reason" : "[parent] Data too large, data for [<http_request>] would be [745522124/710.9mb], which is larger than the limit of [745517875/710.9mb]",
"bytes_wanted" : 745522124,
"bytes_limit" : 745517875
}
],
"type" : "circuit_breaking_exception",
"reason" : "[parent] Data too large, data for [<http_request>] would be [745522124/710.9mb], which is larger than the limit of [745517875/710.9mb]",
"bytes_wanted" : 745522124,
"bytes_limit" : 745517875
},
"status" : 503
}
Initially, I was getting this error while doing simple term queries when I got this circuit_breaking_exception error, To debug this I tried _cat/health query on elasticsearch cluster, but still, the same error, even the simplest query localhost:9200 is giving the same error Not sure what happens to the cluster suddenly.
Her is my circuit breaker status:
"breakers" : {
"request" : {
"limit_size_in_bytes" : 639015321,
"limit_size" : "609.4mb",
"estimated_size_in_bytes" : 0,
"estimated_size" : "0b",
"overhead" : 1.0,
"tripped" : 0
},
"fielddata" : {
"limit_size_in_bytes" : 639015321,
"limit_size" : "609.4mb",
"estimated_size_in_bytes" : 406826332,
"estimated_size" : "387.9mb",
"overhead" : 1.03,
"tripped" : 0
},
"in_flight_requests" : {
"limit_size_in_bytes" : 1065025536,
"limit_size" : "1015.6mb",
"estimated_size_in_bytes" : 560,
"estimated_size" : "560b",
"overhead" : 1.0,
"tripped" : 0
},
"accounting" : {
"limit_size_in_bytes" : 1065025536,
"limit_size" : "1015.6mb",
"estimated_size_in_bytes" : 146387859,
"estimated_size" : "139.6mb",
"overhead" : 1.0,
"tripped" : 0
},
"parent" : {
"limit_size_in_bytes" : 745517875,
"limit_size" : "710.9mb",
"estimated_size_in_bytes" : 553214751,
"estimated_size" : "527.5mb",
"overhead" : 1.0,
"tripped" : 0
}
}
I found a similar issue hereGithub Issue that suggests increasing circuit breaker memory or disabling the same. But I am not sure what to choose. Please help!
Elasticsearch Version 6.3
After some more research finally, I found a solution for this i.e
We should not disable circuit breaker as it might result in OOM error and eventually might crash elasticsearch.
dynamically increasing circuit breaker memory percentage is good but it is also a temporary solution because at the end after solution increased percentage might also fill up.
Finally, we have a third option i.e increase overall JVM heap size which is 1GB by default but as recommended it should be around 30-32 GB on production, also it should be less than 50% of available total memory.
For more info check this for good JVM memory configurations of elasticsearch on production, Heap: Sizing and Swapping
In my case I have an index with large documents, each document has ~30 KB and more than 130 fields (nested objects, arrays, dates and ids).
and I was searching all fields using this DSL query:
query_string: {
query: term,
analyze_wildcard: true,
fields: ['*'], // search all fields
fuzziness: 'AUTO'
}
Since full-text searches are expensive. Searching through multiple fields at once is even more expensive. Expensive in terms of computing power, not storage.
Therefore:
The more fields a query_string or multi_match query targets, the
slower it is. A common technique to improve search speed over multiple
fields is to copy their values into a single field at index time, and
then use this field at search time.
please refer to ELK docs that recommends searching as few fields as possible with the help of copy-to directive.
After I changed my query to search one field:
query_string: {
query: term,
analyze_wildcard: true,
fields: ['search_field'] // search in one field
}
everything worked like a charm.
I got this error with my docker container so I increase the java_opts to 1GB and now it works without any error.
Here are the docker-compose.yml
version: '1'
services:
elasticsearch-cont:
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
container_name: elasticsearch
environment:
- "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
- discovery.type=single-node
ulimits:
memlock:
soft: -1
hard: -1
ports:
- 9200:9200
- 9300:9300
networks:
- elastic
networks:
elastic:
driver: bridge
In my case, I also have an index with large documents which store system running logs and I searched the index with all fields. I use the Java Client API, like this:
TermQueryBuilder termQueryBuilder = QueryBuilders.termQuery("uid", uid);
searchSourceBuilder.query(termQueryBuilder);
When I changed my code like this:
TermQueryBuilder termQueryBuilder = QueryBuilders.termQuery("uid", uid);
searchSourceBuilder.fetchField("uid");
searchSourceBuilder.fetchSource(false);
searchSourceBuilder.query(termQueryBuilder);
the error disappeared.
The laodtest fails to run after changing the arrivalrate from 10 to 100.
Artillery: 1.6.0-27 Artillery Pro: not installed Node.js: v10.15.0 OS: darwin/x64
:test $ artillery run -o report.json artillery.yml
Started phase 0, duration: 10s # 10:01:42(+0000) 2019-03-10
.
<--- Last few GCs --->
[62621:0x102803200] 9478 ms: Mark-sweep 1392.4 (1401.5) -> 1392.3 (1401.5) MB, 20.1 / 0.0 ms (average mu = 0.439, current mu = 0.002)
last resort GC in old space requested
[62621:0x102803200] 9498 ms: Mark-sweep 1392.3 (1401.5) -> 1392.3 (1401.5) MB, 20.6 / 0.0 ms (average mu = 0.277, current mu = 0.001)
last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
0: ExitFrame [pc: 0x38a6205dbe3d]
Security context: 0x1da57481e6e1
1: byteLength [0x1da5274066f1] [buffer.js:526] [bytecode=0x1da597d26509 offset=126](this=0x1da5d7c5fbc1 <JSFunction
Buffer (sfi = 0x1da573a14251)>,string=0x1da597e082b9 ,encoding=0x1da5d92026f1 )
2: arguments adaptor frame: 1->2
3: setContentLength(aka setContentLength) [0x1da5201841e9] [/Users//.nv...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: 0x10003b125 node::Abort() [/Users//.nvm/versions/node/v10.15.0/bin/node]
2: 0x10003b32f node::OnFatalError(char const*, char const*) [/Users//.nvm/versions/node/v10.15.0/bin/node]
3: 0x1001a8e85 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char
const*, bool) [/Users//.nvm/versions/node/v10.15.0/bin/node]
4: 0x1005742a2 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users//.nvm/versions/node/v10.15.0/bin/node]
5: 0x10057d7a4 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment)
[/Users//.nvm/versions/node/v10.15.0/bin/node]
6: 0x10054f055 v8::internal::Factory::NewRawOneByteString(int, v8::internal::PretenureFlag)
[/Users//.nvm/versions/node/v10.15.0/bin/node]
7: 0x1006811a8 v8::internal::String::SlowFlatten(v8::internal::Handlev8::internal::ConsString,
v8::internal::PretenureFlag)
[/Users//.nvm/versions/node/v10.15.0/bin/node]
8: 0x1001c6c1d v8::String::Utf8Length() const [/Users//.nvm/versions/node/v10.15.0/bin/node]
9: 0x10004eaac node::Buffer::(anonymous namespace)::ByteLengthUtf8(v8::FunctionCallbackInfov8::Value const&)
[/Users//.nvm/versions/node/v10.15.0/bin/node]
10: 0x10023170f v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo*)
[/Users//.nvm/versions/node/v10.15.0/bin/node]
11: 0x100230c51 v8::internal::MaybeHandlev8::internal::Object v8::internal::(anonymous
namespace)::HandleApiCallHelper(v8::internal::Isolate*,
v8::internal::Handlev8::internal::HeapObject,
v8::internal::Handlev8::internal::HeapObject,
v8::internal::Handlev8::internal::FunctionTemplateInfo,
v8::internal::Handlev8::internal::Object,
v8::internal::BuiltinArguments)
[/Users//.nvm/versions/node/v10.15.0/bin/node]
12: 0x1002302f0 v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments,
v8::internal::Isolate*) [/Users//.nvm/versions/node/v10.15.0/bin/node]
13: 0x38a6205dbe3d
Abort trap: 6
My tests look like this.
.yml
config:
target: "<URL_REMOVED"
processor: "./getData.js"
phases:
- duration: 10
arrivalRate: 100
scenarios:
- flow:
- function: "getData"
- post:
url: "/api/v2/auth"
json:
productId: "56729b6b77c82288f746c0cf"
capture:
json: "$.data.token"
as: "token"
- post:
url: "/api/v2/sessions"
headers:
Authorization: 'Bearer {{token}}'
json:
productId: "56729b6b77c82288f746c0cf"
jobId: "{{jobId}}"
capture:
json: "$.data.session._id"
as: "sessionId"
- post:
url: "/api/v2/sessions/{{sessionId}}/document"
headers:
Authorization: "Bearer {{token}}"
json:
side: "front"
payload: "{{frontDocument}}"
- get:
url: "/api/v2/sessions/{{sessionId}}/metrics/front"
headers:
Authorization: "Bearer {{token}}"
- get:
url: "/api/v2/sessions/{{sessionId}}/classification"
headers:
Authorization: "Bearer {{token}}"
- get:
url: "/api/v2/sessions/{{sessionId}}/end"
headers:
Authorization: "Bearer {{token}}"
getData.js
'use strict';
var faker = require('faker');
var FRONT_ID = require("./resources/id/front.json");
module.exports = {
getData
};
function getData(userContext, events, done) {
let jobId = faker.random.uuid()
userContext.vars.jobId = jobId;
userContext.vars.frontDocument = FRONT_ID.base64;
return done();
}
Your nodejs, the instance running artillery, is out of RAM. Its default limit is about 1.4G.
Artillery, on *nix, probably gets started from /usr/bin/artillery when you install it with npm install -g.
That file's first line is, very likely,
#!/usr/bin/env node
Try changing that to
#!/usr/bin/env node --max-old-space-size=8192
to get heap space of 8G. But don't grab more heap space than the machine running artillery has physical RAM, or you'll thrash. Read this: how to increase nodejs default memory?
Edit: the question remains: why does your artillery process blow out its heap. It looks like it ran for about 9s before it crashed. It may be that your system under test cannot keep up with the load you place on it with an arrival rate of 100. It may be that artillery keeps creating and queueing in-memory post and get requests, and they don't finish so they don't get released. Do your performance logs give you any hints about this?
The whole point of load-testing is to find the breaking point of your system under test. You wouldn't test a boat by loading ten sandbags into it, then saying "cool it works" and then dumping 100 sandbags into it. The boat would just sink. Why test a server that way? All you learn is it has a capacity somewhere between 10 and 100. Instead, ramp up the load, sandbag by sandbag, until the system starts to stumble.
This is why artillery (and indeed most load-testing systems) has a way to ramp up the load.
Why not try something like this:
- duration: 120
arrivalRate: 10
name: "Two minutes, ten arrivals/sec"
- duration: 600
arrivalRate: 10
rampTo: 100
name: "Ten minutes, gradual ramp to 100 arrivals/sec"
Once you know the arrival rate at which your system stumbles, you can do more detailed tests as needed. And, you can try to rework your code to make it faster, if necessary.
Also, a 10-second test isn't generally long enough to gather worthwhile data. You want to know your system's capacity at a steady high load.
I am trying to get memory resource but for any reason, I cannot. My code is the following:
mmio_base0 = pci_resource_start (dev,0);
mmio_length0 =pci_resource_len(dev,0);
if(!mmio_base0 || ((pci_resource_flags (dev,0) & IORESOURCE_MEM)== 0))
{
dev_err(&(dev->dev), "no mem resource at PCI #0\n");
}
else
{
printk(KERN_ALERT "There is memory at BAR0\n");
if(request_mem_region(mmio_base0,mmio_length0,"driver")==0){
printk(KERN_ALERT "Impossible to get memory\n");
}
}
I always get "Impossible to get memory". So, there are memory but I cannot request it.
cat /proc/iomem
c20000000-c2fffffff : /pcie#ffe250000
c20000000-c2fffffff : PCI Bus 0001:01
c20000000-c200fffff : 0001:01:00.0 //these two memories I want to request
c20100000-c201fffff : 0001:01:00.0
c40000000-c4fffffff : /pcie#ffe270000
c40000000-c4fffffff : PCI Bus 0002:01
There are memories there, but I cannot request them. It always fail! But my uboot detect my device.
PCIe2: Root Complex, x2 gen2, regs # 0xfe250000
02:00.0 - 1957:0808 - Processor
And there is not driver loaded which take these memories:
0001:01:00.0 Power PC: Freescale Semiconductor Inc Device 0808 (rev 10) (prog-if 01)
Flags: bus master, fast devsel, latency 0, IRQ 41
Memory at c20000000 (32-bit, non-prefetchable) [size=1M]
Memory at c20100000 (32-bit, prefetchable) [size=1M]
Capabilities: [44] Power Management version 3
Capabilities: [4c] Express Endpoint, MSI 00
Capabilities: [88] MSI: Enable- Count=1/16 Maskable- 64bit+
Capabilities: [100] Advanced Error Reporting
Any idea what is going on?
BR
I have already solved this problem. I have changed the following:
printk(KERN_ALERT "There is memory at BAR0\n");
if(request_mem_region(mmio_base0,mmio_length0,"driver")==0){
printk(KERN_ALERT "Impossible to get memory\n");
}
to
printk(KERN_ALERT "There is memory at BAR0\n");
if(pci_request_region(dev,0,"mydriver")==0){
printk(KERN_ALERT "Memory reserved properly\n");
}
When the function return 0, the memory is reserved properly. So, you can find in /proc/iomem the memory with a associated name ("mydriver").
Best regards