Why does Apache Ignite use more memory than configured - memory-leaks

Question

When using Ignite 2.8.1-1 version and default configuration(1GB heap, 20% default region for off-heap storage and persistence enabled) on a Linux host with 16GB memory, I notice the ignite process could use up to 11GB of memory(verified by checking the resident size of memory used by the process in top, see attachment). When I check the metrics in the log, the consumed memory(heap+off-heap) doesn't add up to close to 7GB. One possibility is the extra memory could be used by the checkpoint buffer but that shall be by default 1/4 of the default region, that is, only about a quarter of 0.25 * 0.2 * 16GB.
Any hints on what the rest of the memory is used for?
Thanks!
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=a52857f2, name=dummy-ignite-node, uptime=6 days, 10:41:45.541]
^-- H/N/C [hosts=98, nodes=221, CPUs=1481]
^-- CPU [cur=16.17%, avg=16.02%, GC=0%]
^-- PageMemory [pages=788278]
^-- Heap [used=1474MB, free=58.24%, comm=2020MB]
^-- Off-heap [used=3115MB, free=10.39%, comm=3376MB]
^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
^-- default region [used=3115MB, free=1.93%, comm=3176MB]
^-- metastoreMemPlc region [used=0MB, free=99.73%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=57084MB]
^-- sysMemPlc region [used=0MB]
^-- default region [used=57084MB]
^-- metastoreMemPlc region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=8, qSize=0]
=== Some logs regarding settings ===
dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default, *maxSize=3330655027, initSize=268435456,* swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=true, checkpointPageBufSize=0, lazyMemoryAllocation=true], dataRegions=null, storagePath=null, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=FSYNC, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory#285c08c8, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null, walPageCompression=DISABLED, walPageCompressionLevel=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, handshakeTimeout=10000, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null, thinCliCfg=ThinClientConfiguration [maxActiveTxPerConn=100]], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=RestartProcessFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet []]], commFailureRslvr=DefaultCommunicationFailureResolver []]
[2022-01-27T16:41:44,227][INFO ][main][IgniteKernal%ignite-jetstream-prd1] Daemon mode: on
[2022-01-27T16:41:44,227][INFO ][main][IgniteKernal%ignite-jetstream-prd1] OS: Linux 3.10.0-1160.49.1.el7.x86_64 amd64
[2022-01-27T16:41:44,227][INFO ][main][IgniteKernal%ignite-jetstream-prd1] OS user: root
[2022-01-27T16:41:44,227][INFO ][main][IgniteKernal%ignite-jetstream-prd1] PID: 7181
[2022-01-27T16:41:44,228][INFO ][main][IgniteKernal%ignite-jetstream-prd1] Language runtime: Scala ver. 2.11.12
[2022-01-27T16:41:44,228][INFO ][main][IgniteKernal%ignite-jetstream-prd1] VM information: OpenJDK Runtime Environment 1.8.0_312-b07 Red Hat, Inc. OpenJDK 64-Bit Server VM 25.312-b07
[2022-01-27T16:41:44,229][INFO ][main][IgniteKernal%ignite-jetstream-prd1] VM total memory: 0.96GB
[2022-01-27T16:41:44,229][INFO ][main][IgniteKernal%ignite-jetstream-prd1] Remote Management [restart: off, REST: off, JMX (remote: off)]
[2022-01-27T16:41:44,229][INFO ][main][IgniteKernal%ignite-jetstream-prd1] Logger: Log4J2Logger [quiet=true, config=config/ignite-log4j2.xml]
[2022-01-27T16:41:44,229][INFO ][main][IgniteKernal%ignite-jetstream-prd1] IGNITE_HOME=/usr/share/apache-ignite
[2022-01-27T16:41:44,230][INFO ][main][IgniteKernal%ignite-jetstream-prd1] VM arguments: [-XX:+AggressiveOpts, *-Xms1g, -Xmx1g, -XX:MaxPermSize=128M*, -Dfile.encoding=UTF-8, -DIGNITE_QUIET=true, -DIGNITE_HOME=/usr/share/apache-ignite, -DIGNITE_PROG_NAME=/usr/share/apache-ignite/bin/ignitevisorcmd.sh, -DIGNITE_DEPLOYMENT_MODE_OVERRIDE=ISOLATED]
Screen shot of top command output

Answer 1

Yes, the checkpoint buffer size is also taken into account here, if you haven't overridden the defaults, it should be 3GB/4 as you correctly highlighted. I wonder if it might be changed automatically since you have a lot more data ^-- Ignite persistence [used=57084MB] stored than the region capacity is - only 3GB. Also, this might be related to Direct Memory usage which I suppose is not being counted for the Java heap usage.
Anyway, I think it's better to check for Ignite memory metrics explicitly like data region and onheap usage and inspect them in detail.

Answer 2

Apache Ignite RAM necessary:
ON_HEAP_MEMORY (JVM heap max size)
+ OFF_HEAP_MEMORY (default region max size)
+ OFF_HEAP_MEMORY * 0.3 (Indexes also require memory. Basic use cases will add a 30% increase)
+ MIN(256MB, OFF_HEAP_MEMORY) (OFF_HEAP_MEMORY < 1 GB) || OFF_HEAP_MEMORY/4 (OFF_HEAP_MEMORY between 1 GB and 8 GB) || 2GB (OFF_HEAP_MEMORY > 8 GB) (default region checkpointing buffer size)
+ 100MB (sysMemPlc region max size)
+ 100MB (metastoreMemPlc region max size)
+ 100MB (TxLog region max size)
+ 100MB (volatileDsMemPlc region max size)