ArangoDB-aysnchronize-log invokes msync too much - arangodb

Question

Here is the question: why Arango sync data thousands times in one second under async mode? Is it my wrong configuration or expected behavior?
Recently I'm testing async insert of ArangoDB and MongoDB. In my test, the average latency of Arango is 2x of MongoDB. After tuning I found their IO is different. I think it's the root cause for pool async-inert perf of Arango.
Arango: Invoke msync continuously, thousand times in one second, like the following. This causes too much iowait and too much jbd2.
05:42:21.138119 msync(0x7f50fdd75000, 4096, MS_SYNC) = 0 <0.000574>
05:42:21.138843 msync(0x7f50fdd75000, 8192, MS_SYNC) = 0 <0.000558>
05:42:21.139541 msync(0x7f50fdd76000, 4096, MS_SYNC) = 0 <0.000351>
05:42:21.139928 msync(0x7f50fdd76000, , MS_SYNC) = 0 <0.000555>
05:42:21.140532 msync(0x7f50fdd77000, 4096, MS_SYNC) = 0 <0.000318>
05:42:21.141002 msync(0x7f50fdd77000, 8192, MS_SYNC) = 0 <0.000714>
05:42:21.141755 msync(0x7f50fdd78000, 4096, MS_SYNC) = 0 <0.000345>
05:42:21.142133 msync(0x7f50fdd78000, 4096, MS_SYNC) = 0 <0.000725>
Mongo: Invoke fdatasync just several times in one second.
Test Env:
All tests are in one VM: 8vCPU-24GBMem-120GBDisk-Centos6.7
It's single thread async insert test based on java driver with ycsb.
Conf for Arango:
v2.8.7
Server, scheduler, v8-cs's threads are all set to 1.
Create collection with false waitForSync, Send insert request with false waitForSync.
Start cmd:
/usr/sbin/arangod --uid arangodb --gid arangodb --pid-file /var/run/arangodb/arangod.pid --temp-path /var/tmp/arangod --log.tty --supervisor --wal.sync-interval=1000
Collection propertis:
{
"doCompact" : true,
"journalSize" : 33554432,
"isSystem" : false,
"isVolatile" : false,
"waitForSync" : false,
"keyOptions" : {
"type" : "traditional",
"allowUserKeys" : true
},
"indexBuckets" : 8
}
Detailed trace log:
2016-04-19T06:59:36Z [12065] TRACE [arangod/Wal/SynchronizerThread.cpp:213] syncing logfile 6627612014716, region 0x7ff9beef3318 - 0x7ff9beef37f2, length: 1242, wfs: false
2016-04-19T06:59:36Z [12065] TRACE [arangod/Wal/SynchronizerThread.cpp:213] syncing logfile 6627612014716, region 0x7ff9beef37f8 - 0x7ff9beef3cd2, length: 1242, wfs: false
2016-04-19T06:59:36Z [12065] TRACE [arangod/Wal/SynchronizerThread.cpp:213] syncing logfile 6627612014716, region 0x7ff9beef3cd8 - 0x7ff9beef41b2, length: 1242, wfs: false
2016-04-19T06:59:36Z [12065] TRACE [arangod/Wal/SynchronizerThread.cpp:213] syncing logfile 6627612014716, region 0x7ff9beef41b8 - 0x7ff9beef4692, length: 1242, wfs: false
2016-04-19T06:59:36Z [12065] TRACE [arangod/Wal/SynchronizerThread.cpp:213] syncing logfile 6627612014716, region 0x7ff9beef4698 - 0x7ff9beef4b72, length: 1242, wfs: false
2016-04-19T06:59:36Z [12065] TRACE [arangod/Wal/SynchronizerThread.cpp:213] syncing logfile 6627612014716, region 0x7ff9beef4b78 - 0x7ff9beef5052, length: 1242, wfs: false
2016-04-19T06:59:36Z [12065] TRACE [arangod/Wal/SynchronizerThread.cpp:213] syncing logfile 6627612014716, region 0x7ff9beef5058 - 0x7ff9beef5532, length: 1242, wfs: false

Answer 1

ArangoDB as a multi model database can offer more usecases than MongoDB. While it can act as replacement, the other available features also imply different requirements to the default configuration settings and implementation details.
When you work with i.e. graphs, and want to maintain persistency in them, you can alter the probability data is actually lost by doing more frequent syncs.
ArangoDB does these syncs in another thread; When trying to reproduce your setup we found that this thread actualy does more than one would think the sync-interval configuration value in /etc/arangodb/arangod.conf:
[wal]
sync-interval=10000
We fixed this; it improves the perfromance a bit when writing locally via foxx or the arangod rescue console (which you get if you don't start it in daemon mode with the --console parameter)
However, It doesn't significantly change the performance when i.e. using arangosh to sequentially insert 10 k documents:
var time= require("internal").time;
var s = time()
db._drop('test')
db._create('test')
for (i=0; i < 100000; i++) {db.test.save({i: i})}
require("internal").print(time() -s)
In general, your numbers are similar to those in our performance comparison - so thats what was to be expected with ArangoDB 2.8.
Currently you can use the bulk import facility to reduce the overhead you get in the HTTP communication.