ArangoDB batch mode doesn't work with Java Driver - arangodb

Question

I am evaluating ArangoDB using spring batch.
I tried to insert some data and, without batch mode, it works as expected.
However, if batch mode is on, the execution of the program hangs.
I am using arango 2.3.3 and com.arangodb:arangodb-java-driver:[2.2-SNAPSHOT,2.2]
arangoDriver.startBatchMode();
for(Account acc : items){
acc.getRecordHash();
acc.getIdHash();
arangoDriver.createDocument("AccountCollection", acc);
}
arangoDriver.executeBatch();
Any ideas what I am doing wrong?

Answer 1

I tried to reproduce what you are trying, first of all does the collection "AccountCollection" exist ? If not you would get an error in the batch result but still the program should not hang. i created a unittest :
#Test
public void test_StartCancelExecuteBatchMode() throws ArangoException {
driver.startBatchMode();
ArrayList<Account> items = new ArrayList<Account>();
items.add(new Account());
items.add(new Account());
items.add(new Account());
items.add(new Account());
for(Account acc : items){
acc.getRecordHash();
acc.getIdHash();
driver.createDocument("AccountCollection", acc, true, false);
}
driver.executeBatch();
}
This works perfectly and returns:
EOB
16:47:01.862 [main] DEBUG com.arangodb.http.HttpManager - [RES]http-POST: statusCode=200
16:47:01.862 [main] DEBUG com.arangodb.http.HttpManager - [RES]http-POST: text=--dlmtrMLTPRT
Content-Type: application/x-arango-batchpart
Content-Id: request1
HTTP/1.1 202 Accepted
Location: /_db/unitTestDatabase/_api/document/AccountCollection/48033214501
Content-Type: application/json; charset=utf-8
Etag: "48033214501"
Content-Length: 95
{"error":false,"_id":"AccountCollection/48033214501","_rev":"48033214501","_key":"48033214501"}
--dlmtrMLTPRT
Content-Type: application/x-arango-batchpart
Content-Id: request2
HTTP/1.1 202 Accepted
Location: /_db/unitTestDatabase/_api/document/AccountCollection/48033411109
Content-Type: application/json; charset=utf-8
Etag: "48033411109"
Content-Length: 95
{"error":false,"_id":"AccountCollection/48033411109","_rev":"48033411109","_key":"48033411109"}
--dlmtrMLTPRT
Content-Type: application/x-arango-batchpart
Content-Id: request3
HTTP/1.1 202 Accepted
Location: /_db/unitTestDatabase/_api/document/AccountCollection/48033607717
Content-Type: application/json; charset=utf-8
Etag: "48033607717"
Content-Length: 95
{"error":false,"_id":"AccountCollection/48033607717","_rev":"48033607717","_key":"48033607717"}
--dlmtrMLTPRT
Content-Type: application/x-arango-batchpart
Content-Id: request4
HTTP/1.1 202 Accepted
Location: /_db/unitTestDatabase/_api/document/AccountCollection/48033804325
Content-Type: application/json; charset=utf-8
Etag: "48033804325"
Content-Length: 95
{"error":false,"_id":"AccountCollection/48033804325","_rev":"48033804325","_key":"48033804325"}
--dlmtrMLTPRT--
But even when i create intentional errors the application never "hangs".
Frank just sent me your source code, i take a look into it. Can you try to find out where the programm is hanging ? is "executeBatch" reached at all ?

Answer 2

I already imported 1.6 Mio documents with your code and still everything works.
i guess it might be necessary to monitor your system resources during the import, if anything unusual occurs let us now. Generally speaking it does not seem to be the best practice to perform a one-time bulk import like this using the java api. i would recommend to use arangoimp to import the data directly into the database, this will be much faster. it is documented here

Answer 3

You need to increase the number of open file descriptors. The Mac has a very low limit (256). ArangoDB stores the data in datafiles of a certain chunk size. With large datasets more files are need (and some fd are already used for communication and other stuff).
When ArangoDB runs out of file descriptors, it can neither extend the dataset nor answer new questions. Therefore the import process will hang.