Firestore performance issues when compared to Firestore emulator - node.js

Question

My app can have a large amount of writes, reads and updates (can even go above 10000) under certain circumstances.
While developing the application locally, these operations usually take a few seconds at most (great!) however, it can easily take minutes when running the application on google cloud, to the point that the Firebase function times out.
I developed a controlled test in a separate project, whose sole purpose is to write, get and delete thousands of items for bench-marking. These were the results (averaged out from several tests):
Local Emulator:
5000 items, 4.2s write, 2.2s delete
5000 items, batch mode ON, 0.75s write, 0.11s delete
Cloud Firestore:
100 items, 15.8s write, 14.5s delete
1000 items, batch mode ON, 4.8s write, 3.0s delete
5000 items, async mode ON, 10.2s write, 8.0s delete
5000 items, batch & async ON, 4.5s write, 3.9s delete
NOTE: My local emulator crashes whenever I try to perform db operations async (which is a problem for another day) but it is why I was unable to test the write/delete speeds asynchronously locally. Also, write and read values usually vary +-25% between runs.
However, as you can see, the fact that my local emulator is faster in its slowest mode compared to the fastest test in the cloud definitely raises some questions.
Could it be that I have some sort of configuration issue? or is it just that these numbers are standard for firestore? Here is the (summarised) typescript code if you wish to try it:
functions.runWith({ timeoutSeconds: 540, memory: "2GB" }).https.onRequest(async (req, res) => {
//getting the settings from the request
var data = req.body;
var numWrites: number = data.numWrites;
var syncMode: boolean = !data.asyncMode;
var batchMode: boolean = data.batchMode;
var batchLimit: number = data.batchLimit;
//pre-run setup
var dbObj = {
number: 123,
string: "abc",
boolean: true,
object: { var1: "var1", num1: 1 },
array: [1, 2, 3, 4]
};
var collection = db.collection("testCollection");
var startTime = moment();
//insert requested number of items, using requested settings
var allInserts: Promise<any>[] = [];
if (!batchMode) { //sequential writes
for (var i = 0; i < numWrites; i++) {
var set = collection.doc().set(dbObj);
allInserts.push(set);
if (syncMode) await set;
}
} else { //batch writes
var batch = db.batch();
for (var i = 1; i <= numWrites; i++) {
batch.set(collection.doc(), dbObj);
if (i % batchLimit === 0) {
var commit = batch.commit();
allInserts.push(commit);
batch = db.batch();
if (syncMode) await commit;
}
}
}
//some logging information. Getting items to delete
var numInserts = allInserts.length;
await Promise.all(allInserts);
var insertTime = moment();
var alldocs = (await collection.get()).docs;
var numDocs = alldocs.length;
var getTime = moment();
//deletes all of the items in the collection
var allDeletes: Promise<any>[] = [];
if (!batchMode) { //sequential deletes
for (var doc of alldocs) {
var del = doc.ref.delete();
allDeletes.push(del);
if (syncMode) await del;
}
} else { //batch deletes
var batch = db.batch();
for (var i = 1; i <= numDocs; i++) {
var doc = alldocs[i - 1];
batch.delete(doc.ref);
if (i % batchLimit === 0) {
var commit = batch.commit();
allDeletes.push(commit);
batch = db.batch();
if (syncMode) await commit;
}
}
}
var numDeletes = allDeletes.length;
await Promise.all(allDeletes);
var deleteTime = moment();
res.status(200).send(/* a whole bunch of metrics for analysis */);
});
EDIT: just to clarify, the UI does not perform these write operations, so latency between the end-user machine and cloud servers should (in theory) not cause any major latency issues. The communication to the database is handled fully by Firebase Functions
EDIT 2: I have run this test on two deployments, one in Europe and another in US. Both took around the same amount of time to run, even though my ping to these two servers are vastly different

Answer 1

It is normal to have faster response with the local emulator than Cloud Firestore as the remote environment adds the network traffic that takes time.
For large amounts of operations from a single source the recommendation is to use batch operations as these will reduce the transcactions, and with it Round trips.
And the reason for the Async mode to be faster is that the caller is not waiting for each transaction to be completed before sending the next one So it also makes sense that the calls are faster with it.
The Times you have on the table seem normal to me.
Just as an additional thing to optimize make sure that the region where your firestore database is located is the closest one to your location.