Set Firestore documents with partially merged fields - Firestore limitation - python-3.x

Question

I am using the Python library of Firestore to communicate with Firestore.
I have now run into a limitation of Firestore and I am wondering if there is a way around it.
Imagine we have this map / Dict (dictVar1):
dictVar1 = {
"testArray": ["Yes"],
"testMap": {
"test1": 1,
"test2": 1
}
}
To begin with, I used to store my testMap in an array, but due to Firestore query limitations (you can only have a single array-contains operation in a query), I changed my structure to a map instead (as you can see in the dictVar1 structure above). If Firestore queries did not have this limitation, I would not change my structure from an array.
Now I am facing another Firestore limitation due to the new structure.
What I would like to do & other conditions:
I want to add this map / dict to a Firestore document.
I would like to do it in one Firestore operation using Firestore batch
I don't know if the document exists or not before updating/creating
One batch can contain anything between 1 and 500 operations
If the document exists, I do not want to remove any other fields from the existing document if these fields are not present in dictVar1 dict / map.
The fields in dictVar1 dict / map should replace the fields in the document completely
So if the existing document would contain this data:
{
"doNotChange": "String",
"testMap": {
"test0": 1
}
}
It would be updated to ("test0" is removed from the inner map, basically how an array would work):
{
"doNotChange": "String",
"testArray": ["Yes"],
"testMap": {
"test1": 1,
"test2": 1
}
}
And if the document doesn't exist, the document would be set to:
{
"testArray": ["Yes"],
"testMap": {
"test1": 1,
"test2": 1
}
}
I see two ways to do this:
Do this in two operations
Instead of using testMap as a map, replace it with an array.
99% of the time the document exists, therefore I am fine with doing this in two operations if the document doesn't exist, but one operation if the document exists.
This could be done using Firestore's update function, but since I am using batch and potentially updating 100s of documents in one batch, if the document doesn't exist, it would ruin the whole batch operation.
Another potential solution would be to:
Run batch with updates, if it succeeds, then great, if 404 (document not found) is raised then:
Change the operation to set instead of an update for this document and then redo the batch, in a loop until the batch is successful
Two potential problems I see with this:
Will I be fully charged for all the failed batch operations or will I be just be charged 1 read per failed batch operation? If I get fully charged for the batch, then this is still not a good solution.
Is it possible to easily change the operation type for a specific document reference to a different operation type without having to recreate the batch operation totally from scratch?
Do you have any ideas on how I could solve one of these problems?
Here is the Python code to test out:
from json import dumps
from google.cloud import firestore
db = firestore.Client.from_service_account_json("firebaseKeysDev.json")
originalDoc = {
"doNotChange": "String",
"testMap": {
"test0": 1
}
}
dictVar1 = {
"testArray": ["Yes"],
"testMap": {
"test1": 1,
"test2": 1
}
}
prefOutput = {
"doNotChange": "String",
"testArray": [
"Yes"
],
"testMap": {
"test1": 1,
"test2": 1
}
}
# Let's first create the document with the original dict / map
originalSetOp = db.collection("test").document("testDoc").set(originalDoc)
# Now let's get the original map / dict from Firestore
originalOpDoc = db.collection("test").document("testDoc").get()
# Convert to Python Dict
originalOpDocDict = originalOpDoc.to_dict()
# Now let's print out the original document dict
print("Here is the original map:")
print(dumps(originalOpDocDict, ensure_ascii=False, sort_keys=True, indent=4))
# Now let's merge the original dict / map with our dictVar1 dict / map
mergeDictVar1WithODoc = db.collection("test").document("testDoc").set(dictVar1, merge=True)
# Now let's get the new merged map / dict from Firestore
newDictDoc = db.collection("test").document("testDoc").get()
# Convert to Python Dict
newDictDocDict = newDictDoc.to_dict()
# Let's print the new merged dict / map
print("\nHere is the merged map:")
print(dumps(newDictDocDict, ensure_ascii=False, sort_keys=True, indent=4))
print("\nHere is the output we want:")
print(dumps(prefOutput, ensure_ascii=False, sort_keys=True, indent=4))
Output:
Here is the original map:
{
"doNotChange": "String",
"testMap": {
"test0": 1
}
}
Here is the map we want to merge:
{
"testArray": [
"Yes"
],
"testMap": {
"test1": 1,
"test2": 1
}
}
Here is the merged map:
{
"doNotChange": "String",
"testArray": [
"Yes"
],
"testMap": {
"test0": 1,
"test1": 1,
"test2": 1
}
}
Here is the output we want:
{
"doNotChange": "String",
"testArray": [
"Yes"
],
"testMap": {
"test1": 1,
"test2": 1
}
}

Answer 1

You can try using .set() with SetOptions of merge or mergeFields instead of .update() - the field, in this case, would be your map.
Specifically, .set() will create a document if it doesn't exist. It seems (I'm not on the Firebase team) the PURPOSE of .update() failing is to signal the document doesn't already exist.
I use this extensively in a wrapper library I created for Firestore in my app.
Documented Here