Google cloud functions write quota problems with 40+ functions - node.js

Question

We are using Google Cloud Functions quite a bit (around 40 functions currently deployed). They are all in one repository, which is a monorepo for our nodejs backend. They are deployed using Github Actions when a new features/bugfix is merged. Our problem is that all functions are deployed, we dealt with concurrency to deploy more than one function at a time (multiple deploys are run in parallel) but we hit a wall. We are hitting the write quota (which is 80 requests pre 100 seconds) and we are not sure why. It seems that a single function deploy sends around 40 write requests which is isane and while deploying the functions in a slower manner (2 at a time max) it's not acceptable as the deploy would then take 40+ minutes.
While searching info about the quota I found that a single function deploy should do 1 write request (makes sense), but it does multiple for us and I couldn't find any way to debug this.
Example command used for deploying:
gcloud functions deploy functionName --runtime nodejs10 --memory=2048MB --timeout=540s --set-env-vars FN_NAME=functionName --trigger-http --allow-unauthenticated --project our-project --set-env-vars APP_ENV=production
Our functions structure looks like this (names have been replaced):
functions/src
├── fns
│   │  
│   ├── atlas
│   │   ├── some-function.fn.ts
│   │   ├── some-function.fn.ts
│   │   ├── newsletter
│   │   │   ├── some-function.fn.ts
│   │   │   └── some-function.fn.ts
│   │   ├── suggestion
│   │   │   ├── some-function.fn.ts
│   │   │   └── some-function.fn.ts
│   │   ├── some-function.fn.ts
│   │   └── some-function.fn.ts
│   │  
│   ├── leads
│   │   ├── some-function.fn.ts
│   │   ├── some-function.fn.ts
│   │   ├── some-function.fn.ts
│   │   ├── some-function.fn.ts
│   │   ├── some-function.fn.ts
│   │   └── some-function.fn.ts
│   │  
│   ├── utils
│   │   ├── some-function.fn.ts
│   │   ├── some-function.fn.ts
│   │   ├── some-function.fn.ts
│   │   └── some-function.fn.ts
│   └── development.fn.ts
├── utils
│   ├── some-file-used-by-multiple-functions.ts
│   └── some-file-used-by-multiple-functions.ts
└── index.ts
development.fn.ts contains code which is run only on local machine and is ignored during deploy. It basically starts all the functions.
Every .fn.ts exports a single variable named after the function, which is simply a function handling the request request. This is wrapped in our "bootstrap" which handles connecting to databse, PubSub client and others.
index.ts is the entry file for Google Cloud with this content:
import { fns, getFnDefinition } from './bootstrap/get-fns';
// should export util
const ENV_FUNCTION_NAME = process.env.FN_NAME;
const shouldExportFn = (fnName: string) => {
if (!ENV_FUNCTION_NAME) {
return true;
}
return ENV_FUNCTION_NAME === fnName;
};
// export cycle
for (const fn of fns) {
if (shouldExportFn(fn.name)) {
const fnDefinition = getFnDefinition(fn);
exports[fn.name] = fnDefinition.handler;
}
}
export default exports;
Where fns is an array of { name, absolutePath } for our functions. It's read from filesystem (so no imports) and getFnDefinition requires the file and based on the result (exportet object) decides whether the function is triggered by HTTP request or PubSub message.
Also I saw the --entry-point=ENTRY_POINT option, but I'm not sure if that would solve our problem. Would it help if every function had its own entry point instead of the index.js?

Answer 1

The issue is how you are deploying them. You have all 40 functions in one Github repo, but how are you deploying them when one function requires a change? Do you resync / redeploy the whole thing? That would explain the 40 writes since you have 40 functions. I would recommend having them in individual repo or make sure each individual update doesn't cause all the functions to get updated.

Answer 2

Running into this too, but with READS! I hit 150k READS in a few hours just deploying (about 24 functions) about two dozen times. Looks like I have to even optimize my deployment strategy...

Answer 3

Our current solution is kind of dumb, but it works.
So it turns out the limit we were hitting (write quota) can be easily bypassed. What we do now is create a zip of the functions deploy, upload it to gcloud storage and then pass it as a parameter during deploy. This means we now don't reach the write quota (as no files are being uploaded) and everything works. We will however need to solve this in a better way in the future, as there is a limit of 60 when deploying functions and we currently have 48.