Getting "Permission denied" container.clusters.create when deployed but not on localhost - node.js

Question

I'm trying to create a Kubernetes Cluster via the NodeJS client on google App engine. The Kubernetes cluster is on a separate project to where the app engine project is hosted, say "my-node-project" & "my-k8-project".
"my-node-project" has the relevant service account(Owner level access) for the kubernetes project.
I make the cluster create call as follows:
var client = new container.v1.ClusterManagerClient({
projectId: projectId,
key: serviceAccount
});
var zone = 'us-central1-b';
var password = "<some password>";
var clusterConfig = {
"name": clusterName,
"description": "api created cluster",
"initialNodeCount": 3,
"nodeConfig": {
"oauthScopes": [
"https://www.googleapis.com/auth/compute",
"https://www.googleapis.com/auth/devstorage.read_only"
]
}
,
"masterAuth": {
"username": "admin",
"password": password
},
"zone": zone
};
var request = {
projectId: projectId,
zone: zone,
cluster: clusterConfig,
};
return client.createCluster(request)
.then(responses => {
var response = responses[0];
console.log("response: ", response);
return response;
})
.catch(err => {
console.error(err);
return err;
});
In the above code the serviceAccount variable is a json object containing the service account, with all the private key, project id fields etc.
The strange thing is that when I run the code locally, i.e. call the endpoint that runs the above function, the request goes through just fine, i.e. the clusters are created and I can even add workloads via the api.
However, after I deploy the nodejs project to app engine standard and call the same endpoint running on app engine, I get the error:
Error: 7 PERMISSION_DENIED: Required "container.clusters.create" permission(s) for "projects/my-k8-project". See https://cloud.google.com/kubernetes-engine/docs/troubleshooting#gke_service_account_deleted for more info.
at Object.exports.createStatusError (/srv/node_modules/grpc/src/common.js:91:15)
at Object.onReceiveStatus (/srv/node_modules/grpc/src/client_interceptors.js:1204:28)
at InterceptingListener._callNext (/srv/node_modules/grpc/src/client_interceptors.js:568:42)
at InterceptingListener.onReceiveStatus (/srv/node_modules/grpc/src/client_interceptors.js:618:8)
at callback (/srv/node_modules/grpc/src/client_interceptors.js:845:24)
code: 7,
metadata: Metadata { _internal_repr: {} },
details:
'Required "container.clusters.create" permission(s) for "projects/my-k8-project". See https://cloud.google.com/kubernetes-engine/docs/troubleshooting#gke_service_account_deleted for more info.' }
Since I got that troubleshooting link, I tried to create a new service account and use that. In addition I tried disabling and enabling the both the kubernetes and compute APIs. I also tried to place the service account in the root directory of the project and refer to the service account that way.
Unfortunately everything I tried resulted in exactly the same error. But still worked when running from localhost.
Is there a whitelist somewhere I'm missing? Perhaps localhost is whitelisted by default and "my-node-project" app engine project isn't on the list?
Any tips, hints or pointing in the right direction would be very much appreciated.

Answer 1

You need to add service account also to your "my-k8-project" as a member and give relevant role to it.
In the Cloud Console, navigate to project "my-k8-project". Find the "IAM & admin" > "IAM" page. Click the "Add" button. In the "New members" field paste the name of the service account and give it the appropriate role.