SecretManagerServiceClient in Google Cloud Run and authentication via service account - python-3.x

Question

I can create a SecretManagerServiceClient without using a key file successfully in Google Cloud Shell:
from google.cloud import secretmanager
from google.oauth2 import service_account
from google.auth.exceptions import DefaultCredentialsError
import logging
import sys
import os
def list_secrets(client, project_id):
"""
Retrieve all secrets associated with a project
:param project_id: the alpha-numeric name of the project
:return: a generator of Secrets
"""
try:
secret_list = client.list_secrets(request={"parent": "projects/{}".format(project_id)})
except Exception as e:
sys.exit("Did not successfully retrieve secret list.")
return secret_list
def set_env_secrets(client, secret_ids, label=None):
"""
Sets secrets retrieved from Google Secret Manager in the runtime environment
of the Python process
:param secret_ids: a generator of Secrets
:param label: Secrets with this label will be set in the environment
"""
for s in secret_ids:
# we only want secrets with matching labels (or all of them if label wasn't specified)
if not label or label in s.labels:
version = client.access_secret_version(request={'name': '{}/versions/latest'.format(s.name)})
payload_str = version.payload.data.decode("UTF-8")
os.environ[s.name.split('/')[-1]] = payload_str
if __name__ == "__main__":
client = secretmanager.SecretManagerServiceClient()
secrets = list_secrets(client, "myprojectid-123456")
set_env_secrets(client, secrets)
print(os.getenv("DATA_DB_HOST"))
However, when I use similar code as the basis for an entry point of a container in Google Cloud Run, the attempt to retrieve a client using the default service account's credentials fails with
File "entry_point.py", line 27, in get_client
client = secretmanager.SecretManagerServiceClient()
File "/usr/local/lib/python3.6/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", line 274, in __init__
client_info=client_info,
File "/usr/local/lib/python3.6/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/transports/grpc.py", line 162, in __init__
scopes=self.AUTH_SCOPES, quota_project_id=quota_project_id
File "/usr/local/lib/python3.6/site-packages/google/auth/_default.py", line 340, in default
credentials, project_id = checker()
File "/usr/local/lib/python3.6/site-packages/google/auth/_default.py", line 186, in _get_explicit_environ_credentials
os.environ[environment_vars.CREDENTIALS]
File "/usr/local/lib/python3.6/site-packages/google/auth/_default.py", line 97, in load_credentials_from_file
"File {} was not found.".format(filename)
google.auth.exceptions.DefaultCredentialsError: File was not found.
The default service account has the Editor and Secret Manager Admin roles (thanks to #DanielOcando for his comment). Why is it that the ADC library, as described here does not pick up the permissions of the default service account and use them to instantiate the client?
Update 1
#guillaumeblaquiere asked about dependencies. The container is built with Python 3.6.12 and the following libraries:
Django==2.1.15
django-admin-rangefilter==0.3.7
django-extensions==2.1.2
django-ipware==1.1.6
pytz==2017.3
psycopg2==2.7.3.2
waitress==1.4.1
geoip2==2.6
gunicorn==19.9.0
social-auth-app-django==3.1.0
semver==2.8.1
sentry-sdk==0.6.9
google-api-core==1.23.0
google-auth==1.23.0
google-cloud-secret-manager==2.0.0
I created a custom service account, added Editor and Secret Manager Admin roles to it, and then used the Console to deploy a new revision with that account, but the same error resulted.
Update 2
Thinking that matching the CPython version in Cloud Shell would do the trick, I rebuilt the image with Python 3.7. No luck.
Update 3
Taking a different tack, I added Service Account Token Creator role to the default service account of the project and created a terraform file and configured it for service account impersonation. I also ran gcloud auth application-default login in the shell prior to invoking terraform.
provider "google" {
alias = "tokengen"
}
data "google_client_config" "default" {
provider = google.tokengen
}
data "google_service_account_access_token" "sa" {
provider = "google.tokengen"
target_service_account = "XXXXXXXXXXXX-compute#developer.gserviceaccount.com"
lifetime = "600s"
scopes = [
"https://www.googleapis.com/auth/cloud-platform",
]
}
provider "google" {
project = "myprojectid-123456"
region = "us-central1"
zone = "us-central1-f"
#impersonate_service_account = "XXXXXXXXXXXX-compute#developer.gserviceaccount.com
}
resource "google_cloud_run_service" "default" {
name = "myprojectid-123456"
location = "us-central1"
template {
spec {
containers {
image = "us.gcr.io/myprojectid-123456/testimage"
}
}
}
traffic {
percent = 100
latest_revision = true
}
}
This did work to create the service, but again, when the endpoint attempted to instantiate SecretManagerServiceClient, the same error resulted.