Does anybody know if there are any other ways of authentication/authorization for access to Google Cloud Storage besides of Service Account key when I use #google-cloud/storage Node.js module from here?
I have read about “Workload identity federation”, but it seems for me that I cannot use this approach when I use #google-cloud/storage library.
I was not able to find any suitable constructor, only these two:
const {Storage} = require('#google-cloud/storage');
var storage = new Storage({
projectId : `my_google_project_id`,
keyFilename : `my_google_key_file.json` // service account key is inside of this file
});
// or this one:
var storage = new Storage(); // service account key is inside of file specified by environment variable GOOGLE_APPLICATION_CREDENTIALS
Any recommendations?
Thank you
Most Google Clients support a new secrets key file with the type external_account. The following demonstrates how to create this file and setup Application Default Credentials (ADC) to load this file.
To use Workload Identity Federation with Google Client libraries, save the federated credentials to a file and then specify that file via the environment variable GOOGLE_APPLICATION_CREDENTIALS. The Storage client will use ADC and locate the credentials from the environment.
Example for AWS:
# Generate an AWS configuration file.
gcloud iam workload-identity-pools create-cred-config \
projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$POOL_ID/providers/$AWS_PROVIDER_ID \
--service-account $SERVICE_ACCOUNT_EMAIL \
--aws \
--output-file /path/to/generated/config.json
Example for Azure:
# Generate an Azure configuration file.
gcloud iam workload-identity-pools create-cred-config \
projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$POOL_ID/providers/$AZURE_PROVIDER_ID \
--service-account $SERVICE_ACCOUNT_EMAIL \
--azure \
--output-file /path/to/generated/config.json
Note: I generated my credentials on an Azure VM. I added the following command line option to the above command:
--app-id-uri=https://iam.googleapis.com/projects/REDACTED/locations/global/workloadIdentityPools/pool-azure/providers/provider-id
The output-file value is used to set the environment:
set GOOGLE_APPLICATION_CREDENTIALS=/path/to/generated/config.json
The file has the following structure. This example is for Azure:
{
"type": "external_account",
"audience": "//iam.googleapis.com/projects/REDACTED/locations/global/workloadIdentityPools/pool-azure/providers/provider-id",
"subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
"token_url": "https://sts.googleapis.com/v1/token",
"credential_source": {
"url": "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://iam.googleapis.com/projects/REDACTED/locations/global/workloadIdentityPools/pool-azure/providers/provider-id",
"headers": {
"Metadata": "True"
},
"format": {
"type": "json",
"subject_token_field_name": "access_token"
}
},
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/REDACTED#REDACTED.iam.gserviceaccount.com:generateAccessToken"
}
Use this style to create a client:
var storage = new Storage();
Related
I'm writing a cloud function in Nodejs (10), and trying to access a secret like so:
const [secret] = await new SecretManagerServiceClient().accessSecretVersion({
name: `projects/PROJECT_NUMBER/secrets/SECRET_NAME/versions/latest`
})
I created the secret in the web console and the name used in code matches that of the existing secret. On the page for the cloud function details, it states that the service account is PROJECT_ID#appspot.gserviceaccount,com, so I added the secretmanager.secretAccessor role to it. However, I'm still getting the same error every time:
Error: 7 PERMISSION_DENIED: Permission 'secretmanager.versions.access' denied for resource 'projects/PROJECT_NUMBER/secrets/SECRET_NAME/versions/latest' (or it may not exist).
It makes no difference if I specify a concrete version or just use latest.
HTTP cloud function code:
const { SecretManagerServiceClient } = require('#google-cloud/secret-manager');
const secretManagerServiceClient = new SecretManagerServiceClient();
const name = 'projects/shadowsocks-218808/secrets/workflow/versions/latest';
exports.testSecretManager = async (req, res) => {
const [version] = await secretManagerServiceClient.accessSecretVersion({ name });
const payload = version.payload.data.toString();
console.debug(`Payload: ${payload}`);
res.sendStatus(200);
};
Deploy:
gcloud functions deploy testSecretManager --runtime nodejs10 --trigger-http --allow-unauthenticated
Deploying function (may take a while - up to 2 minutes)...done.
availableMemoryMb: 256
entryPoint: testSecretManager
httpsTrigger:
url: https://us-central1-shadowsocks-218808.cloudfunctions.net/testSecretManager
ingressSettings: ALLOW_ALL
labels:
deployment-tool: cli-gcloud
name: projects/shadowsocks-218808/locations/us-central1/functions/testSecretManager
runtime: nodejs10
serviceAccountEmail: shadowsocks-218808#appspot.gserviceaccount.com
sourceUploadUrl: https://storage.googleapis.com/gcf-upload-us-central1-43476143-b555-4cb2-8f6f-1b2d1952a2d7/42c4cda4-98a8-4994-a3be-d2203b9e646a.zip?GoogleAccessId=service-16536262744#gcf-admin-robot.iam.gserviceaccount.com&Expires=1596513795&Signature=kbLw5teN8EoYmj4fEweKKiIaakxcrhlUg2GGHV4jWJjvmeEfXePpRNOn9yz2zLn%2Fba0UqM9qdJMXujs5afBk%2BVBmywPEiptAZe2qgmldpr%2BsYejFu0woNgsPHVqtJ0NoWDo6W2dq4CuNNwO%2BaQ89mnhahUUQTInkJ55Y3wCIe9smk%2BqWtcvta3zICiToA7RQvPKY5MS6NViyj5mLxuJtDlTY9IKPL%2BqG6JAaQJSFYKYVgLyb6JfirXk8Q7%2FMvnHPpXPlhvsBLQksbF6jDPeefp2HyW4%2FSIQYprfpwKV3hlEIQyRQllz5J9yF83%2FxDPh%2BQPc5QmswKP5XAvYaszJPEw%3D%3D
status: ACTIVE
timeout: 60s
updateTime: '2020-08-04T03:34:32.665Z'
versionId: '2'
Test:
gcloud functions call testSecretManager --data '{}'
Got error same as you:
error: |-
Error: function terminated. Recommended action: inspect logs for termination reason. Details:
7 PERMISSION_DENIED: Permission 'secretmanager.versions.access' denied for resource 'projects/shadowsocks-218808/secrets/workflow/versions/latest' (or it may not exist).
solution:
You can find the serviceAccountEmail: shadowsocks-218808#appspot.gserviceaccount.com from the deployment information details of cloud function.
go to IAM & Admin web UI, click ADD ANOTHER ROLE button, add Secret Manager Secret Accessor role to this service account.
Test again:
> gcloud functions call testSecretManager --data '{}'
executionId: 1tsatxl6fndw
result: OK
Read the logs for testSecretManager cloud function:
gcloud functions logs read testSecretManager
You will see the logs for the secret payload string.
I had the same issue and to solve it, I just had to:
Find the Service Account under General of my Google Cloud Function.
It looked like <project-name>#appspot.gserviceaccount.com
In IAM Admin, Add Secret Manager Secret Accessor Role to this Service Account.
After this, everything worked!
I have had similar issues working with secretmanager and the python google-cloud-secretmanager library (2.4). Specifically, after creating a secret and giving my service account the secretmanager.secretAccessor role on this secret (and nothing else, following the principle of least privilege), I was getting the following error when trying to access it:
details = "Permission 'secretmanager.versions.access' denied for resource 'projects/projectid/secrets/keyname/versions/latest' (or it may not exist)."
I could only make it work by also adding the secretmanager.viewer role at the project level, which as far as I can tell is not described in the documentation.
I had similar problem using terraform under gitlab.
I must add two authorizations to the service account which runs the pipeline:
resource "google_project_iam_policy" "gitlab" {
project = "secret_owner_project_id"
policy_data = data.google_iam_policy.iam.policy_data
}
data "google_iam_policy" "iam" {
binding {
role = "roles/secretmanager.secretAccessor"
members = [
"serviceAccount:project_accessing_secret#XYZ.iam.gserviceaccount.com",
]
}
binding {
role = "roles/viewer"
members = [
"serviceAccount:project_accessing_secret#XYZ.iam.gserviceaccount.com",
]
}
}
A bit late, but maybe this answer could be useful for future users. I encountered the same behavior only with Python. I tried lots of things but only thing that worked was creating new service account with zero roles(if I granted it secretmanager.secretAccessor role immediately, I got the same error). Then when empty service account is created, in IAM tab I press +Add, copy my empty service account adress and ONLY then I add secretmanager.secretAccessor role to it. Then I use this account as the account that will execute particular function. You of course may need to add other roles depending on what your function is intended to accomplish.
OAuth scope plays an important role here and please make sure the scope is defined correctly.
To use Secret Manager with workloads running on Compute Engine or GKE, the underlying instance or node must have the cloud-platform OAuth scope. If you receive an error with the following message, it means the instance or node was not provisioned with the correct OAuth scopes.
Request had insufficient authentication scopes
The required OAuth scope to use Secret Manager is:
https://www.googleapis.com/auth/cloud-platform
Example gcloud command to create dataproc with scope
gcloud dataproc clusters create xyz-pqr --region asia-south1 --subnet projects/xyz-pqr/regions/asia-south1/subnetworks/abc-serverless-vpc --zone asia-south1-b --master-machine-type n1-standard-4 --master-boot-disk-size 100 --num-workers 2 --worker-machine-type n1-standard-4 --worker-boot-disk-size 100 --image-version 2.0-debian10 --project xyz-development -scopes https://www.googleapis.com/auth/cloud-platform
I want to get list of all instances in a project using python google client api google-api-python-client==1.7.11
Am trying to connect using method googleapiclient.discovery.build this method required credentials as argument
I read documentation but did not get crdential format and which credential it requires
Can anyone explain what credentials and how to pass to make gcp connection
The credentials that you need are called "Service Account JSON Key File". These are created in the Google Cloud Console under IAM & Admin / Service Accounts. Create a service account and download the key file. In the example below this is service-account.json.
Example code that uses a service account:
from googleapiclient import discovery
from google.oauth2 import service_account
scopes = ['https://www.googleapis.com/auth/cloud-platform']
sa_file = 'service-account.json'
zone = 'us-central1-a'
project_id = 'my_project_id' # Project ID, not Project Name
credentials = service_account.Credentials.from_service_account_file(sa_file, scopes=scopes)
# Create the Cloud Compute Engine service object
service = discovery.build('compute', 'v1', credentials=credentials)
request = service.instances().list(project=project_id, zone=zone)
while request is not None:
response = request.execute()
for instance in response['items']:
# TODO: Change code below to process each `instance` resource:
print(instance)
request = service.instances().list_next(previous_request=request, previous_response=response)
Application default credentials are provided in Google API client libraries automatically. There you can find example using python, also check this documentation Setting Up Authentication for Server to Server Production Applications.
According to GCP most recent documentation:
we recommend you use Google Cloud Client Libraries for your
application. Google Cloud Client Libraries use a library called
Application Default Credentials (ADC) to automatically find your
service account credentials
In case you still want to set it manaully, you could, first create a service account and give all necessary permissions:
# A name for the service account you are about to create:
export SERVICE_ACCOUNT_NAME=your-service-account-name
# Create service account:
gcloud iam service-accounts create ${SERVICE_ACCOUNT_NAME} --display-name="Service Account for ai-platform-samples repo"
# Grant the required roles:
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member serviceAccount:${SERVICE_ACCOUNT_NAME}#${PROJECT_ID}.iam.gserviceaccount.com --role roles/ml.developer
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member serviceAccount:${SERVICE_ACCOUNT_NAME}#${PROJECT_ID}.iam.gserviceaccount.com --role roles/storage.objectAdmin
# Download the service account key and store it in a file specified by GOOGLE_APPLICATION_CREDENTIALS:
gcloud iam service-accounts keys create ${GOOGLE_APPLICATION_CREDENTIALS} --iam-account ${SERVICE_ACCOUNT_NAME}#${PROJECT_ID}.iam.gserviceaccount.com
Once it's done check whether the ADC path has been set properly by checking:
echo $GOOGLE_APPLICATION_CREDENTIALS
Having set the ADC path, you don't need to import from code the service access key, which undesirable, so the code looks as follows:
service = googleapiclient.discovery.build(<API>, <version>,cache_discovery=False)
I want to use Google Cloud Storage in NodeJS but authenticate with google-auth-library
Specifically: I want to host this on heroku, so I want to keep the secret in an environment variable, not in a file (as I'd have to commit the file to deploy to heroku). Basically what is suggested in the auth library:
https://github.com/googleapis/google-auth-library-nodejs#loading-credentials-from-environment-variables
But I can't seem to pass the resulting client to the Storage constructor?
Reading the code [1,2,3,4,5] you should be able to pass in credentials as constructor options:
storageOptions = {
projectId: 'your-project-id',
credentials: {
client_email: 'your-client-email',
private_key: 'your-private-key'
}
};
client = new Storage(storageOptions);
when I try to run firebase functions with cloud vision API and test the functions. I get this error:
ERROR: { Error: 7 PERMISSION_DENIED: Cloud Vision API has not been
used in project 563584335869 before or it is disabled. Enable it by
visiting
https://console.developers.google.com/apis/api/vision.googleapis.com/overview?project=563584335869
then retry. If you enabled this API recently, wait a few minutes for
the action to propagate to our systems and retry.
I do not recognize this project number and I have already enabled the API with the project that I am using. I set the GOOGLE_APPLICATION_CREDENTIALS using the project with the enabled API. What is it that I'm doing wrong?
For those of you who are still having this issue here is what worked for me:
const client = new vision.ImageAnnotatorClient({
keyFilename: 'serviceAccountKey.json'
})
This error message is usually thrown when the application is not being authenticated correctly due to several reasons such as missing files, invalid credential paths, incorrect environment variables assignations, among other causes.
Based on this, I recommend you to validate that the credential file and file path are being correctly assigned, as well as follow the Obtaining and providing service account credentials manually guide in order to explicitly specify your service account file directly into your code; In this way, you will be able to set it permanently and verify if you are passing the service credentials correctly. Additionally, you can take a look on this link that contains a useful step-by-step guide to use Firebase functions with Vision API which includes the Vision object authentication code for Node.js.
Passing the path to the service account key in code example:
// Imports the Google Cloud client library.
const Storage = require('#google-cloud/storage');
// Instantiates a client. Explicitly use service account credentials by
// specifying the private key file. All clients in google-cloud-node have this
// helper, see https://github.com/GoogleCloudPlatform/google-cloud-node/blob/master/docs/authentication.md
const storage = new Storage({
keyFilename: '/path/to/keyfile.json'
});
// Makes an authenticated API request.
storage
.getBuckets()
.then((results) => {
const buckets = results[0];
console.log('Buckets:');
buckets.forEach((bucket) => {
console.log(bucket.name);
});
})
.catch((err) => {
console.error('ERROR:', err);
});
API has not been used in project 563584335869
If you clicked that link has been printed in a console, that guide you to this url.
https://console.developers.google.com/apis/api/vision.googleapis.com/overview?project=firebase-cli
So that project id means you used the project credential of 'firebase-cli' not yours.
If you tried to set the value of the environment, you can find the variables in your directory
~/.config/firebase/...credentials.json
And sometime it would be not replaced after you tried to override.
But, you can set your credential in code.
You can find the way of getting a credential in here.
https://cloud.google.com/iam/docs/creating-managing-service-account-keys
And the credential format is like this one.
{
"type": "service_account",
"project_id":
"private_key_id":
"private_key":
"client_email":
"client_id":
"auth_uri":
"token_uri":
"auth_provider_x509_cert_url":
"client_x509_cert_url":
}
I've faced exactly the same error what you have when I used another google API. and resolved that way including the credential inside code.
const textToSpeech = require("#google-cloud/text-to-speech")
const keyfile = require(".././my-project.json")
const config = {
projectId: keyfile.project_id,
keyFilename: require.resolve(".././my-project.json")
};
const TTS_Client = new textToSpeech.TextToSpeechClient(config)
I'm using Google Cloud Storage and have a few buckets that contain objects which are not shared publicly. Example in screenshot below. Yet I was able to retrieve file without supplying any service account keys or authentication tokens from a local server using NodeJS.
I can't access the files from browser via the url formats (which is good):
https://www.googleapis.com/storage/v1/b/mygcstestbucket/o/20180221164035-user-IMG_0737.JPG
https://storage.googleapis.com/mygcstestbucket/20180221164035-user-IMG_0737.JPG
However, when I tried using retrieving the file from NodeJS without credentials, surprisingly it could download the file to disk. I checked process.env to make sure there were no GOOGLE_AUTHENTICATION_CREDENTIALS or any pem keys, and also even did a gcloud auth revoke --all on the command line just to make sure I was logged out, and still I was able to download the file. Does this mean that the files in my GCS bucket is not properly secured? Or I'm somehow authenticating myself with the GCS API in a way I'm not aware?
Any guidance or direction would be greatly appreciated!!
// Imports the Google Cloud client library
const Storage = require('#google-cloud/storage');
// Your Google Cloud Platform project ID
const projectId = [projectId];
// Creates a client
const storage = new Storage({
projectId: projectId
});
// The name for the new bucket
const bucketName = 'mygcstestbucket';
var userBucket = storage.bucket(bucketName);
app.get('/getFile', function(req, res){
let fileName = '20180221164035-user-IMG_0737.JPG';
var file = userBucket.file(fileName);
const options = {
destination: `${__dirname}/temp/${fileName}`
}
file.download(options, function(err){
if(err) return console.log('could not download due to error: ', err);
console.log('File completed');
res.json("File download completed");
})
})
Client Libraries use Application Default Credentials to authenticate Google APIs. So when you don't explicitly use a specific Service Account via GOOGLE_APPLICATION_CREDENTIALS the library will use the Default Credentials. You can find more details on this documentation.
Based on your sample, I'd assume the Application Default Credentials were used for fetching those files.
Lastly, you could always run echo $GOOGLE_APPLICATION_CREDENTIALS (Or applicable to your OS) to confirm if you've pointed a service account's path to the variable.
Create New Service Account in GCP for project and download the JSON file. Then set environment variable like following:
$env:GCLOUD_PROJECT="YOUR PROJECT ID"
$env:GOOGLE_APPLICATION_CREDENTIALS="YOUR_PATH_TO_JSON_ON_LOCAL"