How to keep google-cloud-auth.json securely in app.yaml as an environmental variable? - node.js

I'm new to deployment/securing keys, and I'm not sure how to securely store the google-cloud-auth.json (auth required for creating the API client) outside of source code to prevent leaking credentials.
I've currently secured my API keys and tokens in my app.yaml file specifying them as environmental variables which successfully work as expected and shown below.
accessruntime: nodejs10
env_variables:
SECRET_TOKEN: "example"
SECRET_TOKEN2: "example2"
However my google-cloud-auth.json is kept as its own file since the parameter used for creating the client requires a path string.
const {BigQuery} = require('#google-cloud/bigquery');
...
const file = "./google-cloud-auth.json";
// Creates a BigQuery client
const bigquery = new BigQuery({
projectId: projectId,
datasetId: datasetId,
tableId: tableId,
keyFilename: file
});

According to the Setting Up Authentication for Server to Server Production Applications:
GCP client libraries will make use of the ADC (Application Default Credentials) to find the credentials meant to be used by the app.
What ADC does is basically to check if the GOOGLE_APPLICATION_CREDENTIALS env variable is set with the path to a service account file.
In case the env variable is not set, ADC will use the default service account provided by App Engine.
With this information I can suggest a couple of solutions to provide these credentials safely:
If you require to use a specific service account, set the path to the file with the GOOGLE_APPLICATION_CREDENTIALS. This section explains how to do that.
If you are not a fan of moving credential files around, I would suggest trying to use the default service account provided by the App Engine.
I just created a new project and deployed a basic app by mixing these 2 guides:
BigQuery Client Libraries
Quickstart for Node.js in the App Engine Standard Environment
My app.yaml had nothing more than the runtime: nodejs10 line, and I was still able to query through the BigQuery client library, using the default service account.
This account comes with the Project/Editor role and you can add any additional roles you need.

Related

How can I integrate a credential json file with GCP bigquery with nodejs?

I have a file type of json. It is a credential file. I want to integrate with GCP bigquery and access to GCP bigquery using this credential file with Nodejs.
How can I do that?
How can integrate with GCP bigquery using credential file in nodejs?
How can I test the result of integration to test integration is valid or not?
You probably want the keyFilename attribute, unless I've misunderstood your question.
This GCP doc talks about authenticating using a service account key file.
So if your credentials file lived in /var/my_credentials.json (dumb path but whatever), your Node.js code would look something like this:
const {BigQuery} = require('#google-cloud/bigquery');
const options = {
keyFilename: '/var/my_credentials.json',
projectId: 'my_project',
};
const bigquery = new BigQuery(options);
Also consider: keep the contents of that credentials file in Google Secret Manager and use gcloud secrets versions access latest, dumping the output into a temporary json file local to the script, then remove the temporary json file after it's no longer needed by the script. No need to have credentials floating around on servers.

How to authenticate with tokens in Nodejs to a private bucket in Cloud Storage

Usually in Python what I do, I get the application default credentials, I get the access token then I refresh it to be able to authenticate to a private environment.
Code in Python:
# getting the credentials and project details for gcp project
credentials, your_project_id = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
#getting request object
auth_req = google.auth.transport.requests.Request();
print(f"Checking Authentication : {credentials.valid}")
print('Refreshing token ....')
credentials.refresh(auth_req)
#check for valid credentials
print(f"Checking Authentication : {credentials.valid}")
access_token = credentials.token
credentials = google.oauth2.credentials.Credentials(access_token);
storage_client = storage.Client(project='itg-ri-consumerloop-gbl-ww-dv',credentials=credentials)
I am entirely new to NodeJS, and I am trying to make the same thing.
My goal later is to create an app engine application that would expose an image that is found in a private bucket, so credentials are a must.
How it is done?
For authentication, you could rely on the default application credentials that are present within the GCP platform (GAE, Cloud Functions, VM, etc.). Then you could just run the following piece of code from the documentation:
const {Storage} = require('#google-cloud/storage');
const storage = new Storage();
const bucket = storage.bucket('albums');
const file = bucket.file('my-existing-file.png');
In most circumstances, there is no need to explicitly use authentication packages since they are already executed underneath the google-cloud/storage package in Nodejs. The same holds for the google-cloud-storage package in Python. It could help to look at the source code of both packages on Github. For me, this really helped to understand the authentication mechanism.
When I develop code on my own laptop, that interacts with google cloud storage, I first tell the gcloud SDK what my credentials are and on which GCP project I am working. I use the following commands for this:
gcloud config set project [PROJECT_ID]
gcloud auth application-default login
You could also set DEFAULT_APPLICATION_CREDENTIALS as an environment variable that points to a credentials file. Then within your code, you could pass the project name when initializing the client. This could be helpful if you are running your code outside of GCP on another server for example.

Why my API calls using google-api-nodejs-client to Google Analytics are not working in production?

I'm calling the Google Analytics Reporting API using google-api-nodejs-client to show the number of visits inside a blog.
This blog is hosted inside Google App Engine Standard Environment.
In development, I'm authenticating my API calls using the Application Default Credentials. I downloaded the JSON file with the credentials from the account service I created exclusively for analytics purposes, set the file to the Google_Application_Credentials environment variable and everything worked. I'm able to get the data from Google Analytics and display it in the website.
But this is not working in production. I suppose getClient() it's not getting the credentials in that environment.
Things to note: 1) I did not upload the downloaded JSON file with the credentials from the service account (I think it would be counter intuitive and unsafe to do that, and from what I understood in the docs, GCP is able to deal automatically with the API authentications);
const {google} = require("googleapis");
async function main () {
// This method looks for the GCLOUD_PROJECT and GOOGLE_APPLICATION_CREDENTIALS
// environment variables.
const auth = await google.auth.getClient({
// Scope of the analytics reporting,
// with only reading access.
scopes: 'https://www.googleapis.com/auth/analytics.readonly',
});
// Create the analytics reporting object
const analyticsreporting = await google.analyticsreporting({
version: 'v4',
auth: auth,
});
// Fetch the analytics reporting
const res = await analyticsreporting.reports.batchGet({...});
return res.data;
}
I already run out of options. Can someone help me with this?
This is a problem with the default scopes and application default credentials. By default, if you don't create a new service account, you are going to get 'application default credentials' from the GCE metadata service:
https://cloud.google.com/docs/authentication/production#auth-cloud-implicit-nodejs
Those credentials usually only have the cloud-platform scope, and the set of scopes cannot be changed (as of today). To make this work, you have a few options.
You could create a new service account, download the service account key, and use the keyFile property in the getClient method options to reference the key. If you do it this way, the scopes you pass into getClient will be respected.
You could play with the scopes available to the service account under which your GAE application is running. I haven't personally tried that, but it theoretically should be possible.
Best of luck!

Google Cloud Vision reverse image search fails on Azure App Service because GOOGLE_APPLICATION_CREDENTIALS file cannot be found

I am attempting to perform a Google reverse image search using Google Cloud Vision on an Azure app service web app.
I have generated a googleCred.json, which the Google client libraries use in order to construct API requests. Google expects it to be available from an environment variable named GOOGLE_APPLICATION_CREDENTIALS.
The Azure app service that runs the web app has settings that mimic environment variables for the Google client libraries. The documentation is here, and I have successfully set the variable here:
Furthermore, the googleCred.json file has been uploaded to the app service. Here is the documentation I followed to use FTP and FileZilla to upload the file:
Also, the file permissions are as open as they can be:
However, when I access the web app in the cloud, I get the following error message:
Error reading credential file from location D:\site\wwwroot\Statics\googleCred.json: Could not find a part of the path 'D:\site\wwwroot\Statics\googleCred.json'. Please check the value of the Environment Variable GOOGLE_APPLICATION_CREDENTIALS
What am I doing wrong? How can I successfully use the Google Cloud Vision API on an Azure web app?
This error message is usually thrown when the application is not being authenticated correctly due to several reasons such as missing files, invalid credential paths, incorrect environment variables assignations, among other causes.
Based on this, I recommend you to validate that the credential file and file path are being correctly assigned, as well as follow the Obtaining and providing service account credentials manually guide in order to explicitly specify your service account file directly into your code; In this way, you will be able to set it permanently and verify if you are passing the service credentials correctly.
Passing the path to the service account key in code example:
// Imports the Google Cloud client library.
const Storage = require('#google-cloud/storage');
// Instantiates a client. Explicitly use service account credentials by
// specifying the private key file. All clients in google-cloud-node have this
// helper, see https://github.com/GoogleCloudPlatform/google-cloud-node/blob/master/docs/authentication.md
const storage = new Storage({
keyFilename: '/path/to/keyfile.json'
});
// Makes an authenticated API request.
storage
.getBuckets()
.then((results) => {
const buckets = results[0];
console.log('Buckets:');
buckets.forEach((bucket) => {
console.log(bucket.name);
});
})
.catch((err) => {
console.error('ERROR:', err);
});
I'm writing here since i can't comment, but at a quick glance, is the "D:" in the path necessary? I assume you uploaded the file to the app service so try with this value for the path "\site\wwwroot\Statics\googleCred.json"

Firebase serve --only functions, admin.auth() locally does not work unless I manually set the key.json

I am trying to run cloud functions locally using firebase serve --only functions. It works if I manually specify the key.json that is downloaded when I create a service account through firebase console here: https://console.firebase.google.com/project/project-id/settings/serviceaccounts/adminsdk.
And then doing export GOOGLE_APPLICATION_CREDENTIALS=key.json. Why do I have to do this? Shouldn't firebase or gcloud handle this correctly? I thought credential in functions.config().firebase was ApplicationDefault. https://developers.google.com/identity/protocols/application-default-credentials. According to that, the env variable like above is checked first, so it works if that is there. But shouldn't #2 validate me correctly?
I have gcloud for app engine so I have done gcloud init and gcloud auth application-default login.
I initialize my app in index.js
const admin = require('firebase-admin');
const functions = require('firebase-functions');
admin.initializeApp(functions.config().firebase);
firebase serve --only functions deploys the functions fine. But when I execute one which uses this code admin.auth().getUser(uid)... it breaks with the following error:
Error: An internal error has occurred. Raw server response: "{"error":{"errors":[{"domain":"usageLimits","reason":"accessNotConfigured","message":"Access Not Configured. Google Identity Toolkit API has not been used in project 7640...50 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/identitytoolkit.googleapis.com/overview?project=7640...50 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.","extendedHelp":"https://console.developers.google.com/apis/api/identitytoolkit.googleapis.com/overview?project=7640...50"}],"code":403,"message":"Access Not Configured. Google Identity Toolkit API has not been used in project 7640...50 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/identitytoolkit.googleapis.com/overview?project=7640...50 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry."}}
The project number doesn't correspond to my firebase project or project id/number in console.cloud.google.com
Without gcloud, I get this error
Error: Credential implementation provided to initializeApp() via the "credential" property failed to fetch a valid Google OAuth2 access token with the following error: "Error fetching access token: invalid_grant (Token has been expired or revoked.)". There are two likely causes: (1) your server time is not properly synced or (2) your certificate key file has been revoked. To solve (1), re-sync the time on your server. To solve (2), make sure the key ID for your key file is still present at https://console.firebase.google.com/iam-admin/serviceaccounts/project. If not, generate a new key file at https://console.firebase.google.com/project/_/settings/serviceaccounts/adminsdk
similar to https://github.com/urish/firebase-server/issues/81
I already have a key file generated. I've tried deleting my existing ones and making new ones, the only solution is to download this key that is generated and manually export the environment variable
Is there any other way than that?
Technically yes, but you can build your code so that it uses a local key when testing locally, and uses the service key when running on the server.
For this I set the env var FB_SERVICE_ACCOUNT_KEY to the path of the json secret. When it is unset (i.e. deployed within Firebase functions), the standard init will be used.
Here is a basic example
if(process.env.FB_SERVICE_ACCOUNT_KEY){
var serviceAccount = require(process.env.FB_SERVICE_ACCOUNT_KEY);
admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
databaseURL: `https://${process.env.GCLOUD_PROJECT}.firebaseio.com`
});
}
else{
admin.initializeApp(functions.config().firebase);
}

Resources