Google FCM firebase-admin initializeApp() error in K8S cluster - node.js

I am building NodeJs server and trying to leverage FCM for push notifications. Locally everything works as expected, but in the K8S cluster I receive the following error
FirebaseAppError: Credential implementation provided to initializeApp() via the "credential" property failed to fetch a valid Google OAuth2 access token with the following error: "Error fetching access token: Missing error payload".
errorInfo: {
code: 'app/invalid-credential',
message: 'Credential implementation provided to initializeApp() via the "credential" property failed to fetch a valid Google OAuth2 access token with the following error: "Error fetching access token: Missing error payload".'
},
codePrefix: 'app'
I tried to initialize app without arguments admin.initializeApp(), keeping env var with path json key in a config object, the path is valid, the file is present, the env is printed out ok.
I tried like this const firebase = admin.initializeApp({ credential: admin.credential.applicationDefault(), }); which is obviuoely the same thing. I Tried to explicitly pass credential prop with json certificate like so const firebase = initializeApp({ credential: admin.credential.cert(cert) });
cert is present. I even tried to downgrade from firebase-admin from 10 to 9.
Node 12
please help

I found an issue. It was DNS settings of my K8S cluster. So it didn't have proper access to the outside world (check /etc/resolv.conf inside the pod)
Had to add dns settings to my deployment declaration like so
containers:
...
dnsPolicy: "None"
dnsConfig:
nameservers:
- 10.96.0.10 // for inner services lookup
- 8.8.8.8
searches:
- default.svc.cluster.local
- svc.cluster.local
- cluster.local

Related

Firebase Cloud messaging : app/invalid-credential - failed to fetch a valid Google OAuth2 access token

I've been facing the same issue randomly i.e. this issue occurs sometime while working fine most of the time. Running nodejs version 14.x over AWS Lambda.
Error sending message: FirebaseAppError: Credential implementation provided to initializeApp() via the "credential" property failed to fetch a valid Google OAuth2 access token with the following error: "Error fetching access token: Error while making request: connect ETIMEDOUT ".
at /opt/nodejs/node_modules/firebase-admin/lib/app/firebase-app.js:85:19
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at async fetchVideosAndUsers (/var/task/src/videos-notifications.js:180:17)
at async doAction (/var/task/src/videos-notifications.js:233:30)
at async Runtime.exports.handler (/var/task/src/) {
errorInfo: {
code: 'app/invalid-credential',
message: 'Credential implementation provided to initializeApp() via the "credential" property failed to fetch a valid Google OAuth2 access token with the following error: "Error fetching access token: Error while making request: connect ETIMEDOUT".'
},
codePrefix: 'app'
}
Please advise.
This error might be caused by an issue with your application’s credentials. You can try checking the following:
Ensure that you’re requesting the scope: https://www.googleapis.com/auth/firebase.messaging, as you’ll need this to authorize access to FCM for your app.
Ensure that you obtain the proper service account. Then generate a private key file(in JSON format) to authorize access to Firebase services.

Connect to Cloud Storage through kubernetes pod with NodeJS

what I want to achieve, is that for my pods that live inside GKE to share files. So what I'm thinking is using the GoogleCloudStorage to write and read the files.
I have created a service account in my kubetcl
kubectl create serviceaccount myxxx-svc-account --namespace
myxxx
Then I also created the service account in my GCP console
Then, I added the roles of roles/iam.workloadIdentityUser in my GCP account
Next, I annotated my kubectl account with my GCP service account
kubectl annotate serviceaccount --namespace myxxx
myxxx-svc-account
iam.gke.io/gcp-service-account=myxxx-svc-account#myxxx-xxxxx.iam.gserviceaccount.com
I also added the roles of Storage Admin and Storage Object Admin in the GCP - IAM page
Then, in my deployment.yaml, I included my service account
spec:
serviceAccountName: myxxx-account
Bellow is how I try to upload a file to the storage
const {Storage} = require('#google-cloud/storage');
const storage = new Storage();
const bucket = storage.bucket('bucket-name');
const options = {
destination: '/folder1/folder2/123456789'
};
bucket.upload("./index.js", options, function(uploadError, file, apiResponse) {
console.log(uploadError.message)
console.log(uploadError.stack)
});
I deploy my node application to the GKE pods through docker. In the dockerFile, im using
FROM node
...
...
...
CMD ["node", "index.js"]
But I always get unauthorized 403 error
Could not refresh access token: A Forbidden error was returned while
attempting to retrieve an access token for the Compute Engine built-in
service account. This may be because the Compute Engine instance does
not have the correct permission scopes specified: Could not refresh
access token: Unsuccessful response status code. Request failed with
status code 403
Error: Could not refresh access token: A Forbidden
error was returned while attempting to retrieve an access token for
the Compute Engine built-in service account. This may be because the
Compute Engine instance does not have the correct permission scopes
specified: Could not refresh access token: Unsuccessful response
status code. Request failed with status code 403
at Gaxios._request (/opt/app/node_modules/gaxios/build/src/gaxios.js:130:23)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async metadataAccessor (/opt/app/node_modules/gcp-metadata/build/src/index.js:68:21)
at async Compute.refreshTokenNoCache (/opt/app/node_modules/google-auth-library/build/src/auth/computeclient.js:54:20)
at async Compute.getRequestMetadataAsync (/opt/app/node_modules/google-auth-library/build/src/auth/oauth2client.js:298:17)
at async Compute.requestAsync (/opt/app/node_modules/google-auth-library/build/src/auth/oauth2client.js:371:23)
at async Upload.makeRequest (/opt/app/node_modules/#google-cloud/storage/build/src/resumable-upload.js:574:21)
at async retry.retries (/opt/app/node_modules/#google-cloud/storage/build/src/resumable-upload.js:306:29)
at async Upload.createURIAsync (/opt/app/node_modules/#google-cloud/storage/build/src/resumable-upload.js:303:21)
What I'm doing wrong? seems like I have given the permission already? how can I troubleshoot it? Is it related with the docker image?

Authenticating as a service account outside of the Google Cloud environment using firebase-admin

I'm having trouble on authenticating as a service account in my Next.js app hosted on Vercel. My code is working fine in my dev environment, but it fails with the following error message when I try to run it on Vercel Node.js v14 environment. I guess that it runs on my local machine because I'm logged in gcloud with my email (project owner).
This is the error I'm getting:
Error: Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information.
at GoogleAuth.getApplicationDefaultAsync (/var/task/node_modules/google-auth-library/build/src/auth/googleauth.js:173:19)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async GoogleAuth.getClient (/var/task/node_modules/google-auth-library/build/src/auth/googleauth.js:551:17)
at async GrpcClient._getCredentials (/var/task/node_modules/google-gax/build/src/grpc.js:109:24)
at async GrpcClient.createStub (/var/task/node_modules/google-gax/build/src/grpc.js:252:23)
I've created the following service account to use it with my Next.js APIs.
It has all the necessary roles. I've created a JSON key and download it.
I'm using firebase-admin and this is how I'm initializing it:
export const initializeFirebaseAdmin = (): FirebaseAdmin => {
const account = getServiceAccount(); // THIS IS THE SERVICE ACCOUNT JSON KEY (ALREADY PARSED AS AN OBJECT)
if (!admin.apps.length) {
admin.initializeApp({
credential: admin.credential.cert(account),
});
}
return admin;
};
This is what I think it's happening:
From: https://cloud.google.com/docs/authentication/production#automatically
From the image above:
I'm not setting any GOOGLE_APPLICATION_CREDENTIALS environment variable
So I should be on number 2. It will "try to use the service account that is attached to the resource that is running your code."
It's obviously failing and I getting the error
But which resource is it referring to? Which code?
The resource that is running my code is Firebase Admin? Am I not initializing it correctly?
This code should work to authenticate the firebase-admin package:
export const initializeFirebaseAdmin = (): FirebaseAdmin => {
const account = getServiceAccount(); // THIS IS THE SERVICE ACCOUNT JSON KEY (ALREADY PARSED AS AN OBJECT)
if (!admin.apps.length) {
admin.initializeApp({
credential: admin.credential.cert(account),
});
}
return admin;
};
My problem was related to the fact that I was using a client exposed from the firebase-admin package, that doesn't have access to the authentication from the main package. So I needed to pass the credentials to it as well. Like:
const client = new admin.firestore.v1.FirestoreAdminClient({
credentials: SERVICE_ACCOUNT as CredentialBody // <<<<<< THIS IS THE SERVICE ACCOUNT JSON KEY
});
Refer also to: Trying to export backup via admin.firestore.v1.FirestoreAdminClient on a Next.js API route. Error: Could not load the default credentials

Firebase serve --only functions, admin.auth() locally does not work unless I manually set the key.json

I am trying to run cloud functions locally using firebase serve --only functions. It works if I manually specify the key.json that is downloaded when I create a service account through firebase console here: https://console.firebase.google.com/project/project-id/settings/serviceaccounts/adminsdk.
And then doing export GOOGLE_APPLICATION_CREDENTIALS=key.json. Why do I have to do this? Shouldn't firebase or gcloud handle this correctly? I thought credential in functions.config().firebase was ApplicationDefault. https://developers.google.com/identity/protocols/application-default-credentials. According to that, the env variable like above is checked first, so it works if that is there. But shouldn't #2 validate me correctly?
I have gcloud for app engine so I have done gcloud init and gcloud auth application-default login.
I initialize my app in index.js
const admin = require('firebase-admin');
const functions = require('firebase-functions');
admin.initializeApp(functions.config().firebase);
firebase serve --only functions deploys the functions fine. But when I execute one which uses this code admin.auth().getUser(uid)... it breaks with the following error:
Error: An internal error has occurred. Raw server response: "{"error":{"errors":[{"domain":"usageLimits","reason":"accessNotConfigured","message":"Access Not Configured. Google Identity Toolkit API has not been used in project 7640...50 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/identitytoolkit.googleapis.com/overview?project=7640...50 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.","extendedHelp":"https://console.developers.google.com/apis/api/identitytoolkit.googleapis.com/overview?project=7640...50"}],"code":403,"message":"Access Not Configured. Google Identity Toolkit API has not been used in project 7640...50 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/identitytoolkit.googleapis.com/overview?project=7640...50 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry."}}
The project number doesn't correspond to my firebase project or project id/number in console.cloud.google.com
Without gcloud, I get this error
Error: Credential implementation provided to initializeApp() via the "credential" property failed to fetch a valid Google OAuth2 access token with the following error: "Error fetching access token: invalid_grant (Token has been expired or revoked.)". There are two likely causes: (1) your server time is not properly synced or (2) your certificate key file has been revoked. To solve (1), re-sync the time on your server. To solve (2), make sure the key ID for your key file is still present at https://console.firebase.google.com/iam-admin/serviceaccounts/project. If not, generate a new key file at https://console.firebase.google.com/project/_/settings/serviceaccounts/adminsdk
similar to https://github.com/urish/firebase-server/issues/81
I already have a key file generated. I've tried deleting my existing ones and making new ones, the only solution is to download this key that is generated and manually export the environment variable
Is there any other way than that?
Technically yes, but you can build your code so that it uses a local key when testing locally, and uses the service key when running on the server.
For this I set the env var FB_SERVICE_ACCOUNT_KEY to the path of the json secret. When it is unset (i.e. deployed within Firebase functions), the standard init will be used.
Here is a basic example
if(process.env.FB_SERVICE_ACCOUNT_KEY){
var serviceAccount = require(process.env.FB_SERVICE_ACCOUNT_KEY);
admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
databaseURL: `https://${process.env.GCLOUD_PROJECT}.firebaseio.com`
});
}
else{
admin.initializeApp(functions.config().firebase);
}

How to connect to Firebase by Firebase Admin against a proxy?

Currently, I am using Firebase Admin SDK to connect a Firebase database in a NodeJS server side application.
But I do not find an option to connect Firebase via proxy settings, or it can detect my system HTTP_PROXY environment variable.
When I run the node script by node index.js, and got some timeout messages like this(I know in my work network, I can not connect to Firebase directly).
Error: Credential implementation provided to initializeApp() via the "credential
" property failed to fetch a valid Google OAuth2 access token with the following
error: "connect ETIMEDOUT 216.58.200.237:443".
at ....erver\node_modules\firebase-adm
in\lib\firebase-app.js:74:23
at process._tickCallback (internal/process/next_tick.js:103:7)
I also use browser to access the firebase console via proxy, it works.
But how to resolve this issue in NodeJS server side scripts?
This error also happens if the date and time on your host machine where you run NodeJS process is not set right. Make sure to keep the server time synced.
The full error message:
Error: Credential implementation provided to initializeApp() via the "credential" property failed to fetch a valid Google OAuth2 access token with the following error: "Error fetching access token: invalid_grant (Invalid JWT: Token must be a short-lived token and in a reasonable timeframe)". The most likely cause of this error is using a certificate key file which has been revoked. Make sure the key ID for your key file is still present at https://console.firebase.google.com/iam-admin/serviceaccounts/project. If not, generatea new key file at https://console.firebase.google.com/project/_/settings/serviceaccounts/adminsdk.
Faced with the exact same problem yesterday, got it solved.
Let's get it straight, you are getting this error because Google services are banned in your region, therefore you should access firebase through a proxy. Here's a blog explaining how it is done.
For this specific case, you should
Prepare a proxy server that allows you to access Google services, then
Install the https-proxy-agent package through npm or yarn, then
Include the proxy in your firebase app initilization code like this
import HttpsProxyAgent from 'https-proxy-agent';
import * as admin from 'firebase-admin';
...
const agent = new HttpsProxyAgent('url to your proxy server');
admin.initializeApp({
credential: admin.credential.applicationDefault(agent),
// Or any function you would like to use to provide your application's credentials
// But remember to include the proxy agent in the parameter
httpAgent: agent
});
But remember not to commit this change into your repo, since this problem appears specifically in regions without access to Google services.

Resources