Google Cloud Vision OCR Error Code 7 - Permission Denied - node.js

I am building a webapp that utilizes Google Cloud Vision's OCR. The OCR works fine for about 7-8 requests, after which I get an error like so:
Error: 7 PERMISSION_DENIED: Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the vision.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/.
The problem is, I have already set up a billing account and a service account.
I have tried using multiple GCloud commands to fix this, and when I run gcloud auth list, I can see that my service account is the active account. I have also tried generating a JSON key and setting path to that key in my enviroment variables - as instructed here: https://cloud.google.com/docs/authentication/getting-started
Has anyone encountered this issue before? For reference, I am running Windows 10 and using Node.js for the webapp. Thanks!

You are authenticating using end user credentials from the Google Cloud SDK or Google Cloud Shell and not service account credentials.
1.Make a new directory
mkdir ocr
cd ocr
2.Download an image.
curl https://www.python.org/static/apple-touch-icon-144x144-precomposed.png > image.png
3.Install the client library.
sudo pi3 install --upgrade google-cloud-vision
4.Create a service account.
gcloud iam service-accounts create ocr-vision \
--description "ocr-vision" \
--display-name "ocr-vision"
gcloud iam service-accounts list
5.Create a key.json file.
gcloud iam service-accounts keys create key.json \
--iam-account ocr-vision#your-project.iam.gserviceaccount.com
6.Assign the owner role to the service account.
gcloud projects add-iam-policy-binding your-project \
--member serviceAccount:ocr-vision#your-project.iam.gserviceaccount.com \
--role roles/owner
7.Export the env variable
export GOOGLE_APPLICATION_CREDENTIALS=key.json
8.Run the script
python script.py
import io
import os
# Imports the Google Cloud client library
from google.cloud import vision
from google.cloud.vision import types
# Instantiates a client
client = vision.ImageAnnotatorClient()
# The name of the image file to annotate
file_name = os.path.abspath('image.png')
# Loads the image into memory
with io.open(file_name, 'rb') as image_file:
content = image_file.read()
image = types.Image(content=content)
# Performs label detection on the image file
response = client.label_detection(image=image)
labels = response.label_annotations
print('Labels:')
for label in labels:
print(label.description)
9.Output
Labels:
Yellow
Font
Line
Material property
Clip art
Logo
Symbol
Icon
Graphics
Illustration

Related

How to access Files in Google Cloud Storage through GKE pods

I'm trying get image files of Google Cloud Storage (GCS) in my Node.js application using Axios client. On develop mode using my PC I pass a Bearer Token and all works properly.
But, I need to use this in production in a cluster hosted on Google Kubernetes Engine (GKE).
I made recommended tuturials to create a service account (GSA), then I vinculed with kubernetes account (KSA), via Workload identity approach, but when I try get files througt one endpoint on my app, I'm receiving:
{"statusCode":401,"message":"Unauthorized"}
What is missing to make?
Update: What I've done:
Create Google Service Account
https://cloud.google.com/iam/docs/creating-managing-service-accounts
Create Kubernetes Service Account
# gke-access-gcs.ksa.yaml file
apiVersion: v1
kind: ServiceAccount
metadata:
name: gke-access-gcs
kubectl apply -f gke-access-gcs.ksa.yaml
Relate KSAs and GSAs
gcloud iam service-accounts add-iam-policy-binding \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:cluster_project.svc.id.goog[k8s_namespace/ksa_name]" \
gsa_name#gsa_project.iam.gserviceaccount.com
Note the KSA and complete the link between KSA and GSA
kubectl annotate serviceaccount \
--namespace k8s_namespace \
ksa_name \
iam.gke.io/gcp-service-account=gsa_name#gsa_project.iam.gserviceaccount.com
Set Read and Write role:
gcloud projects add-iam-policy-binding project-id \
--member=serviceAccount:gsa-account#project-id.iam.gserviceaccount.com \
--role=roles/storage.objectAdmin
Test access:
kubectl run -it \
--image google/cloud-sdk:slim \
--serviceaccount ksa-name \
--namespace k8s-namespace \
workload-identity-test
The above command works correctly. Note that was passed --serviceaccount and workload-identity. Is this necessary to GKE?
PS: I don't know if this influences, but I am using SQL Cloud with proxy in the project.
EDIT
Issue portrayed in the question is related to the fact that axios client does not use the Application Default Credentials (as official Google libraries) mechanism that Workload Identity takes advantage of. The ADC checks:
If the environment variable GOOGLE_APPLICATION_CREDENTIALS is set, ADC uses the service account file that the variable points to.
If the environment variable GOOGLE_APPLICATION_CREDENTIALS isn't set, ADC uses the default service account that Compute Engine, Google Kubernetes Engine, App Engine, Cloud Run, and Cloud Functions provide.
-- Cloud.google.com: Authentication: Production
This means that axios client will need to fall back to the Bearer token authentication method to authenticate against Google Cloud Storage.
The authentication with Bearer token is described in the official documentation as following:
API authentication
To make requests using OAuth 2.0 to either the Cloud Storage XML API or JSON API, include your application's access token in the Authorization header in every request that requires authentication. You can generate an access token from the OAuth 2.0 Playground.
Authorization: Bearer OAUTH2_TOKEN
The following is an example of a request that lists objects in a bucket.
JSON API
Use the list method of the Objects resource.
GET /storage/v1/b/example-bucket/o HTTP/1.1
Host: www.googleapis.com
Authorization: Bearer ya29.AHES6ZRVmB7fkLtd1XTmq6mo0S1wqZZi3-Lh_s-6Uw7p8vtgSwg
-- Cloud.google.com: Storage: Docs: Api authentication
I've included basic example of a code snippet using Axios to query the Cloud Storage (requires $ npm install axios):
const Axios = require('axios');
const config = {
headers: { Authorization: 'Bearer ${OAUTH2_TOKEN}' }
};
Axios.get(
'https://storage.googleapis.com/storage/v1/b/BUCKET-NAME/o/',
config
).then(
(response) => {
console.log(response.data.items);
},
(err) => {
console.log('Oh no. Something went wrong :(');
// console.log(err) <-- Get the full output!
}
);
I left below example of Workload Identity setup with a node.js official library code snippet as it could be useful to other community members.
Posting this answer as I've managed to use Workload Identity and a simple nodejs app to send and retrieve data from GCP bucket.
I included some bullet points for troubleshooting potential issues.
Steps:
Check if GKE cluster has Workload Identity enabled.
Check if your Kubernetes service account is associated with your Google Service account.
Check if example workload is using correct Google Service account when connecting to the API's.
Check if your Google Service account is having correct permissions to access your bucket.
You can also follow the official documentation:
Cloud.google.com: Kubernetes Engine: Workload Identity
Assuming that:
Project (ID) named: awesome-project <- it's only example
Kubernetes namespace named: bucket-namespace
Kubernetes service account named: bucket-service-account
Google service account named: google-bucket-service-account
Cloud storage bucket named: workload-bucket-example <- it's only example
I've included the commands:
$ kubectl create namespace bucket-namespace
$ kubectl create serviceaccount --namespace bucket-namespace bucket-service-account
$ gcloud iam service-accounts create google-bucket-service-account
$ gcloud iam service-accounts add-iam-policy-binding --role roles/iam.workloadIdentityUser --member "serviceAccount:awesome-project.svc.id.goog[bucket-namespace/bucket-service-account]" google-bucket-service-account#awesome-project.iam.gserviceaccount.com
$ kubectl annotate serviceaccount --namespace bucket-namespace bucket-service-account iam.gke.io/gcp-service-account=google-bucket-service-account#awesome-project-ID.iam.gserviceaccount.com
Using the guide linked above check the service account authenticating to API's:
$ kubectl run -it --image google/cloud-sdk:slim --serviceaccount bucket-service-account --namespace bucket-namespace workload-identity-test
The output of $ gcloud auth list should show:
Credentialed Accounts
ACTIVE ACCOUNT
* google-bucket-service-account#AWESOME-PROJECT.iam.gserviceaccount.com
To set the active account, run:
$ gcloud config set account `ACCOUNT`
Google service account created earlier should be present in the output!
Also it's required to add the permissions for the service account to the bucket. You can either:
Use Cloud Console
Run: $ gsutil iam ch serviceAccount:google-bucket-service-account#awesome-project.iam.gserviceaccount.com:roles/storage.admin gs://workload-bucket-example
To download the file from the workload-bucket-example following code can be used:
// Copyright 2020 Google LLC
/**
* This application demonstrates how to perform basic operations on files with
* the Google Cloud Storage API.
*
* For more information, see the README.md under /storage and the documentation
* at https://cloud.google.com/storage/docs.
*/
const path = require('path');
const cwd = path.join(__dirname, '..');
function main(
bucketName = 'workload-bucket-example',
srcFilename = 'hello.txt',
destFilename = path.join(cwd, 'hello.txt')
) {
const {Storage} = require('#google-cloud/storage');
// Creates a client
const storage = new Storage();
async function downloadFile() {
const options = {
// The path to which the file should be downloaded, e.g. "./file.txt"
destination: destFilename,
};
// Downloads the file
await storage.bucket(bucketName).file(srcFilename).download(options);
console.log(
`gs://${bucketName}/${srcFilename} downloaded to ${destFilename}.`
);
}
downloadFile().catch(console.error);
// [END storage_download_file]
}
main(...process.argv.slice(2));
The code is exact copy from:
Googleapis.dev: NodeJS: Storage
Github.com: Googleapis: Nodejs-storage: downloadFile.js
Running this code should produce an output:
root#ubuntu:/# nodejs app.js
gs://workload-bucket-example/hello.txt downloaded to /hello.txt.
root#ubuntu:/# cat hello.txt
Hello there!

Issue while recreating AWS authentication profile in Terraform

Followed below mentioned step for creating the sso profile in AWS using command prompt and leverage it to create S3 bucket:
Steps to Setup Profile:
In PowerShell ran: aws configure sso
sso_start_url = url
sso_region = us-east-1
Note: It will take to browser for login and verification. Once verified, it retrieves the role, After that i have selected the role
CLI default client Region [eu-west-1]: us-east-1
CLI default output format [json]: json
CLI profile name : <<Provide your choice of name>>
This will create .aws folder and config file under your home directory (under \User\\<<username>>)
Steps to use profile:
Ran below command to set profile in PowerShell
setx AWS_PROFILE <<profile name>> and configure the access key and secret key using command aws configure
After setting profile in current PowerShell session, trying to create the S3 bucket, but unable to create it.
Got error like
No valid credentials for AWS provider
Profile recreation steps:
To recreate profile I have deleted the .aws folder which was created in the home directory path (under \User\\<<username>>\\.aws)
Now, while trying to create new profile using aws configure sso command, it is showing error like
The config profile (profile name) could not be found

Authenticating to Google Cloud Firestore from GKE with Workload Identity

I'm trying to write a simple backend that will access my Google Cloud Firestore, it lives in the Google Kubernetes Engine. On my local I'm using the following code to authenticate to Firestore as detailed in the Google Documentation.
if (process.env.NODE_ENV !== 'production') {
const result = require('dotenv').config()
//Additional error handling here
}
This pulls the GOOGLE_APPLICATION_CREDENTIALS environment variable and populates it with my google-application-credentals.json which I got from creating a service account with the "Cloud Datastore User" role.
So, locally, my code runs fine. I can reach my Firestore and do everything I need to. However, the problem arises once I deploy to GKE.
I followed this Google Documentation to set up a Workload Identity for my cluster, I've created a deployment and verified that the pods all are using the correct IAM Service Account by running:
kubectl exec -it POD_NAME -c CONTAINER_NAME -n NAMESPACE sh
> gcloud auth list
I was under the impression from the documentation that authentication would be handled for my service as long as the above held true. I'm really not sure why but my Firestore() instance is behaving as if it does not have the necessary credentials to access the Firestore.
In case it helps below is my declaration and implementation of the instance:
const firestore = new Firestore()
const server = new ApolloServer({
schema: schema,
dataSources: () => {
return {
userDatasource: new UserDatasource(firestore)
}
}
})
UPDATE:
In a bout of desperation I decided to tear down everything and re-build it. Following everything over step by step I appear to have either encountered a bug or (more likely) I did something mildly wrong the first time. I'm now able to connect to my backend service. However, I'm now getting a different error. Upon sending any request (I'm using GraphQL, but in essence it's any REST call) I get back a 404.
Inspecting the logs yields the following:
'Getting metadata from plugin failed with error: Could not refresh access token: A Not Found error was returned while attempting to retrieve an accesstoken for the Compute Engine built-in service account. This may be because the Compute Engine instance does not have any permission scopes specified: Could not refresh access token: Unsuccessful response status code. Request failed with status code 404'
A cursory search for this issue doesn't seem to return anything related to what I'm trying to accomplish, and so I'm back to square one.
I think your initial assumption was correct! Workload Identity is not functioning properly if you still have to specify scopes. In the Workload article you have linked, scopes are not used.
I've been struggling with the same issue and have identified three ways to get authenticated credentials in the pod.
1. Workload Identity (basically the Workload Identity article above with some deployment details added)
This method is preferred because it allows each pod deployment in a cluster to be granted only the permissions it needs.
Create cluster (note: no scopes or service account defined)
gcloud beta container clusters create {cluster-name} \
--release-channel regular \
--identity-namespace {projectID}.svc.id.goog
Then create the k8sServiceAccount, assign roles, and annotate.
gcloud container clusters get-credentials {cluster-name}
kubectl create serviceaccount --namespace default {k8sServiceAccount}
gcloud iam service-accounts add-iam-policy-binding \
--member serviceAccount:{projectID}.svc.id.goog[default/{k8sServiceAccount}] \
--role roles/iam.workloadIdentityUser \
{googleServiceAccount}
kubectl annotate serviceaccount \
--namespace default \
{k8sServiceAccount} \
iam.gke.io/gcp-service-account={googleServiceAccount}
Then I create my deployment, and set the k8sServiceAccount.
(Setting the service account was the part that I was missing)
kubectl create deployment {deployment-name} --image={containerImageURL}
kubectl set serviceaccount deployment {deployment-name} {k8sServiceAccount}
Then expose with a target of 8080
kubectl expose deployment {deployment-name} --name={service-name} --type=LoadBalancer --port 80 --target-port 8080
The googleServiceAccount needs to have the appropriate IAM roles assigned (see below).
2. Cluster Service Account
This method is not preferred, because all VMs and pods in the cluster will have permissions based on the defined service account.
Create cluster with assigned service account
gcloud beta container clusters create [cluster-name] \
--release-channel regular \
--service-account {googleServiceAccount}
The googleServiceAccount needs to have the appropriate IAM roles assigned (see below).
Then deploy and expose as above, but without setting the k8sServiceAccount
3. Scopes
This method is not preferred, because all VMs and pods in the cluster will have permisions based on the scopes defined.
Create cluster with assigned scopes (firestore only requires "cloud-platform", realtime database also requires "userinfo.email")
gcloud beta container clusters create $2 \
--release-channel regular \
--scopes https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/userinfo.email
Then deploy and expose as above, but without setting the k8sServiceAccount
The first two methods require a Google Service Account with the appropriate IAM roles assigned. Here are the roles I assigned to get a few Firebase products working:
FireStore: Cloud Datastore User (Datastore)
Realtime Database: Firebase Realtime Database Admin (Firebase Products)
Storage: Storage Object Admin (Cloud Storage)
Going to close this question.
Just in case anyone stumbles onto it here's what fixed it for me.
1.) I re-followed the steps in the Google Documentation link above, this fixed the issue of my pods not launching.
2.) As for my update, I re-created my cluster and gave it the Cloud Datasource permission. I had assumed that the permissions were seperate from what Workload Identity needed to function. I was wrong.
I hope this helps someone.

Google Cloud Platform Authentification: Recognized as end user anthentification despite using a service account

Anyone can HELP? This one is really driving me crazy... Thank you!
I tried to use a google cloud platform API Speech-to-text.
Tools: WINDOWS 10 && GCP &&Python(Pycharm IDE)
I've created a service account as a owner for my speech-to-test project and generated a key from GCP console in json, then I set the environment variables.
Code I ran on WIN10 Powershell && CMD:
$env:GOOGLE_APPLICATION_CREDENTIALS="D:\GCloud speech-to-text\Speech
To Text Series-93e03f36bc9d.json"
set GOOGLE_APPLICATION_CREDENTIALS=D:\GCloud speech-to-text\Speech To
Text Series-93e03f36bc9d.json
PS: the added environment variables disappear in CMD and Powershell after reboot my laptop but do show in the env list if added again.
I've enabled the google storage api and google speech-to-text api in GCP console.
I've tried the explicitely showing credential method via python, same problem.
I've installed the google cloud SDK shell and initialized by using command to log in my account.
PYTHON SPEECH-TO-TEXT CODE(from GCP demo)
import io
import os
# Imports the Google Cloud client library
from google.cloud import speech
from google.cloud.speech import enums
from google.cloud.speech import types
# Instantiates a client
client = speech.SpeechClient()
# The name of the audio file to transcribe
file_name = os.path.join(
os.path.dirname(__file__),
'test_cre.m4a')
# Loads the audio into memory
with io.open(file_name, 'rb') as audio_file:
content = audio_file.read()
audio = types.RecognitionAudio(content=content)
config = types.RecognitionConfig(
encoding=enums.RecognitionConfig.AudioEncoding.LINEAR16,
sample_rate_hertz=16000,
language_code='en-US')
# Detects speech in the audio file
response = client.recognize(config, audio)
for result in response.results:
print('Transcript: {}'.format(result.alternatives[0].transcript))
----Expected to receive a "200OK" and the transcribed text when runing the code above (a demo of short speech to text api from GCP Document)
----But got:
D:\Python\main program\lib\site-packages\google\auth_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/
warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for quota metric 'speech.googleapis.com/default_requests' and limit 'DefaultRequestsPerMinutePerProject' of service 'speech.googleapis.com' for consumer 'project_number:764086051850'.
ANOTHER WEIRD THING: the error info shows that 'project_number:764086051850', which is different from my speech-to-text project_number on GCP (I do distinguish project number and project ID), the project_number shown in the error info also varies every time the code runs. It seems I was sending cloud requirement of the wrong project?
My GOOGLE_APPLICATION_CREDENTIALS system environment variables disappear after I restart my laptop next time. After adding again, it will appear in the env list but can't be stored after reboot again.
Appreciate it if someone can help, thank you!
try to do this:
Run gcloud init -> authenticate with your account and choose your project
Run gcloud auth activate-service-account <service account email> --key-file=<JSON key file>
Run gcloud config list to validate your configuration.
Run your script and see if it's better.
Else, try to do the same thing on a micro-vm for validating your code, service account and environment (and for validating that there is a problem only with Windows)
For Windows issues, I'm on ChromeBook, I can't test and help you on this. However, I checked about EnvVar on internet, and this update the registry. Check if you don't have stuff which protect Registry update (Antivirus,....)
D:\Python\main program\lib\site-packages\google\auth_default.py:66:
UserWarning: Your application has authenticated using end user
credentials from Google Cloud SDK. We recommend that most server
applications use service accounts instead. If your application
continues to use end user credentials from Cloud SDK, you might
receive a "quota exceeded" or "API not enabled" error. For more
information about service accounts, see
https://cloud.google.com/docs/authentication/
warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
This error means that your code is not using a service account. Your code is configured to use ADC (Application Default Credentials). Most likely your code is using the Google Cloud SDK credentials configured and stored by the CLI gcloud.
To determine what credentials the Cloud SDK is using, execute this command:
gcloud auth list
The IAM Member ID, displayed as ACCOUNT, with the asterisk is the account used by the CLI and any applications that do not specify credentials.
To learn more about ADC, read this article that I wrote:
Google Cloud Application Default Credentials
google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for
quota metric 'speech.googleapis.com/default_requests' and limit
'DefaultRequestsPerMinutePerProject' of service
'speech.googleapis.com' for consumer 'project_number:764086051850'.
The Cloud SDK has the concept of default values. Execute gcloud config list. This will display various items. Look for project. Most likely this project does not have the API Cloud Speech-to-Text enabled.
ANOTHER WEIRD THING: the error info shows that
'project_number:764086051850', which is different from my
speech-to-text project_number on GCP (I do distinguish project number
and project ID), the project_number shown in the error info also
varies every time the code runs. It seems I was sending cloud
requirement of the wrong project?
To see the list of projects, Project IDs and Project Numbers that your current credentials can see (access) execute:
gcloud projects list.
This command will display the Project Number given a Project ID:
gcloud projects list --filter="REPLACE_WITH_PROJECT_ID" --format="value(PROJECT_NUMBER)"
My GOOGLE_APPLICATION_CREDENTIALS system environment variables
disappear after I restart my laptop next time. After adding again, it
will appear in the env list but can't be stored after reboot again.
When you execute this command in a Command Prompt, it only persists for the life of the Command Prompt: set GOOGLE_APPLICATION_CREDENTIALS=D:\GCloud speech-to-text\Speech To
Text Series-93e03f36bc9d.json. When you exit the Command Prompt, reboot, etc. the environment variable is destroyed.
To create persistent environment variables on Windows, edit the System Properties -> Environment Variables. You can launch this command as follows from a Command Prompt:
SystemPropertiesAdvanced.exe
Suggestions to make your life easier:
Do NOT use long path names with spaces for your service account files. Create a directory such as C:\Config and place the file there with no spaces in the file name.
Do NOT use ADC (Application Default Credentials) when developing on your desktop. Specify the actual credentials that you want to use.
Change this line:
client = speech.SpeechClient()
To this:
client = speech.SpeechClient().from_service_account_json('c:/config/service-account.json')
Service Accounts have a Project ID inside them. Create the service account in the same project that you intend to use them (until you understand IAM and Service Accounts well).

How do I create HMAC credentials for IBM Cloud Object Storage using the CLI?

I am using the IBM Cloud CLI and tried to generate credentials for my cloud object storage service. However, the following command does not create HMAC credentials needed for using some S3 tools and APIs:
ibmcloud resource service-key-create cos-hmac-cli Writer --instance-name myobjectstorage
How can I create HMAC credentials using the command line interface?
The trick is to provide an additional parameter that tells the service to generate the HMAC part, too:
ibmcloud resource service-key-create cos-hmac-cli Writer \
--instance-name myobjectstorage --parameters '{"HMAC":true}'
The --parameters '{"HMAC":true}' adds the feature request in JSON format.

Resources