Authenticate calls to Google Cloud Functions programmatically - groovy

I am trying to authenticate to Google Cloud Functions from SAP CPI to fetch some data from a database. To push data, we use pub/sub, with a service account access token, and it works perfectly. But for the functions, it needs an identity token instead of an access token. We get the previous token with a groovy script (No Jenkins). Is it possible to authenticate to the functions also with an access token? Or to get the identity token without building a whole IAP layer?

You have to call your Cloud Functions (or Cloud Run, it's the same) with a signed identity token.
So you can use a groovy script for generating a signed identity token. Here an example
import com.google.api.client.http.GenericUrl
import com.google.api.client.http.HttpRequest
import com.google.api.client.http.HttpRequestFactory
import com.google.api.client.http.HttpResponse
import com.google.api.client.http.javanet.NetHttpTransport
import com.google.auth.Credentials
import com.google.auth.http.HttpCredentialsAdapter
import com.google.auth.oauth2.IdTokenCredentials
import com.google.auth.oauth2.IdTokenProvider
import com.google.auth.oauth2.ServiceAccountCredentials
import com.google.common.base.Charsets
import com.google.common.io.CharStreams
String myUri = "YOUR_URL";
Credentials credentials = ServiceAccountCredentials
.fromStream(new FileInputStream(new File("YOUR_SERVICE_ACCOUNT_KEY_FILE"))).createScoped("https://www.googleapis.com/auth/cloud-platform");
String token = ((IdTokenProvider) credentials).idTokenWithAudience(myUri, Collections.EMPTY_LIST).getTokenValue();
System.out.println(token);
IdTokenCredentials idTokenCredentials = IdTokenCredentials.newBuilder()
.setIdTokenProvider((ServiceAccountCredentials) credentials)
.setTargetAudience(myUri).build();
HttpRequestFactory factory = new NetHttpTransport().createRequestFactory(new HttpCredentialsAdapter(idTokenCredentials));
HttpRequest request = factory.buildGetRequest(new GenericUrl(myUri));
HttpResponse httpResponse = request.execute();
System.out.println(CharStreams.toString(new InputStreamReader(httpResponse.getContent(), Charsets.UTF_8)));
Service account key file is required only if you are outside GCP. Else, the default service account is enough, but must be a service account. Your personal user account won't work
Add this dependency (here in Maven)
<dependency>
<groupId>com.google.auth</groupId>
<artifactId>google-auth-library-oauth2-http</artifactId>
<version>0.20.0</version>
</dependency>
Or you can use a tool that I wrote and open sourced. I also wrote a Medium article for explaining the use cases

You can only access your secured cloud function using Identity token.
1.Create a service account with roles/cloudfunctions.invoker
2.Create a cloud function that allows only authenticated requests
https://REGION-PROJECT_ID.cloudfunctions.net/FUNCTION_NAME
from google.oauth2 import service_account
from google.auth.transport.requests import AuthorizedSession
target_audience = 'https://REGION-PROJECT_ID.cloudfunctions.net/FUNCTION_NAME'
creds = service_account.IDTokenCredentials.from_service_account_file(
'/path/to/svc.json', target_audience=target_audience)
authed_session = AuthorizedSession(creds)
# make authenticated request and print the response, status_code
resp = authed_session.get(target_audience)
print(resp.status_code)
print(resp.text)

Related

I can't properly authorize my Python web app to access Google API

I'm using google api to integrate some google sheet data into my app. I have a server where I can only connect via SSH, as it has no user interface. When I start my app, it gives me this message:
Please visit this URL to authorize this application: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=XXX-XXX.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A57731%2F&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fspreadsheets.readonly&state=XXX&access_type=offline
Since I don't have a UI on my server, how can I authorize the application? I've just started using the google api, I still can't figure out how to go about it.
The only thing I've tried and that works is to run the code from a windows pc for example and then transfer the token.json file to my server. The problem is that after 30 days it expires and I have to redo the whole procedure again.
Is there something I'm doing wrong? Here is my code:
from __future__ import print_function
from google.auth.transport.requests import Request
from google.oauth2.credentials import Credentials
from google_auth_oauthlib.flow import InstalledAppFlow
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError
import os.path
# If modifying these scopes, delete the file token.json.
SCOPES = ['https://www.googleapis.com/auth/spreadsheets.readonly']
# The ID of a sample spreadsheet.
SPREADSHEET_ID = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
# Google API credentials
googleCredentials = None
def main():
print("Loadin google API...")
global googleCredentials
if os.path.exists("./json/token.json"):
googleCredentials = Credentials.from_authorized_user_file("./json/token.json", SCOPES)
# If there are no (valid) credentials available, let the user log in.
if not googleCredentials or not googleCredentials.valid:
if googleCredentials and googleCredentials.expired and googleCredentials.refresh_token:
googleCredentials.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file("./json/credentials.json", SCOPES)
googleCredentials = flow.run_local_server(port=0)
# Save the credentials for the next run
with open("./json/token.json", 'w') as token:
token.write(googleCredentials.to_json())
if __name__ == "__main__":
main()
The only thing I've tried and that works is to run the code from a windows pc for example and then transfer the token.json file to my server. The problem is that after 30 days it expires and I have to redo the whole procedure again.
It sounds like you have done exactly what you should do. The token.json file should be just fine stored on your machine. The question is why is your refresh token expireing.
If your app is still in the testing phase the refresh token will expire after seven days. The solution for that is to go to google cloud console for your app then to the Oauth2 consent screen and set it to production the next time you create a refresh token it will not expire.
I'm a bit curious as to why you say this lasts 30 days which me think you might be getting a new refresh token back every time it refreshes your access token and its not being saved but that's not supposed to happen with the python library its supposed to handle that I will have to test it.
If you are only connecting to this single sheet, Have you considered using a service account instead?
service account
Just make sure to share the sheet with the service accounts email address, just like you would any other user via the google drive web app. It will have access to your sheet without any user intervension.
credentials = ServiceAccountCredentials.from_json_keyfile_name(
KEYFILE,
scopes=SCOPES,
)
How to create service account credentials
Should you be using a Google service account
Token.json explained
Below is the contense of a token.json file, this is exactly what an authorization server returns to you as the final step in the oauth flow.
The first on Token is actually your access token, the expire says when that access token will expire. When your code loads it will check if the access token in the file has expired if it has it will request a new one using the refresh token. look for: googleCredentials.expired
{
"token": "[Redacted Access Token]",
"refresh_token": "[Redacted refresh Token]",
"token_uri": "https://oauth2.googleapis.com/token",
"client_id": "[REDACTED]",
"client_secret": "[REDACTED]",
"scopes": [
"https://www.googleapis.com/auth/calendar.readonly"
],
"expiry": "2022-10-25T12:06:40.318779Z"
}

Connect to GCP SQL using the .json credentials file

I have a PostgreSQL DB at GCP. Right now I can login using a username, password e.g
import pandas as pd
import pyodbc
conn_str = (
"DRIVER={PostgreSQL Unicode};"
"DATABASE=test;"
"UID=user;"
"PWD=a_very_strong_password;"
"SERVER=34.76.yy.xxxx;"
"PORT=5432;"
)
with pyodbc.connect(conn_str) as con:
print(pd.read_sql("SELECT * from entries",con=con))
Is there a way to use the .json credentialsfile which is downloaded when I created my IAM user, instead of "hard typing" the credentials like above? I recon I can use the file to connect to a GCP storage, where I then can save my credentials for the DB thus I can write a script which loads the username,password etc. from the storage, but I feel it is a kinda "clunky" workaround.
From the guide here it seems like you can create IAM roles for such, but you only grants access for an hour at a time, and you need to create a token-pair each time.
Short answer: Yes, you can connect to a Cloud SQL instance using SA keys (json file) but only with PostgreSQL but you need to refresh the token every hour.
Long answer: The purpouse of the json is more intended to make operations in the instance at resource level or when using the Cloud SQL proxy.
For example when you use the Cloud SQL proxy with a service account you make a "magical bridge" to the instance but at the end you need to authenticate the way you're doing right now but using as SERVER = 127.0.0.1. This is the recommended method in most cases.
As well you've mentioned that using IAM authentication can work, this approach works for 1 hour since you depend on token refresh. If you're okay with this, just keep in mind you need to be refreshing the token.
Another approach I can think of for now is to use Secret Manager. The steps can be as follows:
Create a service account and a key for that.
Create a secret which contains your password.
Grant access to this particular secret to the SA created in step 1:
Go to Secret Manager.
Select the secret and click on Show info panel
Click on Add member and type or paste the email of the SA
Grant Secret Manager Secret Accessor
Click on Save
Now in your code you can get the secret content (which is the password) with maybe this sample code:
import pandas as pd
import pyodbc
from google.cloud import secretmanager
from google.oauth2 import service_account
credentials = service_account.Credentials.from_service_account_file('/path/to/key.json')
client = secretmanager.SecretManagerServiceClient(credentials=credentials)
name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"
response = client.access_secret_version(request={"name": name})
secret_password = response.payload.data.decode("UTF-8")
conn_str = (
"DRIVER={PostgreSQL Unicode};"
"DATABASE=test;"
"UID=user;"
"PWD=" + secret_password + ";"
"SERVER=34.76.yy.xxxx;"
"PORT=5432;"
)
with pyodbc.connect(conn_str) as con:
print(pd.read_sql("SELECT * from entries",con=con))
BTW, you can install the lib using pip install google-cloud-secret-manager.
Finally, you can also use this approach to keep the instance IP, user, DB name, etc. creating more secrets if you prefer

Generating Cloud Storage Signed URL from Google Cloud Function without using explicit key file

I'd like to create a pre-signed upload URL to a storage bucket, and would like to avoid an explicit reference to a json key.
Currently, I'm attempting to do this with the Default App Engine Service Account
I'm attempting to follow along with this answer but am getting this error:
AttributeError: you need a private key to sign credentials.the
credentials you are currently using <class
'google.auth.compute_engine.credentials.Credentials'> just contains a
token. see
https://googleapis.dev/python/google-api-core/latest/auth.html#setting-up-a-service-account
for more details.
My Cloud Function code looks like this:
from google.cloud import storage
import datetime
import google.auth
def generate_upload_url(blob_name, additional_metadata: dict = {}):
credentials, project_id = google.auth.default()
# Perform a refresh request to get the access token of the current credentials (Else, it's None)
from google.auth.transport import requests
r = requests.Request()
credentials.refresh(r)
client = storage.Client()
bucket = client.get_bucket("my_bucket")
blob = bucket.blob(blob_name)
service_account_email = credentials.service_account_email
print(f"attempting to create signed url for {service_account_email}")
url = blob.generate_signed_url(
version="v4",
service_account_email=service_account_email,
access_token=credentials.token,
# This URL is valid for 120 minutes
expiration=datetime.timedelta(minutes=120),
# Allow PUT requests using this URL.
method="PUT",
content_type="application/octet-stream",
)
return url
def get_upload_url(request):
blob_name = get_param(request, "blob_name")
url = generate_upload_url(blob_name)
return url
When you use version v4 of signed URL, the first line of the method calls ensure_signed_credentialsmethod that check if the current service account can generate a signature in standalone mode (so with a private key). And so, that's break the current behavior.
In the comment of the function, it's clearly describe that a service account JSON file is required
If you are on Google Compute Engine, you can't generate a signed URL.
Follow `Issue 922`_ for updates on this. If you'd like to be able to
generate a signed URL from GCE, you can use a standard service account
from a JSON file rather than a GCE service account.
So, use v2 version instead.

Google Drive API Python Service Account Example

Is it possible to authenticate against the Google Drive API using a Google Service Account, rather than an OAuth flow ?
The Python examples for Google Drive use OAuth - Google drive Python Quick start
However I can't find any Service Account examples.
The majority of the other Google APIs I use (Translate, Cloud Vision) do use Service Account however, so I'd like to deprecate my Google Drive OAuth code for consistency.
The best service account python example that i know of is the one for Google analytics
It should be something like this.
from apiclient.discovery import build
from oauth2client.service_account import ServiceAccountCredentials
SCOPES = ['https://www.googleapis.com/auth/drive.readonly']
KEY_FILE_LOCATION = '<REPLACE_WITH_JSON_FILE>'
VIEW_ID = '<REPLACE_WITH_VIEW_ID>'
def initialize_drive():
"""Initializes an service object.
Returns:
An authorized service object.
"""
creds = ServiceAccountCredentials.from_json_keyfile_name(
KEY_FILE_LOCATION, SCOPES)
# Build the service object.
service = build('drive', 'v3', credentials=creds)
return service
Once you have the drive service you should be able to use the rest of the code you have from the other tutorial. Its just the auth method that is diffrent.
Just remember to create service account credentials and not Oauth credentials.
You can use Credentials oauth2 object using the following,
import json
from googleapiclient.discovery import build
from google.oauth2.service_account import Credentials
SCOPES = ['https://www.googleapis.com/auth/drive']
service_account_info = json.load(open('service_account.json'))
creds=Credentials.from_service_account_info(
service_account_info, scopes=SCOPES)
drive_service = build('drive', 'v3', credentials=creds)

Youtube Data API v.3 - fully automated oAuth flow (Python)?

I have been exploring the YouTube Data API. The premise of my project is simple: using the API, authenticate (yes, I have the credentials for the account) and then simply retrieve the list of all my videos, public and private.
I have been able to accomplish this successfully, except for the fully automated part. I have used code from various sources and when I run it on the command line, it provides me a link to be used in a browser so that the authorization takes place.
It looks something like this:
Please visit this URL to authorize this application: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=7932902759886-cb8ai84grcqshe24nn459ka46uh45ssj.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fyoutube.readonly&state=zNVvgEyO47nmacvdEEAhDsQipY194k&prompt=consent&access_type=offline&code_challenge=aF7uTCghjwgwjg49o3fgiIU-_ryK19rDeX4l1uzr37w&code_challenge_method=S256
Enter the authorization code:
....
Here's a snippet of my python code:
import google_auth_oauthlib.flow
import googleapiclient.discovery
import googleapiclient.errors
...
...
# Get credentials and create an API client
flow = google_auth_oauthlib.flow.InstalledAppFlow.from_client_secrets_file(
client_secrets_file, scopes)
credentials = flow.run_console()
youtube = googleapiclient.discovery.build(
api_service_name, api_version, credentials=credentials)
## MAKE youtube SEARCH REQUEST
last_date = '2018-10-01T00:00:00Z'
request = youtube.search().list(
part="snippet",
forMine=True,
maxResults=50,
order="date",
type="video"
)
all_items = []
response = request.execute()
My question here is the following: Is it possible to programatically perform the authorization so that the app can run standalone and not have to wait for this user action (to literally copy the URL from CMD, visit to get the token, and the copy and paste the token again)? I'd like to schedule this and therefore would like it to run and authenticate without human intervention. Is this possible at all? If so, can someone please point me to some working examples and/or other resources to help me get there? Thanks a million.
# -*- coding: utf-8 -*-
# Sample Python code for youtube.channels.list
# See instructions for running these code samples locally:
# https://developers.google.com/explorer-help/guides/code_samples#python
#!/usr/bin/python3.7
import os
import pickle
import google_auth_oauthlib.flow
import googleapiclient.discovery
import googleapiclient.errors
scopes = ["https://www.googleapis.com/auth/youtube.readonly"]
client_secrets_file = "client_secret.json"
api_service_name = "youtube"
api_version = "v3"
def main():
# Disable OAuthlib's HTTPS verification when running locally.
# *DO NOT* leave this option enabled in production.
os.environ["OAUTHLIB_INSECURE_TRANSPORT"] = "1"
# Get credentials and create an API client
flow = google_auth_oauthlib.flow.InstalledAppFlow.from_client_secrets_file(
client_secrets_file, scopes)
youtube = get_authenticated_service()
request = youtube.channels().list(
part="contentDetails",
mine=True
)
response = request.execute()
print(response)
def get_authenticated_service():
if os.path.exists("CREDENTIALS_PICKLE_FILE"):
with open("CREDENTIALS_PICKLE_FILE", 'rb') as f:
credentials = pickle.load(f)
else:
flow = google_auth_oauthlib.flow.InstalledAppFlow.from_client_secrets_file(client_secrets_file, scopes)
credentials = flow.run_console()
with open("CREDENTIALS_PICKLE_FILE", 'wb') as f:
pickle.dump(credentials, f)
return googleapiclient.discovery.build(
api_service_name, api_version, credentials=credentials)
if __name__ == "__main__":
main()
The Credentials instance from credentials = flow.run_console() has a built-in functionality to refresh token.
It'll will refresh the token when a request being execute if needed.
Therefore you can save the credentials object into pickle, and read it back when need it
A few alteration on Google python sample code:
def get_authenticated_service():
if os.path.exists(CREDENTIALS_PICKLE_FILE):
with open(CREDENTIALS_PICKLE_FILE, 'rb') as f:
credentials = pickle.load(f)
else:
flow = InstalledAppFlow.from_client_secrets_file(CLIENT_SECRETS_FILE, SCOPES)
credentials = flow.run_console()
with open(CREDENTIALS_PICKLE_FILE, 'wb') as f:
pickle.dump(credentials, f)
return build(API_SERVICE_NAME, API_VERSION, credentials = credentials)
copied from https://developers.google.com/identity/protocols/OAuth2InstalledApp
Step 3: Google prompts user for consent
In this step, the user decides whether to grant your application the requested access. At this stage, Google displays a consent window that shows the name of your application and the Google API services that it is requesting permission to access with the user's authorization credentials. The user can then consent or refuse to grant access to your application.
Your application doesn't need to do anything at this stage as it waits for the response from Google's OAuth 2.0 server indicating whether the access was granted. That response is explained in the following step.
Where this is important:
At this stage, Google displays a consent window that shows the name of your application and the Google API services that it is requesting permission to access with the user's authorization credentials.
So, at least as I interpret it, what you want to do should not be done for security reasons.
However: you can "simulate" a browser by how ever python have libs for do such. On the other hand: Once you got the auth-token you can re-use it instead of request a new token each time. I couldn't find it in provided doc on GitHub, but Java as example supports to store an obtained token along with its refresh token so it can be reused once obtained and auto-refreshed. Maybe python provides some way to store the obtained token (check if it contains a refresh token) and re-load it. Also: if you load such token, first you have to do is to refresh it before using it. Java provieds a way to just save a refresh token instead of the whole auth-token wich can be used in a later run to automatic obtain a new auth-token.
As response is a JSON maybe you could build some yourself if the lib doesn't already offer this.
// edit
In addition from https://github.com/googleapis/google-auth-library-python/blob/master/google/oauth2/credentials.py
There are methods to load a credential object either from an "authorized user info" (wich I also somewhere found can be loaded from file) or to load it directly from file. So, I guess you just have to figure out how to store the token. As doc says for from_authorized_user_file:
Creates a Credentials instance from an authorized user json file.
I guess that means you just have to save the token response you get after the initial authorization was done.

Resources