How to use google-api-client for Google Cloud Logging - python-3.x

Question

I want to access to Google Cloud Platform Logging from a python script.
I have get to access to this logs from https://cloud.google.com/logging/docs/reference/v2/rest/v2/entries/list --> Try this API
Now I want to get the same, but from a Python script. I saw that in step before, is created an authorization token automatically.
I am trying with this code sample, but then I don't know how to POST https://logging.googleapis.com/v2/entries:list using discovery:
from google.oauth2 import service_account
import googleapiclient.discovery
credentials = service_account.Credentials.from_service_account_file(service_account_file)
logging = googleapiclient.discovery.build('logging', 'v2', credentials=credentials)
Then I have tried with this code sample:
import requests
payload = {
"projectIds": [
"my-proyect"
],
"resourceNames": [],
"filter": "resource.type=cloudiot_device",
"orderBy": "timestamp desc",
"pageSize": 1
}
headers = {"Authorization": "Bearer AAAAAAA"}
r = requests.post("https://logging.googleapis.com/v2/entries:list", params=payload, headers=headers)
That code sample works correctly but where it puts AAAAAAA token I copy and paste the code that I saw in https://cloud.google.com/logging/docs/reference/v2/rest/v2/entries/list but I don't know how to generate this token from a python script.
Thanks!

Answer 1

This is less easy to find because many of Google's Cloud (!) services now prefer Cloud Client libraries.
However...
import google.auth
from googleapiclient import discovery
credentials, project = google.auth.default()
service = discovery.build("logging", "v2", credentials=credentials)
Auth: https://pypi.org/project/google-auth/
Now, this uses Google Application Default credentials and I recommend you create a service account, generate a key and grant the account the permission needed. You will then need to export GOOGLE_APPLICATION_CREDENTIALS before running your code.
PROJECT=[[YOUR-PROJECT]]
BILLING=[[YOUR-BILLING]]
ACCOUNT=[[YOUR-ACCOUNT]]
gcloud projects create ${PROJECT}
gcloud beta billing projects link ${PROJECT} \
--billing-account=${BILLING}
gcloud iam service-accounts create ${ACCOUNT} \
--project=${PROJECT}
EMAIL="${ACCOUNT}#${PROJECT}.iam.gserviceaccount.com"
gcloud iam service-accounts keys create ${PWD}/${ACCOUNT}.json \
--iam-account=${EMAIL} \
--project=${PROJECT}
# See: https://cloud.google.com/iam/docs/understanding-roles#logging-roles
gcloud projects add-iam-policy-binding ${PROJECT} \
--member=serviceAccount:${EMAIL} \
--role=roles/logging.viewer
export GOOGLE_APPLICATION_CREDENTIALS=${PWD}/${ACCOUNT}.json
python3 your-code.py

Answer 2

Ok, thanks to the Google Engineer, the first part of the solution is to disable the SDK's use of gRPC and force HTTP so that page_size is respected:
client = logging.Client(_use_grpc=0)
Alternatively, you can GOOGLE_CLOUD_DISABLE_GRPC="{{anything}}"
And the second part of the solution is to only iterate over the first page of page_size results:
iterator = logger.list_entries(
order_by=DESCENDING,
page_size=page_size,
)
print(type(iterator))
for entry in next(iterator.pages):
timestamp = entry.timestamp.isoformat()
print("{}".format(timestamp))
NOTE forcing HTTP entails logger.list_entries returning an HTTPIterator instead of a (gRPC) generator hence the ability to use next() and the pages property.
NOTE The 'trick' is to only enumerate the first page of n results. There may be multiple pages but we ignore subsequent ones.

Answer 3

I am using the following code sample to extract log information from Google Cloud Logging.
import os
from google.cloud import logging
from google.cloud.logging import DESCENDING
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "my-service-account-file"
def list_entries(logger_name):
"""Lists the most recent entries for a given logger."""
logging_client = logging.Client()
logger = logging_client.logger(logger_name)
print("Listing entries for logger {}:".format(logger.name))
filter_str = "resource.type=cloudiot_device AND resource.labels.device_num_id=00000000000 AND jsonPayload.eventType=PUBLISH"
for entry in logger.list_entries(filter_=filter_str, order_by=DESCENDING, page_size=10):
timestamp = entry.timestamp.isoformat()
print(" {}: {}".format(, timestamp, entry.payload))
list_entries("cloudiot.googleapis.com%2Fdevice_activity")
My goal is to run this python script every 5 minutes and get the last 5 entries from the Logging. My problem is that this code sample start extracting entries, but it never stoppes. How can I limit the number of entries?
Thanks!