i would like to send more expressive log entries to stackdriver logging from my app engine standard python3 app.
By following the official documentation i was able to send my logs to stackdriver and it seems that the timestamp is parsed correctly.
But i'm missing the severity levels. In addition i see no way to link logs for a certain request together to a operation. Something that the java logging seems to be doing out of the box.
For reference here is my code:
import logging
import os
from flask import Flask
from google.cloud import logging as glog
app = Flask(__name__)
log_client = glog.Client(os.getenv('GOOGLE_CLOUD_PROJECT'))
# Attaches a Google Stackdriver logging handler to the root logger
log_client.setup_logging()
#app.route('/_ah/push-handlers/cloudbuild', methods=['POST'])
def pubsub_push_handle():
logging.info("stdlib info")
logging.warning("stdlib warn")
logging.error("stdlib error")
logs resulting in stackdriver:
As you can see the timestamps and message are available while the severity is strangely missing and it gets classified as 'Any'
Can someone point me in the right direction or is this level of incorporation not yet available?
Thanks for your time!
Carsten
You need to create your own logger and add the google-cloud-logging default handler to it:
import logging
from flask import Flask
from google.cloud import logging as cloudlogging
log_client = cloudlogging.Client()
log_handler = log_client.get_default_handler()
cloud_logger = logging.getLogger("cloudLogger")
cloud_logger.setLevel(logging.INFO)
cloud_logger.addHandler(log_handler)
app = Flask(__name__)
#app.route('/_ah/push-handlers/cloudbuild', methods=['POST'])
def pubsub_push_handle():
cloud_logger.info("info")
cloud_logger.warning("warn")
cloud_logger.error("error")
return 'OK'
Produces:
Related
I have some code that is using logger package to log to a file on a Lambda function. The problem is that it also sends everything to CloudWatch, and since there are a lot of logs, is very expensive.
When I was using Python 3.7, this was working and logging only to the file:
import os
import sys
import logging
LOG_LEVEL = os.getenv('LOG_LEVEL', 'INFO')
root_logger = logging.getLogger()
root_logger.disabled = True
# create custom logger
logger = logging.getLogger('my_logger')
logger.removeHandler(sys.stdout)
logger.setLevel(logging.getLevelName(LOG_LEVEL))
for handler in logger.handlers:
logger.removeHandler(handler)
handler = logging.FileHandler('file.log', encoding='utf-8')
handler.setLevel(logging.getLevelName(LOG_LEVEL))
logger.addHandler(handler)
sys.stdout = open(os.devnull, 'w')
run_code_that_logs_stuff()
But after upgrading to Python 3.9, the logs started to show in CloudWatch again.
I have tried changing the last 2 lines with:
with open(os.devnull, 'w') as f, contextlib.redirect_stdout(f):
run_code_that_logs_stuff()
but same result:
START RequestId: 6ede89b6-26b0-4fac-872a-48ecf64c41d1 Version: $LATEST
2022-05-29T05:46:50.949+02:00 [INFO] 2022-05-29T03:46:50.949Z 6ede89b6-26b0-4fac-872a-48ecf64c41d1 Printing stuff I don't want to appear
The easiest way to prevent Lambda from writing to CloudWatch Logs is to remove the IAM policy that allows it to do so.
If you look at your Lambda's configuration, you'll see an execution role. If you look at that role's configuration, you'll see that it references the managed policy AWSLambdaBasicExecutionRole (or, if it's running inside a VPC, AWSLambdaVPCAccessExecutionRole). If you look at those managed policies, you'll see that they grant permissions logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents.
Replace those managed policies with your own managed policies that grant everything except those three permissions.
Or, alternatively, create a managed policy that explicitly denies those three permissions (and nothing else), and attach it to the Lambda's execution role.
Update:
I noted in the comments that you can remove all handlers from the logging module at the start of your Lambda code. However, I think that's an incredibly bad idea, as it means that you will have no visibility into your function if things go wrong.
A much better approach would be to leverage the capabilities of the Python logging framework, and log your output at different levels. At the top of your function, use this code to set the level of the logger:
LOGGER = logging.getLogger(__name__)
LOGGER.setLevel(os.environ.get('LOG_LEVEL', 'WARNING'))
Then, in your code, use the appropriate level when logging:
LOGGER.debug("this message won't normally appear")
LOGGER.warning("this message will", exc_info=True)
This should trim your log messages considerably, while still giving you the ability to see what's happening in your function when things go wrong.
Docs: https://docs.python.org/3/library/logging.html
Update 2:
If you're determined to disable logging entirely (imo a bad idea), then I recommend doing it within the logging framework, rather than hacking streams:
import logging
logging.root.handlers = [logging.NullHandler()]
def lambda_handler(event, context):
logging.warn("this message won't appear")
I am trying to write logs to Logging from Python applications by using Cloud Logging API Cloud client library with "execution ID" that as same as google's default value.
logger setup:
from google.cloud import logging
from google.cloud.logging.resource import Resource
log_client = logging.Client()
# This is the resource type of the log
log_name = 'cloudfunctions.googleapis.com%2Fcloud-functions'
# Inside the resource, nest the required labels specific to the resource type
res = Resource(type="cloud_function",
labels={
"function_name": "my-function",
"region": "asia-east2"
})
logger = log_client.logger(log_name.format("my-project"))
write log:
logger.log_struct({"message": request.remote_addr}, resource=res, severity='INFO')
It's currently not possible to do this using the purely the Cloud Function Framework itself, but you can try to extract the executionId from the request itself by using the following:
request.headers.get('function-execution-id')
I found an issue in Cloud Functions Github tracking the implementation of a native way to get those values, you can follow this thread for updates, if you'd like.
I had the same issue using an older version of google-cloud-logging. I was able to get this functional using the default python logging module. In a cloud function running python 3.8 and google-cloud-logging==2.5.0, the executionId is correctly logged with logs, as well as the severity within stackdriver.
main.py:
# Imports the Cloud Logging client library
import google.cloud.logging
# Instantiates a client
client = google.cloud.logging.Client()
# Retrieves a Cloud Logging handler based on the environment
# you're running in and integrates the handler with the
# Python logging module. By default this captures all logs
# at INFO level and higher
client.get_default_handler()
client.setup_logging()
# Imports Python standard library logging
import logging
def hello_world(req):
# Emits the data using the standard logging module
logging.info('info')
logging.warning('warn')
logging.error('error')
return ""
requirements.txt:
google-cloud-logging==2.5.0
Triggering this cloud function results in the following in stackdriver:
I'm developing a web application in Flask, using GAE.
My issue here is: Every time that my application tries to log, i got multiple entries on log file:
log viewer
.
My dbconnection class only imports a default logger class that i created and calls unexpected_error_log() to write whenever it needed.
My logger class:
import logging
from google.cloud import logging as cloudlogging
class LoggerDB:
def __init__(self):
log_client = cloudlogging.Client()
log_handler = log_client.get_default_handler()
self.cloud_logger = logging.getLogger("cloudLogger")
self.cloud_logger.setLevel(logging.INFO)
self.cloud_logger.addHandler(log_handler)
def unexpected_error_log(self, name, error="Unhandled Exception"):
self.cloud_logger.error("Unexpected Error on %s: %s", name, error)
Code when executed:
def insertVenda(self, venda):
try:
query = "xxxxx"
self.cursor.execute(query)
self.connection.commit()
return "Success"
except Exception as error:
self.logger.unexpected_error_log(__name__, error)
self.connection.rollback()
return "Error"
I suspect that gunicorn/app logging is duplicating my logs, but i don't know how to handle this case.
Did someone had the same problem?
I am struggling with this at the moment, my suspicion at the moment if you include something like this:
# Imports Python standard library logging
import logging
import google.cloud.logging
# Instantiates a client
client = google.cloud.logging.Client()
# Retrieves a Cloud Logging handler based on the environment
# you're running in and integrates the handler with the
# Python logging module. By default this captures all logs
# at INFO level and higher
client.get_default_handler()
client.setup_logging()
I get logs OK, but multiple duplicates.
If I omit I just get single stdout print statements going to the stackdriver logs.
I am trying to use the standard logging module in my AzureML run but nothing gets printed out to stdout.
I tried following instructions from here but I have not been successful.
This is what I am running:
import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
logger.info('WHY IS THIS NOT WORKING?')
Am I doing something wrong?
stdout is generally redirected to a log file called "70_driver_log.txt", and with no configuration a local run submission should honor the basicConfig and you should see the output there.
You can fetch the logs from the UI or by streaming the logs in the SDK via run.wait_for_completion(show_output=True)
This is the link to the UI screenshot (but don't have the rep to post an image): driver log in the ML studio
Following piece of code worked for me. I guess, setting the handler is the key here.
import logging
handler = logging.StreamHandler()
logger = logging.getLogger(__name__)
logger.addHandler(handler)
logger.setLevel(logging.INFO)
More detailed response as well alternative approach about the issue can be found here.
I'm not sure how to say this but, I'm feeling like there is something under the hood that was changed by Google without me knowing about it. I used to get my logs from my python Cloud Functions in the Google Cloud Console within the logging dashboard. And now, it just stopped working.
So I went investigating for a long time, I just made a log hello world python Cloud Function:
import logging
def cf_endpoint(req):
logging.debug('log debug')
logging.info('log info')
logging.warning('log warning')
logging.error('log error')
logging.critical('log critical')
return 'ok'
So this is my main.py that I deploy as a Cloud Function with an http trigger.
Since I was having a log ingestion exclusion filter with all the "debug" level logs I wasn't seeing anything in the logging dashboard. But when I removed it I discovered this :
So it seems like something that was parsing the python built-in log records into stackdriver stopped parsing the log severity parameter! I'm sorry if I look stupid but that's the only thing I can think about :/
Do you guys have any explanations or solutions for this ? am I doing it the wrong way ?
Thank you in advance for your help.
UPDATE 2022/01:
The output now looks for example like:
[INFO]: Connecting to DB ...
And the drop-down menu for the severity looks like:
With "Default" as the filter that is needed to show the Python logging logs, which means to show just any log available, and all of the Python logs are under "Default", the severity is still dropped.
Stackdriver Logging severity filters are no longer supported when using the Python native logging module.
However, you can still create logs with certain severity by using the Stackdriver Logging Client Libraries. Check this documentation in reference to the Python libraries, and this one for some usage-case examples.
Notice that in order to let the logs be under the correct resource, you will have to manually configure them, see this list for the supported resource types.
As well, each resource type has some required labels that need to be present in the log structure.
As an example, the following code will write a log to the Cloud Function resource, in Stackdriver Logging, with an ERROR severity:
from google.cloud import logging
from google.cloud.logging.resource import Resource
log_client = logging.Client()
# This is the resource type of the log
log_name = 'cloudfunctions.googleapis.com%2Fcloud-functions'
# Inside the resource, nest the required labels specific to the resource type
res = Resource(type="cloud_function",
labels={
"function_name": "YOUR-CLOUD-FUNCTION-NAME",
"region": "YOUR-FUNCTION-LOCATION"
},
)
logger = log_client.logger(log_name.format("YOUR-PROJECT-ID"))
logger.log_struct(
{"message": "message string to log"}, resource=res, severity='ERROR')
return 'Wrote logs to {}.'.format(logger.name) # Return cloud function response
Notice that the strings in YOUR-CLOUD-FUNCTION-NAME, YOUR-FUNCTION-LOCATION and YOUR-PROJECT-ID, need to be specific to your project/resource.
I encountered the same issue.
In the link that #joan Grau shared, I also see there is a way to integrate cloud logger with Python logging module, so that you could use Python root logger as usually, and all logs will be sent to StackDriver Logging.
https://googleapis.github.io/google-cloud-python/latest/logging/usage.html#integration-with-python-logging-module
...
I tried it and it works. In short, you could do it two ways
One simple way that bind cloud logger to root logging
from google.cloud import logging as cloudlogging
import logging
lg_client = cloudlogging.Client()
lg_client.setup_logging(log_level=logging.INFO) # to attach the handler to the root Python logger, so that for example a plain logging.warn call would be sent to Stackdriver Logging, as well as any other loggers created.
Alternatively, you could set logger with more fine-grain control
from google.cloud import logging as cloudlogging
import logging
lg_client = cloudlogging.Client()
lg_handler = lg_client.get_default_handler()
cloud_logger = logging.getLogger("cloudLogger")
cloud_logger.setLevel(logging.INFO)
cloud_logger.addHandler(lg_handler)
cloud_logger.info("test out logger carrying normal news")
cloud_logger.error("test out logger carrying bad news")
Not wanting to deal with cloud logging libraries, I created a custom Formatter that emits a structured log with the right fields, as cloud logging expects it.
class CloudLoggingFormatter(logging.Formatter):
"""Produces messages compatible with google cloud logging"""
def format(self, record: logging.LogRecord) -> str:
s = super().format(record)
return json.dumps(
{
"message": s,
"severity": record.levelname,
"timestamp": {"seconds": int(record.created), "nanos": 0},
}
)
Attaching this handler to a logger results in logs being parsed and shown properly in the logging console. In cloud functions I would configure the root logger to send to stdout and attach the formatter to it.
# setup logging
root = logging.getLogger()
handler = logging.StreamHandler(sys.stdout)
formatter = CloudLoggingFormatter(fmt="[%(name)s] %(message)s")
handler.setFormatter(formatter)
root.addHandler(handler)
root.setLevel(logging.DEBUG)
From Python 3.8 onwards you can simply print a JSON structure with severity and message properties. For example:
print(
json.dumps(
dict(
severity="ERROR",
message="This is an error message",
custom_property="I will appear inside the log's jsonPayload field",
)
)
)
Official documentation: https://cloud.google.com/functions/docs/monitoring/logging#writing_structured_logs
To use the standard python logging module on GCP (tested on python 3.9), you can do the following:
import google.cloud.logging
logging_client = google.cloud.logging.Client()
logging_client.setup_logging()
import logging
logging.warning("A warning")
See also: https://cloud.google.com/logging/docs/setup/python
I use a very simple custom logging function to log to Cloud Logging:
import json
def cloud_logging(severity, message):
print(json.dumps({"severity": severity, "message": message}))
cloud_logging(severity="INFO", message="Your logging message")