I am trying to use the standard logging module in my AzureML run but nothing gets printed out to stdout.
I tried following instructions from here but I have not been successful.
This is what I am running:
import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
logger.info('WHY IS THIS NOT WORKING?')
Am I doing something wrong?
stdout is generally redirected to a log file called "70_driver_log.txt", and with no configuration a local run submission should honor the basicConfig and you should see the output there.
You can fetch the logs from the UI or by streaming the logs in the SDK via run.wait_for_completion(show_output=True)
This is the link to the UI screenshot (but don't have the rep to post an image): driver log in the ML studio
Following piece of code worked for me. I guess, setting the handler is the key here.
import logging
handler = logging.StreamHandler()
logger = logging.getLogger(__name__)
logger.addHandler(handler)
logger.setLevel(logging.INFO)
More detailed response as well alternative approach about the issue can be found here.
Related
I have some code that is using logger package to log to a file on a Lambda function. The problem is that it also sends everything to CloudWatch, and since there are a lot of logs, is very expensive.
When I was using Python 3.7, this was working and logging only to the file:
import os
import sys
import logging
LOG_LEVEL = os.getenv('LOG_LEVEL', 'INFO')
root_logger = logging.getLogger()
root_logger.disabled = True
# create custom logger
logger = logging.getLogger('my_logger')
logger.removeHandler(sys.stdout)
logger.setLevel(logging.getLevelName(LOG_LEVEL))
for handler in logger.handlers:
logger.removeHandler(handler)
handler = logging.FileHandler('file.log', encoding='utf-8')
handler.setLevel(logging.getLevelName(LOG_LEVEL))
logger.addHandler(handler)
sys.stdout = open(os.devnull, 'w')
run_code_that_logs_stuff()
But after upgrading to Python 3.9, the logs started to show in CloudWatch again.
I have tried changing the last 2 lines with:
with open(os.devnull, 'w') as f, contextlib.redirect_stdout(f):
run_code_that_logs_stuff()
but same result:
START RequestId: 6ede89b6-26b0-4fac-872a-48ecf64c41d1 Version: $LATEST
2022-05-29T05:46:50.949+02:00 [INFO] 2022-05-29T03:46:50.949Z 6ede89b6-26b0-4fac-872a-48ecf64c41d1 Printing stuff I don't want to appear
The easiest way to prevent Lambda from writing to CloudWatch Logs is to remove the IAM policy that allows it to do so.
If you look at your Lambda's configuration, you'll see an execution role. If you look at that role's configuration, you'll see that it references the managed policy AWSLambdaBasicExecutionRole (or, if it's running inside a VPC, AWSLambdaVPCAccessExecutionRole). If you look at those managed policies, you'll see that they grant permissions logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents.
Replace those managed policies with your own managed policies that grant everything except those three permissions.
Or, alternatively, create a managed policy that explicitly denies those three permissions (and nothing else), and attach it to the Lambda's execution role.
Update:
I noted in the comments that you can remove all handlers from the logging module at the start of your Lambda code. However, I think that's an incredibly bad idea, as it means that you will have no visibility into your function if things go wrong.
A much better approach would be to leverage the capabilities of the Python logging framework, and log your output at different levels. At the top of your function, use this code to set the level of the logger:
LOGGER = logging.getLogger(__name__)
LOGGER.setLevel(os.environ.get('LOG_LEVEL', 'WARNING'))
Then, in your code, use the appropriate level when logging:
LOGGER.debug("this message won't normally appear")
LOGGER.warning("this message will", exc_info=True)
This should trim your log messages considerably, while still giving you the ability to see what's happening in your function when things go wrong.
Docs: https://docs.python.org/3/library/logging.html
Update 2:
If you're determined to disable logging entirely (imo a bad idea), then I recommend doing it within the logging framework, rather than hacking streams:
import logging
logging.root.handlers = [logging.NullHandler()]
def lambda_handler(event, context):
logging.warn("this message won't appear")
i would like to send more expressive log entries to stackdriver logging from my app engine standard python3 app.
By following the official documentation i was able to send my logs to stackdriver and it seems that the timestamp is parsed correctly.
But i'm missing the severity levels. In addition i see no way to link logs for a certain request together to a operation. Something that the java logging seems to be doing out of the box.
For reference here is my code:
import logging
import os
from flask import Flask
from google.cloud import logging as glog
app = Flask(__name__)
log_client = glog.Client(os.getenv('GOOGLE_CLOUD_PROJECT'))
# Attaches a Google Stackdriver logging handler to the root logger
log_client.setup_logging()
#app.route('/_ah/push-handlers/cloudbuild', methods=['POST'])
def pubsub_push_handle():
logging.info("stdlib info")
logging.warning("stdlib warn")
logging.error("stdlib error")
logs resulting in stackdriver:
As you can see the timestamps and message are available while the severity is strangely missing and it gets classified as 'Any'
Can someone point me in the right direction or is this level of incorporation not yet available?
Thanks for your time!
Carsten
You need to create your own logger and add the google-cloud-logging default handler to it:
import logging
from flask import Flask
from google.cloud import logging as cloudlogging
log_client = cloudlogging.Client()
log_handler = log_client.get_default_handler()
cloud_logger = logging.getLogger("cloudLogger")
cloud_logger.setLevel(logging.INFO)
cloud_logger.addHandler(log_handler)
app = Flask(__name__)
#app.route('/_ah/push-handlers/cloudbuild', methods=['POST'])
def pubsub_push_handle():
cloud_logger.info("info")
cloud_logger.warning("warn")
cloud_logger.error("error")
return 'OK'
Produces:
I'm not sure how to say this but, I'm feeling like there is something under the hood that was changed by Google without me knowing about it. I used to get my logs from my python Cloud Functions in the Google Cloud Console within the logging dashboard. And now, it just stopped working.
So I went investigating for a long time, I just made a log hello world python Cloud Function:
import logging
def cf_endpoint(req):
logging.debug('log debug')
logging.info('log info')
logging.warning('log warning')
logging.error('log error')
logging.critical('log critical')
return 'ok'
So this is my main.py that I deploy as a Cloud Function with an http trigger.
Since I was having a log ingestion exclusion filter with all the "debug" level logs I wasn't seeing anything in the logging dashboard. But when I removed it I discovered this :
So it seems like something that was parsing the python built-in log records into stackdriver stopped parsing the log severity parameter! I'm sorry if I look stupid but that's the only thing I can think about :/
Do you guys have any explanations or solutions for this ? am I doing it the wrong way ?
Thank you in advance for your help.
UPDATE 2022/01:
The output now looks for example like:
[INFO]: Connecting to DB ...
And the drop-down menu for the severity looks like:
With "Default" as the filter that is needed to show the Python logging logs, which means to show just any log available, and all of the Python logs are under "Default", the severity is still dropped.
Stackdriver Logging severity filters are no longer supported when using the Python native logging module.
However, you can still create logs with certain severity by using the Stackdriver Logging Client Libraries. Check this documentation in reference to the Python libraries, and this one for some usage-case examples.
Notice that in order to let the logs be under the correct resource, you will have to manually configure them, see this list for the supported resource types.
As well, each resource type has some required labels that need to be present in the log structure.
As an example, the following code will write a log to the Cloud Function resource, in Stackdriver Logging, with an ERROR severity:
from google.cloud import logging
from google.cloud.logging.resource import Resource
log_client = logging.Client()
# This is the resource type of the log
log_name = 'cloudfunctions.googleapis.com%2Fcloud-functions'
# Inside the resource, nest the required labels specific to the resource type
res = Resource(type="cloud_function",
labels={
"function_name": "YOUR-CLOUD-FUNCTION-NAME",
"region": "YOUR-FUNCTION-LOCATION"
},
)
logger = log_client.logger(log_name.format("YOUR-PROJECT-ID"))
logger.log_struct(
{"message": "message string to log"}, resource=res, severity='ERROR')
return 'Wrote logs to {}.'.format(logger.name) # Return cloud function response
Notice that the strings in YOUR-CLOUD-FUNCTION-NAME, YOUR-FUNCTION-LOCATION and YOUR-PROJECT-ID, need to be specific to your project/resource.
I encountered the same issue.
In the link that #joan Grau shared, I also see there is a way to integrate cloud logger with Python logging module, so that you could use Python root logger as usually, and all logs will be sent to StackDriver Logging.
https://googleapis.github.io/google-cloud-python/latest/logging/usage.html#integration-with-python-logging-module
...
I tried it and it works. In short, you could do it two ways
One simple way that bind cloud logger to root logging
from google.cloud import logging as cloudlogging
import logging
lg_client = cloudlogging.Client()
lg_client.setup_logging(log_level=logging.INFO) # to attach the handler to the root Python logger, so that for example a plain logging.warn call would be sent to Stackdriver Logging, as well as any other loggers created.
Alternatively, you could set logger with more fine-grain control
from google.cloud import logging as cloudlogging
import logging
lg_client = cloudlogging.Client()
lg_handler = lg_client.get_default_handler()
cloud_logger = logging.getLogger("cloudLogger")
cloud_logger.setLevel(logging.INFO)
cloud_logger.addHandler(lg_handler)
cloud_logger.info("test out logger carrying normal news")
cloud_logger.error("test out logger carrying bad news")
Not wanting to deal with cloud logging libraries, I created a custom Formatter that emits a structured log with the right fields, as cloud logging expects it.
class CloudLoggingFormatter(logging.Formatter):
"""Produces messages compatible with google cloud logging"""
def format(self, record: logging.LogRecord) -> str:
s = super().format(record)
return json.dumps(
{
"message": s,
"severity": record.levelname,
"timestamp": {"seconds": int(record.created), "nanos": 0},
}
)
Attaching this handler to a logger results in logs being parsed and shown properly in the logging console. In cloud functions I would configure the root logger to send to stdout and attach the formatter to it.
# setup logging
root = logging.getLogger()
handler = logging.StreamHandler(sys.stdout)
formatter = CloudLoggingFormatter(fmt="[%(name)s] %(message)s")
handler.setFormatter(formatter)
root.addHandler(handler)
root.setLevel(logging.DEBUG)
From Python 3.8 onwards you can simply print a JSON structure with severity and message properties. For example:
print(
json.dumps(
dict(
severity="ERROR",
message="This is an error message",
custom_property="I will appear inside the log's jsonPayload field",
)
)
)
Official documentation: https://cloud.google.com/functions/docs/monitoring/logging#writing_structured_logs
To use the standard python logging module on GCP (tested on python 3.9), you can do the following:
import google.cloud.logging
logging_client = google.cloud.logging.Client()
logging_client.setup_logging()
import logging
logging.warning("A warning")
See also: https://cloud.google.com/logging/docs/setup/python
I use a very simple custom logging function to log to Cloud Logging:
import json
def cloud_logging(severity, message):
print(json.dumps({"severity": severity, "message": message}))
cloud_logging(severity="INFO", message="Your logging message")
I am using the websockets package to create a websocket server in python; at the same time I am heavily using the logging in different log-levels.
The websockets documentation mentions the logger configuration and that it could be changed to the log-level ERROR using
logger = logging.getLogger('websockets.server')
logger.setLevel(logging.ERROR)
logger.addHandler(logging.StreamHandler())
This, however, does not have any effect in my code, wherever I place it (below the import, before the websockets import, within __main__). I'd like to have two configurations - one global logging configuration and the logger-configuration of the websockets server.
Another option would be to disable the logging completely for the websockets module but I can't figure out how to do that. Any ideas?
It turns out it is not only the websockets.server that is emitting lots of debug messages, there's other ones too. I've ended up hunting down those packages by using the %(name)s field in the root logger configuration and could find websockets.protocol being responsible for the additional messages.
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s %(name)s %(levelname)-8s %(message)s',
datefmt='(%H:%M:%S)')
# disable all loggers from different files
logging.getLogger('asyncio').setLevel(logging.ERROR)
logging.getLogger('asyncio.coroutines').setLevel(logging.ERROR)
logging.getLogger('websockets.server').setLevel(logging.ERROR)
logging.getLogger('websockets.protocol').setLevel(logging.ERROR)
When you start a websocket, set:
websocket.enableTrace(False)
No DEBUG and hart beat messages will be logged or shown in the console.
My example code:
def start_websocket(url, listen_method):
"""
Start a websocket connection.
:param url: string
:param listen_method: method
"""
# Enable or Disable logging
websocket.enableTrace(False)
ws = websocket.WebSocketApp(
url,
on_message=listen_method,
on_error=on_error,
on_close=on_close
)
ws.on_open = on_open
ws.run_forever(ping_interval=3, ping_timeout=2)
I can't find much on logging onto Google App Engine using Flexible environment and NodeJS.
As the docs says, one is able to write its own logging messages using the standard stdout and stderr. But this is simple logging, and I would like to have something a little bit more refined.
In particular, the Log Views in Google Cloud Platform Console allows the user to filter logs on their severity levels:
Critical
Error
Warning
Info
Debug
Any log level
I would like to find a way to use those levels in my applications, so that I can read logs much better.
By default, if I print to stdout using console.log() the logs will only appear if I filter by "Any log level", and I have no option to select a severity level.
I have tried to use winston-gae as reported on the docs, but without any success. Maybe I have configured it wrong?
To be more clear, I would like to be able to do something like this in Python (source):
import logging
import webapp2
class MainPage(webapp2.RequestHandler):
def get(self):
logging.debug('This is a debug message')
logging.info('This is an info message')
logging.warning('This is a warning message')
logging.error('This is an error message')
logging.critical('This is a critical message')
self.response.out.write('Logging example.')
app = webapp2.WSGIApplication([
('/', MainPage)
], debug=True)
I would recommend to look at the Google Cloud Node.js client library, that can help you call the Stackdriver Logging API in order to log structured log entries: https://github.com/GoogleCloudPlatform/google-cloud-node#google-stackdriver-logging-beta