I'm trying to configure some logging across multiple modules within a package. These modules may be executed in different containers, possibly in different regions, so I wanted to explicitly use GMT / UTC time for logging.
When I'm reading about the Formatter class in logging, it indicates you can specify converter to use either local-time or GMT. I'd like to utilize this feature, in conjunction with dictConfig (or possibly fileConfig) to specify the configurations for the different modules, but the documentation is sparse with respect to this feature. Everything specified in the config is working, except for the timezone. The log always uses local time. I can include the '%z' formatting specification in datefmt to specify offset from GMT, but that breaks the .%(msecs)03d formatting.
Below is my code using a defined dictionary and dictConfig. Has anyone had any success specifying timezone in the config? is this possible?
import json
import logging
from logging.config import dictConfig
import time
DEFAULT_CONFIG = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'standard': {
'format': '%(levelname)s : %(name)s : %(asctime)s.%(msecs)03d : %(message)s',
'datefmt': '%Y-%m-%d %H.%M.%S',
'converter': time.gmtime # also fails with 'time.gmtime', 'gmtime'
}
},
'handlers' : {
'default': {
'level': 'NOTSET',
'class': 'logging.StreamHandler',
'formatter': 'standard',
}
},
'loggers': {
'TEST': { # logging from this module should be logged in DEBUG level
'handlers' : ['default'],
'level': 'INFO',
'propagate': False,
},
},
'root': {
'level': 'INFO',
'handlers': ['default']
},
'incremental': False
}
if __name__ == '__main__':
dictConfig(DEFAULT_CONFIG)
logger = logging.getLogger('TEST')
logger.debug('debug message') # this should not be displayed if level==INFO
logger.info('info message')
logger.warning('warning message')
logger.error('error message')
Output:
INFO : TEST : 2022-07-07 17.20.13.434 : info message
WARNING : TEST : 2022-07-07 17.20.13.435 : warning message
ERROR : TEST : 2022-07-07 17.20.13.435 : error message
Related
I have a Python FASTAPI application which I run through Uvicorn.
I have a dependent service and I generate a python client using the swaggen-codegen provided by the openapi.json of the dependent service.
My logger dictconfig is:
def get_logging_config():
return {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'standard': {
'()': ColoredFormatter
},
},
'handlers': {
'console': {
'level': logging.getLevelName(logging.DEBUG),
'formatter': 'standard',
'class': 'logging.StreamHandler',
'stream': 'ext://sys.stdout'
}
},
'loggers': {
'': { # root logger
'handlers': ['console'],
'level': logging.getLevelName(get_logging_level()),
'propagate': False
},
'uvicorn': { # uvicorn logger
'handlers': ['console'],
'level': logging.getLevelName(logging.WARN),
'propagate': False
},
'urllib3': { # urllib3 logger
'handlers': ['console'],
'level': logging.getLevelName(logging.DEBUG),
'propagate': False
},
'urllib3.connectionpool': { # urllib3 logger for HTTP calls
'handlers': ['console'],
'level': logging.getLevelName(logging.DEBUG),
'propagate': False
},
'requests.packages.urllib3': {
'handlers': ['console'],
'level': logging.getLevelName(logging.DEBUG),
'propagate': True
}
}
}
My logging module init.py contains:
import logging
import logging.config
from .http_client_logger import httpclient_logging_patch
from .logging_config import get_logging_config
httpclient_logging_patch()
logging.config.dictConfig(get_logging_config())
My http_client_logger.py file looks like:
import logging
import http.client
httpclient_logger = logging.getLogger("http.client")
def httpclient_logging_patch(level=logging.DEBUG):
"""Enable HTTPConnection debug logging to the logging framework"""
def httpclient_log(*args):
httpclient_logger.log(level, "test-" + "".join(args))
# mask the print() built-in in the http.client module to use
# logging instead
http.client.print = httpclient_log
# enable debugging
http.client.HTTPConnection.debuglevel = 1
http.client.HTTPSConnection.debuglevel = 1
The logs which I could see are
2023-01-14 20:18:02.384 - DEBUG - https://test-dependent-service-url.com:443 "POST /test-endpoint HTTP/1.1" 200 None (urllib3.connectionpool:_make_request:456)
2023-01-14 20:18:02.385 - DEBUG - response body: b'{"data":[],"id":"93d0f81a-3d4e-4ae6-9c47-07adde47400a","timestamp":"2023-01-14T19:18:02.489899239Z"}' (dependent_service.client.swagger_client.rest:request:219)
I would like to log the request sent (POST, GET, PUT, DELETE) to the dependent service and response received from the dependent service which contains the body of the request and response. Note that I am using an HTTPS connection. But in the logs, the request body is not being logged. Also, in none of the cases, the headers are getting logged.
Please, anyone could help here?
I have tried the following:
Log all requests from the python-requests module
I have also tried to put the debuglevel to 2 but nothing helped
http.client.HTTPConnection.debuglevel = 2
http.client.HTTPSConnection.debuglevel = 2
https://bhoey.com/blog/better-debug-logging-for-the-python-requests-library/ It talks about the file handler but it same as previous and I am using StreamHandler.
Although it works for GET but not for POST.
I stucked with problem during Google Cloud Logging setup for Python3.8 in Google App Engine Standard environment.
I'm using FastAPI with unicorn. My code logging configuration:
import logging.config
import sys
from google.cloud import logging as google_logging
from app.settings import ENV, _settings
if _settings.ENV == ENV.LOCAL:
MAIN_LOGGER = 'console'
LOGGER_CONF_DICT = {
'class': 'logging.StreamHandler',
'formatter': 'verbose',
'stream': sys.stdout,
'level': _settings.LOG_LEVEL.upper(),
}
else:
log_client = google_logging.Client()
MAIN_LOGGER = 'stackdriver_logging'
LOGGER_CONF_DICT = {
'class': 'app.gcloud_logs.GCLHandler',
'client': log_client,
'name': 'appengine.googleapis.com%2Frequest_log'
# I've tried other names: stdout, %2FA instead of / symbol, appengine.googleapis.com/stdout
# the same result or no logs at all
}
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'verbose': {
'format': '%(log_color)s%(asctime)s [%(levelname)s] [%(name)s] %(message)s (%(filename)s:%(lineno)d)',
'()': 'colorlog.ColoredFormatter',
'log_colors': {
'DEBUG': 'cyan',
'INFO': 'green',
'WARNING': 'yellow',
'ERROR': 'red',
'CRITICAL': 'bold_red',
},
}
},
'handlers': {
MAIN_LOGGER: {**LOGGER_CONF_DICT},
'blackhole': {'level': 'DEBUG', 'class': 'logging.NullHandler'},
},
'loggers': {
'fastapi': {'level': 'INFO', 'handlers': [MAIN_LOGGER]},
'uvicorn.error': {'level': 'INFO', 'handlers': [MAIN_LOGGER], 'propagate': False},
'uvicorn.access': {'level': 'INFO', 'handlers': [MAIN_LOGGER], 'propagate': False},
'uvicorn': {'level': 'INFO', 'handlers': [MAIN_LOGGER], 'propagate': False},
'google.cloud.logging.handlers.transports.background_thread': {'level': 'DEBUG', 'handlers': ['blackhole'],
'propagate': False},
'': {
'level': _settings.LOG_LEVEL.upper(),
'handlers': [MAIN_LOGGER],
'propagate': True,
},
}
}
logging.config.dictConfig(LOGGING)
And my logging handler code:
import os
from typing import Any, Dict, Optional
from google.cloud.logging.handlers import CloudLoggingHandler
from google.cloud.logging.resource import Resource
from starlette.requests import Request
from starlette_context import context
from app.settings import _settings
class GCLHandler(CloudLoggingHandler):
def emit(self, record):
message = super(GCLHandler, self).format(record)
request: Optional[Request] = None
trace: Optional[str] = None
span_id: Optional[str] = None
user_id: Optional[int] = None
resource = Resource(
type='gae_app',
labels={
'module_id': os.environ['GAE_SERVICE'],
'project_id': _settings.PROJECT_NAME,
'version_id': os.environ['GAE_VERSION'],
'zone': 'us16' # tried without zone - the same result
}
)
labels: Dict[str, Any] = {}
if context.exists(): # I'm sure that it works
request = context.get('request') # I'm sure that it works
user_id = context.get('user_id') # I'm sure that it works
if user_id is not None:
labels['user_id'] = user_id
if request:
if request.headers.get('X-Cloud-Trace-Context'):
cloud_trace = request.headers.get('X-Cloud-Trace-Context').split('/')
if len(cloud_trace) > 1:
span_id = cloud_trace[1].split(';')[0]
trace = f'projects/{_settings.PROJECT_NAME}/traces/{cloud_trace[0]}'
labels['logging.googleapis.com/trace'] = cloud_trace[0] # Found in some guides, not sure that its neccessary
labels['appengine.googleapis.com/trace_id'] = cloud_trace[0] # Found in some guides, not sure that its neccessary
self.transport.send(
record,
message,
resource=resource,
labels=labels,
trace=trace,
span_id=span_id
)
I've got some strange results in logs viewer that my log has the same trace as request log, but they're not grouped
Any ideas?
There are 2 types of logs in App Engine :
Request log: A log of the requests that are sent to your app. App Engine automatically creates entries in the request log.
App log: log entries that you write to a supported framework or file as described on this page.
The both logs are send to the Cloud Logging Agent automatically by App Engine Standard.
On first request, app logs and request logs are not correlated and that's why there are not shown in a group , this is a known issue stated in App Engine Official Documentation. However in the second request, you can see that the logs are shown in a group.
A feature request in Public Issue Tracker has already been created for this behavior where you will get all the updates regarding the fix.
log_client = google_logging.Client()
MAIN_LOGGER = 'stackdriver_logging'
LOGGER_CONF_DICT = {
'class': 'app.gcloud_logs.GCLHandler',
'client': log_client,
'name': 'app'
}
Change name to app is helped
I am trying to configure my logger using dictConfig to use a different format but it does not seem to be taking effect. Following is the code that I have (I am also simultaneously trying to suppress the logs from imported modules)-
import logging.config
import requests
logging.config.dictConfig({
'version': 1,
'disable_existing_loggers': True,
'formatters':{'standard': { 'format': "[%(asctime)s] [%(levelname)8s] - %(message)s", 'datefmt':"%d-%b-%Y %I:%M:%S %p"}},
'handlers': {'default': {'level': 'DEBUG', 'formatter': 'standard', 'class': 'logging.StreamHandler', 'stream': 'ext://sys.stdout'}},
'loggers':{'__main__': {'handlers': ['default'], 'level': 'DEBUG', 'propagate': False }}
})
req = requests.get('https://www.google.com')
logging.debug("Only thing that should be printed")
Output -
DEBUG:root:Only thing that should be printed
Expected Output -
[2020-04-04 22:46:24,866] [ DEBUG] - Only thing that should be printed
I learnt how to use dictConfig from this SO post.
If you look at the post post that you've mentioned, you will see that you forget a line :)
log = logging.getLogger(__name__)
log.debug("Only thing that should be printed")
The logger hierarchy must be defined explicitly in the logger name, using dot-notation.
When using the __name__ :
This means that logger names track the package/module hierarchy, and
it’s intuitively obvious where events are logged just from the logger
name.
For more explanations, the docs are pretty complete.
When trying to log SQL statements in Django I can see params, duration but not the SQL statement itself. The context parameter sql returns None
I am using MySQL as backend
Settings setup:
DEBUG=True
...
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'verbose': {
'format': '{levelname} SQL:{sql} {duration} params:{params}',
'style': '{',
}},
'handlers': {
'console': {
'level': 'DEBUG',
'class': 'logging.StreamHandler',
'formatter': 'verbose'
}
},
'loggers': {
'django.db.backends': {
'handlers': ['console'],
'level': 'DEBUG',
},
}
}
Sample output:
DEBUG SQL:None 0.036980390548706055 params:(1,)
So I can the duration, the parameters but not the SQL statements. And I do not know why.
I have based my configuration on https://docs.djangoproject.com/en/dev/topics/logging/#django-db-backends
During a long long search found out this is a bug in version 2.1.4 upgrading to 2.1.5 fixed the problem. I based this on this
https://github.com/jazzband/django-debug-toolbar/issues/1124
Checked and verified. Output after update:
DEBUG SQL:SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED 0.035988569259643555 params:None
I'm trying to code a dictionary based logging configuration and have been stumped by a ValueError that occurs when I run the program. I've stripped it down to the essentials and the problem remains. I've read the 3.5 docs, logging HOWTO, Logging Cookbook, etc. but unfortunately, the solution has not presented itself. Any help would be appreciated.
Also, I'm only 3 weeks into python so I may just be out of my depth at this point. Here's the code...
import logging.config
log_config = {
'version': 1,
'disable_existing_loggers': False,
'formatters':{
'verbose_formatter':{
'format':'%(levelname)s: %(name)s: %(asctime)s.%(msecs).03d : '\
'%(message)s: %(process)s: %(processName)s',
'datefmt':'%Y-%m-%d %H:%M:%S'
},
'precise_formatter':{
'format':'%(levelname)s: %(name)s: %(asctime)s.%(msecs).03d : '\
'%(message)s',
'datefmt':'%Y-%m-%d %H:%M:%S'
},
'brief_formatter':{
'format':'%(levelname)s: %(message)s'
}
},
'handlers':{
'con_handler':{
'class':'logging.StreamHandler',
'level':'DEBUG',
'formatter':'precise_formatter',
'stream':'ext://sys.stdout'
},
'file_handler':{
'class':'logging.handlers.RotatingFileHandler',
'filename':'logger.log',
'maxBytes':1048576,
'backupCount':4,
'level':'DEBUG',
'formatter':'precise_formatter',
'encoding':'utf8'
}
},
'loggers':{
'level':'DEBUG',
'handlers':['con_handler', 'file_handler']
}
}
logging.config.dictConfig(log_config)
logger = logging.getLogger(__name__)
logger.critical('This should always be seen!')
When run, I receive the follow:
ValueError was unhandled by user code
Message: Unable to configure logger 'handlers': 'ConvertingList' object has no attribute 'get'
or sometimes this...
ValueError was unhandled by user code
Message: Unable to configure logger 'level': 'str' object has no attribute 'get'
I suspect that the different errors may have to do with the sometimes changing order of the dictionary?
Change the loggers section to
'loggers':{
'': {
'level':'DEBUG',
'handlers':['con_handler', 'file_handler']
}
}
The '' (empty string) refers to the root logger. you can add more loggers for different components:
'loggers':{
'': {
'level':'DEBUG',
'handlers':['con_handler', 'file_handler']
}
'bottle': { #I only want error level from bottle :)
'level':'ERROR',
'handlers':['con_handler', 'file_handler']
}
}
To config the root logger use a root key of your log_config dictionary.
root - this will be the configuration for the root logger.
Source: Dictionary Schema Details
Following this description your config should look something like this:
log_config = {
...
'handlers': {
'con_handler': ...,
'file_handler': ...
},
'loggers': {
'other_logger': ...
},
'root': {
'level': 'DEBUG',
'handlers': ['con_handler', 'file_handler']
}
}