Python - Request and Response logging using HTTPS Connection - Swagger Codegen - OpenAPI - python-3.x

I have a Python FASTAPI application which I run through Uvicorn.
I have a dependent service and I generate a python client using the swaggen-codegen provided by the openapi.json of the dependent service.
My logger dictconfig is:
def get_logging_config():
return {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'standard': {
'()': ColoredFormatter
},
},
'handlers': {
'console': {
'level': logging.getLevelName(logging.DEBUG),
'formatter': 'standard',
'class': 'logging.StreamHandler',
'stream': 'ext://sys.stdout'
}
},
'loggers': {
'': { # root logger
'handlers': ['console'],
'level': logging.getLevelName(get_logging_level()),
'propagate': False
},
'uvicorn': { # uvicorn logger
'handlers': ['console'],
'level': logging.getLevelName(logging.WARN),
'propagate': False
},
'urllib3': { # urllib3 logger
'handlers': ['console'],
'level': logging.getLevelName(logging.DEBUG),
'propagate': False
},
'urllib3.connectionpool': { # urllib3 logger for HTTP calls
'handlers': ['console'],
'level': logging.getLevelName(logging.DEBUG),
'propagate': False
},
'requests.packages.urllib3': {
'handlers': ['console'],
'level': logging.getLevelName(logging.DEBUG),
'propagate': True
}
}
}
My logging module init.py contains:
import logging
import logging.config
from .http_client_logger import httpclient_logging_patch
from .logging_config import get_logging_config
httpclient_logging_patch()
logging.config.dictConfig(get_logging_config())
My http_client_logger.py file looks like:
import logging
import http.client
httpclient_logger = logging.getLogger("http.client")
def httpclient_logging_patch(level=logging.DEBUG):
"""Enable HTTPConnection debug logging to the logging framework"""
def httpclient_log(*args):
httpclient_logger.log(level, "test-" + "".join(args))
# mask the print() built-in in the http.client module to use
# logging instead
http.client.print = httpclient_log
# enable debugging
http.client.HTTPConnection.debuglevel = 1
http.client.HTTPSConnection.debuglevel = 1
The logs which I could see are
2023-01-14 20:18:02.384 - DEBUG - https://test-dependent-service-url.com:443 "POST /test-endpoint HTTP/1.1" 200 None (urllib3.connectionpool:_make_request:456)
2023-01-14 20:18:02.385 - DEBUG - response body: b'{"data":[],"id":"93d0f81a-3d4e-4ae6-9c47-07adde47400a","timestamp":"2023-01-14T19:18:02.489899239Z"}' (dependent_service.client.swagger_client.rest:request:219)
I would like to log the request sent (POST, GET, PUT, DELETE) to the dependent service and response received from the dependent service which contains the body of the request and response. Note that I am using an HTTPS connection. But in the logs, the request body is not being logged. Also, in none of the cases, the headers are getting logged.
Please, anyone could help here?
I have tried the following:
Log all requests from the python-requests module
I have also tried to put the debuglevel to 2 but nothing helped
http.client.HTTPConnection.debuglevel = 2
http.client.HTTPSConnection.debuglevel = 2
https://bhoey.com/blog/better-debug-logging-for-the-python-requests-library/ It talks about the file handler but it same as previous and I am using StreamHandler.
Although it works for GET but not for POST.

Related

Python logging with dictConfig and using GMT / UTC zone time

I'm trying to configure some logging across multiple modules within a package. These modules may be executed in different containers, possibly in different regions, so I wanted to explicitly use GMT / UTC time for logging.
When I'm reading about the Formatter class in logging, it indicates you can specify converter to use either local-time or GMT. I'd like to utilize this feature, in conjunction with dictConfig (or possibly fileConfig) to specify the configurations for the different modules, but the documentation is sparse with respect to this feature. Everything specified in the config is working, except for the timezone. The log always uses local time. I can include the '%z' formatting specification in datefmt to specify offset from GMT, but that breaks the .%(msecs)03d formatting.
Below is my code using a defined dictionary and dictConfig. Has anyone had any success specifying timezone in the config? is this possible?
import json
import logging
from logging.config import dictConfig
import time
DEFAULT_CONFIG = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'standard': {
'format': '%(levelname)s : %(name)s : %(asctime)s.%(msecs)03d : %(message)s',
'datefmt': '%Y-%m-%d %H.%M.%S',
'converter': time.gmtime # also fails with 'time.gmtime', 'gmtime'
}
},
'handlers' : {
'default': {
'level': 'NOTSET',
'class': 'logging.StreamHandler',
'formatter': 'standard',
}
},
'loggers': {
'TEST': { # logging from this module should be logged in DEBUG level
'handlers' : ['default'],
'level': 'INFO',
'propagate': False,
},
},
'root': {
'level': 'INFO',
'handlers': ['default']
},
'incremental': False
}
if __name__ == '__main__':
dictConfig(DEFAULT_CONFIG)
logger = logging.getLogger('TEST')
logger.debug('debug message') # this should not be displayed if level==INFO
logger.info('info message')
logger.warning('warning message')
logger.error('error message')
Output:
INFO : TEST : 2022-07-07 17.20.13.434 : info message
WARNING : TEST : 2022-07-07 17.20.13.435 : warning message
ERROR : TEST : 2022-07-07 17.20.13.435 : error message

Google Cloud logging, Python3.8 standard environment, group request related logs by trace id

I stucked with problem during Google Cloud Logging setup for Python3.8 in Google App Engine Standard environment.
I'm using FastAPI with unicorn. My code logging configuration:
import logging.config
import sys
from google.cloud import logging as google_logging
from app.settings import ENV, _settings
if _settings.ENV == ENV.LOCAL:
MAIN_LOGGER = 'console'
LOGGER_CONF_DICT = {
'class': 'logging.StreamHandler',
'formatter': 'verbose',
'stream': sys.stdout,
'level': _settings.LOG_LEVEL.upper(),
}
else:
log_client = google_logging.Client()
MAIN_LOGGER = 'stackdriver_logging'
LOGGER_CONF_DICT = {
'class': 'app.gcloud_logs.GCLHandler',
'client': log_client,
'name': 'appengine.googleapis.com%2Frequest_log'
# I've tried other names: stdout, %2FA instead of / symbol, appengine.googleapis.com/stdout
# the same result or no logs at all
}
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'verbose': {
'format': '%(log_color)s%(asctime)s [%(levelname)s] [%(name)s] %(message)s (%(filename)s:%(lineno)d)',
'()': 'colorlog.ColoredFormatter',
'log_colors': {
'DEBUG': 'cyan',
'INFO': 'green',
'WARNING': 'yellow',
'ERROR': 'red',
'CRITICAL': 'bold_red',
},
}
},
'handlers': {
MAIN_LOGGER: {**LOGGER_CONF_DICT},
'blackhole': {'level': 'DEBUG', 'class': 'logging.NullHandler'},
},
'loggers': {
'fastapi': {'level': 'INFO', 'handlers': [MAIN_LOGGER]},
'uvicorn.error': {'level': 'INFO', 'handlers': [MAIN_LOGGER], 'propagate': False},
'uvicorn.access': {'level': 'INFO', 'handlers': [MAIN_LOGGER], 'propagate': False},
'uvicorn': {'level': 'INFO', 'handlers': [MAIN_LOGGER], 'propagate': False},
'google.cloud.logging.handlers.transports.background_thread': {'level': 'DEBUG', 'handlers': ['blackhole'],
'propagate': False},
'': {
'level': _settings.LOG_LEVEL.upper(),
'handlers': [MAIN_LOGGER],
'propagate': True,
},
}
}
logging.config.dictConfig(LOGGING)
And my logging handler code:
import os
from typing import Any, Dict, Optional
from google.cloud.logging.handlers import CloudLoggingHandler
from google.cloud.logging.resource import Resource
from starlette.requests import Request
from starlette_context import context
from app.settings import _settings
class GCLHandler(CloudLoggingHandler):
def emit(self, record):
message = super(GCLHandler, self).format(record)
request: Optional[Request] = None
trace: Optional[str] = None
span_id: Optional[str] = None
user_id: Optional[int] = None
resource = Resource(
type='gae_app',
labels={
'module_id': os.environ['GAE_SERVICE'],
'project_id': _settings.PROJECT_NAME,
'version_id': os.environ['GAE_VERSION'],
'zone': 'us16' # tried without zone - the same result
}
)
labels: Dict[str, Any] = {}
if context.exists(): # I'm sure that it works
request = context.get('request') # I'm sure that it works
user_id = context.get('user_id') # I'm sure that it works
if user_id is not None:
labels['user_id'] = user_id
if request:
if request.headers.get('X-Cloud-Trace-Context'):
cloud_trace = request.headers.get('X-Cloud-Trace-Context').split('/')
if len(cloud_trace) > 1:
span_id = cloud_trace[1].split(';')[0]
trace = f'projects/{_settings.PROJECT_NAME}/traces/{cloud_trace[0]}'
labels['logging.googleapis.com/trace'] = cloud_trace[0] # Found in some guides, not sure that its neccessary
labels['appengine.googleapis.com/trace_id'] = cloud_trace[0] # Found in some guides, not sure that its neccessary
self.transport.send(
record,
message,
resource=resource,
labels=labels,
trace=trace,
span_id=span_id
)
I've got some strange results in logs viewer that my log has the same trace as request log, but they're not grouped
Any ideas?
There are 2 types of logs in App Engine :
Request log: A log of the requests that are sent to your app. App Engine automatically creates entries in the request log.
App log: log entries that you write to a supported framework or file as described on this page.
The both logs are send to the Cloud Logging Agent automatically by App Engine Standard.
On first request, app logs and request logs are not correlated and that's why there are not shown in a group , this is a known issue stated in App Engine Official Documentation. However in the second request, you can see that the logs are shown in a group.
A feature request in Public Issue Tracker has already been created for this behavior where you will get all the updates regarding the fix.
log_client = google_logging.Client()
MAIN_LOGGER = 'stackdriver_logging'
LOGGER_CONF_DICT = {
'class': 'app.gcloud_logs.GCLHandler',
'client': log_client,
'name': 'app'
}
Change name to app is helped

How to configure 'dictConfig' properly to use a different logging format?

I am trying to configure my logger using dictConfig to use a different format but it does not seem to be taking effect. Following is the code that I have (I am also simultaneously trying to suppress the logs from imported modules)-
import logging.config
import requests
logging.config.dictConfig({
'version': 1,
'disable_existing_loggers': True,
'formatters':{'standard': { 'format': "[%(asctime)s] [%(levelname)8s] - %(message)s", 'datefmt':"%d-%b-%Y %I:%M:%S %p"}},
'handlers': {'default': {'level': 'DEBUG', 'formatter': 'standard', 'class': 'logging.StreamHandler', 'stream': 'ext://sys.stdout'}},
'loggers':{'__main__': {'handlers': ['default'], 'level': 'DEBUG', 'propagate': False }}
})
req = requests.get('https://www.google.com')
logging.debug("Only thing that should be printed")
Output -
DEBUG:root:Only thing that should be printed
Expected Output -
[2020-04-04 22:46:24,866] [ DEBUG] - Only thing that should be printed
I learnt how to use dictConfig from this SO post.
If you look at the post post that you've mentioned, you will see that you forget a line :)
log = logging.getLogger(__name__)
log.debug("Only thing that should be printed")
The logger hierarchy must be defined explicitly in the logger name, using dot-notation.
When using the __name__ :
This means that logger names track the package/module hierarchy, and
it’s intuitively obvious where events are logged just from the logger
name.
For more explanations, the docs are pretty complete.

Python Zeep WSDL Unexpected Elements Traceback

I can't handle this below.
XMLParseError: Unexpected element 'metadata', expected 'id' error.
I also try strict=False settings but that time zeep returned as none.
from zeep import Client
import zeep
wsdl = 'https://api.n11.com/ws/CategoryService.wsdl'
cl = Client(wsdl=wsdl)
request_data = {"auth":{'appKey' : ***,
'appSecret' : **},'categoryId':int(1003221),"pagingData":{'pageSize':1,'currentPage':0}}
print(cl.service.GetCategoryAttributes(**request_data))
Thanks.
I faced the same problem with the same API. My solution is changing zeep's settings.
You should use xsd_ignore_sequence_order=True.
I hope it's not too late.
settings = Settings(strict=False, xml_huge_tree=True, xsd_ignore_sequence_order=True)
read more:
https://docs.python-zeep.org/en/master/settings.html#module-zeep.settings
Solution:
firstly, to check logs:
import logging.config
logging.config.dictConfig({
'version': 1,
'formatters': {
'verbose': {
'format': '%(name)s: %(message)s'
}
},
'handlers': {
'console': {
'level': 'DEBUG',
'class': 'logging.StreamHandler',
'formatter': 'verbose',
},
},
'loggers': {
'zeep.transports': {
'level': 'DEBUG',
'propagate': True,
'handlers': ['console'],
},
}
})
Second:
from zeep import Client
from zeep.plugins import HistoryPlugin
wsdl = 'https://api.n11.com/ws/CategoryService.wsdl'
from zeep import Client, Settings
settings = Settings(strict=False, xml_huge_tree=True)
history = HistoryPlugin()
from zeep.transports import Transport
transport = Transport(timeout=10)
key={'appKey' : '**',
'appSecret' : '**',
}
cat= {'categoryId':1003221}
dat= {'pageSize':1,'currentPage':1}
client = Client(wsdl=wsdl, transport=transport, plugins=[history], settings=settings)
with client.settings(raw_response=False,strict=False, xml_huge_tree=True):
response = client.service.GetCategoryAttributes(key,1003221,pagingData=dat)
#assert response.status_code == 200
#assert response.content
#node = client.create_message(client.service, 'GetCategoryAttributes',key,1003221,pagingData=dat)
from lxml import etree
try:
for hist in [history.last_sent, history.last_received]:
print(etree.tostring(hist["envelope"], encoding="unicode", pretty_print=True))
except (IndexError, TypeError):
# catch cases where it fails before being put on the wire
pass
lastly, you can parse with bs4 for xml content:
from bs4 import BeautifulSoup
soup = BeautifulSoup(ass, 'lxml')
soup.find('category').find('metadata')

Cannot see SQL statements when logging in Django 2.1.4

When trying to log SQL statements in Django I can see params, duration but not the SQL statement itself. The context parameter sql returns None
I am using MySQL as backend
Settings setup:
DEBUG=True
...
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'verbose': {
'format': '{levelname} SQL:{sql} {duration} params:{params}',
'style': '{',
}},
'handlers': {
'console': {
'level': 'DEBUG',
'class': 'logging.StreamHandler',
'formatter': 'verbose'
}
},
'loggers': {
'django.db.backends': {
'handlers': ['console'],
'level': 'DEBUG',
},
}
}
Sample output:
DEBUG SQL:None 0.036980390548706055 params:(1,)
So I can the duration, the parameters but not the SQL statements. And I do not know why.
I have based my configuration on https://docs.djangoproject.com/en/dev/topics/logging/#django-db-backends
During a long long search found out this is a bug in version 2.1.4 upgrading to 2.1.5 fixed the problem. I based this on this
https://github.com/jazzband/django-debug-toolbar/issues/1124
Checked and verified. Output after update:
DEBUG SQL:SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED 0.035988569259643555 params:None

Resources