How can I keep sensitive data out of logs?
Currently I have an exception which is raised in a method A. In method B this exception is re-raised adding further information. In method C I want to log the exception with a further information.
My first attempt was to add a string replace method before logging the exception, but this does not affect the whole traceback. Especially because I call the Python library request in methodA. The first exception takes place in this library.
first excepton in requests: urllib3.exceptions.MaxRetryError
second exception in requests: requests.exceptions.ProxyError
Both exceptions within the request library contain already the sensitive data in the traceback.
def methodA():
try:
connect_to_http_with_request_lib()
except requests.exceptions.ProxyError as err:
raise MyExceptionA(f"this log contains sensitive info in err: {err}")
def methodB():
try:
methodA()
except MyExceptionA as err:
raise MyExceptionB (f"add some more info to: {err}")
def methodC():
try:
methodB()
return True
except MyExceptionB as err:
err = re.sub(r"(?is)password=.+", "password=xxxx", str(err))
logger.exception(f("methodB failed exception {err}")
return False
How can I parse the whole traceback before logging the exception in order to mask out sensitive data?
I use loguru as logging library.
The Django framework seems to address the same problem with their own methods. See here
Related
I want to wrap my database-operations into sessions (transactions). For a good user experience I want to try to handle as many possible errors inside my code. My basic structure of a database transaction is in the code block.
There are two kinds of error:
My code can try to handle them, for ex. a user try to insert a key that already exists.
Some other exception raise, that was not handled by the code.
In both situations, I want to show the user a message.
How can I give out the error, so that I can display it after the session was closed?
# Transaction
session = Session()
try:
db_operation1(session, ...)
db_operation2(session, ...)
...
session.commit()
except:
session.rollback()
finally:
session.close()
# Show error
# ???
Edit:
After some investigation, I change my code to this:
# Transaction
session = Session()
try:
db_operation1(session, ...)
db_operation2(session, ...)
...
session.commit()
except Exception as e:
session.rollback()
# Show Error
print(str(e))
finally:
session.close()
But in the case, my code can handle the exception, I need some more user friendly message.
Edit2:
As snakecharmerb points me to, it is better to handle the exception outside the session-scope, I change the code again:
try:
# Transaction
session = Session()
try:
db_operation1(session, ...)
db_operation2(session, ...)
...
session.commit()
except:
session.rollback()
raise
finally:
session.close()
except Exception as e:
# Show Error
print(str(e))
I have a piece of code that I don't control and while running it raises an error. I'd like to capture the value of exc object inside the exc_func method.
As you can see exc_func raises two exceptions, one of which is handled. What I care about is the value of the exc object, but so far have little luck retrieving it. The value does not exist in exc_traceback object and the exception message is not very helpful.
import traceback
import sys
def exc_func():
try:
a = 1
a.length()
except Exception as exc:
exc.with_not_existing()
def main():
try:
exc_func()
except Exception as exc:
exc_type, exc_value, exc_traceback = sys.exc_info()
tb_walk = traceback.walk_tb(exc_traceback)
# Need this in order to pickle traceback
tb_summary = traceback.StackSummary.extract(tb_walk, capture_locals=True)
if __name__ == '__main__':
main()
EDIT:
For instance, the exc object in main is AttributeError("'AttributeError' object has no attribute 'with_not_existing'"). What I really want to see is the exc object inside exc_func. Just to be clear, I need the exc object itself, something like traceback.format_exc() is not helpful in my case, due to the nature of the exception (it's a C lib that raises this exception)
When an exception is raised during handling another exception, the initial exception is stored as the __context__. It can be extracted when handling the new exception.
try:
exc_func()
except Exception as exc:
parent = exc.__context__ # the previously handled exception
print(type(parent), parent)
Note that an exception handler may also explicitly chain exceptions via __cause__.
Built-in Exceptions
[...]
When raising (or re-raising) an exception in an except or finally clause __context__ is automatically set to the last exception caught; if the new exception is not handled the traceback that is eventually displayed will include the originating exception(s) and the final exception.
The raise statement
[...]
The from clause is used for exception chaining: if given, the second expression must be another exception class or instance, which will then be attached to the raised exception as the __cause__ attribute (which is writable).
[...]
A similar mechanism works implicitly if an exception is raised inside an exception handler or a finally clause: the previous exception is then attached as the new exception’s __context__ attribute:
folks.
I'm trying to configure logging from an external yaml configuration file which may or may not have the necessary options forcing me to check and fail over in several different ways. I wrote two solutions doing same thing, but in different styles:
More traditional "C-like":
try:
if config['log']['stream'].lower() == 'console':
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter(fmt='scheduler: (%(levelname).1s) %(message)s'))
elif config['log']['stream'].lower() == 'syslog':
raise ValueError
else:
print('scheduler: (E) Failed to set log stream: Unknown stream: \'' + config['log']['stream'] + '\'. Failing over to syslog.', file=sys.stderr)
raise ValueError
except (KeyError, ValueError) as e:
if type(e) == KeyError:
print('scheduler: (E) Failed to set log stream: Stream is undefined. Failing over to syslog.', file=sys.stderr)
handler = logging.handlers.SysLogHandler(facility=logging.handlers.SysLogHandler.LOG_DAEMON, address = '/dev/log')
handler.setFormatter(logging.Formatter(fmt='scheduler[%(process)d]: (%(levelname).1s) %(message)s'))
finally:
log.addHandler(handler)
And "pythonic" with internal procedure:
def setlogstream(stream):
if stream == 'console':
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter(fmt='scheduler: (%(levelname).1s) %(message)s'))
elif stream == 'syslog':
handler = logging.handlers.SysLogHandler(facility=logging.handlers.SysLogHandler.LOG_DAEMON, address = '/dev/log')
handler.setFormatter(logging.Formatter(fmt='scheduler[%(process)d]: (%(levelname).1s) %(message)s'))
else:
raise ValueError
log.addHandler(handler)
try:
setlogstream(config['log']['stream'].lower())
except KeyError:
print('scheduler: (E) Failed to set log stream: Stream is undefined. Failing over to syslog.', file=sys.stderr)
setlogstream('syslog')
except ValueError:
print('scheduler: (E) Failed to set log stream: Unknown stream: \'' + config['log']['stream'] + '\'. Failing over to syslog.', file=sys.stderr)
setlogstream('syslog')
They both do what I need, both short, both extendible in case I need more streams, but now I wonder which one is better and why?
Saying one is "better" is mostly a matter of personal preference; if it accomplishes the task it needs to, then pick whichever way you prefer. That said, I think the second one should be used, and here's why:
defining setlogstream() both makes it clear what that section of your code does, and allows you to use it again later, if you need to.
using separate except cases makes your code more readable and easier to follow. this could be especially useful if somehow another error occurred in the handling of the first.
Overall, the second one is far more readable, and your future self will thank you for writing it that way.
My code tries to do something, but it triggers an Error... which triggers another Error. So, the error message looks something like this:
SillyError: you can`t do that becuz blablabla
The above exception was the direct cause of the following exception:
LoopyError: you can`t do that becuz blobloblo
I want to create a try except block that only catches this specific duo of errors. However, I am only able to catch the first one, because once I do, the second one never gets a chance to trigger.
This other question is about catching either exception, but I want to catch only if both are triggered in succession. Is there a way?
If you have a try\except, you will always catch the error based on the outer exception. However you do have the option to pass on any exceptions you don't want to process.
In this code, the ZeroDivisionError is caught and wrapped in another exception which is then caught by the calling code. The calling code checks for the inner exception and decides whether to re-raise the exception up the stack.
def xtest():
try:
a = 1/0 # exception - division by zero
except ZeroDivisionError as e:
raise Exception("Outer Exception") from e # wrap exception (e not needed for wrap)
try:
xtest()
except Exception as ex:
print(ex) # Outer Exception
print(ex.__cause__) # division by zero
if (str(ex) == "Outer Exception" and str(ex.__cause__) == "division by zero"):
print("Got both exceptions")
else:
raise # pass exception up the stack
Just for completion, you can do the check based on the exception class name also:
if (type(ex).__name__ == "Exception" and type(ex.__cause__).__name__ == "ZeroDivisionError"):
print("Got both exceptions")
#ShadowRanger pointed out that it may be quicker to just check the class type instead of the class name:
if (type(ex) == Exception and type(ex.__cause__) == ZeroDivisionError):
print("Got both exceptions")
I am newbie to Python. I am trying to mock exception in my unit test and test my block of code; however exception message is always empty. Is this below recommended way to mock exception? Also how can I make sure exception message is not empty?
import pytest
import requests
from unittest.mock import patch
#Unit Test
#patch('requests.get')
def test_exception(mock_run):
mock_run.side_effect = requests.exceptions.ConnectionError()
with pytest.raises(SystemExit) as sys_exit:
method_to_test()
assert 'Error ' in str(sys_exit.value) # Here sys_exit.value is always empty
#Method to Test
def method_to_test():
try:
response = requests.get('some_url', verify=False, stream=True)
response.raise_for_status()
except (requests.exceptions.HTTPError,
requests.exceptions.ConnectionError,
requests.exceptions.Timeout) as err:
msg = f'Failure: {err}' # Here err is always empty
raise SystemExit(msg)
Long story short: You don't get a message because you don't specify one.
You probably want to check for 'Failure: ' instead of 'Error: ', since this is what you prefix the original exception message with. This might be the real problem in your code, not the empty string representation of the exception you raise in your test.
Why is str(err) empty?
Take a look at the class hierarchy:
BaseException
Exception
IOError
requests.RequestException
requests.ConnectionError
IOError overrides __str__ if more than one argument is given to the constructor, but otherwise the behavior of BaseException applies:
If str() is called on an instance of this class, the representation of the argument(s) to the instance are returned, or the empty string when there were no arguments.
https://docs.python.org/3/library/exceptions.html#BaseException
>>> import requests
>>> str(requests.exceptions.ConnectionError())
''
>>> str(requests.exceptions.ConnectionError('foo'))
'foo'
>>> str(requests.exceptions.ConnectionError('foo', 'bar'))
'[Errno foo] bar'
The last example is the behavior defined by the IOError exception.
If you want to simulate the exception, raise the exception in your try block, just like you're raising an exception in your except block. When you do this, you can pass a string argument to be the exception message -
raise requests.exceptions.HTTPError("Test this error")
That will filter the message to the except block.
With your example:
def method_to_test():
try:
# ADD EXCEPTION HERE
raise requests.exceptions.HTTPError('Throwing an exception here!')
response = requests.get('some_url', verify=False, stream=True)
response.raise_for_status()
except (requests.exceptions.HTTPError,
requests.exceptions.ConnectionError,
requests.exceptions.Timeout) as err:
# err will now be 'Throwing an exception here!'
msg = f'Failure: {err}' # Here err is always empty
raise SystemExit(msg)