I want to write a hypothesis.stateful.RuleBasedStateMachine which asserts that an exception is raised under certain circumstances. pytest provides the raises context manager for writing tests about exceptions. If I use pytest.raises inside a hypothesis.stateful.rule, the sequence of steps that lead to the test failure is not reported.
Rewriting the rule without pytest.raises results in the desired behaviour: the sequence of steps is displayed.
Here is some sample code:
from os import getenv
from pytest import raises
from hypothesis.stateful import RuleBasedStateMachine, rule
SHOW_PROBLEM = getenv('SHOW_PROBLEM') == 'yes'
# A state machine which asserts that an exception is raised in under some condition
class FifthCallShouldRaiseValueError(RuleBasedStateMachine):
def __init__(self):
super().__init__()
self.model = Model()
self.count = 0
if SHOW_PROBLEM:
# This version does NOT report the rule sequence
#rule()
def the_rule(self):
self.count += 1
if self.count > 4:
with raises(ValueError):
self.model.method()
else:
# This version DOES report the rule sequence
#rule()
def the_rule(self):
self.count += 1
if self.count > 4:
try:
self.model.method()
except ValueError: assert True
except : assert False
else : assert False
T = FifthCallShouldRaiseValueError.TestCase
# A model that deliberately fails the test, triggering reporting of
# the sequence of steps which lead to the failure.
class Model:
def __init__(self):
self._count = 0
def method(self):
self._count += 1
if self._count > 4:
# Deliberate mistake: raise wrong exception type
raise TypeError
To observe the difference in behaviour, exectue the test with
SHOW_PROBLEM=yes pytest <...>
SHOW_PROBLEM=no pytest <...>
In the second case the output will show
state = FifthCallShouldRaiseValueError()
state.the_rule()
state.the_rule()
state.the_rule()
state.the_rule()
state.the_rule()
state.teardown()
This sequence of steps is missing from the output in the first case. This is usdesirable: the sequence should be show in both cases.
pytest.raises raises Failed: DID NOT RAISE <class 'ValueError'> while the hand-written version raises AssertionError. The former is more informative when it comes to the failure to raise the required exception, but somehow seems to prevent hypothesis.stateful from reporting the sequence of steps, which tells us how we got into that state, and is often the most interesting part of the output.
What can be done to mitigate this, i.e. to ensure that the sequence of steps is printed out, besides not using pytest.raises?
It turns out that the steps are not printed if a rule raises BaseException or a non-Exception subclass. pytest.raises(...) raises just such an error if it doesn't get the expected exception, and there you are.
https://github.com/HypothesisWorks/hypothesis/issues/1372
It's not a particularly gnarly bug now that it's been identified - and thanks for your part in that, by reporting a reproducible case! - so we should get a fix out soon.
Update: this bug was fixed in Hypothesis 3.65.1, on 2018-07-03.
Related
I am trying to learn test driven development but I ran into such a silly issue and for the life of me I do not understand what is going on. I am following the book clean craftsmanship and am on the bowling example where classes get tested, this book is in java but I rather do python. Here is the example code I have written:
class BowlingGame:
def __init__(self) -> None:
self.frames = [Frame for _ in range(10)]
class Frame:
def __init__(self) -> None:
self.rolls = [Roll]
class Roll:
pass
and the test suit is:
from bowling import BowlingGame, Frame
game = BowlingGame()
def test_game_has_ten_frames():
assert len(game.frames) == 10
assert type(game.frames[0]) == type(Frame)
assert type(game.frames[9]) == type(Frame)
def test_frame_has_atleast_one_roll():
len(game.frames[0].rolls) == 1
the results give:
collected 2 items
test_bowling.py .F [100%]
===================================================================================== FAILURES ======================================================================================
__________________________________________________________________________ test_frame_has_atleast_one_roll __________________________________________________________________________
def test_frame_has_atleast_one_roll():
> len(game.frames[0].rolls) == (1)
E AttributeError: type object 'Frame' has no attribute 'rolls'
test_bowling.py:13: AttributeError
============================================================================== short test summary info ==============================================================================
FAILED test_bowling.py::test_frame_has_atleast_one_roll - AttributeError: type object 'Frame' has no attribute 'rolls'
I have tested that it is of type frame in the first test. Yet it claims that Frame doesn't have rolls but this doesn't make any sense to me as its the only attribute it has.
Can someone please explain what is happening before I lose my mind?
I have made a Frame class outside of the game and it was able to locate the rolls so I know the attribute exists. Yet when I try to access it from the game class this is when issues are occurring. What in the python is happening ?????
let's say that i have a bunch of instruction which might all raise exceptions and I'd like to simply ignore those that fail.
failable_1
failable_2
...
failable_n
Ignoring them with the usual exception pattern might quickly become cumbersome:
try:
failable_1
except SomeError:
pass
try:
failable_2
except SomeError:
pass
...
try:
failable_n
except SomeError:
pass
This is especially true if it is about declaring a list of possibly non existing symbols:
my_list=[optional_1, optional_2, ..., optional_n]
(Let's axiomatically assume that somewhere else in the code, there was something like:
for (var,val) in zip(optional_variable_names_list, values):
exec(var+"="+repr(val))
)...
Because in this case, you cannot even write the name in the code.
my_list=[]
for variable in [optional_1, optional_2, ..., optional_n]: # problem remains here
try:
my_list.append(variable)
except:
pass
wouldn't work. You have to use eval():
my_list=[]
for variable in ["optional_1", "optional_2", ..., "optional_n"]:
try:
my_list.append(eval(variable))
except:
pass
So my question is :
Isn't there a way to write something like the on error next or on error ignore that existed in some old time languages. some kind of :
ignore SomeError:
failable_1
failable_2
...
failable_n
or
ignore NameError:
my_list=[optional_1, optional_2, ..., optional_n]
And if not, why would it be a bad idea ?
You can simplify the pattern somewhat by using a context manager that suppresses an exception by returning True if the exception that occurs is one the those specified with the constructor, or otherwise re-raises the exception:
class Ignore:
def __init__(self, *ignored_exceptions):
self.ignored_exceptions = ignored_exceptions
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
if isinstance(exc_value, self.ignored_exceptions):
return True
raise
so that:
with Ignore(NameError, ValueError):
a = b
outputs nothing, while:
with Ignore(NameError, ValueError):
1 / 0
raises ZeroDivisionError: division by zero as expected.
Was experimenting with applying memoization decorator to recursive functions using standard python #decorator notation, which worked beautifully. According to much of the documentation I've read, these code examples are supposed to be equivalent:
# letting python decorator mechanism take care of wrapping your function with the decorator function
#decorator
def func(): . . .
print(func(arg))
AND
# manually wrapping the decorator function around the function
func = decorator(func)
print(func(arg))
When I do this, however, I get an UnboundLocalError on func.
If I change the code to
new_name = decorator(func)
print(new_name(func))
The code runs, but the decorator is only applied to the first call and not to any of the recursive calls (this did NOT surprise me), but I don't get any error messages either.
What seems weird to me, however, is the original error message itself.
If I experiment further and try the following code:
new_name = decorator(func)
func = new_name
print(func(arg))
I get the same error ON THE SAME LINE AS BEFORE (????)
In fact, if I form a chain of these assignments, with func = several_names_later
I still get the same error on the ORIGINAL line
Can anyone explain what is going on, and why I'm getting the error, and why it seems that the error is disconnected from the location of the variable in question?
As requested, here is most of the actual code (with just one of the recursive functions), hope it's not too much . . .
import functools
def memoize(fn):
cache = dict()
#functools.wraps(fn)
def memoizer(*args):
print(args)
if args not in cache:
cache[args] = fn(*args)
return cache[args]
return memoizer
##memoize
def number_sum(n):
'''Returns the sum of the first n numbers'''
assert(n >= 0), 'n must be >= 0'
if n == 0:
return 0
else:
return n + number_sum(n-1)
def main():
# Book I'm reading claims this can be done but I get error instead:
number_sum = memoize(number_sum) # this is the flagged line
print(number_sum(300))
#UnboundLocalError: local variable 'number_sum' referenced before assignment
# When I do this instead, only applies decorator to first call, as I suspected
# but works otherwise: no errors and correct - but slow - results
# No complaints about number_sum
wrapped_number_sum = memoize(number_sum)
print(wrapped_number_sum(300))
# This is what is odd:
# When I do any version of this, I get the same error as above, but always on
# the original line, flagging number_sum as the problem
wrapped_number_sum = memoize(number_sum) # the flagged line, no matter what
number_sum = wrapped_number_sum
print(number_sum(300))
OR even:
wrapped_number_sum = memoize(number_sum) # still the flagged line
another_variable = wrapped_number_sum
number_sum = another_variable
print(number_sum(300))
if __name__ == '__main__':
main()
I am more than a little mystified at this.
Consider you have something like the following:
import pytest
#pytest.fixture
def foo_fixture():
assert x == 1, "X is not 1"
if x != 1:
pytest.fail("X is not 1")
def test_foo():
assert x == 1, "X is not 1"
if x != 1:
pytest.fail("X is not 1")
Can someone please shed light on differences between raising errors via pytest.fail vs assert,
in both contexts - inside pytest.fixture or inside regular test function.
What is the difference, what is more right\common to do in each case?
Thank in advance
The biggest difference is that pytest.fail() can't be caught and assert can:
for example - this test is going to pass:
def test_assert():
try:
assert False
except:
pass
and this test is going to fail:
def test_pytest_fail():
try:
pytest.fail('failed')
except Exception:
pass
Other than that I don't think there are more major differences because they both raise an AssertionError.
Anyway, I would suggest you use this library https://pypi.org/project/assertpy/ for more cleaner and readable assertions.
**** EDIT: ****
Guess I was wrong:
pytest.fail() can be caught by catching the specific exception Failed or just 'BaseException.
So - they both are ways to raise exceptions but from some reason Failed does not inherit from the generic Exception type.
so pytest.fail() raises Failed exception and not AssertionError
I want to define a custom Exception class for the NoneType error in python inheriting from Exception class. I expect the code below to do such a thing for me but no success. Is there any way to do this?
class NoneTypeException(Exception):
pass
try:
image = cv2.imread("/path/")
gray = cv2.cvtColor(image.copy(), cv2.COLOR_BGR2GRAY) #exception should be raised here
...
except NoneTypeException:
raise NoneTypeException("Image not found") # and should be caught here
I want to the try except block raise the NoneTypeException. Any way?
While you can declare an exception to represent whatever you want, existing code won't raise it for you. In specific, operations on None already raise some well-defined errors and you cannot replace them. Strictly speaking, you cannot get the desired behaviour.
If you know that a certain section of code is vulnerable to None values, you can catch the generic Exception and raise a specific one. This hinges on the assumption that None is the only bogus value you might get:
class NoneTypeException(Exception):
pass
try:
image = cv2.imread("/path/")
gray = cv2.cvtColor(image.copy(), cv2.COLOR_BGR2GRAY)
...
except AttributeError as err: # handle generic exception from action on ``None``
raise NoneTypeException("Image not found")
Take note that this is rarely what you should be doing - the None value is a symptom, not the cause of the error. It would be more appropriate to raise a FileNotFoundError.
Since None can trigger errors anywhere, it is easier to protect against it at its origin. Checking for identity with None is very cheap:
try:
image = cv2.imread("/path/")
if image is None: # verify value immediately
raise NoneTypeException()
# safely work with image
gray = cv2.cvtColor(image.copy(), cv2.COLOR_BGR2GRAY)
...
except NoneTypeException:
raise FileNotFoundError('image not found')