I know that we can use ErrorCollector or soft assertions (AssertJ or TestNG) that do not fail a unit test immediately.
How they can be used with Mockito assertions? Or if they can't, does Mockito provide any alternatives?
Code sample
verify(mock).isMethod1();
verify(mock, times(1)).callMethod2(any(StringBuilder.class));
verify(mock, never()).callMethod3(any(StringBuilder.class));
verify(mock, never()).callMethod4(any(String.class));
Problem
In this snippet of code if a verification will fail, then the test will fail which will abort the remaining verify statements (it may require multiple test runs until all the failures from this unit test are revealed, which is time-consuming).
Since Mockito 2.1.0 you can use VerificationCollector rule in order to collect multiple verification failures and report at once.
Example
import static org.mockito.Mockito.verify;
import org.junit.Rule;
import org.mockito.junit.MockitoJUnit;
import org.mockito.junit.VerificationCollector;
// ...
#Rule
public final VerificationCollector collector = MockitoJUnit.collector();
#Test
public void givenXWhenYThenZ() throws Exception {
// ...
verify(mock).isMethod1();
verify(mock, times(1)).callMethod2(any(StringBuilder.class));
verify(mock, never()).callMethod3(any(StringBuilder.class));
verify(mock, never()).callMethod4(any(String.class));
}
Known issues
This rule cannot be used with ErrorCollector rule in the same test method. In separate tests it works fine.
Using Soft assertions you could do :
softly.assertThatThrownBy(() -> verify(mock).isMethod1()).doesNotThrowAnyException();
softly.assertThatThrownBy(() -> verify(mock, times(1)).callMethod2(any(StringBuilder.class))).doesNotThrowAnyException();
softly.assertThatThrownBy(() -> verify(mock, never()).callMethod3(any(StringBuilder.class))).doesNotThrowAnyException();
softly.assertThatThrownBy(() -> verify(mock, never()).callMethod4(anyString())).doesNotThrowAnyException();
If one or more of your mockito assertions fail, it will trigger an exception, and softAssertion will do the reporting job.
Related
We are trying to define a UnitTest where we mock an object which I here called x for simplicity:
...
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.doNothing;
import org.kubek2k.springockito.annotations.SpringockitoContextLoader;
import org.kubek2k.springockito.annotations.WrapWithSpy;
...
#ContextConfiguration(
loader = SpringockitoContextLoader.class,
inheritLocations = true)
public class SyncServiceIntegrationTest extends AbstractIntegrationTest {
#Autowired
#WrapWithSpy
private EventDrivenIssueDeliveryConfirmer x;
...
#Before
public void setUp() {
...
doNothing().when(x).foobar(any(Event.class));
}
...
i.e. we want our UT (not shown here) to later NOT call the method foobar on that object x.
Strange enough we get an NPE during initialization of this UT-class. The NPE is thrown by method foobar(), when the passed argument is null.
As turned out this call with argument null happens in the line doNothing()... in the setup-method which in our understanding is supposed to just define the mock-object's stubbing. But instead it evaluates the any(Event.class)-expression which apparently yields null and with that result it then calls the foobar(...)-method on x which causes the NPE.
Besides the NullPointerException we also get an error message from Mockito:
java.lang.NullPointerException: null
... <stack trace omitted for brevity>
org.mockito.exceptions.misusing.InvalidUseOfMatchersException:
Misplaced or misused argument matcher detected here:
-> at ch.sst.integration.SyncServiceIntegrationTest .setUp(SyncServiceIntegrationTest.java:69)
You cannot use argument matchers outside of verification or stubbing.
Examples of correct usage of argument matchers:
... <examples omitted for brevity>
org.mockito.exceptions.misusing.UnfinishedStubbingException:
Unfinished stubbing detected here:
-> at ch.sst.integration.SyncServiceIntegrationTest .setUp(SyncServiceIntegrationTest.java:69)
...
Why is that so??? Why is our stubbing considered "unfinished"? What are we missing here?
Later addition:
The issue seems to have to do with the fact that class
EventDrivenIssueDeliveryConfirmer is marked with #Transactional. Removing/commenting that lets the UT succeed. But of course that's no workaround - we need that annotation.
At least this provides a hint in which direction to search. The wrapping caused by #Transactional and the wrapping done by Mockito seem to step on each other's foot here.
I have the same issue but with a totally different setup: kotlin, mockito and, of course, mockito-kotlin.
I comment on this issue because maybe somebody will come to this question with the kotlin mockito problem in the future? I sure did. Anyhow.
When not declaring a method as open in kotlin it's compiled as a final method which can't be mocked by mockito-kotlin. As a result the method gets executed which to me is kind of weird but that's what it does. It's mentioned in the mockito-kotlin github issues under https://github.com/mockito/mockito-kotlin/issues/314
Im using Spock for my tests and I have multiple classes to test. I want to tell Spock to test each class in specific order. Is there a way to do this? I noticed TestNG has got #AfterTest annotation, so does Spock have something similar?
You can't specify Spock test classes execution order. However Spock allows you to specify methods execution order when you add #Stepwise annotation to your test class - in this case Spock will execute all methods in order from top to bottom and if one method fails it will ignore remaining methods in this test class.
Indicates that a spec's feature methods should be run sequentially in their declared order (even in the presence of a parallel spec runner), always starting from the first method. If a method fails, the remaining methods will be skipped. Feature methods declared in super- and subspecs are not affected.
#Stepwise is useful for specs with (logical) dependencies between methods. In particular, it helps to avoid consecutive errors after a method has failed, which makes it easier to understand what really went wrong.
Reference: http://spockframework.org/spock/javadoc/1.1/spock/lang/Stepwise.html
#Stepwise
class StepwiseExample extends Specification {
def "first test method to run"() {
// ....
}
def "second test method to run"() {
// ....
}
def "if previous method failed this one will be ignored"() {
// ....
}
}
Using org.junit.runners.Suite
Jeff Scott Brown gave a good comment about JUnit's #Suite.SuiteClasses. In this case you create a class where you can aggregate your test classes (specifications) into a single test suite and those classes will be executed in the order they have been defined in the suite. Consider following example:
import org.junit.runner.RunWith
import org.junit.runners.Suite
#RunWith(Suite)
#Suite.SuiteClasses([
Test2Specification,
Test1Specification
])
class TestSuiteSpecification { }
This suite executes two specifications Test2Specification and Test1Specification in the defined order:
In the http://mockito.org/ website the following sample code is given to describe the usage of the mockito framework. However i am not able to understand what the following code is trying to test. Please comment on the following code as what is it exactly testing. If it is testing the add method of the List, what does it expect?
import static org.mockito.Mockito.*;
// mock creation
List mockedList = mock(List.class);
// using mock object - it does not throw any "unexpected interaction" exception
mockedList.add("one");
mockedList.clear();
// selective, explicit, highly readable verification
verify(mockedList).add("one");
verify(mockedList).clear();
In your case, you've a test preparation
// mock creation
List mockedList = mock(List.class);
a test execution
// using mock object - it does not throw any "unexpected interaction" exception
mockedList.add("one");
mockedList.clear();
and a test verification (without mockito mostly done with something like assertEquals)
// selective, explicit, highly readable verification
verify(mockedList).add("one");
verify(mockedList).clear();
The last block (the test verifaction) is testing, if
the add method of the mockedList was called with a parameter that equals the String "one"
the clear method of the mockedList was called
It is just a basic demo of how the interaction with a mocked class can be verified (since you cannot verify the contents of the List because it has been mocked and the calls of add("one") and clear() don't change the contents of the List).
Thanks in advance for the help -
I am new to mockito but have spent the last day looking at examples and the documentation but haven't been able to find a solution to my problem, so hopefully this is not too dumb of a question.
I want to verify that deleteLogs() calls deleteLog(Path) NUM_LOGS_TO_DELETE number of times, per path marked for delete. I don't care what the path is in the mock (since I don't want to go to the file system, cluster, etc. for the test) so I verify that deleteLog was called NUM_LOGS_TO_DELETE times with any non-null Path as a parameter. When I step through the execution however, deleteLog gets passed a null argument - this results in a NullPointerException (based on the behavior of the code I inherited).
Maybe I am doing something wrong, but verify and the use of isNotNull seems pretty straight forward...here is my code:
MonitoringController mockController = mock(MonitoringController.class);
// Call the function whose behavior I want to verify
mockController.deleteLogs();
// Verify that mockController called deleteLog the appropriate number of times
verify(mockController, Mockito.times(NUM_LOGS_TO_DELETE)).deleteLog(isNotNull(Path.class));
Thanks again
I've never used isNotNull for arguments so I can't really say what's going wrong with you code - I always use an ArgumentCaptor. Basically you tell it what type of arguments to look for, it captures them, and then after the call you can assert the values you were looking for. Give the below code a try:
ArgumentCaptor<Path> pathCaptor = ArgumentCaptor.forClass(Path.class);
verify(mockController, Mockito.times(NUM_LOGS_TO_DELETE)).deleteLog(pathCaptor.capture());
for (Path path : pathCaptor.getAllValues()) {
assertNotNull(path);
}
As it turns out, isNotNull is a method that returns null, and that's deliberate. Mockito matchers work via side effects, so it's more-or-less expected for all matchers to return dummy values like null or 0 and instead record their expectations on a stack within the Mockito framework.
The unexpected part of this is that your MonitoringController.deleteLog is actually calling your code, rather than calling Mockito's verification code. Typically this happens because deleteLog is final: Mockito works through subclasses (actually dynamic proxies), and because final prohibits subclassing, the compiler basically skips the virtual method lookup and inlines a call directly to the implementation instead of Mockito's mock. Double-check that methods you're trying to stub or verify are not final, because you're counting on them not behaving as final in your test.
It's almost never correct to call a method on a mock directly in your test; if this is a MonitoringControllerTest, you should be using a real MonitoringController and mocking its dependencies. I hope your mockController.deleteLogs() is just meant to stand in for your actual test code, where you exercise some other component that depends on and interacts with MonitoringController.
Most tests don't need mocking at all. Let's say you have this class:
class MonitoringController {
private List<Log> logs = new ArrayList<>();
public void deleteLogs() {
logs.clear();
}
public int getLogCount() {
return logs.size();
}
}
Then this would be a valid test that doesn't use Mockito:
#Test public void deleteLogsShouldReturnZeroLogCount() {
MonitoringController controllerUnderTest = new MonitoringController();
controllerUnderTest.logSomeStuff(); // presumably you've tested elsewhere
// that this works
controllerUnderTest.deleteLogs();
assertEquals(0, controllerUnderTest.getLogCount());
}
But your monitoring controller could also look like this:
class MonitoringController {
private final LogRepository logRepository;
public MonitoringController(LogRepository logRepository) {
// By passing in your dependency, you have made the creator of your class
// responsible. This is called "Inversion-of-Control" (IoC), and is a key
// tenet of dependency injection.
this.logRepository = logRepository;
}
public void deleteLogs() {
logRepository.delete(RecordMatcher.ALL);
}
public int getLogCount() {
return logRepository.count(RecordMatcher.ALL);
}
}
Suddenly it may not be so easy to test your code, because it doesn't keep state of its own. To use the same test as the above one, you would need a working LogRepository. You could write a FakeLogRepository that keeps things in memory, which is a great strategy, or you could use Mockito to make a mock for you:
#Test public void deleteLogsShouldCallRepositoryDelete() {
LogRepository mockLogRepository = Mockito.mock(LogRepository.class);
MonitoringController controllerUnderTest =
new MonitoringController(mockLogRepository);
controllerUnderTest.deleteLogs();
// Now you can check that your REAL MonitoringController calls
// the right method on your MOCK dependency.
Mockito.verify(mockLogRepository).delete(Mockito.eq(RecordMatcher.ALL));
}
This shows some of the benefits and limitations of Mockito:
You don't need the implementation to keep state any more. You don't even need getLogCount to exist.
You can also skip creating the logs, because you're testing the interaction, not the state.
You're more tightly-bound to the implementation of MonitoringController: You can't simply test that it's holding to its general contract.
Mockito can stub individual interactions, but getting them consistent is hard. If you want your LogRepository.count to return 2 until you call delete, then return 0, that would be difficult to express in Mockito. This is why it may make sense to write fake implementations to represent stateful objects and leave Mockito mocks for stateless service interfaces.
Is there any way to continue executing Cucumber Steps even when one of the steps fails. In my current setup when a step fails , cucumber skips remaining steps....I wonder if there is some way to twick cucumber runner setup..
I could comment out failing steps but its not practical when you dont know which step will fail...If i could continue with remaining step i would know complete set of failing Tests in one shot....rather than going in cycle over cycle...
Environment: Cucumber JVM , R , Java , Ibatis , Spring Framework, Maven
It is not a good idea to continue executing steps after a step failure because a step failure can leave the World with an invariant violation. A better strategy is to increase the granularity of your scenarios. Instead of writing a single scenario with several "Then" statements, use a list of examples to separately test each postconditions. Sometimes a scenario outline and list of examples can consolidate similar stories. https://docs.cucumber.io/gherkin/reference/#scenario-outline
There is some discussion about adding a feature to tag certain steps to continue after failure. https://github.com/cucumber/cucumber/issues/79
One way would be to catch all the assertion errors and decide in the last step whether to fail or pass the test case. In this case, you can tailor it, say, check at any step to see if there is more than n errors and fail the test, if so.
Here's what I have done:
initialize a StringBuffer for Errors in your #Before for the test cases
catch the Assertion Errors and add to the StringBuffer, so that, they do not get thrown and terminate the test case.
Check the StringBuffer to determine whether to fail the test case.
StringBuffer verificationErrors;
// Initialize your error SringBuffer here
#Before
public void initialize() {
verificationErrors = new StringBuffer();
}
// The following is one of the steps in the test case where I need to assert something
#When("^the value is (\\d+)$")
public void the_result_should_be(int arg1) {
try {
assertEquals(arg1, 0);
}
catch(AssertionError ae) {
verificationErrors.append("Value is incorrect- "+ae.getMessage());
}
Check the StringBuffer in #After or in the last step of test case to determine if you can pass it or fail it, as follows:
if (!(verificationErrors.size()==0)) {
fail(verificationErrors.toString());
}
The only issue would be that, in the report, all the steps would look green but the test case looks failed. Then you might have to look through the Errors String to know which step(s) failed. You could add extra information to the String whenever there is an Assertion Error to help you locate the step easily.
Use SoftAssert to accumulate all assertion failures. Then tag your step definitions class as #ScenarioScoped and in step definitions class add a method tagged #After where you do mySoftAssert.assertAll();
i.e.
import io.cucumber.guice.ScenarioScoped;
import io.cucumber.java.After;
import io.cucumber.java.Before;
import io.cucumber.java.en.Then;
#ScenarioScoped
public class MyStepDefinitions {
SoftAssert mySoftAssert=new SoftAssert();
#Then("check something")
public void checkSomething() {
mySoftAssert.assertTrue(actualValue>expectedMinValue);
}
#After
public void afterScenario(Scenario scenario) throws Exception {
mySoftAssert.assertAll();
}
}