I have to restart IDE to run tests - android-studio

Question

Weird bug with Android Studio 2.2 when running a single test directly in IDE - I'll either have "No tests found" or run my previously compiled test.
Example:
#Test
fun testyTest() {
}
I'll create this test, and get the "No tests found". So I restart, and test is found! Obviously, the above test passes because it's empty. Next, I want my test to fail. So I manually throw an exception and add one line of code, below.
#Test
fun testyTest() {
throw RuntimeException()
}
Now I recompile, and this test passes too, because IntelliJ is for some reason has cached the previous test and reruns that. To get this test to properly fail - I must restart the IDE.
Anyone know what's going on? Note - I'm using Kotlin pretty heavily in this project.
EDIT 2: This is a JVM test (non-android, src/test/java) in an android library project.
EDIT: here's a full test class
package com.example.zak
import org.junit.Before
import org.junit.Test
class ExampleTest {
#Before
fun setUp() {
}
#Test
fun testyTest() {
//throw RuntimeException()
}
}

Answer 1

Found the fix...the issue only occurs when there's a variant selection conflict. Removing the conflict fixed the issue.

Answer 2

Check if you're have all these dependencies:
dependencies {
compile "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
testCompile 'junit:junit:4.11'
testCompile "org.jetbrains.kotlin:kotlin-test-junit:$kotlin_version"
}
Also as you're using Kotlin please disable entirely Instant Run in File -> Preferences.
EDIT: You're talking about tests in library - do modules and project has these dependencies and proper configuration.
It looks like after Gradle rebuilding, test works fine.

Answer 3

I found this bug description in the JetBrains IntelliJ bug tracking system. Here is the URL:
https://youtrack.jetbrains.com/issue/IDEA-149275
It discusses a change that was made to classloading that may impact Junit test execution. A workaround is suggested by adding a line to the options.xml config file. Here is an example workaround for Windows boxes - change as necessary for your platform. The version of Android below is the current 2.2 release:
1) exit Android studio
2) navigate to the config/options directory for your Android version:
e.g. C:\Users\YourNameHere\.AndroidStudio2.2\config\options
3) then add the "<property name="idea.dynamic.classpath.jar" value="false" />" line inside the component block of the options.xml file:
<component name="PropertiesComponent">
<property name="idea.dynamic.classpath.jar" value="false" />
Start Android Studio and try running local Junit tests.

Answer 4

I guess you use gradle in your project? I had the exact same problem and the solution was to use gradle to execute tests (inside the gradle task list) instead of using the menu (right click on the test file and run tests).