Kotest Shrinking

In property-based testing, the first failure case found may contain a lot of complexity that is not actually responsible for making the test fail.

Shrinking

In property-based testing, the first failure case found may contain a lot of complexity that is not actually responsible for making the test fail. Shrinking is the mechanism by which a property-based testing framework simplifies a failing case to find the smallest reproducible case. In Kotest, the way generators shrink failing cases is defined by implementations of the shrinking interface. Built-in generators usually have default shrinkers defined by the framework, and custom generators can specify custom shrinker implementations.

Shrinking Built-in Generators

Built-in generators, see the generators list, have default shrink functions defined by the framework. A shrink function takes the value that caused the test to fail as input and returns a list of new values that Kotest can apply to the test. The exact behavior depends on the data type. For example, for strings, shrinking can remove the first or last character; for integers, it can reduce the value or halve it. Shrinking behavior is also defined for edge cases such as empty strings or integer zero. Shrinking is performed when a test using these generators fails.

Arb.positiveInt().checkAll { i ->
    calculateProperty(i) shouldBe true
}

If the test fails for one of the generated inputs, the shrink result is displayed:

Property test failed for inputs

0) 1792716902

Caused by io.kotest.assertions.AssertionFailedError: expected:<1792716902> but was:<0> at
    PropertyBasedTest$1$1$3$1.invokeSuspend(PropertyBasedTest.kt:54)
    PropertyBasedTest$1$1$3$1.invoke(PropertyBasedTest.kt)
    PropertyBasedTest$1$1$3$1.invoke(PropertyBasedTest.kt)
    io.kotest.property.internal.ProptestKt$proptest$3$2.invokeSuspend(proptest.kt:45)

Attempting to shrink arg 1792716902
Shrink #1: 1 pass
Shrink #2: 597572300 fail
Shrink #3: 199190766 fail
Shrink #4: 66396922 fail
Shrink #5: 22132307 fail
Shrink #6: 7377435 fail
Shrink #7: 2459145 fail

[...]

Shrink #999: 29948 pass
Shrink #1000: 44922 pass
Shrink #1001: 59896 pass
Shrink #1002: 89839 fail
Shrink result (after 1002 shrinks) => 89839

Caused by io.kotest.assertions.AssertionFailedError: expected:<89839> but was:<0> at
    PropertyBasedTest$1$1$3$1.invokeSuspend(PropertyBasedTest.kt:54)
    PropertyBasedTest$1$1$3$1.invoke(PropertyBasedTest.kt)
    PropertyBasedTest$1$1$3$1.invoke(PropertyBasedTest.kt)
    io.kotest.property.internal.ShrinkfnsKt$shrinkfn$1$1$smallestA$1.invokeSuspend(shrinkfns.kt:19)

By default, Kotest shrinks 1000 times. This behavior is configurable. For example, if you want shrinking to continue without a limit:

Arb.positiveInt().checkAll(PropTestConfig(shrinkingMode = ShrinkingMode.Unbounded)) { i ->
    calculateProperty(i) shouldBe true
}

Shrinking Custom Generators

Custom generators do not have shrinkers defined by Kotest. Instead, you can implement a custom shrinker. The example below returns the coordinates next to the value itself.

data class Coordinate(val x: Int, val y: Int)

class CoordinateTest : FunSpec({
    context("Coordinate Transformations") {
        // Shrinker takes the four neighbouring coordinates
        val coordinateShrinker = Shrinker<Coordinate> { c ->
            listOf(
                Coordinate(c.x - 1, c.y),
                Coordinate(c.x, c.y - 1),
                Coordinate(c.x + 1, c.y),
                Coordinate(c.x, c.y + 1),
            )
        }
        val coordinateArb = arbitrary(coordinateShrinker) {
            Coordinate(Arb.nonNegativeInt().bind(), Arb.nonNegativeInt().bind())
        }

        test("Coordinates are always positive after transformation") {
            coordinateArb.checkAll {
                transform(it).x shouldBeGreaterThanOrEqualTo 0
                transform(it).y shouldBeGreaterThanOrEqualTo 0
            }
        }
    }
})

References

Shrinking | Kotest