Kotest Test Ordering

Ordering in Kotest is a feature for controlling the order of test execution. It is useful when you want to run tests in a specific order or filter and run tests according to specific conditions.
Tags:

Controlling Test Order

Kotest can control test order in several ways. By using the Ordering feature to adjust tests, you can prevent problems caused by dependencies or interactions between tests and build a more predictable test execution environment.

There are two main ways to control Ordering in Kotest:

  1. Spec Ordering: Controls the execution order of tests inside a specific test spec or test class. This lets you control the order of tests within a spec.
  2. Test Ordering: Controls the execution order of tests within a specific test spec or between specific test specs. This lets you control execution order across multiple test specs.

Spec Ordering

By default, the order of spec classes is not defined. In other words, it is effectively random depending on the order in which the discovery mechanism finds them.

Even without ordering this is usually not a major issue, but if you need to control the execution order of specs, specify the order in the project configuration.

class OrderingConfig: AbstractProjectConfig() {
    override val specExecutionOrder = SpecExecutionOrder.Undefined
}

SpecExecutionOrder has several options:

  • Undefined: The default. The order is not defined, and specs run in the order they are discovered at runtime. For example, they may run in the order discovered on the JVM classpath or in the order shown in JavaScript files.
  • Lexicographic: Sorts specs lexicographically.
  • Random: Explicitly runs specs in a random order.
  • Annotated: Uses the @Order annotation added to classes, running from the lowest value first. If this annotation is missing, the class is considered “last”. This option works only on the JVM. Classes with the same value run in an arbitrary order.

Ordering Lexicographically

The following example is defined to run in lexicographic order.

package com.devkuma.kotest.tutorial.ordering

import io.kotest.core.config.AbstractProjectConfig
import io.kotest.core.spec.SpecExecutionOrder
import io.kotest.core.spec.style.FunSpec

class LexicographicConfig: AbstractProjectConfig() {
    override val specExecutionOrder = SpecExecutionOrder.Lexicographic
}

class AlphaTest : FunSpec({
    test("alpha") {
        println("AlphaTest")
    }
})

class BetaTest : FunSpec({
    test("beta") {
        println("BetaTest")
    }
})

class GammaTest : FunSpec({
    test("gamma") {
        println("GammaTest")
    }
})


class DeltaTest : FunSpec({
    test("delta") {
        println("DeltaTest")
    }
})

When the test runs, it prints as follows:

AlphaTest
BetaTest
DeltaTest
GammaTest

You can confirm that it prints in alphabetical order: AlphaTest, BetaTest, DeltaTest, GammaTest.

Ordering with Annotated

The following example is annotated with @Order.

package com.devkuma.kotest.tutorial.ordering

import io.kotest.core.config.AbstractProjectConfig
import io.kotest.core.spec.Order
import io.kotest.core.spec.SpecExecutionOrder
import io.kotest.core.spec.style.FunSpec

class AnnotatedConfig: AbstractProjectConfig() {
    override val specExecutionOrder = SpecExecutionOrder.Annotated
}

@Order(1)
class FooTest : FunSpec({
    test("foo") {
        println("FooTest")
    }
})

@Order(0)
class BarTest: FunSpec({
    test("bar") {
        println("BarTest")
    }
})

@Order(1)
class FarTest : FunSpec({
    test("far") {
        println("FarTest")
    }
})

class BooTest : FunSpec({
    test("boo") {
        println("BooTest")
    }
})

When the test runs, it prints as follows:

BarTest
FarTest
FooTest
BooTest

BarTest, which has the lowest Order value, runs first. FarTest and FooTest have the next lowest value, so they run next, but since both values are 1, their order relative to each other is not defined. Finally, BooTest has no annotation, so it runs last.

Test Ordering

When running multiple tests according to a spec, there is a defined way they run.

By default, sequential order is used, meaning tests run in the order defined in the spec, but they can also be configured to run in random or lexicographic order.

This setting can be configured by overriding the testCaseOrder function in a Spec or ProjectConfig. If both exist, the Spec setting takes precedence.

Sequential Order

The example below runs tests in the order they are defined:

package com.devkuma.kotest.tutorial.ordering.testordering

import io.kotest.core.spec.style.StringSpec
import io.kotest.core.test.TestCaseOrder

class SequentialSpecTest : StringSpec() {

    override fun testCaseOrder(): TestCaseOrder = TestCaseOrder.Sequential

    init {
        "foo" {
            // I run first as I'm defined first
            println("foo")
        }

        "bar" {
            // I run second as I'm defined second
            println("bar")
        }
    }
}

Random Order

The example below runs tests in random order:

package com.devkuma.kotest.tutorial.ordering.testordering

import io.kotest.core.spec.style.StringSpec
import io.kotest.core.test.TestCaseOrder

class RandomSpecTest : StringSpec() {

    override fun testCaseOrder(): TestCaseOrder = TestCaseOrder.Random

    init {
        "foo" {
            // This test may run first or second
            println("foo")
        }

        "bar" {
            // This test may run first or second
            println("bar")
        }
    }
}

When the test runs, it may print as follows:

foo
bar

Or it may print as follows:

bar
foo

Lexicographic Order

The example below runs tests in lexicographic order:

package com.devkuma.kotest.tutorial.ordering.testordering

import io.kotest.core.spec.style.StringSpec
import io.kotest.core.test.TestCaseOrder

class LexicographicSpecTest : StringSpec() {

    override fun testCaseOrder(): TestCaseOrder = TestCaseOrder.Lexicographic

    init {

        "beta" {
            // I run second as gamma < beta < alpha
            println("beta")
        }

        "alpha" {
            // I run first as gamma < beta < alpha
            println("alpha")
        }

        "gamma" {
            // I run third as gamma < beta < alpha
            println("gamma")
        }
    }
}

When the test runs, it prints as follows:

alpha
beta
gamma

References