Kotest Timeouts

This page explains how to test with invocation time limits.
Tags:

Test Timeouts

Kotest supports two types of test time limits. The first is the total time for all invocations of a test. This is called a timeout. The second is per individual execution of the test, and this is called an invocation timeout.

Test Timeout

To set a test timeout, use test configuration:

class TimeoutTest : FunSpec({
   test("this test will timeout quickly!").config(timeout = 100.milliseconds) {
      // test here
   }
})

You can also apply a test timeout to every test in a spec file:

class TimeoutTest : FunSpec({

   timeout = 100.milliseconds

   test("this test will timeout quickly!") {
      // test here
   }

   test("so will this one!") {
      // test here
   }
})

Invocation Timeout

Kotest can be configured to invoke a test multiple times. For example:

class TimeoutTest : DescribeSpec({

   describe("my test context") {
        it("run me three times").config(invocations = 3) {
            // this test will be invoked three times
        }
   }

})

You can then apply a timeout per invocation by using the invocationTimeout property.

class TimeoutTest : DescribeSpec({

   describe("my test context") {
        it("run me three times").config(invocations = 3, invocationTimeout = 60.milliseconds) {
            // this test will be invoked three times and each has a timeout of 60 milliseconds
        }
   }

})

In the previous example, each invocation must complete within 60 milliseconds. You can combine this with an overall test timeout:

class TimeoutTest : DescribeSpec({

   describe("my test context") {
        it("run me three times").config(timeout = 100.milliseconds, invocations = 3, invocationTimeout = 60.milliseconds) {
            // this test will be invoked three times
        }
   }

})

Here, all three tests must complete within 100 milliseconds, while a specific invocation can take up to 60 milliseconds.

As with test timeouts, invocation timeouts can be applied at the spec level:

class TimeoutTest : FunSpec({

   invocationTimeout = 25.milliseconds

   test("foo") {
      // test here
   }

   test("bar") {
      // test here
   }
})

Project-wide Settings

You can use project configuration to apply test and/or invocation timeouts to every test in a module.

object ProjectConfig : AbstractProjectConfig {
    override val timeout = 100.milliseconds
    override val invocationTimeout = 33.milliseconds
}

These values apply unless overridden at the spec or test level.

System Properties

Both test timeouts and invocation timeouts can be set using system properties, with values specified in milliseconds.

  • kotest.framework.timeout sets the combined test timeout.
  • kotest.framework.invocation.timeout sets the invocation test timeout.

Project Timeout

Kotest supports project-level timeouts. This timeout applies to all tests in a module and includes setup and teardown time for all specs and tests in the module.

To enable it, use ProjectConfig.

class ProjectConfig : AbstractProjectConfig() {
  override val projectTimeout: Duration = 10.minutes
}

In the example above, the project timeout is set to 10 minutes. All specs and tests must complete within 10 minutes, otherwise the build fails.

Blocking Tests

When you specify a timeout in a test, Kotest uses the withTimeout coroutine function provided by the Kotlin coroutines library. These timeouts are cooperative by nature, and timeouts are detected when a coroutine suspends, resumes, or calls yield.

However, when running blocking code, the thread is blocked, so the cooperative approach does not work. In this scenario, it must fall back to interrupting the thread using Thread.interrupt or something similar. For this interrupt to work safely, the test must run on a dedicated thread.

Therefore, it is up to you to tell Kotest that a specific test should run on a dedicated thread where interruption can be used safely. Enable the blocking test flag in the test configuration.

For example:

class MyBlockingTest : FunSpec() {
  init {

    test("interrupt me!").config(blockingTest = true, timeout = 10.seconds) {
       Thread.sleep(100000000)
    }

    test("uses suspension").config(timeout = 10.seconds) {
      delay(100000000)
    }
  }
}

In the example above, the first test needs the blocking test flag because it uses a thread-blocking operation. The second test does not need it because it uses a suspendable operation.

References