Kotest Non-deterministic Testing

Eventually

Testing non-deterministic code can be difficult. You may need to consider threads, timeouts, race conditions, and the unpredictability of when events occur.

For example, when testing whether an asynchronous file write has completed successfully, you need to wait until the write operation finishes and is flushed to disk.

Some common approaches to these problems are:

• Use a callback that is invoked when the operation completes. You can then use the callback to verify that the system state is as expected. However, not all operations provide callback functionality.

• Use Thread.sleep to stop a thread, or delay to suspend a function and wait until the operation completes. The sleep threshold must be set high enough so the operation can finish on both fast and slow computers. Also, even if the code completes quickly on a fast computer, the test still waits for the timeout.

• Use a loop with retry after sleep, but you need to write boilerplate such as tracking the number of iterations, handling specific exceptions while failing on others, and checking that the total elapsed time has not exceeded the maximum.

• Use a countdown latch and block the thread until the latch is released by the non-deterministic operation. If you can insert the latch in the right place, this can work well, but like callbacks, it is not always possible to integrate the code under test with a latch.

As an alternative to the solutions above, Kotest provides the eventually function for the common use case: “I expect this code to pass after a short time.”

The eventually function periodically invokes the given lambda while ignoring specified exceptions until the lambda passes, the timeout is reached, or too many iterations have elapsed. It is flexible and suitable for testing non-deterministic code. You can customize it by specifying which exception types to handle, how to determine lambda success or failure, whether to use listeners, and more.

API

There are two ways to use eventually. The first is to provide a duration using Kotlin’s Duration type and then enter the code that should eventually pass without throwing an exception.

For example:

eventually(5.seconds) {
  userRepository.getById(1).name shouldBe "bob"
}

The second method is to provide a configuration block. Use this when you need to set more options than just a duration. You can also share the configuration across multiple eventually calls.

For example:

val config = eventuallyConfig {
  duration = 1.seconds
  interval = 100.milliseconds
}

eventually(config) {
  userRepository.getById(1).name shouldBe "bob"
}

Configuration Options

Duration and Interval

The duration is the total time to keep trying for the test to pass. The interval lets you specify how often to try the test. Therefore, if the duration is 5 seconds and the interval is 250 ms, the test is attempted up to 5000/250 = 20 times.

val config = eventuallyConfig {
  duration = 5.seconds
  interval = 250.milliseconds
}

Alternatively, instead of specifying a fixed number for the interval, you can pass a function. This lets you perform a kind of backoff or any other required behavior.

For example, you can use a Fibonacci-increasing interval starting at 100 ms:

val config = eventuallyConfig {
  duration = 5.seconds
  intervalFn = 100.milliseconds.fibonacci()
}

Initial Delay

Usually, the test block starts executing immediately, but you can add an initial delay before the first iteration by using initialDelay as follows:

val config = eventuallyConfig {
  initialDelay = 1.seconds
}

Retries

In addition to limiting the number of calls by time, you can also limit them by the number of iterations. In the following example, the operation is retried 10 times or until 8 seconds expire.

val config = eventuallyConfig {
  initialDelay = 8.seconds
  retries = 10
}

eventually(config) {
  userRepository.getById(1).name shouldBe "bob"
}

Specifying Exceptions to Trap

By default, eventually ignores all AssertionErrors thrown inside the function, meaning it does not catch Error. More specifically, you can configure eventually to ignore certain exceptions and immediately fail the test for others. We call these expected exceptions.

For example, when testing whether a user should exist in a database, a UserNotFoundException may be thrown if the user does not exist. You know that the user will eventually exist. However, if an IOException occurs, it indicates a problem bigger than timing, so you do not want to keep retrying.

To do this, specify UserNotFoundException as the exception to suppress.

val config = eventuallyConfig {
  duration = 5.seconds
  expectedExceptions = setOf(UserNotFoundException::class)
}

eventually(config) {
  userRepository.getById(1).name shouldBe "bob"
}

Instead of passing a set of exceptions, you can provide a function that is called with the thrown exception. This function should return true if the exception should be ignored, and false if the exception should bubble up.

val config = eventuallyConfig {
  duration = 5.seconds
  expectedExceptions = { it is UserNotFoundException }
}

eventually(config) {
  userRepository.getById(1).name shouldBe "bob"
}

Listeners

You can attach a listener that is called on each iteration with the current iteration count and the exception that caused that iteration to fail. Note that the listener is not executed when the call succeeds.

val config = eventuallyConfig {
  duration = 5.seconds
  listener = { k, throwable -> println("Iteration $k, with cause $throwable") }
}

eventually(config) {
  userRepository.getById(1).name shouldBe "bob"
}

Sharing Configuration

The eventuallyConfig builder makes it easy to share eventually configuration. Suppose operations in your system are classified as “slow” and “fast”. Instead of remembering timing values for slow and fast operations, you can set up a few objects shared across tests and customize them per operation. This is also a good time to show the listener feature of eventually, which provides insight into the current producer result and iteration state.

val slow = eventuallyConfig {
  duration = 5.minutes
  interval = 25.milliseconds.fibonacci()
  listener = { i, t -> logger.info("Current $i after ${t.times} attempts") }
}

val fast = slow.copy(duration = 5.seconds)

class FooTests : FunSpec({
  test("server eventually provides a result for /foo") {
    eventually(slow) {
      fooApi()
    }
  }
})

class BarTests : FunSpec({
  test("server eventually provides a result for /bar") {
    eventually(fast) {
      barApi()
    }
  }
})

Continually

In the other sense of eventually, continually verifies that a code block succeeds and continues to succeed for a certain period. For example, you may want to verify that an HTTP connection remains open for 60 seconds after the last packet is received. You could sleep for 60 seconds and then check, but if the connection closes after 5 seconds, the test sits idle for another 55 seconds before failing. Failing quickly is better.

class MyTests : ShouldSpec() {
  init {
    should("pass for 60 seconds") {
      continually(60.seconds) {
        // code here that should succeed and continue to succeed for 60 seconds
      }
    }
  }
}

The function passed to the continually block is executed every 10 ms. If desired, you can specify the polling interval:

class MyTests: ShouldSpec() {
  init {
    should("pass for 60 seconds") {
      continually(60.seconds, 5.seconds) {
        // code here that should succeed and continue to succeed for 60 seconds
      }
    }
  }
}

Until

When testing non-deterministic code, a common use case is: “I expect this code to pass after a short time.”

For example, you may want to test whether a broker has received a message. You could set a timeout and repeatedly poll until the message is received, but this blocks the thread. You also need to write loop boilerplate.

As an alternative, Kotest provides the until function, which periodically runs a function until it returns true or the specified duration expires.

until is the predicate counterpart to eventually.

Duration

Suppose you have a function that polls a broker and returns a list of messages. When a message is sent, you want to test that the broker receives the message within 5 seconds.

class MyTests : ShouldSpec() {

  private val broker = createBrokerClient()

  init {
    should("broker should receive a message") {
      sendMessage()
      until(5.seconds) {
        broker.poll().size > 0
      }
    }
  }
}

Interval

By default, the predicate is checked every second. You can specify an interval to control the delay between calls. The following is the same example, but this time it fixes the interval more aggressively at 250 ms.

class MyTests : ShouldSpec() {

  private val broker = createBrokerClient()

  init {
    should("broker should receive a message") {
      sendMessage()
      until(5.seconds, 250.milliseconds.fixed()) {
        broker.poll().size > 0
      }
    }
  }
}

You can also specify a Fibonacci interval to increase the delay after each failure.

class MyTests : ShouldSpec() {

  private val broker = createBrokerClient()

  init {
    should("broker should receive a message") {
      sendMessage()
      until(5.seconds, 100.milliseconds.fibonacci()) {
        broker.poll().size > 0
      }
    }
  }
}

Retry

retry is similar to eventually, but instead of trying a code block for a period of time, it tries the code block up to a maximum number of times. You still provide a timeout period to prevent the loop from running forever.

class MyTests: ShouldSpec() {
  init {
    should("retry up to 4 times") {
      retry(4, 10.minutes) {
        // ...
      }
    }
  }
}

Additional options include a delay during execution, a multiplier for exponential delays such as 1, 2, 4, 8, 16, and exception classes if you want to retry only for certain exceptions and fail for others.

References

• Eventually | Kotest