Kotest Configuration

Kotest Configuration is used to customize and control test execution behavior. It provides many configuration options so users can adjust the test environment as needed.
Tags:

Test Case Configuration

Each test can be configured with various parameters. Call the configuration function after the test name and pass the parameters you want to set. The available parameters are:

  • invocations: The number of times this test is executed. This is useful when you have a non-deterministic test and want to run a specific test a fixed number of times to see whether it eventually fails. The test succeeds only if all invocations succeed. The default is 1.
  • threads: Sets the number of threads used to parallelize invocations of this test. The value must be less than or equal to the invocation value. Likewise, if invocations is set to the same value as the number of threads, each invocation gets its own thread.
  • enabled: If set to false, this test is disabled. This can be useful when you need to temporarily ignore a test. You can also use this parameter with a boolean expression to run the test only under specific conditions.
  • enabledIf: Provides the same function as enabled, but is a lazily evaluated function when the test case is due to execute.
  • timeout: Sets the timeout for this test. If the test does not complete within that time, it fails. This is useful for code that is non-deterministic and may not complete. The timeout is a kotlin.Duration type that can be instantiated as 2.seconds, 3.minutes, and so on.
  • tags: A set of tags that can be used to group tests. See the detailed description below.
  • listeners: Registers test listeners that run only for this test.
  • extensions: Registers extensions that run only for this test.

The following are examples of configuration settings in tests:

import io.kotest.core.spec.style.ShouldSpec
import io.kotest.matchers.shouldBe

class MyTests : ShouldSpec() {
    init {
        should("return the length of the string").config(invocations = 10, threads = 2) {
            "sammy".length shouldBe 5
            "".length shouldBe 0
        }
    }
}

import io.kotest.core.spec.style.WordSpec
import io.kotest.matchers.shouldBe
import kotlin.time.Duration.Companion.seconds

class MyTests : WordSpec() {
    init {
        "String.length" should {
            "return the length of the string".config(timeout = 2.seconds) {
                "sammy".length shouldBe 5
                "".length shouldBe 0
            }
        }
    }
}

import io.kotest.core.spec.style.FunSpec
import io.kotest.core.Tag

class FunSpecTest : FunSpec() {
    init {
        test("FunSpec should support config syntax").config(tags = setOf(Database, Linux)) {
            // ...
        }
    }
}

object Database : Tag()
object Linux : Tag()

You can also specify a default TestCaseConfig for all test cases in a spec.

Override the defaultTestCaseConfig function:

import io.kotest.core.spec.style.StringSpec
import io.kotest.core.test.config.TestCaseConfig

class MySpec : StringSpec() {

    override fun defaultTestCaseConfig() = TestCaseConfig(invocations = 3)

    init {
        // your test cases ...
    }
}

Or assign a value to defaultTestConfig:

import io.kotest.core.spec.style.FunSpec
import io.kotest.core.test.config.TestCaseConfig

class FunSpecTest : FunSpec() {
    init {

        defaultTestConfig = TestCaseConfig(enabled = true, invocations = 3)

        test("FunSpec should support Spec config syntax in init{} block") {
            // ...
        }
    }
}

Project Configuration

Kotest is flexible and lets you configure tests in many ways, such as configuring test order inside specs or how test classes are created. Sometimes you may want to configure settings globally, and for that you should use project-level config.

Project-level configuration can be used by creating an object or class that extends AbstractProjectConfig.

Any configuration set directly at the Spec level or on a test overrides configuration specified at the project level.

Some configuration options available in KotestProjectConfig include test parallelism, failing specs with ignored tests, global AssertSoftly, and reusable listeners or extensions.

Runtime Detection

At runtime, Kotest searches for classes that extend AbstractProjectConfig, instantiates them, and uses all configuration values defined in those classes.

You can create more than one configuration class in different modules, and all classes on the current classpath are detected and their settings are merged. This is useful because common settings can be placed in the root module. If conflicts occur, one value is selected arbitrarily, so it is not recommended to add competing settings to different configuration classes.

For large projects, if startup cost is high, you can disable automatic discovery of these configuration classes. Set the system property or environment variable kotest.framework.classpath.scanning.config.disable to true.

If you still want to use project configuration after automatic scanning is disabled, specify a well-known class name so Kotest can instantiate it reflectively. The system property or environment variable to use is kotest.framework.config.fqn.

For example, the following settings:

kotest.framework.classpath.scanning.config.disable=true
kotest.framework.config.fqn=com.wibble.KotestConfig

disable runtime scanning and look for the com.wibble.KotestConfig class. This class must still extend AbstractProjectConfig.

Parallelism

You can ask Kotest to run specs in parallel to take advantage of modern CPUs with multiple cores by setting the parallelism level, whose default is 1. Tests within a spec always run sequentially.

To do this, override parallelism in the configuration and set it to a value greater than 1. The configured number is the number of specs that run at the same time. For example:

object KotestProjectConfig : AbstractProjectConfig() {
    override val parallelism = 3
}

Another way to enable this is with the system property kotest.framework.parallelism, which always takes precedence over this value if defined.

Some tests may not be suitable for parallel execution, so you can opt out individual specs and run them separately by using the @DoNotParallelize annotation on the spec.

Assertion Mode

You can request that the build fail, or warn to standard error, when a test that does not use Kotest assertions is executed.

To do this, set assertionMode to AssertionMode.Error or AssertionMode.Warn in the configuration. Another way to enable this is with the system property kotest.framework.assertion.mode, which always takes precedence over this value if defined.

object KotestProjectConfig : AbstractProjectConfig() {
    override val assertionMode = AssertionMode.Error
}

Global Assert Softly

Assert Softly is very useful for batching multiple errors into a single failure. To enable this automatically for all tests, you can configure it here. Another way to enable it is to set the system property kotest.framework.assertion.globalassertsoftly to true, which always takes precedence over this value if defined.

object KotestProjectConfig : AbstractProjectConfig() {
    override val globalAssertSoftly = true
}

Handling Duplicate Test Names

By default, Kotest renames a test if it has the same name as another test in the same scope. It appends _1, _2, and so on to the test name. This is useful for automatically generated tests.

This behavior can be changed globally by setting duplicateTestNameMode to DuplicateTestNameMode.Error or DuplicateTestNameMode.Warn.

Error fails the test suite on repeated names, while warning renames the tests but prints a warning.

Failing on Ignored Tests

You may want to treat ignored tests as failures. To enable this feature, set failOnIgnoredTests to true in the project configuration.

For example:

object KotestProjectConfig : AbstractProjectConfig() {
    override val failOnIgnoredTests = true
}

Test Ordering

Kotest supports applying order independently to specs and tests.

Test Order

When running multiple tests in a single spec, there is a specific order for how they run.

By default, sequential order, the order in which tests are defined in the spec, is used, but this can be changed. See Test Ordering for the available options.

Spec Order

By default, the order of spec classes is not defined. This is enough if you do not need a default setting, but if you need to control the execution order of specs, use Spec Ordering.

Test Naming

Test names can be adjusted in several ways.

Test Cases

Test name case can be controlled by changing the value of testNameCase.

By default, the value is TestNameCase.AsIs, which does not change names.

If the value is set to TestNameCase.Lowercase, test names are printed in lowercase.

If you use a spec that adds a prefix to test names, which must be WordSpec or BehaviorSpec, the TestNameCase.Sentence and TestNameCase.InitialLowercase values can be useful.

Test Name Tags

Another option for test names is testNameAppendTags. If set to true, all tags associated with the test are included in the test name. For example, if a test foo is defined in a spec with linux and spark tags, the test name is adjusted to foo [linux, spark].

This setting can also be enabled by using the system property or environment variable kotest.framework.testname.append.tags with the value true.

Test Name Whitespace

removeTestNameWhitespace can be useful when test names are defined over multiple lines. Consider this example:

"""this is
   my test case""" {
  // test here
}

In this case, the test name in the output is this is my test case. If removeTestNameWhitespace is set to true, whitespace is removed so the name becomes this is my test case.

Another way to enable this feature is to set the system property kotest.framework.testname.multiline to true, which always takes precedence over this value if defined.

object KotestProjectConfig : AbstractProjectConfig() {
    override val testNameRemoveWhitespace = true
}

Framework Configuration Properties

KotestEngineProperties.kt

package io.kotest.core.internal

object KotestEngineProperties {

   const val scriptsEnabled = "kotest.framework.scripts.enabled"

   const val dumpConfig = "kotest.framework.dump.config"

   /**
    * Sets the tag expression that determines included/excluded tags.
    */
   const val tagExpression = "kotest.tags"

   const val excludeTags = "kotest.tags.exclude"

   const val includeTags = "kotest.tags.include"

   /**
    * A regex expression that is used to match the test [io.kotest.core.descriptors.Descriptor]'s path
    * to determine if a test should be included in the test plan or not.
    */
   const val filterTests = "kotest.filter.tests"

   /**
    * A regex expression that is used to match the [io.kotest.mpp.bestName] of a class
    * to determine if a spec should be included in the test plan or not.
    */
   const val filterSpecs = "kotest.filter.specs"

   const val propertiesFilename = "kotest.properties.filename"

   /**
    * If set to true, then source ref's will not be created for test cases.
    * This may speed up builds (as the engine will not need to create stack traces to
    * generate line numbers) but will also reduce functionality in the intellij plugin
    * (by limiting the ability to drill directly into the test inside a file).
    */
   const val disableSourceRef = "kotest.framework.sourceref.disable"

   /**
    * If set to true, disables the use of '!' as a prefix to disable tests.
    */
   const val disableBangPrefix = "kotest.bang.disable"

   /**
    * The default [io.kotest.core.spec.IsolationMode] for specs.
    */
   const val isolationMode = "kotest.framework.isolation.mode"

   /**
    * The default [io.kotest.core.test.AssertionMode] for tests.
    */
   const val assertionMode = "kotest.framework.assertion.mode"

   /**
    * The default parallelism for specs.
    */
   const val parallelism = "kotest.framework.parallelism"

   /**
    * The default timeout for test cases.
    */
   const val timeout = "kotest.framework.timeout"

   /**
    * The default timeout for the entire test suite.
    */
   const val projectTimeout = "kotest.framework.projecttimeout"

   const val logLevel = "kotest.framework.loglevel"

   /**
    * The default timeout for each invocation of a test case.
    */
   const val invocationTimeout = "kotest.framework.invocation.timeout"

   const val disableTestNestedJarScanning = "kotest.framework.disable.test.nested.jar.scanning"

   const val concurrentSpecs = "kotest.framework.spec.concurrent"

   const val concurrentTests = "kotest.framework.test.concurrent"

   /**
    * Disable scanning the classpath for configuration classes by setting this property to true
    */
   const val disableConfigurationClassPathScanning = "kotest.framework.classpath.scanning.config.disable"

   /**
    * Specify a fully qualified name to use for project config.
    * This class will be instantiated via reflection.
    */
   const val configurationClassName = "kotest.framework.config.fqn"

   /**
    * Disable scanning the classpath for listeners with @AutoScan by setting this property to true
    */
   const val disableAutoScanClassPathScanning = "kotest.framework.classpath.scanning.autoscan.disable"

   const val allowMultilineTestName = "kotest.framework.testname.multiline"

   /**
    *  If set -> filter testCases by this severity level and higher, else running all
    */
   const val testSeverity = "kotest.framework.test.severity"

   /**
    * Enable assert softly globally.
    * */
   const val globalAssertSoftly = "kotest.framework.assertion.globalassertsoftly"

   /**
    * Appends all tags associated with a test case to its display name.
    * */
   const val testNameAppendTags = "kotest.framework.testname.append.tags"

   /**
    * Controls whether classes will inherit tags from their supertypes. Default false
    */
   const val tagInheritance = "kotest.framework.tag.inheritance"

   /**
    * Controls the [io.kotest.core.names.DuplicateTestNameMode] mode.
    */
   const val duplicateTestNameMode = "kotest.framework.testname.duplicate.mode"

   const val disableJarDiscovery = "kotest.framework.discovery.jar.scan.disable"
}

References