Kotest Statistics

Sometimes you may want to know what kinds of values Kotest generates to check that a generator is configured as expected.

Statistics

Sometimes you may want to know what kinds of values Kotest generates to check that a generator is configured the way you want. Property test statistics are designed for this need.

The collect function is the starting point for statistics and is used to count categories of values. Use it by calling the function inside a property test with the category you want to increment.

For example, suppose you want to collect statistics for the RoundingMode values used by BigDecimal. Call checkAll as usual and pass the rounding mode to the collect function.

checkAll(Arb.enum<RoundingMode>(), Arb.bigDecimal()) { mode, decimal ->
  collect(mode)
  // test here
}

When the test completes, Kotest prints the test name and the count/percentage for each category:

Statistics: [collecting stats] (1000 iterations, 1 args)

HALF_DOWN                                                     142 (14%)
HALF_UP                                                       141 (14%)
CEILING                                                       132 (13%)
FLOOR                                                         122 (12%)
UP                                                            119 (12%)
UNNECESSARY                                                   119 (12%)
HALF_EVEN                                                     118 (12%)
DOWN                                                          107 (11%)

The categories you use do not have to be enums. They can be any object, and if you want more control, you can wrap them conditionally.

checkAll(Arb.int()) { k ->
  when {
    k % 2 == 0 -> collect("EVEN")
    else -> collect("ODD")
  }
  // test here
}

Labels

Sometimes you may need orthogonal sets of statistics. For example, in a simple numeric test, you may want to check both that a certain percentage is even and that a certain percentage is negative. One way is to use EVEN_POS, EVEN_NEG, ODD_POS, and ODD_NEG.

checkAll(Arb.int()) { k ->
  when {
    k > 0 && k % 2 == 0 -> collect("EVEN_POS")
    k % 2 == 0 -> collect("EVEN_NEG")
    k > 0 -> collect("ODD_POS")
    else -> collect("ODD_NEG")
  }
  // test here
}

This gives one output set:

EVEN_POS                                                       142 (27%)
EVEN_NEG                                                       141 (23%)
ODD_POS                                                        132 (24%)
ODD_NEG                                                        122 (26%)

However, as combinations increase, this becomes hard to manage, so Kotest supports labelled statistics. You can think of these as separate statistics sets. To use labels, simply pass the label name as the first argument to the collect method.

checkAll(Arb.int()) { k ->
  when {
    k % 2 == 0 -> collect("even_odd", "EVEN")
    else -> collect("even_odd", "ODD")
  }
  when {
    k > 0 -> collect("pos_neg", "POS")
    else -> collect("pos_neg", "NEG")
  }
  // test here
}

Kotest now prints several statistics sets with the label name in the title:

Statistics: [collecting labelled stats] (1000 iterations, 1 args) [even_odd]

ODD                                                           520 (52%)
EVEN                                                          480 (48%)


Statistics: [collecting labelled stats] (1000 iterations, 1 args) [pos_neg]

NEG                                                           527 (53%)
POS                                                           473 (47%)

Report Mode

By default, statistics for all property tests are printed. There are four modes that can be configured with the global configuration object PropertyTesting.

The possible options are:

Mode	Function
`PropertyTesting.statisticsReportMode = StatisticsReportMode.OFF`	Disables all statistics reporting.
`PropertyTesting.statisticsReportMode = StatisticsReportMode.ALL`	Enables all statistics reporting.
`PropertyTesting.statisticsReportMode = StatisticsReportMode.SUCCESS`	Outputs statistics only on successful tests.
`PropertyTesting.statisticsReportMode = StatisticsReportMode.FAILED`	Outputs statistics only on failed tests.

Checking Statistics Coverage

To programmatically assert that certain values are being generated, you can specify constraints that must be satisfied.

For example, in the previous rounding example, you can use withCoveragePercentages to verify that at least 10% of inputs cover HALF_DOWN and 10% cover FLOOR.

withCoveragePercentages(mapOf(RoundingMode.HALF_DOWN to 10.0, RoundingMode.FLOOR to 10.0)) {
  checkAll(Arb.enum<RoundingMode>(), Arb.bigDecimal()) { mode, decimal ->
    collect(mode)
    // use the mode / decimal
  }
}

To check absolute counts instead of percentages, use withCoverageCounts.

withCoverageCounts(mapOf(RoundingMode.HALF_DOWN to 75, RoundingMode.FLOOR to 75)) {
  checkAll(Arb.enum<RoundingMode>(), Arb.bigDecimal()) { mode, decimal ->
    collect(mode)
    // use the mode / decimal
  }
}

Custom Reports

You can customize the report format or generate reports from raw data by using your own instance of a statistics reporter. This is configured through the global configuration object PropertyTesting.

For example:

object MyStatisticsReporter : object : StatisticsReporter { ... }
PropertyTesting.statisticsReporter = MyStatisticsReporter

References

Statistics | Kotest