Thresholds

You can define when to consider a test flaky, broken, or back to a good state. And you can do that for each of the different test suites type you upload to TestPulse (see parameter -tt in the Uploader).

More specifically:

• Flaky: thresholds (in percentage) after which a test will be considered flaky. For example, if you specify 5, it means that any test that fails more than 5% of the time will be considered flaky. Default: 3.
• Broken: number of consecutive failures after which the test will be considered broken. For example, if you specify 3, the test will be considered broken if it fails consecutively 3 times in master. If you specify 1, every time it will fail in master it will be considered broken. Default: 2.
• Recovery: number of consecutive passes after which the test will be considered “recovered” and put back in good state. For example, if a unit test was broken, and you specify from_broken: unit: 3, if the test passes consecutively 3 times, it will be considered in good state. Default:
  • from_broken: 5
  • from_flaky: 10

Example:

settings:
  flaky:
    thresholds:
      unit: 3
      integration: 5
  broken:
    consecutive_failures:
      unit: 1
      integration: 3
  recovery:
    consecutive_passes:
      from_broken:
        unit: 5
        integration: 5
      from_flaky:
        unit: 5
        integration: 5