Define what counts as success
RunFP reports a run group's success rate with a Wilson confidence interval. The result depends on the outcome mode you select and the evidence actually recorded. Define the task's success condition before collecting trials.
Repeated trials must share a group label. The baseline is the reference artifact and is excluded from the set of attempts. Reimporting identical evidence does not add an independent trial.
Choose the outcome mode explicitly
| Mode | How an attempt passes | Missing recorded outcome |
|---|---|---|
recorded | The captured process or SDK outcome indicates success | Excluded and counted as missing outcome evidence |
diff | The comparison meets the selected finding threshold | Uses comparison evidence |
auto (default) | A successful recorded outcome and acceptable comparison, when an outcome exists | Falls back to comparison criteria |
For task success, use recorded outcomes that your application has validated:
runfp certify pick-place --outcome recorded --json
For comparison consistency, select --outcome diff. Do not describe a comparison-only pass rate as physical task success.
Read the interval alongside the rate
The default confidence level is 95%. A confidence interval communicates sampling uncertainty, even when all observed attempts passed. Report the trial count, task conditions, outcome mode, and interval with the observed rate.
These intervals do not correct for biased sampling, mislabeled outcomes, dependent trials, or untested deployment conditions. Your test design must address those limits separately.
Configure a release threshold
This example gates on an observed rate of 90%; choose a threshold appropriate to your application:
runfp certify pick-place --outcome recorded --min-success-rate 0.9 --json
To compare the interval's lower bound with that threshold instead:
runfp certify pick-place --outcome recorded --min-success-rate 0.9 --strict-interval --json
Strict interval gating needs enough trials to support the threshold. With an enabled gate, a group with no comparable attempts fails. Exit codes are 0 for pass, 1 for tool error, and 3 for gate failure.
Keep coverage claims scoped
runfp coverage pick-place --json
This reports bin coverage of recorded scalar state variables. It does not measure every physical operating condition or establish deployment safety. Track conditions such as payload, lighting, surface, and human interaction explicitly when they matter to your application.
Use these criteria in your release checks.