Choose the release question
RunFP can gate a robot release by comparing a new recording with a named baseline or by checking a group's outcome rate. Install RunFP on the runner and provide the required recordings, baseline evidence, and evaluation criteria before enabling either check.
A comparison gate asks whether recorded evidence regressed. An outcome-rate gate asks whether the selected success metric meets your threshold. Use both when both questions matter.
1. Prepare a baseline
Use getting started to initialize a workspace and capture a reference episode. Set it as the group's baseline:
runfp baseline set <baseline-episode-id> --name pick-place
Keep the baseline and workspace accessible to the runner. A fresh runner cannot use an episode ID whose artifact has not been supplied. Imported baseline archives and recordings must refer to the intended robot task.
2. Check a new recording
runfp check artifacts/nightly_run.mcap --group pick-place --json
This command captures the new recording into the group and compares it with the named baseline. Save its machine-readable output and associated evidence as build artifacts so a failed gate can be investigated.
The recording must already exist. This command does not schedule a physical robot trial or supply the task's ground truth.
3. Gate on outcomes when needed
For a group with validated recorded task outcomes:
runfp certify pick-place --outcome recorded --min-success-rate 0.9 --strict-interval --json
The 90% threshold is an example policy, not an OORB reliability promise. The strict interval option evaluates the Wilson interval's lower bound. See success rate evaluation before choosing your sample size and gate.
4. Preserve exit codes
| Exit code | Meaning | Runner action |
|---|---|---|
0 | Check passed | Continue the release workflow |
1 | Tool or input error | Investigate installation, paths, or evidence |
3 | Evaluation gate failed | Block the release and inspect findings |
Let the CI runner treat a nonzero exit code as failure. If you pipe output through tee in Bash, enable set -o pipefail so the logging command does not hide a failed evaluation.
Repeatable checks also require a stable task protocol, comparable conditions, and explicit baseline updates. Review run comparisons before changing the reference used by a release gate.