# Check robot regressions in CI

Compare a new MCAP recording with a named baseline and gate releases using explicit comparison or outcome criteria.

Canonical: https://oorb.io/docs/robot-regression-ci/
Maintained by: OORB
Updated: 2026-10-14

## Choose the release question

RunFP can gate a robot release by comparing a new recording with a named baseline or by checking a group's outcome rate. Install RunFP on the runner and provide the required recordings, baseline evidence, and evaluation criteria before enabling either check.

A comparison gate asks whether recorded evidence regressed. An outcome-rate gate asks whether the selected success metric meets your threshold. Use both when both questions matter.

## 1. Prepare a baseline

Use [getting started](/docs/getting-started/) to initialize a workspace and capture a reference episode. Set it as the group's baseline:

```sh
runfp baseline set <baseline-episode-id> --name pick-place
```

Keep the baseline and workspace accessible to the runner. A fresh runner cannot use an episode ID whose artifact has not been supplied. Imported baseline archives and recordings must refer to the intended robot task.

## 2. Check a new recording

```sh
runfp check artifacts/nightly_run.mcap --group pick-place --json
```

This command captures the new recording into the group and compares it with the named baseline. Save its machine-readable output and associated evidence as build artifacts so a failed gate can be investigated.

The recording must already exist. This command does not schedule a physical robot trial or supply the task's ground truth.

## 3. Gate on outcomes when needed

For a group with validated recorded task outcomes:

```sh
runfp certify pick-place --outcome recorded --min-success-rate 0.9 --strict-interval --json
```

The 90% threshold is an example policy, not an OORB reliability promise. The strict interval option evaluates the Wilson interval's lower bound. See [success rate evaluation](/docs/robot-success-rate/) before choosing your sample size and gate.

## 4. Preserve exit codes

| Exit code | Meaning | Runner action |
| --- | --- | --- |
| `0` | Check passed | Continue the release workflow |
| `1` | Tool or input error | Investigate installation, paths, or evidence |
| `3` | Evaluation gate failed | Block the release and inspect findings |

Let the CI runner treat a nonzero exit code as failure. If you pipe output through `tee` in Bash, enable `set -o pipefail` so the logging command does not hide a failed evaluation.

Repeatable checks also require a stable task protocol, comparable conditions, and explicit baseline updates. [Review run comparisons](/docs/compare-robot-runs/) before changing the reference used by a release gate.