Validate the candidate under the conditions you intend to deploy
Validate a robot policy update by recording its model identity, comparing candidate runs with a fixed deployed baseline, and checking physical task outcomes under defined conditions. OORB RunFP supports recorded-run provenance and evaluation; telemetry differences and task success answer separate release questions.
Use your own recordings and an OORB-provided installation. This workflow evaluates available evidence; your team performs the physical trials and owns the acceptance policy.
1. Define the release comparison before collecting runs
| Keep explicit | Example for a pick-and-place task |
|---|---|
| Task objective | Place the intended object in the target location under the defined completion rules |
| Deployment conditions | The lighting, object variety, payload, and calibration states in scope |
| Candidate identity | The checkpoint and software configuration the robot actually loaded |
| Reference identity | The deployed or accepted model and its fixed recorded baseline |
| Required observations | The topics and outcome records needed to judge this task |
| Acceptance policy | The findings and outcome-rate criteria that block this release |
The examples describe a test plan, not universal thresholds. Decide how to handle missing outcomes and untested conditions before observing the candidate's results.
2. Preserve the deployed reference
For an initialized workspace, declare the model record and import an appropriate recording:
runfp model set --name pick-place-policy --version deployed --file models/deployed.pt
runfp model show
runfp snapshot recordings/deployed.mcap --label pick-place --json
runfp baseline set <deployed-episode-id> --name pick-place-candidate
Use real version labels and paths. Verify the robot's loaded model through your deployment process. A declared model record documents the workspace identity available at capture time; it does not independently verify historical recordings or replace runtime verification.
3. Capture the candidate without moving the baseline
runfp model set --name pick-place-policy --version candidate --file models/candidate.pt
runfp model show
runfp snapshot recordings/candidate.mcap --label pick-place-candidate --protocol protocols/pick-place.yaml --json
runfp diff <deployed-episode-id> <candidate-episode-id> --protocol protocols/pick-place.yaml --json
runfp model history
Review missing or incompatible signals before interpreting changes. Keep the reference fixed while deciding whether the candidate passes. Baseline replacement is a separate release decision.
If you use an external artifact registry, follow model provenance to record the intended artifact reference and verify that it matches your deployment.
4. Evaluate physical outcomes separately
Imported telemetry may have no recorded success or failure outcome. Confirm the capture path supplies validated task outcomes before using the recorded-outcome metric. An application status is meaningful only when it corresponds to your task definition.
For a group with valid recorded outcomes:
runfp certify pick-place-candidate --outcome recorded --min-success-rate 0.9 --strict-interval --json
The 90% threshold is an example policy. With the strict-interval option, the gate uses the Wilson interval's lower bound rather than just the observed rate. Read success-rate evaluation for missing-outcome handling and coverage limits.
The candidate uses its own pick-place-candidate label, while its named baseline refers to the deployed episode. RunFP collects attempts by label, so this keeps the baseline trial out of the candidate outcome cohort. A label is not a condition filter: use distinct candidate groups for different model versions or conditions, and inspect which attempts are included before interpreting a group-level score.
5. Preserve the release decision
Save the candidate and reference identities, task protocol, comparable trial conditions, comparison findings, outcome definition, interval, and untested conditions. Record who accepted the remaining limitations.
For later recorded checks against the approved reference, use the CI workflow. Preserve failed-gate output and the associated artifacts for investigation.
What if the policy improves one condition but regresses another?
Report results by the conditions relevant to deployment. A pooled improvement can hide a failing condition. Gather comparable trials for each required condition and keep unsupported conditions outside the release claim until they have evidence.