Keep your scripts or adopt a tool?
Keep custom robot log-analysis scripts when they answer a narrow question and your team can maintain their assumptions. Consider OORB RunFP when you need shared recorded episodes, provenance, named baselines, and repeatable gates; preserve the task-specific checks that make your existing analysis useful.
This is a decision guide, not a claim that custom software is less accurate. A well-tested script can be the right evaluation tool.
Signs that your current approach fits
- The inputs and task definition are stable and owned by a small team.
- The script's output has an explicit interpretation, including missing signals and invalid inputs.
- Changes to schemas, thresholds, and outcome definitions have review and validation.
- Another engineer can reproduce a result from preserved artifacts and the exact script version.
If those conditions hold, document the workflow and keep using it. Adding another tool should solve an identified problem.
Compare the maintenance responsibilities
| Responsibility | A custom script must provide | RunFP's documented workflow |
|---|---|---|
| Reference selection | A preserved input and a rule for replacing it | A named baseline referring to stored evidence |
| Software and model identity | Metadata capture and its association with each run | Available workspace provenance and a declared model record captured with episodes |
| Missing or incompatible signals | Explicit handling rather than silent omissions | Review comparison findings and task-protocol requirements |
| Sharing an investigation | A portable package of data, code, metadata, and instructions | Episode export and import for exchanging stored evidence |
| A release gate | Documented result semantics and reliable exit handling | CLI comparison gates and outcome-rate gates with distinct meanings |
These are responsibilities, not an automatic feature checklist. Your script may already meet them. RunFP's documented capabilities still require comparable recordings, task criteria, and an OORB-provided installation.
What RunFP does not remove from the team's work
Your team still defines the physical task, records the relevant signals, validates success labels, and chooses tolerances. A built-in comparison cannot recover absent measurements or establish causation from a changed signal alone.
If your script computes a task-specific metric that the public RunFP guides do not document, keep it. This guide does not promise that RunFP imports arbitrary script metrics or reproduces your custom algorithm.
Move one evaluation workflow at a time
1. Freeze a reference case
Save a representative healthy recording, a known failed recording, the script version, thresholds, and expected interpretation. Include an incomplete recording if missing data is a realistic failure mode.
2. Import the same evidence
After initializing a workspace, use a common task label:
runfp snapshot recordings/healthy.mcap --label pick-place --json
runfp snapshot recordings/failed.mcap --label pick-place --json
Capture from the intended robot project so the available workspace provenance is relevant. Importing an old recording today does not reconstruct the historical software that originally ran on the robot.
3. Compare interpretations
runfp diff <healthy-episode-id> <failed-episode-id> --protocol protocols/pick-place.yaml --json
Check where the script and RunFP agree, where they examine different evidence, and where either result is unavailable. Avoid treating different metric definitions as contradictory measurements.
4. Introduce a gate after reviewing the differences
runfp baseline set <healthy-episode-id> --name pick-place
runfp check recordings/candidate.mcap --group pick-place --json
Keep your existing evaluation alongside the new check while validating the change. When a gate fails, preserve its output and artifacts. The CI guide explains exit handling.
5. Preserve a shared investigation artifact
runfp episodes export <failed-episode-id>
A teammate can import the resulting episode archive and inspect the same stored evidence. The archive is a captured artifact, not a complete recreation of the original physical environment.
How should we compare costs?
Estimate time spent maintaining decoders, metadata, artifact sharing, and gate behavior. Add onboarding and validation time for any new tool. RunFP's public guides do not publish a price, and we make no claim about time saved or lower total cost without a measured trial.