Robot regression testing

Robot Regression Testing: Catch What an Update Broke

Published

software does not ship because the new feature works. it ships because the old ones still do. robots rarely get that check : a new checkpoint demos better, it goes out, and the field finds what it broke.

what robot regression testing means :

Robot regression testing is re-running a fixed set of tasks after every change to a robot and comparing the results with a recorded baseline, to find behavior that used to work and no longer does.

this page is about physical robots. regression suites written with Robot Framework, the test automation tool, are a different subject.

what counts as a change :

  • the model : a new policy checkpoint or a fine-tune.
  • the code : the task logic, a controller, a perception stack.
  • the dependencies : a ROS package, a driver, a runtime version.
  • the hardware : a replaced gripper, a new motor, a firmware update, a sensor mount that moved.
  • the setup : a calibration, a parameter file, the success criteria themselves.
  • the environment : a new site, a new layout, different lighting.

most teams re-test after the first one and assume the rest are safe. the rest are where the quiet regressions come from.

why one green run is not a pass :

a unit test gives the same answer every time. a robot running a learned behavior does not. one successful run after an update says the task is still possible, not that it is still reliable.

and the average hides the damage. a fine-tune that lifts the overall rate can still sink one condition : better grasps in daylight, worse grasps under warehouse LEDs. a regression test compares rates per condition, with enough runs to tell a drop from noise. the numbers are in robot reliability testing.

how to set one up :

  • fix the baseline. record the accepted version on each task and keep those runs with the code, model, dependencies and hardware behind them.
  • fix the protocol. the task, the success criteria and the conditions are written down, so two runs months apart mean the same thing.
  • re-run on every change. the same tasks, the same conditions, the same number of runs.
  • compare per condition. look at each cell against its baseline, not at one pooled score.
  • gate the release. a cell that got worse blocks the rollout for that condition until someone decides otherwise.
  • move the baseline on purpose. replacing the reference is its own decision, never a side effect of a passing run.

two gates, two questions :

  • a comparison gate asks whether the recorded evidence regressed against the named baseline.
  • an outcome-rate gate asks whether the task success rate meets your threshold, judged on the lower bound of its interval and not on the observed rate alone.

use both when both questions matter. RunFP runs them from the command line and returns an exit code a CI runner can act on, as described in check MCAP robot regressions in CI. the check reads recordings that already exist : it does not schedule the physical trial, and your team still runs the robot.

how runs are compared is in compare MCAP robot runs against a baseline, and how a rate and its interval are reported is in measure robot task success rates.

when a regression shows up :

the test tells you something got worse. the next question is what changed. with every run recorded alongside its context, the difference between the last good run and the first bad one can be ranked : code, model, dependencies, hardware, environment or setup. the workflow is in investigate a robot field failure.

regression testing a policy, and a fleet :

for a learned policy, the regression suite is the last stage of robot policy evaluation, and for a vision-language-action model the axes to cover are in VLA evaluation.

on a fleet, run the candidate on a few robots first and widen only for the conditions the evidence covers. a drift that appears after rollout is a job for robot fleet monitoring.

a regression testing checklist :

  • a named baseline per task, recorded and kept
  • a written task protocol with numeric success criteria
  • the list of conditions every re-run has to cover
  • the same number of runs per condition, every time
  • results compared per condition, never only pooled
  • a gate that blocks the release when a cell gets worse
  • failed runs and gate output saved for investigation
  • baseline updates made as their own decision, with a name on it

frequently asked questions :

What is robot regression testing ?

Robot regression testing is re-running a fixed set of tasks after every change to a robot and comparing the results with a recorded baseline, to find behavior that used to work and no longer does.

How is it different from software regression testing ?

A software test gives the same answer every time, so one run is enough. A robot running a learned behavior does not act the same way twice, so a regression test is a comparison of success rates over many runs per condition, not a single pass or fail.

Can robot regression tests run in CI without a robot ?

The check can, the trial cannot. A CI job can compare a recording that already exists with a named baseline and fail the build when it regressed. Someone still has to run the robot to produce that recording.

How often should a robot regression suite run ?

Before every change that ships: a new policy checkpoint, a code change, a dependency update, a calibration, a replaced part or a new site layout. Anything that can change behavior is a reason to re-run the suite.

"we think the new version is better" is a feeling. "it is better everywhere except low-light bin picks" is a release decision.

bring the runs from before and after one change and we will show you which conditions moved, and what was different.

Robot failing in the field? Get help now