Robot Reliability Testing Beyond the Demo

the demo trap :

every robotics team has a great demo. the lighting is right, the objects are placed where the policy likes them, the calibration was done that morning, and the one run that gets recorded is the one where everything worked.

then a customer asks the only question that matters : "how often does it work ?"

and suddenly there is no number. there is a feeling. "it's pretty reliable." "it almost always works." "we haven't seen that failure in a while."

this article is about turning that feeling into a number you can defend, it's the testing half of robot deployment reliability, which we covered more broadly in a separate piece.

what robot reliability testing actually means :

Robot reliability testing is running a behavior enough times, under enough controlled variations of real conditions, to state a success rate you would defend in front of a paying customer.

every word in that definition is doing work. "enough times" is a statistics question. "controlled variations" is a coverage question. "real conditions" is what separates this from a lab benchmark. and "defend" is the difference between a marketing claim and an engineering claim.

why one run tells you almost nothing :

with traditional software, one passing test means the code path works, and it will keep working until someone changes it.

a learned policy is not like that. it is a stochastic system. the same behavior, on the same robot, in the same room, can succeed 96 times and fail on the 97th because the light shifted, the object landed in a slightly different pose, or the gripper calibration drifted a millimeter.

so a single successful run is not evidence of reliability. it is one sample from a distribution you have not measured yet.

how many runs is enough :

this is the part most teams get wrong, because the honest numbers are bigger than anyone wants them to be.

say you run the behavior 10 times and it succeeds 10 times. it feels like 100%. statistically, with 95% confidence, all you can actually claim is a success rate somewhere above roughly 74%.

• 10 clean runs out of 10 -> you can defend "above ~74%"
• 30 clean runs out of 30 -> you can defend "above ~90%"
• 59 clean runs out of 59 -> you can defend "above ~95%"

and that is the optimistic case where nothing fails. the moment you have real failures mixed in, you need more runs to narrow the interval, not fewer.

these are not our numbers, this is the textbook lower confidence bound for a binomial proportion (the Clopper-Pearson interval). with n successes out of n runs, the 95% lower bound is 0.05^(1/n), you can check every figure above with one line of python. medicine has leaned on the same math since the 80s as the rule of three : zero failures observed in n trials still means the true failure rate can be as high as about 3/n.

and robotics research is waking up to the same problem, recent work like Vincent et al. on statistical performance bounds for behavior cloning policies and Snyder et al. on statistically rigorous policy comparison exists precisely because the 10 to 30 hardware rollouts most teams run are too few to support the claims being made from them.

the point is not to memorize the math. the point is that "we tested it a bunch and it looked good" usually translates to a number nobody would put in a contract.

test the conditions, not just the task :

a success rate without a condition scope is not a real claim. "91% success" means nothing until you say under what lighting, which calibration state, which object poses, and which surfaces.

in practice this means building a condition matrix and running the behavior across it :

• lighting variations, including the ugly ones
• calibration bands, fresh and drifted
• object poses, positions, and orientations
• surfaces and floor friction
• battery levels and thermal state
• sensor noise and occlusions

and just as important : recording what you did not test. "untested : low-light, 0 runs" is one of the most valuable lines a test report can contain, because it tells the deployment team exactly where not to trust the robot yet.

regression testing when the model changes :

every new model checkpoint, every retrain, every fine-tune is a new stochastic system wearing the old one's name.

a new model that improves the average success rate can still silently regress a specific condition. better carpet vacuuming, worse corner handling. better grasps in daylight, worse grasps under warehouse LEDs.

which means reliability testing is not a one-time gate before the first deployment, it's a regression suite you re-run against a baseline every time anything changes : the model, the calibration, the hardware, the environment, the task definition.

if you cannot compare this week's numbers to last week's baseline under the same conditions, you are not doing regression testing, you are collecting anecdotes.

what a defensible test report looks like :

• number of runs, per behavior, per condition
• success rate with a confidence interval
• conditions that were covered
• conditions that were not covered
• failure modes observed, with counts
• regressions vs the previous baseline
• improvements vs the previous baseline
• conditions where deployment should be blocked

robot reliability testing checklist :

before trusting a learned policy in a customer workflow, these are the questions I would want answered, not from memory, from records :

• how many runs were actually executed, not watched once ?
• what success rate can we claim, and at what confidence ?
• which conditions were covered, and how many runs each ?
• which conditions have zero runs ?
• what were the failure modes, and how often did each occur ?
• how do these numbers compare to the previous baseline ?
• did anything regress, even if the average improved ?
• what would block this deployment if we were being honest ?

none of this requires exotic tooling to start. it requires discipline about capturing every run and what state the robot was in when it happened. the tooling is what makes it survivable at scale.