Robot Reliability Testing Beyond the Demo

the demo trap :

every robotics team has a great demo. the lighting is right, the objects are placed where the policy likes them, the calibration was done that morning, and the one run that gets recorded is the one where everything worked.

then a customer asks the only question that matters : "how often does it work ?"

and suddenly there is no number. there is a feeling. "it's pretty reliable." "it almost always works." "we haven't seen that failure in a while."

the industry-wide version of this gap has been measured : a16z's essay on the physical AI deployment gap puts the typical numbers at 95% success in the lab degrading to around 60% in the field, while production customers expect three nines. the distance between those numbers is exactly what reliability testing is supposed to measure before the customer does.

this article is about turning that feeling into a number you can defend, it's the testing half of robot deployment reliability, which we covered more broadly in a separate piece.

what robot reliability testing actually means :

Robot reliability testing is running a behavior enough times, under enough controlled variations of real conditions, to state a success rate you would defend in front of a paying customer.

every word in that definition is doing work. "enough times" is a statistics question. "controlled variations" is a coverage question. "real conditions" is what separates this from a lab benchmark. and "defend" is the difference between a marketing claim and an engineering claim.

why one run tells you almost nothing :

with traditional software, one passing test means the code path works, and it will keep working until someone changes it.

a learned policy is not like that. it is a stochastic system. the same behavior, on the same robot, in the same room, can succeed 96 times and fail on the 97th because the light shifted, the object landed in a slightly different pose, or the gripper calibration drifted a millimeter.

so a single successful run is not evidence of reliability. it is one sample from a distribution you have not measured yet.

how many runs is enough :

this is the part most teams get wrong, because the honest numbers are bigger than anyone wants them to be.

say you run the behavior 10 times and it succeeds 10 times. it feels like 100%. statistically, with 95% confidence, all you can actually claim is a success rate somewhere above roughly 74%.

• 10 clean runs out of 10 -> you can defend "above ~74%"
• 30 clean runs out of 30 -> you can defend "above ~90%"
• 59 clean runs out of 59 -> you can defend "above ~95%"

and that is the optimistic case where nothing fails. the moment you have real failures mixed in, you need more runs to narrow the interval, not fewer.

these are not our numbers, this is the textbook lower confidence bound for a binomial proportion (the Clopper-Pearson interval). with n successes out of n runs, the 95% lower bound is 0.05^(1/n), you can check every figure above with one line of python. medicine has leaned on the same math since the 80s as the rule of three : zero failures observed in n trials still means the true failure rate can be as high as about 3/n.

and robotics research is waking up to the same problem, recent work like Vincent et al. on statistical performance bounds for behavior cloning policies and Snyder et al. on statistically rigorous policy comparison exists precisely because the 10 to 30 hardware rollouts most teams run are too few to support the claims being made from them. a 2026 survey of evaluation practice (PhAIL) found that real-world VLA evaluations typically run 25 or fewer rollouts per condition, almost always without confidence intervals. the field is grading itself on homework it never checked.

the point is not to memorize the math. the point is that "we tested it a bunch and it looked good" usually translates to a number nobody would put in a contract.

test the conditions, not just the task :

a success rate without a condition scope is not a real claim. "91% success" means nothing until you say under what lighting, which calibration state, which object poses, and which surfaces.

in practice this means building a condition matrix and running the behavior across it :

• lighting variations, including the ugly ones
• calibration bands, fresh and drifted
• object poses, positions, and orientations
• surfaces and floor friction
• battery levels and thermal state
• sensor noise and occlusions

and just as important : recording what you did not test. "untested : low-light, 0 runs" is one of the most valuable lines a test report can contain, because it tells the deployment team exactly where not to trust the robot yet.

regression testing when the model changes :

every new model checkpoint, every retrain, every fine-tune is a new stochastic system wearing the old one's name.

a new model that improves the average success rate can still silently regress a specific condition. better carpet vacuuming, worse corner handling. better grasps in daylight, worse grasps under warehouse LEDs.

which means reliability testing is not a one-time gate before the first deployment, it's a regression suite you re-run against a baseline every time anything changes : the model, the calibration, the hardware, the environment, the task definition.

if you cannot compare this week's numbers to last week's baseline under the same conditions, you are not doing regression testing, you are collecting anecdotes.

the full evaluation pipeline, from sim benchmarks to the deployment sign-off, is its own topic : we walk it in robot policy evaluation.

the three classic categories, updated for learned systems :

the textbook taxonomy for robot reliability testing has been around since Dhillon's Robot Reliability and Safety (1991). it is still the right skeleton, it just predates robots that learn. here is the modern version :

• development and demonstration testing : prove the design can work at all. for learned systems this now includes simulation policy evaluation and failure injection, hunting for the conditions that break the policy before any customer finds them.
• qualification and acceptance testing : the formal "is it ready" gate. for learned systems this is the statistical trial campaign, the condition matrix, enough runs for a defensible confidence interval, and a written record of what was not tested.
• operational testing : watching the deployed system. for learned systems this is where regression testing lives, re-running the baseline suite on every model update and capturing every field run so failures can be replayed instead of re-argued.

most robotics teams do the first category, gesture at the second, and improvise the third. the second and third are where customer trust actually comes from.

why MTBF is the wrong number for a learned policy :

traditional robot reliability has one king metric : MTBF, mean time between failures. a robot that ran 10,000 hours with 5 failures has a 2,000 hour MTBF. clean arithmetic, and for mechanical wear it mostly works, because mechanical failures are roughly independent : something breaks, you fix it, the clock resets.

learned policy failures are not independent. they cluster. those 5 failures might all be the same poorly-lit aisle at the same time of evening. the 2,000 hour MTBF hides that completely : the honest description is "nearly perfect in most conditions, unusable in one", and no single averaged number can say that.

so keep MTBF for the hardware, and describe the learned behavior with per-condition success rates, confidence intervals, and intervention rate, how often a human had to step in. those are the numbers that survive contact with a customer's procurement team.

what a defensible test report looks like :

• number of runs, per behavior, per condition
• success rate with a confidence interval
• conditions that were covered
• conditions that were not covered
• failure modes observed, with counts
• regressions vs the previous baseline
• improvements vs the previous baseline
• conditions where deployment should be blocked

robot reliability testing checklist :

before trusting a learned policy in a customer workflow, these are the questions I would want answered, not from memory, from records :

• how many runs were actually executed, not watched once ?
• what success rate can we claim, and at what confidence ?
• which conditions were covered, and how many runs each ?
• which conditions have zero runs ?
• what were the failure modes, and how often did each occur ?
• how do these numbers compare to the previous baseline ?
• did anything regress, even if the average improved ?
• what would block this deployment if we were being honest ?

none of this requires exotic tooling to start. it requires discipline about capturing every run and what state the robot was in when it happened. the tooling is what makes it survivable at scale.

frequently asked questions :