Robot policy evaluation

Robot Policy Evaluation in Practice

from sim benchmarks to deployment sign-off. the research community has benchmarks, the papers have methods, and the team shipping a robot next quarter has neither a playbook nor a number they can put in a contract. this is the missing middle.

the benchmark trap :

every learned-robotics team knows the leaderboards. LIBERO, CALVIN, SimplerEnv, ManiSkill, and a new one every quarter. your checkpoint scores 84%, the last one scored 79%, the team celebrates, the checkpoint ships.

and then the robot meets a warehouse where the lighting is nothing like the render, the gripper calibration is three weeks old, and the objects are greasy in a way no simulator has ever modeled.

benchmarks are how research measures progress. they are not, and were never designed to be, how a deployment gets approved. even the best open evaluation harnesses are simulation-only, which means the thing they measure ends exactly where your deployment begins.

what robot policy evaluation actually means :

Robot policy evaluation is measuring whether a learned policy is fit to deploy : not whether it scores well on a benchmark, but whether you can state, with confidence, how it will behave under the conditions it will actually face.

the difference between those two things is the difference between a research result and an engineering decision. one gets you a paper. the other gets a robot into a customer's building without an engineer sleeping next to it.

the three questions, in order :

  • does it work at all ? answered in simulation, because sim is cheap and parallel. this is where you reject weak checkpoints, hunt for failure modes, and inject the perturbations you hope never happen.
  • does it work here ? answered on the real robot, under the condition matrix of this specific deployment, with enough runs per condition to put a confidence interval on the answer.
  • does it still work ? answered continuously. every new checkpoint, calibration, or environment change gets compared against the previous baseline before it ships, and every field run gets captured so the next failure can be replayed instead of re-argued.

most teams do the first, sample the second, and improvise the third. the second and third are where deployments live or die.

why sim scores don't transfer :

it is not that simulation lies, it's that it answers a different question. a sim benchmark measures the policy against the distribution the benchmark authors modeled. your deployment measures it against a distribution nobody modeled : this building, this fixture, this seasonal sunlight through this window.

the research community knows this. work on policy evaluation with imperfect simulators exists precisely because raw sim scores are not trustworthy predictors of real performance, and the emerging methods use sim to sharpen real-world estimates, not replace them.

the practical translation : use sim to decide what to test on hardware. never use it to decide you don't need to.

the statistics you cannot skip :

once you are on the real robot, sample size is the whole game. a 2026 survey (PhAIL) found that real-world VLA evaluations typically run 25 or fewer rollouts per condition, almost always without confidence intervals. Toyota Research Institute has written the best practitioner introduction to the statistics, and the short version is : the numbers you need are bigger than you want them to be.

we walk the exact math, how many clean runs support which claims, in robot reliability testing. the one-line summary : 10 clean runs out of 10 only supports claiming "above roughly 74%" at 95% confidence. if your eval report has no interval, it's a mood, not a measurement.

regression testing : the part nobody owns :

software engineering solved this decades ago : you do not ship a release because the new features work, you ship it because the old ones still do. learned robotics has not caught up. a new checkpoint arrives, it demos better, it ships.

but a fine-tune that lifts the average can still sink a condition. better grasps in daylight, worse grasps under warehouse LEDs. the average hides the regression, and the field finds it for you.

treating policies like software artifacts means : a fixed baseline suite, re-run on every change, compared per condition, with the delta reviewed before rollout. it's not glamorous. it's the difference between "we think the new model is better" and "the new model is better everywhere except low-light bin picks, and we're holding those cells back on the old version."

the deployment sign-off :

the output of evaluation is not a score, it's a decision with evidence attached. before a policy ships, the sign-off should contain :

  • sim evaluation results, and what they were checked against
  • real-robot runs per condition, with success rates and intervals
  • conditions with zero runs, listed, not implied
  • failure modes observed, with counts and replayable recordings
  • comparison against the previous deployed baseline
  • the do-not-deploy list : conditions that block rollout until tested

this is the same discipline we describe from the reliability side in robot deployment reliability : stop arguing from memory, start deciding from records.

frequently asked questions :

What is robot policy evaluation ?

Robot policy evaluation is the process of measuring whether a learned robot policy (a VLA, an RL policy, a learned controller) is fit to deploy. It answers three questions in order: does the policy work at all (simulation), does it work under the conditions of this deployment (statistical real-robot trials), and does it still work after every change (regression testing against a baseline).

Can I trust simulation results for my robot policy ?

As a filter, yes; as a sign-off, no. Simulation is the cheapest way to hunt for failure modes and to reject weak checkpoints early, but sim scores systematically diverge from field performance because the deployment's lighting, calibration drift, object variety, and sensor noise are not in the simulator. Use simulation to decide what to test on hardware, not to skip hardware testing.

How do you evaluate a VLA on a real robot ?

Define the condition matrix the deployment will face (lighting, object poses, calibration states, surfaces), run enough trials per condition to state a success rate with a confidence interval, record every run with the full system state so failures can be replayed, and compare results against the previous baseline before shipping. Typical published practice, 25 or fewer rollouts without confidence intervals, is not enough to support a deployment decision.

What is robot policy regression testing ?

Re-running a fixed baseline evaluation suite every time anything changes, the model checkpoint, the calibration, the hardware, the environment, and comparing per-condition results against the previous version. A new checkpoint can raise the average success rate while silently breaking a specific condition; regression testing is how you catch that before the field does.

benchmarks tell you the policy is promising. evaluation tells you it's ready.

we are building the infrastructure that makes the second one routine : capture every run, score reliability per condition, and show what changed when the numbers move.

Get a free audit today