what we are learning about deploying learned policies on physical machines : testing, validation, regressions, and the discipline of putting a defensible number on a robot.
the leaderboard says the model is capable, the matrix says whether it is ready for your floor. which conditions to vary for a VLA, how many rollouts, and how to compare a fine-tune against the last checkpoint.
software ships because the old features still work. robots rarely get that check. what counts as a change, why one green run is not a pass, and how to gate a release against a recorded baseline.
monitoring tells you a robot has stopped. it rarely tells you the pick has been getting worse for two weeks. three layers to watch, the metrics worth a dashboard, and the blind spots.
the demo shows the robot can do it, the deployment shows it keeps doing it. six steps from first pilot to full fleet, where deployments slip, and the checklist to run before go-live.
a fleet manager tells you where every robot is. it does not tell you how often the task succeeds. what fleet management software does, and the reliability evidence to add beside it.
benchmarks tell you a policy is promising, evaluation tells you it's ready. from sim scores to statistical real-robot campaigns to regression testing : the playbook for deployment sign-off.
a robot's logs tell you what the robot believes happened. a supervisor is the second witness : what the OORB Mobile Station carries, and what an outside record of every run is for.
a demo proves the robot can do the task once. reliability testing proves it will keep doing it. how many runs are enough, which conditions to cover, and what a defensible test report looks like.
the robots mostly worked, the deployments died anyway. verified numbers, documented post-mortems from Walmart to Zume to the robot hotel, and the four failure causes that keep repeating.
the market has named the gap between lab success and field reliability. this is the engineering playbook for closing it : six gaps, six concrete answers.
learned robot policies are not software releases, they are stochastic systems pushed into messy physical environments. the missing layer between a working demo and a defensible deployment is reliability engineering.