what VLA evaluation means :
VLA evaluation is measuring how a vision-language-action model performs on defined robot tasks : how often it succeeds, under which conditions, and how that changes when the scene, the instruction or the model changes.
a VLA is one kind of learned policy, so everything in robot policy evaluation applies. this page covers what is specific to a model that reads language and images.
why a VLA is harder to evaluate :
- the instruction is an input. the same task phrased another way is a different test, and operators do not read from a script.
- the scene is an input. lighting, the camera's pose, an object it has never seen and the clutter around the target all change what the model does.
- the output varies. the same checkpoint in the same scene does not produce the same motion twice, so one run proves nothing.
- a fine-tune changes everything at once. there is no single function to unit-test : the only check is to run the task again.
what the research measures :
the papers already name the axes. VLATest generates manipulation scenes to test VLA models, varying the number of confounding objects, the lighting, the camera pose, unseen objects and the wording of the instruction. its study of seven models found they lack the robustness needed for practical deployment.
a 2025 benchmarking study ran ACT, OpenVLA-OFT, RDT-1B and π0 on four manipulation tasks, in simulation and on ALOHA Mobile. it scores three things : success rate and time-to-success, adaptability from in-distribution to out-of-distribution settings, and how accurately the model follows the instruction.
the gap is in the sample sizes. a 2026 survey (PhAIL) found that real-world VLA evaluations typically run 25 or fewer rollouts per condition, almost always without confidence intervals.
an evaluation matrix for a VLA :
a benchmark picks its conditions for you. a deployment has its own. write them down before the first rollout, and vary them on purpose :
- lighting : the levels the site actually has, including the night shift
- camera pose : the mount as installed, and after it has been knocked
- object instance : the objects it trained on, and ones it has not seen
- object position : inside the trained workspace, and at its edges
- clutter : the number of other objects in the scene
- instruction wording : the phrasing operators will really use
- robot state : gripper wear, calibration age, start pose
- checkpoint : the candidate, and the version it would replace
you will not cover every combination. list the cells you ran, and list the ones you did not as untested. an untested cell is not a pass.
how many rollouts :
10 clean runs out of 10 only supports claiming "above roughly 74%" at 95% confidence. each cell of the matrix needs its own runs, and the number depends on the reliability you need to claim. the math is in robot reliability testing, and the way to report a rate with its interval is in measure robot task success rates.
comparing a fine-tune to the last checkpoint :
most VLA evaluation in a company is not "is this model good". it is "is this fine-tune better than the one on the robot". a fine-tune that lifts the average can still sink a condition : better grasps in daylight, worse grasps under warehouse LEDs.
keep the deployed checkpoint as a fixed baseline, run the candidate through the same cells, and compare per condition, not pooled. that is robot regression testing, and the step-by-step workflow is in validate a robot policy update before deployment.
what to keep with every rollout :
a success rate nobody can trace is an opinion. each rollout should keep the recording, the checkpoint the robot actually loaded, the code and dependencies, the robot and sensor configuration, the conditions and the outcome. see track robot model provenance and robot run provenance.
RunFP keeps that record from the data your robots already produce : ROS bags, MCAP files and test-rig exports. it runs on your own machines, so the rollouts stay on your network.
frequently asked questions :
What is VLA evaluation ?
VLA evaluation is measuring how a vision-language-action model performs on defined robot tasks: how often it succeeds, under which conditions, and how that changes when the scene, the instruction or the model changes.
How many rollouts do you need to evaluate a VLA ?
Enough per condition to put an interval on the result. Ten clean runs out of ten only supports claiming a success rate above roughly 74% at 95% confidence, and every condition in the matrix needs its own runs. A single pooled number over a handful of rollouts does not support a deployment decision.
Can a VLA be evaluated in simulation only ?
Simulation is the right place to reject weak checkpoints and to find failure modes cheaply. It does not replace runs on the real robot under the conditions of the actual site. Use simulation to decide what to test on hardware, not to decide that hardware testing is unnecessary.
How is VLA evaluation different from robot policy evaluation ?
A VLA is one kind of learned robot policy, so the method is the same: defined tasks, a condition matrix, enough runs and a baseline. A VLA adds two inputs that have to be varied on purpose, the wording of the instruction and the contents of the scene.
the leaderboard says the model is capable. the matrix says whether it is ready for your floor.
share the rollouts from one VLA behavior and we will show you how reliable it is by condition, where it fails, and what changed when it did.
Robot failing in the field? Get help now