The JEV foundation
Studying JEV reveals a useful control primitive: \(s\) is the observed state, \(q\) is a question, and \(\mathcal{O}=\{o_1,\ldots,o_K\}\) is a finite set of admissible commands.
ISPAMM Lab · Sapienza Università di Roma
Experiment 01 · From games to robotics
At each step PlayJev answers seven questions. The bars are the probabilities over the 25 bins of each axis, for the frame on screen. The highlighted bin is the most likely one; the number is the value executed.
We study what it takes to give Jev the ability of controlling a robot, where every decision is a precise, multidimensional movement.
01 · The architecture behind the study
JEV is the new model architecture from which this study begins. Instead of producing an unconstrained answer, it evaluates a finite set of admissible choices and returns a calibrated probability distribution over them. The result is a fast, inspectable decision layer that can sit between an observed state and an action.[1]
The JEV foundation
Studying JEV reveals a useful control primitive: \(s\) is the observed state, \(q\) is a question, and \(\mathcal{O}=\{o_1,\ldots,o_K\}\) is a finite set of admissible commands.
Choice readout
The language model runs once. Only the logits \(\ell_i\) of the valid answer letters are kept and normalized over the option set.
Decision
The output is a choice and its full probability distribution. No unconstrained token can become an action.
From architecture to robotics
The JEV architecture provides the decision mechanism. The open question is how to use it when the output is no longer “left”, “right”, or “jump”. At every step, a robot must coordinate position, orientation, and its gripper with continuous precision. The question addressed here is how far the JEV architecture can be extended across that gap.
02 · A visual child of JEV
JEV defines the parent interface: a state, a question, a finite set of options, and a probability for each valid answer. PlayJev is its visual specialization. It keeps that contract intact, but replaces a textual state with raw pixels and trains the decision model on interaction.[2]
raw game frame
Which valid move should the agent take?
Only logits assigned to valid answer letters become the action distribution.
Training procedure. PlayJev learns first from supervised cloning, then from three DAgger rounds that add teacher labels on states reached by the model itself.[3] Across 2.2 million game frames, one shared policy learns ten different games and reaches a mean teacher-normalized score of 0.57.
05 · The policy
The best-performing policy fuses each camera with its own previous frame: external \((t{-}1,t)\) and wrist \((t{-}1,t)\), two independent temporal pairs instead of one shared instant. PlayJev encodes both streams once, answers seven axis-specific Choice questions in parallel, and reconstructs one continuous robot action.
01 · two temporal pairs




02 · shared model
03 · one batched forward
7 Choice questions · shared visual state04 · distributions
05 · execute
The result
Every number on this page comes from the official LIBERO-Spatial evaluation protocol, the same one used throughout the published literature for this benchmark (OpenVLA, OpenVLA-OFT and the other systems in the table below): fifty fixed initial states per task — suite.get_task_init_states() — across the suite's ten tasks, five hundred episodes total, every one of the 432 demonstrations used as training data with nothing held out for validation. Each episode seeds the simulator (env.seed(0)), resets to its fixed initial state rather than a random one, runs ten no-op steps to let the scene settle, then allows up to 220 control steps at 256×256 camera resolution; success is read directly from env.check_success(), the suite's own goal predicate, with no custom scoring. We verified this byte-for-byte against OpenVLA-OFT's own evaluation code before running a single episode, so the comparison in this table is not approximate.
| Policy | Params | Successes | Rate | 95% interval |
|---|---|---|---|---|
| Octo, published | 93M | — | 78.9% | — |
| Diffusion Policy, published | ~260M | — | 78.3% | — |
| SpatialVLA, published | 4B | — | 88.2% | — |
| OpenVLA, our local reproduction | 7B | 442 / 500 | 88.4% | 85.6–91.0% |
| pi0, published | ~3.3B | — | 96.8% | — |
| CogACT, published | ~7.3B* | — | 97.2% | — |
| OpenVLA-OFT, our local reproduction | 7B | 488 / 500 | 97.6% | 96.2–98.8% |
| PlayJev, 8 steps ahead, both cameras temporally fused (ours) | 0.8B | 492 / 500 | 98.4% | 96.9–99.2% |
Our own rows (OpenVLA, OpenVLA-OFT, PlayJev) are 500-episode closed-loop evaluations run locally on the official initial states. The other rows are each author's own published number for LIBERO-Spatial, not re-run by us — different training runs, sometimes different episode counts or seeds, so treat cross-paper digits as approximate. *CogACT states its backbone (7B) and action-head size separately, not a single published total; ~7.3B is our own sum of the two, not a number from its paper. Every PlayJev ablation arm (camera combinations, proprioception, chunking, the same-instant camera pairing this arm improves on) is in Appendix C, not here.
Eight steps ahead, with both cameras each fused to their own previous frame, reaches 98.4% — above our own reproduction of OpenVLA-OFT (97.6%), with a 0.8B model against its 7B, and above every published number we could find for LIBERO-Spatial at any parameter count. By task it is close to flawless: 45 to 50 out of 50 on every one of the ten tasks, including the one that was hardest for every other arm in this study (36–39/50 elsewhere; 45/50 here, still its worst task but no longer far behind). Appendix C has the full arm-by-arm breakdown behind this number.
The parameter chart has no clean trend: a 93M-parameter policy and a 7B one land within a point of each other (78.9% vs. 88.4%), and the two largest systems we could find a number for (CogACT, OpenVLA-OFT, both ~7B) sit a point apart from each other. Scale correlates loosely with success across this set, but it does not explain PlayJev's position — the smallest model on the chart (0.8B) is also the highest. The latency chart, restricted to what we actually measured on one machine — the same RTX A6000 for all three systems — shows the same shape OpenVLA-OFT's own chunking argument makes: querying once for eight actions (134.7ms) is both faster and more accurate than querying once per action, whether that single query belongs to OpenVLA (245.7ms, 88.4%) or to PlayJev without chunked execution (181ms forward pass, 98.4% — the accuracy gain here comes from the training horizon and the camera fusion, not from chunking, since the closed loop above still executes one action per query; Appendix C's chunked variant shows what happens once it also executes in blocks of eight). PlayJev's number is forward-pass time only, not the full request (OpenVLA's and OpenVLA-OFT's figures include their own image and action preprocessing); image preprocessing and HTTP overhead add roughly 50ms on top in our own evaluation, which the chart does not count.
Examples
Every clip below is a real rollout of the eight-step arm with both cameras temporally fused (98.4%, 492/500) on the official protocol, replayed from the exact logged seed and initial state. Nothing is cherry-picked past the task spread: hover a tile for the task and outcome.
Hover an element to see the task's textual instruction.
05b · Horizon
The same bin25 policy can also be asked about the next actions of each state, not just the current one: train it to answer about four or eight steps ahead instead of one, while the closed loop still executes only the current action every time it queries the model. The horizon changes what the policy learns from the same demonstrations; it does not, on its own, change what one inference step costs.
On the official five-hundred-episode protocol (§ above), asking about four steps ahead moves success from 83.4% to 88.4%; asking about eight moves it further, to 90.8%, and combined with the wrist camera, to 98.2%. The gain is not free forever: on the external camera alone, going from one to four steps ahead adds 5 points, from four to eight adds another 2.4, a shrinking return for a more demanding training target. With the wrist camera the curve is not even monotonic — M=4 (91.0%) dips slightly below M=1 (91.6%) before M=8 jumps to 98.2% — a reminder that horizon and camera input interact rather than simply add. This sweep keeps the camera pairing fixed (both views at the same instant) so M=1, M=4 and M=8 stay comparable; the strongest result in this study, 98.4%, comes from changing that pairing too — fusing each camera with its own previous frame — tested only at M=8, where this sweep says the wrist camera matters most. The result section above and Appendix C have that arm.
By default the closed loop queries the model and executes one action at a time, even for an arm trained to answer eight steps ahead — the training horizon buys accuracy but not speed on its own. We built the open-loop version — one query, eight actions executed before asking again, the same pattern OpenVLA-OFT uses — and it costs nothing in success (90.4%, no real change from 90.8%) while cutting model queries by 87%, from about 117 per episode to about 15.
06 · Future work
LIBERO · Goal
Tasks with different goals: drawers, stoves, cabinets, bowls, and plates. This directly tests dependence on Spatial’s ten fixed sentences.
Replicate the winning Spatial arm unchanged — bin25, eight steps ahead, external + wrist camera each fused with its own previous frame — then train with 3–5 validated paraphrases per instruction. Evaluate canonical prompts, seen paraphrases, and held-out language templates separately.
LIBERO · Long
Multi-stage sequences: two objects, drawers to close, stoves to switch on. A purely reactive policy is no longer enough; action chunking provides a relevant temporal baseline.[11]
The codec question is already settled by the oracle replay gate above — bin25 unchanged, not re-run. Start from the winning Spatial arm and compare it, as-is, against a history-conditioned variant and a recovery dataset built from states visited after a missed grasp.
LIBERO · Object
The same pick-and-place structure with different objects. A clean control for separating motor control from visual generalization.
Compare Spatial-only, Object-only, and joint training, all starting from the same winning architecture. Sample by task rather than by frame, with robot replay to measure and limit catastrophic forgetting.
LIBERO · Spatial, revisited
Proprioception as text cost 6 points even after removing the gripper-reading confound (Appendix C) — the open question is whether that gap is the text representation itself, not the information.
Replace the proprioception sentence with a small MLP projector over ee_pos/ee_ori/gripper_states, concatenated as one continuous token after the image tokens — the same pattern OpenVLA-OFT's own projector uses, never rendered as text. Compare against the camera-only baseline (83.4%) and the text-sentence arms (42.4% / 77.4%) at matched training budget.
The lesson
A finite-choice model does not need more parameters to control a continuous robot.
It needs the right alphabet.
The discretization choice carried the whole study. At the representation level, a coarse, redundant scheme — 25 uniform bins per axis, seven questions per step — beat a learned codebook that reconstructed every demonstration exactly; precision that a policy cannot reliably hit is not an advantage. At the system level, the same choice of alphabet let an 0.8B model answering multiple-choice questions reach 98.4% on the official protocol, above our own reproduction of a published 7B VLA system, with no architecture change beyond the interface itself and how it reads its two cameras. Scale did not do that work; the alphabet did.
This is a preliminary study, not a final one — LIBERO-Spatial only, one training seed per arm, a single robot and a fixed set of ten tasks. What it establishes is narrower but still useful: that a JEV-style finite-choice model, the same kind built for picking a move in a game, can be adapted to continuous low-level control at all, and that a small one can be competitive while doing it. Appendix C and the roadmap above are what's left before this becomes more than a first result: proprioception as a real continuous token instead of text, and the same interface tested against language and memory, not spatial reasoning alone.
Appendix A · The representation that follows
This is where the alphabet becomes necessary: it translates a continuous robot action into choices PlayJev can make. Each of the seven control dimensions is quantized separately.
The control variable
At each control step \(t\), LIBERO expects six continuous end-effector deltas and one binary gripper command.[4] A single 26-letter question cannot represent this space at useful precision, so we factor the action by dimension.
The first attempt
We first clustered the expert actions into a small codebook. Here, \(K\) is the number of 7D action prototypes in that codebook: each prototype contains all six motion deltas and the gripper command. At every step, the continuous expert action is replaced by its nearest prototype, selected with a single letter.
A larger \(K\) offers more candidate movements, but it still forces the seven control dimensions to vary together. The replay below isolates that representation error: no learned policy is involved.
loading…The original command from the demonstration.
loading…0 / 48 successful oracle replays.
loading…2 / 48 successful oracle replays.
All three rollouts start from the same simulator state and remain synchronized by control step. Their images diverge as each method executes its own command; the vectors below show the expert action and its K16 and K26 reconstructions at that step.
Even the larger codebook fails: choosing one whole-action prototype cannot preserve the independent precision required by each control axis. This motivates the factorized alphabet tested next.
Appendix B · Oracle replay
Before PlayJev can learn an action alphabet, that alphabet must preserve the control signal it is meant to describe. Every codec replaces the expert’s continuous trajectory with symbols, then reconstructs an approximation. If that approximation already misses the grasp or drifts away from the goal, the learning problem is ill-posed: the model is being asked to predict letters that cannot reproduce the correct behavior.
Oracle replay isolates this representation bottleneck. We give each codec the expert actions directly—without vision, language, or policy prediction—and execute only its reconstruction in the simulator. Success is a necessary gate: first prove that the alphabet can carry the movement, then ask PlayJev to learn it.
Three ways to write motion
The methods differ in what they treat as a token. BIN encodes one scalar at one time step. FAST encodes patterns across time. OAT learns a short, ordered latent description of the whole chunk.
Each action dimension is independently assigned to one of \(B\) uniform levels at every time step. Decoding maps the index back to the corresponding scalar. This is the direct action-token approach used by RT-2 and OpenVLA.[5][6]
FAST applies a discrete cosine transform along time, scales and rounds the coefficients, then compresses the resulting integer stream with byte-pair encoding. Smooth trajectories concentrate their information in a few low-frequency coefficients.[7]
OAT uses a transformer autoencoder and finite scalar quantization. Nested dropout trains each prefix of the latent sequence to reconstruct the chunk, ordering tokens from coarse to fine.[8][9][10]
Every replay succeeds. No codec training required.
Seven tokens per step × 32 steps. Precise, predictable, but verbose.
Oracle replay removes perception and policy learning from the experiment. It asks one necessary—but not sufficient—question: if the tokenizer is given the expert’s own actions, can its reconstruction still complete the task?
The decisive clue
OAT can make only a small error at each step. But when those errors share the same sign, they add up. By the end of the trajectory, the robot is no longer where it thinks it is.[8]
Integrated drift predicts failure better than per-step error.
Median maximum deviation in failed OAT replays.
Median maximum deviation in successful replays.
Appendix C · Ablations
The main results compare PlayJev's best arm against other published systems. This appendix is the arm-by-arm record behind that one number: every camera, horizon and proprioception variant on the same 500-episode official protocol, with the reasoning behind why each one lands where it does.
| Arm | Horizon | Successes | Rate | 95% interval |
|---|---|---|---|---|
| External camera (reference) | M=1 | 417 / 500 | 83.4% | 79.9–86.4% |
| + wrist camera | M=1 | 458 / 500 | 91.6% | 88.8–93.7% |
| External camera, 4 steps ahead | M=4 | 442 / 500 | 88.4% | 85.3–90.9% |
| External camera, 8 steps ahead | M=8 | 454 / 500 | 90.8% | 87.9–93.0% |
| ↳ executed in open-loop blocks of 8 | M=8 | 452 / 500 | 90.4% | 87.5–92.7% |
| 4 steps ahead + wrist | M=4 | 455 / 500 | 91.0% | 88.2–93.2% |
| 8 steps ahead + wrist, same instant t | M=8 | 491 / 500 | 98.2% | 96.6–99.1% |
| ↳ both cameras fused (t-1,t) instead (best arm) | M=8 | 492 / 500 | 98.4% | 96.9–99.2% |
| + proprioception (text, with gripper reading) | M=1 | 212 / 500 | 42.4% | 38.1–46.8% |
| + proprioception (text, gripper reading removed) | M=1 | 387 / 500 | 77.4% | 73.5–80.8% |
| + wrist camera + proprioception (text, with gripper reading) | M=4 | 211 / 500 | 42.2% | 37.9–46.6% |
One training seed per arm. A second seed has not run yet.
Pairing both cameras at the same instant, exactly as trained, the wrist camera helps every time it is added: +8.2 points alone, +2.6 points on top of four steps ahead, and the combination with eight steps ahead was the best result in the study — until fusing each camera with its own previous frame, below, pushed it 0.2 points further. The per-task breakdown is uniformly strong across all ten tasks, with no single weak point.
The fixed arm above still reads both cameras at the same instant: neither stream carries motion on its own, only the combination of two static views. We built a genuinely temporal variant instead — external paired with its own previous frame, wrist paired with its own previous frame, each pair fused independently before the two streams are concatenated — so both cameras carry real motion information rather than none. Token budget is unchanged (128 visual tokens either way, confirmed against the real image processor, not just computed by hand), so this is not a bigger model, only a different use of the same input.
At M=8 this reaches 492/500 (98.4%), a further 0.2 points over the same-instant arm's 491/500 — a small, plausibly real gain rather than a dramatic one, but it moves in the direction the architecture predicts and its per-task floor rises too (45/50 on the hardest task, against 46/50 same-instant and 36–39/50 for every external-only arm). We did not re-run the M=1/M=4 points on this fused architecture, so the table above still uses the same-instant arm for the clean M=1→4→8 sweep; this row is the one additional, more literal reading of "two cameras," tested only at the horizon where the wrist camera's effect has been largest.
Comparing the same 500 episodes step by step against the camera-only baseline, the failure is not spread across the whole policy: it is almost entirely the gripper. 36% of its failed episodes never close the gripper at all, against 2% for the baseline; when it does close, it takes until step 71 on average instead of 43. The likely reason: the sentence includes the gripper's own raw finger-position reading, written in text, immediately before asking the model to decide whether to open or close that same gripper — two different things sitting right next to each other in the prompt.
Remove just that one field and performance recovers from 42.4% to 77.4%, and the never-closes-the-gripper failure disappears completely (zero of its failed episodes, better than even the baseline's 2%). The remaining 6-point gap below the camera-only baseline is still open: writing numbers as text loses the metric structure a dedicated continuous encoder would keep; every arm shares the same training budget, so proprioception arms get no extra steps to learn the added parsing task; the extra text competes with the image for a small model's attention; or this task, solvable almost entirely from the external camera alone, may simply not need it.
Adding the wrist camera and more horizon on top (M=4, same gripper-reading sentence) does not rescue it: 42.2%, statistically indistinguishable from the M=1 camera-only-external number above, and the gripper failure is if anything worse — 74% of its failed episodes never close the gripper at all, against 36% for the simpler arm. Whatever the text sentence confuses, a second camera and a longer training horizon do not fix it; only removing the confusing field did.
PlayJev was trained to answer about eight steps ahead, but by default the closed loop still queried the model and executed one action at a time — the same cost per episode as the one-step policy, so the training horizon bought accuracy but not speed. We built the open-loop version — one query, eight actions executed before asking again, the same pattern OpenVLA-OFT uses — and it costs nothing in success (90.4%, no real change from 90.8%) while cutting model queries by 87%, from about 117 per episode to about 15.
Sources
Primary papers and project sources linked from the relevant claims above.
System One finite-choice request and response contract. OmniJev, project repository.
OmniJev, model, training code, evaluation and weights, 2026.
S. Ross, G. Gordon, J. A. Bagnell, AISTATS 2011.
B. Liu, Y. Zhu, C. Gao et al., 2023.
A. Brohan, N. Brown, J. Carbajal et al., 2023.
M. J. Kim, K. Pertsch, S. Karamcheti et al., 2024.
K. Pertsch, K. Stachowicz, B. Ichter et al., 2025.
C. Liu, X. Han, J. Gao et al., 2026.
F. Mentzer, D. Minnen, E. Agustsson, M. Tschannen, ICLR 2024.
O. Rippel, M. A. Gelbart, R. P. Adams, ICML 2014.
T. Z. Zhao, V. Kumar, S. Levine, C. Finn, RSS 2023. Introduces Action Chunking with Transformers.
X. Zhou, Y. Xu, G. Tie et al., 2025.