ISPAMM Lab · Sapienza Università di Roma

Experiment 01 · From games to robotics

Jev for continuous robot control.

One robot · pick up the black bowl · success

At each step PlayJev answers seven questions. The bars are the probabilities over the 25 bins of each axis, for the frame on screen. The highlighted bin is the most likely one; the number is the value executed.

We study what it takes to give Jev the ability of controlling a robot, where every decision is a precise, multidimensional movement.

0.8Bparameters
98.4%success · 492/500, official LIBERO-Spatial protocol
181 msforward pass, p50
01

01 · The architecture behind the study

JEV turns language-model prediction into a decision.

JEV is the new model architecture from which this study begins. Instead of producing an unconstrained answer, it evaluates a finite set of admissible choices and returns a calibrated probability distribution over them. The result is a fast, inspectable decision layer that can sit between an observed state and an action.[1]

I

The JEV foundation

\[\mathcal{D}=(s,q,\mathcal{O})\]

Studying JEV reveals a useful control primitive: \(s\) is the observed state, \(q\) is a question, and \(\mathcal{O}=\{o_1,\ldots,o_K\}\) is a finite set of admissible commands.

II

Choice readout

\[p_\theta(o_i\mid\mathcal{D})=\frac{\exp(\ell_i)}{\sum_{j\in\mathcal{O}}\exp(\ell_j)}\]

The language model runs once. Only the logits \(\ell_i\) of the valid answer letters are kept and normalized over the option set.

III

Decision

\[o^*=\operatorname*{arg\,max}_{o_i\in\mathcal{O}}p_\theta(o_i\mid\mathcal{D})\]

The output is a choice and its full probability distribution. No unconstrained token can become an action.

STATE \(s\)pixels or text
QUESTION \(q\)what should happen?
OPTIONS \(\mathcal{O}\)A · B · C · …
one forward passJEV-style model
simplex \(\Delta^{K-1}\)\([0.08,\,0.71,\,0.21]\)

From architecture to robotics

A finite choice is simple.
A robot action is not.

The JEV architecture provides the decision mechanism. The open question is how to use it when the output is no longer “left”, “right”, or “jump”. At every step, a robot must coordinate position, orientation, and its gripper with continuous precision. The question addressed here is how far the JEV architecture can be extended across that gap.

02

02 · A visual child of JEV

PlayJev inherits the decision contract—and learns to see.

JEV defines the parent interface: a state, a question, a finite set of options, and a probability for each valid answer. PlayJev is its visual specialization. It keeps that contract intact, but replaces a textual state with raw pixels and trains the decision model on interaction.[2]

PARENT CONTRACTJEVfinite-choice decision interface
VISUAL SPECIALIZATIONPlayJevpixels in · action probabilities out
MODELQwen3.5 · 0.8B vision-language transformer STATEone visual frame no game name or internal state OPTIONSshuffled moves position cannot reveal the answer READOUTvalid-letter logits normalized into a policy
Frames from ten games controlled by PlayJev raw game frame
ONE FRAME · ONE FORWARD PASS

Which valid move should the agent take?

Only logits assigned to valid answer letters become the action distribution.

Training procedure. PlayJev learns first from supervised cloning, then from three DAgger rounds that add teacher labels on states reached by the model itself.[3] Across 2.2 million game frames, one shared policy learns ten different games and reaches a mean teacher-normalized score of 0.57.

05 · The policy

Seven questions.
One movement.

The best-performing policy fuses each camera with its own previous frame: external \((t{-}1,t)\) and wrist \((t{-}1,t)\), two independent temporal pairs instead of one shared instant. PlayJev encodes both streams once, answers seven axis-specific Choice questions in parallel, and reconstructs one continuous robot action.

The result

Official evaluation protocol

Every number on this page comes from the official LIBERO-Spatial evaluation protocol, the same one used throughout the published literature for this benchmark (OpenVLA, OpenVLA-OFT and the other systems in the table below): fifty fixed initial states per task — suite.get_task_init_states() — across the suite's ten tasks, five hundred episodes total, every one of the 432 demonstrations used as training data with nothing held out for validation. Each episode seeds the simulator (env.seed(0)), resets to its fixed initial state rather than a random one, runs ten no-op steps to let the scene settle, then allows up to 220 control steps at 256×256 camera resolution; success is read directly from env.check_success(), the suite's own goal predicate, with no custom scoring. We verified this byte-for-byte against OpenVLA-OFT's own evaluation code before running a single episode, so the comparison in this table is not approximate.

PolicyParamsSuccessesRate95% interval
Octo, published93M—78.9%—
Diffusion Policy, published~260M—78.3%—
SpatialVLA, published4B—88.2%—
OpenVLA, our local reproduction7B442 / 50088.4%85.6–91.0%
pi0, published~3.3B—96.8%—
CogACT, published~7.3B*—97.2%—
OpenVLA-OFT, our local reproduction7B488 / 50097.6%96.2–98.8%
PlayJev, 8 steps ahead, both cameras temporally fused (ours)0.8B492 / 50098.4%96.9–99.2%

Our own rows (OpenVLA, OpenVLA-OFT, PlayJev) are 500-episode closed-loop evaluations run locally on the official initial states. The other rows are each author's own published number for LIBERO-Spatial, not re-run by us — different training runs, sometimes different episode counts or seeds, so treat cross-paper digits as approximate. *CogACT states its backbone (7B) and action-head size separately, not a single published total; ~7.3B is our own sum of the two, not a number from its paper. Every PlayJev ablation arm (camera combinations, proprioception, chunking, the same-instant camera pairing this arm improves on) is in Appendix C, not here.

Success rate vs. parametersall rows above, log-scale x-axis
75% 80% 85% 90% 95% 100% 100M 300M 1B 3B 7B parameters (log scale) Octo Diffusion Policy PlayJev (ours) pi0 SpatialVLA OpenVLA CogACT OpenVLA-OFT
Success rate vs. inference latencyonly the three systems measured on the same local GPU, an RTX A6000
85% 90% 95% 100% 120ms 160ms 200ms 240ms 280ms median latency per model query OpenVLA-OFT 8 actions / query PlayJev (ours) forward pass only OpenVLA 1 action / query

Eight steps ahead, with both cameras each fused to their own previous frame, reaches 98.4% — above our own reproduction of OpenVLA-OFT (97.6%), with a 0.8B model against its 7B, and above every published number we could find for LIBERO-Spatial at any parameter count. By task it is close to flawless: 45 to 50 out of 50 on every one of the ten tasks, including the one that was hardest for every other arm in this study (36–39/50 elsewhere; 45/50 here, still its worst task but no longer far behind). Appendix C has the full arm-by-arm breakdown behind this number.

The parameter chart has no clean trend: a 93M-parameter policy and a 7B one land within a point of each other (78.9% vs. 88.4%), and the two largest systems we could find a number for (CogACT, OpenVLA-OFT, both ~7B) sit a point apart from each other. Scale correlates loosely with success across this set, but it does not explain PlayJev's position — the smallest model on the chart (0.8B) is also the highest. The latency chart, restricted to what we actually measured on one machine — the same RTX A6000 for all three systems — shows the same shape OpenVLA-OFT's own chunking argument makes: querying once for eight actions (134.7ms) is both faster and more accurate than querying once per action, whether that single query belongs to OpenVLA (245.7ms, 88.4%) or to PlayJev without chunked execution (181ms forward pass, 98.4% — the accuracy gain here comes from the training horizon and the camera fusion, not from chunking, since the closed loop above still executes one action per query; Appendix C's chunked variant shows what happens once it also executes in blocks of eight). PlayJev's number is forward-pass time only, not the full request (OpenVLA's and OpenVLA-OFT's figures include their own image and action preprocessing); image preprocessing and HTTP overhead add roughly 50ms on top in our own evaluation, which the chart does not count.

Examples

The best policy, watched directly.
Sixteen successes — and all eight times it failed.

Every clip below is a real rollout of the eight-step arm with both cameras temporally fused (98.4%, 492/500) on the official protocol, replayed from the exact logged seed and initial state. Nothing is cherry-picked past the task spread: hover a tile for the task and outcome.

Hover an element to see the task's textual instruction.

05b · Horizon

How far ahead should it plan?

The same bin25 policy can also be asked about the next actions of each state, not just the current one: train it to answer about four or eight steps ahead instead of one, while the closed loop still executes only the current action every time it queries the model. The horizon changes what the policy learns from the same demonstrations; it does not, on its own, change what one inference step costs.

80% 85% 90% 95% 100% M=1 M=4 M=8 action horizon (steps asked ahead per query) 83.4% 88.4% 90.8% external camera 91.6% 91.0% 98.2% external + wrist

On the official five-hundred-episode protocol (§ above), asking about four steps ahead moves success from 83.4% to 88.4%; asking about eight moves it further, to 90.8%, and combined with the wrist camera, to 98.2%. The gain is not free forever: on the external camera alone, going from one to four steps ahead adds 5 points, from four to eight adds another 2.4, a shrinking return for a more demanding training target. With the wrist camera the curve is not even monotonic — M=4 (91.0%) dips slightly below M=1 (91.6%) before M=8 jumps to 98.2% — a reminder that horizon and camera input interact rather than simply add. This sweep keeps the camera pairing fixed (both views at the same instant) so M=1, M=4 and M=8 stay comparable; the strongest result in this study, 98.4%, comes from changing that pairing too — fusing each camera with its own previous frame — tested only at M=8, where this sweep says the wrist camera matters most. The result section above and Appendix C have that arm.

By default the closed loop queries the model and executes one action at a time, even for an arm trained to answer eight steps ahead — the training horizon buys accuracy but not speed on its own. We built the open-loop version — one query, eight actions executed before asking again, the same pattern OpenVLA-OFT uses — and it costs nothing in success (90.4%, no real change from 90.8%) while cutting model queries by 87%, from about 117 per episode to about 15.

06 · Future work

Spatial built the alphabet.
Now we test language and memory.

01

LIBERO · Goal

Does it understand what we ask?

Tasks with different goals: drawers, stoves, cabinets, bowls, and plates. This directly tests dependence on Spatial’s ten fixed sentences.

  • Controlled paraphrases
  • Minimal instruction pairs
  • Language consistency loss

Replicate the winning Spatial arm unchanged — bin25, eight steps ahead, external + wrist camera each fused with its own previous frame — then train with 3–5 validated paraphrases per instruction. Evaluate canonical prompts, seen paraphrases, and held-out language templates separately.

02

LIBERO · Long

Does it remember what it already did?

Multi-stage sequences: two objects, drawers to close, stoves to switch on. A purely reactive policy is no longer enough; action chunking provides a relevant temporal baseline.[11]

  • Recent action history
  • Subgoals and recovery
  • Same bin25 codec

The codec question is already settled by the oracle replay gate above — bin25 unchanged, not re-run. Start from the winning Spatial arm and compare it, as-is, against a history-conditioned variant and a recovery dataset built from states visited after a missed grasp.

03

LIBERO · Object

Does it see an object—or “the black bowl”?

The same pick-and-place structure with different objects. A clean control for separating motor control from visual generalization.

  • Cross-suite transfer
  • Texture, shape, category
  • Balanced joint training

Compare Spatial-only, Object-only, and joint training, all starting from the same winning architecture. Sample by task rather than by frame, with robot replay to measure and limit catastrophic forgetting.

04

LIBERO · Spatial, revisited

Does a real encoder close the last six points?

Proprioception as text cost 6 points even after removing the gripper-reading confound (Appendix C) — the open question is whether that gap is the text representation itself, not the information.

  • Small continuous MLP projector
  • One token, not a sentence
  • Same training budget

Replace the proprioception sentence with a small MLP projector over ee_pos/ee_ori/gripper_states, concatenated as one continuous token after the image tokens — the same pattern OpenVLA-OFT's own projector uses, never rendered as text. Compare against the camera-only baseline (83.4%) and the text-sentence arms (42.4% / 77.4%) at matched training budget.

Then · evaluation only

LIBERO-PRO

Freeze the policy and perturb one variable at a time: position, object, language, task, and environment. First measure zero-shot generalization; only then adapt the training.[12]

positionobjectsemantictaskenvironment

The lesson

A finite-choice model does not need more parameters to control a continuous robot.
It needs the right alphabet.

The discretization choice carried the whole study. At the representation level, a coarse, redundant scheme — 25 uniform bins per axis, seven questions per step — beat a learned codebook that reconstructed every demonstration exactly; precision that a policy cannot reliably hit is not an advantage. At the system level, the same choice of alphabet let an 0.8B model answering multiple-choice questions reach 98.4% on the official protocol, above our own reproduction of a published 7B VLA system, with no architecture change beyond the interface itself and how it reads its two cameras. Scale did not do that work; the alphabet did.

This is a preliminary study, not a final one — LIBERO-Spatial only, one training seed per arm, a single robot and a fixed set of ten tasks. What it establishes is narrower but still useful: that a JEV-style finite-choice model, the same kind built for picking a move in a game, can be adapted to continuous low-level control at all, and that a small one can be competitive while doing it. Appendix C and the roadmap above are what's left before this becomes more than a first result: proprioception as a real continuous token instead of text, and the same interface tested against language and memory, not spatial reasoning alone.

Appendix A · The representation that follows

Continuous motion needs
a usable alphabet.

This is where the alphabet becomes necessary: it translates a continuous robot action into choices PlayJev can make. Each of the seven control dimensions is quantized separately.

The control variable

\[a_t=(\Delta x,\Delta y,\Delta z,\Delta\alpha,\Delta\beta,\Delta\gamma,g)\in\mathbb{R}^6\times\{-1,+1\}\]

At each control step \(t\), LIBERO expects six continuous end-effector deltas and one binary gripper command.[4] A single 26-letter question cannot represent this space at useful precision, so we factor the action by dimension.

The first attempt

One letter for an entire action.

We first clustered the expert actions into a small codebook. Here, \(K\) is the number of 7D action prototypes in that codebook: each prototype contains all six motion deltas and the gripper command. At every step, the continuous expert action is replaced by its nearest prototype, selected with a single letter.

A larger \(K\) offers more candidate movements, but it still forces the seven control dimensions to vary together. The replay below isolates that representation error: no learned policy is involved.

frame 1 / 110

Yellow intervals mark where the expert command closes the gripper. Hover or focus a marker for its exact frame range.

REFERENCE

Expert action

Expert rollout at the selected control step
7D CONTROLloading…

The original command from the demonstration.

ONE LETTER

\(K=16\)

K equals 16 rollout at the selected control step
NEAREST PROTOTYPEloading…

0 / 48 successful oracle replays.

ONE LETTER

\(K=26\)

K equals 26 rollout at the selected control step
NEAREST PROTOTYPEloading…

2 / 48 successful oracle replays.

All three rollouts start from the same simulator state and remain synchronized by control step. Their images diverge as each method executes its own command; the vectors below show the expert action and its K16 and K26 reconstructions at that step.

Even the larger codebook fails: choosing one whole-action prototype cannot preserve the independent precision required by each control axis. This motivates the factorized alphabet tested next.

Appendix B · Oracle replay

Can the alphabet express
the right movement?

Before PlayJev can learn an action alphabet, that alphabet must preserve the control signal it is meant to describe. Every codec replaces the expert’s continuous trajectory with symbols, then reconstructs an approximation. If that approximation already misses the grasp or drifts away from the goal, the learning problem is ill-posed: the model is being asked to predict letters that cannot reproduce the correct behavior.

Oracle replay isolates this representation bottleneck. We give each codec the expert actions directly—without vision, language, or policy prediction—and execute only its reconstruction in the simulator. Success is a necessary gate: first prove that the alphabet can carry the movement, then ask PlayJev to learn it.

Three ways to write motion

Same trajectory.
Different symbolic structure.

The methods differ in what they treat as a token. BIN encodes one scalar at one time step. FAST encodes patterns across time. OAT learns a short, ordered latent description of the whole chunk.

01

BIN

local · deterministic
\[q_B(a)=\operatorname{round}\!\left(\frac{a-a_{\min}}{a_{\max}-a_{\min}}(B-1)\right)\]

Each action dimension is independently assigned to one of \(B\) uniform levels at every time step. Decoding maps the index back to the corresponding scalar. This is the direct action-token approach used by RT-2 and OpenVLA.[5][6]

Unit
axis × step
Training
none
Risk
long sequences
02

FAST

frequency · variable length
\[z=\operatorname{BPE}\!\left(\operatorname{round}(s\,\operatorname{DCT}(a_{1:H}))\right)\]

FAST applies a discrete cosine transform along time, scales and rounds the coefficients, then compresses the resulting integer stream with byte-pair encoding. Smooth trajectories concentrate their information in a few low-frequency coefficients.[7]

Unit
temporal frequency
Training
BPE only
Risk
rounding overshoot
03

OAT

learned · ordered latent
\[z_{1:L}=Q(E_\phi(a_{1:H})),\qquad \hat a_{1:H}=D_\psi(z_{1:L})\]

OAT uses a transformer autoencoder and finite scalar quantization. Nested dropout trains each prefix of the latent sequence to reconstruct the chunk, ordering tokens from coarse to fine.[8][9][10]

Unit
ordered latent
Training
autoencoder
Risk
integrated drift
Per-axis binning · 256 40/40

Every replay succeeds. No codec training required.

Replay success
100%
Token / chunk
224
RMSE xyz
0.002

Seven tokens per step × 32 steps. Precise, predictable, but verbose.

expert actions \(a_{1:T}\)encode \(E\) \(\rightarrow\)tokens \(z\) \(\rightarrow\)decode \(D\) \(\rightarrow\)\(\hat a_{1:T}\) \(\rightarrow\)
\[\hat a_{1:T}=D\!\left(E(a_{1:T})\right)\]

Oracle replay removes perception and policy learning from the experiment. It asks one necessary—but not sufficient—question: if the tokenizer is given the expert’s own actions, can its reconstruction still complete the task?

The decisive clue

It is not only how much you miss.
It is missing in the same direction.

OAT can make only a small error at each step. But when those errors share the same sign, they add up. By the end of the trajectory, the robot is no longer where it thinks it is.[8]

EXPERT OAT REPLAY
integrated drift0.0 cm
AUC0.84

Integrated drift predicts failure better than per-step error.

FAIL6.0 cm

Median maximum deviation in failed OAT replays.

SUCCESS2.9 cm

Median maximum deviation in successful replays.

Appendix C · Ablations

Every arm we trained,
and what each change actually did.

The main results compare PlayJev's best arm against other published systems. This appendix is the arm-by-arm record behind that one number: every camera, horizon and proprioception variant on the same 500-episode official protocol, with the reasoning behind why each one lands where it does.

ArmHorizonSuccessesRate95% interval
External camera (reference)M=1417 / 50083.4%79.9–86.4%
+ wrist cameraM=1458 / 50091.6%88.8–93.7%
External camera, 4 steps aheadM=4442 / 50088.4%85.3–90.9%
External camera, 8 steps aheadM=8454 / 50090.8%87.9–93.0%
↳ executed in open-loop blocks of 8M=8452 / 50090.4%87.5–92.7%
4 steps ahead + wristM=4455 / 50091.0%88.2–93.2%
8 steps ahead + wrist, same instant tM=8491 / 50098.2%96.6–99.1%
↳ both cameras fused (t-1,t) instead (best arm)M=8492 / 50098.4%96.9–99.2%
+ proprioception (text, with gripper reading)M=1212 / 50042.4%38.1–46.8%
+ proprioception (text, gripper reading removed)M=1387 / 50077.4%73.5–80.8%
+ wrist camera + proprioception (text, with gripper reading)M=4211 / 50042.2%37.9–46.6%

One training seed per arm. A second seed has not run yet.

The wrist camera is a consistent win, paired at the same instant as the external camera.

Pairing both cameras at the same instant, exactly as trained, the wrist camera helps every time it is added: +8.2 points alone, +2.6 points on top of four steps ahead, and the combination with eight steps ahead was the best result in the study — until fusing each camera with its own previous frame, below, pushed it 0.2 points further. The per-task breakdown is uniformly strong across all ten tasks, with no single weak point.

"Two cameras" can mean something more literal — and it is worth the extra 0.2 points.

The fixed arm above still reads both cameras at the same instant: neither stream carries motion on its own, only the combination of two static views. We built a genuinely temporal variant instead — external paired with its own previous frame, wrist paired with its own previous frame, each pair fused independently before the two streams are concatenated — so both cameras carry real motion information rather than none. Token budget is unchanged (128 visual tokens either way, confirmed against the real image processor, not just computed by hand), so this is not a bigger model, only a different use of the same input.

At M=8 this reaches 492/500 (98.4%), a further 0.2 points over the same-instant arm's 491/500 — a small, plausibly real gain rather than a dramatic one, but it moves in the direction the architecture predicts and its per-task floor rises too (45/50 on the hardest task, against 46/50 same-instant and 36–39/50 for every external-only arm). We did not re-run the M=1/M=4 points on this fused architecture, so the table above still uses the same-instant arm for the clean M=1→4→8 sweep; this row is the one additional, more literal reading of "two cameras," tested only at the horizon where the wrist camera's effect has been largest.

Proprioception is the one change that clearly hurts — and we now know exactly where.

Comparing the same 500 episodes step by step against the camera-only baseline, the failure is not spread across the whole policy: it is almost entirely the gripper. 36% of its failed episodes never close the gripper at all, against 2% for the baseline; when it does close, it takes until step 71 on average instead of 43. The likely reason: the sentence includes the gripper's own raw finger-position reading, written in text, immediately before asking the model to decide whether to open or close that same gripper — two different things sitting right next to each other in the prompt.

Remove just that one field and performance recovers from 42.4% to 77.4%, and the never-closes-the-gripper failure disappears completely (zero of its failed episodes, better than even the baseline's 2%). The remaining 6-point gap below the camera-only baseline is still open: writing numbers as text loses the metric structure a dedicated continuous encoder would keep; every arm shares the same training budget, so proprioception arms get no extra steps to learn the added parsing task; the extra text competes with the image for a small model's attention; or this task, solvable almost entirely from the external camera alone, may simply not need it.

Adding the wrist camera and more horizon on top (M=4, same gripper-reading sentence) does not rescue it: 42.2%, statistically indistinguishable from the M=1 camera-only-external number above, and the gripper failure is if anything worse — 74% of its failed episodes never close the gripper at all, against 36% for the simpler arm. Whatever the text sentence confuses, a second camera and a longer training horizon do not fix it; only removing the confusing field did.

Chunked open-loop execution: the latency advantage was there; it just needed an evaluator that used it.

PlayJev was trained to answer about eight steps ahead, but by default the closed loop still queried the model and executed one action at a time — the same cost per episode as the one-step policy, so the training horizon bought accuracy but not speed. We built the open-loop version — one query, eight actions executed before asking again, the same pattern OpenVLA-OFT uses — and it costs nothing in success (90.4%, no real change from 90.8%) while cutting model queries by 87%, from about 117 per episode to about 15.

Sources

References

Primary papers and project sources linked from the relevant claims above.

  1. 01
    OpenJev

    System One finite-choice request and response contract. OmniJev, project repository.

  2. 02
    PlayJev: A Multimodal JEV-Like Model for Small Games

    OmniJev, model, training code, evaluation and weights, 2026.

  3. 03
    A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

    S. Ross, G. Gordon, J. A. Bagnell, AISTATS 2011.

  4. 04
    LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning

    B. Liu, Y. Zhu, C. Gao et al., 2023.

  5. 05
    RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

    A. Brohan, N. Brown, J. Carbajal et al., 2023.

  6. 06
    OpenVLA: An Open-Source Vision-Language-Action Model

    M. J. Kim, K. Pertsch, S. Karamcheti et al., 2024.

  7. 07
    FAST: Efficient Action Tokenization for Vision-Language-Action Models

    K. Pertsch, K. Stachowicz, B. Ichter et al., 2025.

  8. 08
    OAT: Ordered Action Tokenization

    C. Liu, X. Han, J. Gao et al., 2026.

  9. 09
    Finite Scalar Quantization: VQ-VAE Made Simple

    F. Mentzer, D. Minnen, E. Agustsson, M. Tschannen, ICLR 2024.

  10. 10
    Learning Ordered Representations with Nested Dropout

    O. Rippel, M. A. Gelbart, R. P. Adams, ICML 2014.

  11. 11
    Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware

    T. Z. Zhao, V. Kumar, S. Levine, C. Finn, RSS 2023. Introduces Action Chunking with Transformers.

  12. 12
    LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization

    X. Zhou, Y. Xu, G. Tie et al., 2025.