Each policy is a websocket server. Per request it receives four RGB images (two static cameras, two wrist cameras), 16 joint positions, two gripper widths and the task variant, and returns 16 rows of joint commands at 30 Hz.
Tasks
- T1–T3 tower transfer, cross-tower transfer and vertical stacking: pick and place blocks, two- and three-layer variants.
- T4 full-tower push-down: push the blocks of a tower out one by one without dragging the blocks below.
- T5 tower insertion: insert blocks into a tower with about 1 mm of clearance per side, without moving its walls and roof.
Scoring and data
- Per variant
s = 60·LHPR + 30·TSR + 10·P/(1+ODR): progress first, then success, then a penalty for disturbing other blocks. - Tasks 4–5 count three times, two thirds of the overall score.
- Ten official episodes per variant. Training data: the official demonstrations only, about 40–50 per variant.
Samples from the official dataset
Each demonstration is one HDF5 episode recorded at 30 Hz in Isaac Sim: four 640×480 camera streams, joint states and commands, end-effector poses, the initial block poses, a language instruction and the simulator configuration that generated it. One episode per task below, shown at 4× speed with the static cameras on top and the wrist cameras below.
From an end-to-end VLA to a modular learned policy
Each turn had a concrete reason.
Pick a backbone
ACT, our own diffusion model and π0.5 with LoRA all lost to a full fine-tune of π0.5. We also set a rule we kept to the end: the environment is fixed, only the policy changes.
A replica we can trust
We rebuilt the evaluation simulator in Isaac Sim from the configuration stored in the official HDF5 files. Comparing it with official results exposed that the official Task 1–3 static cameras were rotated by 180° relative to the training data.
Precision and multitask π0.5
Averaging eight flow samples (mean8) raised the official Task 3-2 success rate from 0.30 to 0.80 without retraining. A multitask model reached 56.7% success on Tasks 1–3, and 30 Hz interpolation stopped the left arm from dropping blocks.
Tasks 4–5 change the plan
π0.5 behaviour cloning was weak on push-down and insertion (0/12 on Task 4 in our replica). Coding agents explored the tasks in our replica and helped us write a program that reuses demonstrations retargeted to the perceived block poses (the RAX program); we reviewed every change. It was strong on all three task families. On Oct 27 the organisers published the scoring formula and banned replay and fixed scripted solutions.
From program to learned policy
In five days we replaced the program's decision and motion parts by learned networks (RAX-AM) and kept the classical perception. Successive versions (v1 → v6d) took it from 77 to 91 successful episodes out of 100, ahead of the program it replaced (90/100).
RAX-AM
Each task is a sequence of skills taken from the demonstrations' phase labels (approach, close, lift, transport, place, release, …). Perception measures where the next block is; two small networks decide which skill is active and produce the motion. In dual-process terms, coding agents working with us were a design-time System 2, and the plan we worked out was trained into a fast run-time System 1. Nothing deliberates at inference.
Learned
- Skill selector. From joints, gripper widths, the policy's own command history, time in the skill and the perceived offsets, it predicts the current skill and when it ends.
- Executor. Conditioned on the skill, its progress and the offsets, it outputs the next 16 command rows as increments.
- Training. Behaviour cloning on the official training demonstrations, augmented by kinematic retargeting: each skill segment's tool path is moved to a random block offset and re-solved by inverse kinematics, so the networks learn to follow offsets the demonstrations rarely show.
Classical
- Perception. An edge-based template fit on the two static cameras, once per block, gives the block's offset (±22 mm, ±12°) from a reference pose.
- Safety filter. Joint and step limits; a rejected chunk is held or slowed, never replaced by stored motion.
- Task-specific modules. A 45-row start transition on T1-2, and four corrections on T5-3: a height loop, a slowed push window with a height offset, and a slot-corner check.
- No demonstration rows are stored or replayed at inference.
Most gains came from understanding a failure, not from more training
Numbers marked “replica” come from our local simulator; the others from official evaluations.
The simulator was hidden in the data
Our first local evaluations ranked models in the wrong order.
The official HDF5 files carry the simulator configuration (gains, latency, gripper, tolerances). A replica built from it matched the official result of our Task 3-2 model (0.52 vs 0.45) and stayed reliable for Tasks 1–3.
Upside-down cameras
Models that worked locally failed officially on Tasks 1–3.
The official static images were rotated by 180° relative to the training data. Rotating them back (camfix) fixed it; every Task 1–3 evaluation before that had been affected.
Good grasps, poor placement
π0.5 grasped reliably but placed poorly on Task 3-2 (success 0.30).
Averaging eight flow samples per request (mean8) raised success to 0.80 with the same weights.
The left arm dropped blocks
Commands at 10 Hz excited the left arm and shook blocks loose.
Interpolating to 30 Hz removed the drops (replica: 14/21 → 0/21); training on all three 30 Hz phases used the rest of the data.
A start pose no demonstration had
RAX-AM v1 succeeded on T1-2 in only 2 of 10 episodes.
The official start differed from the demonstrated start by up to 0.165 rad. A 45-row transition to the demonstrated start gave 10/10.
A safety filter that froze the robot
After rejecting one oversized step, the policy kept proposing it and never advanced.
After repeated holds, the chunk is slewed from the last command within the unchanged limits.
Insertion with 1 mm of clearance
On T5-3 the held block met the walls, and the first push dragged the roof.
Failure modes found in official videos and reproduced in the replica led to four declared modules; T5-3 went from 5/10 to 8/10 with the same network weights.
A rule change five days before the end
Our strongest policy, the RAX program, had no learned part when scripted solutions were banned.
We moved its decisions and motions into learned networks trained on the demonstrations, and declared every classical component that remained.
Most of the work did not reach the final policy. What it taught us did.
| Approach | What happened | Outcome |
|---|---|---|
| ACT, own diffusion model, π0.5 LoRA | All below a full π0.5 fine-tune at the start | Dropped |
| π0.5 single-task | Task 3-2 official success 0.80 with mean8 | Replaced by multitask |
| π0.5 multitask (MT3a2) | Official 100 on T2-2 and 96 on T1-3, but 76 on T3-3 | Counted on T2-2 |
| Simulated data generation, DAgger | Best nominal precision, but did not learn to follow a displaced block; DAgger made placement worse | Dropped |
| MolmoAct2, auxiliary losses, Spatial Forcing, RL fine-tuning (RLinf) | None beat π0.5 with mean8 | Dropped |
| π0.5 behaviour cloning on Tasks 4–5 | Task 4 best 12/20 in the replica, more steps did not help; the Task 5 long fine-tune stayed far below baseline | Not submitted |
| Diffusion Policy on Task 5 | 0/3 on both variants in the replica | Dropped |
| RAX program (agent-guided) | Strong on all tasks, 90/100 officially; no learned component | Reference; counted on T5-3 |
| RAX-AM | Learned selector and executor on classical perception, 91/100 officially | Final policy |
Official variant scores
Submissions that count, 10 episodes per variant. Underlined: per-variant best.
| Variant | RAX-AM v6d | RAX-AM v6b | π0.5 MT3a2 | RAX program |
|---|---|---|---|---|
| T1-2 / T1-3 | 100 / 100 | 100 / 100 | 85.1 / 96.1 | 100 / 100 |
| T2-2 | 91.7 | 91.7 | 100 | 91.4 |
| T2-3 | 100 | 100 | 91.1 | 100 |
| T3-2 / T3-3 | 100 / 100 | 100 / 95.6 | 88.3 / 75.6 | 100 / 95.6 |
| T4-2 | 77.3 | 83.9 | — | 80.9 |
| T4-3 | 91.9 | 79.2 | — | 81.3 |
| T5-2 | 100 | 100 | — | 100 |
| T5-3 | 85.15 | 85.06 | — | 85.65 |
| Overall | 91.93 | 90.65 | Tasks 1–3 only | 90.59 |
Per-variant best: 93.58. Counting learned submissions only (T5-3 from v6d): 93.50. v6b and v6d differ only in the Task 4 model; their identical Task 1–3 backends still gave 10/10 and 9/10 on T3-3, so one-episode differences are noise.
What the final policy still cannot do
Perception
- Vision is used once per block, from the static cameras only; a block that moves after it was measured is not tracked.
- The wrist cameras only serve a blank-image check.
- The template fit is tuned to the simulator's rendering, and its search range (±22 mm, ±12°) bounds what the policy can handle.
Learning
- Deterministic MLPs imitate near-identical demonstrations and their retargeted copies; generalisation comes mostly from the augmentation and the perception, not from the networks.
- On Tasks 1–3 the executor builds on its own commands rather than the measured joints, so it is open-loop within a skill.
- No recovery from states far from the demonstrations, such as a dragged lower block on Task 4.
- No run-time System 2: a model-driven agent watching the cameras could re-plan or pick a recovery skill. The rules allowed one; we did not build it.
Engineering
- T5-3 relies on four classical corrections tuned in our replica; T1-2 needs a scripted start transition.
- The Task 4–5 replica was never calibrated against official results and ranked our Task 4 models in the opposite order.
- Ten episodes per variant leave large uncertainty, and our final version choice used official scores.
- Calibrate the evaluator before trusting it. An uncalibrated replica is good for rejecting broken versions, not for close calls.
- Look for the mechanism. Configuration hidden in the data, a rotated camera, a 10 Hz excitation and an unseen start pose each became a fix.
- Inference and execution moved the score more than training. Holding after done, sample averaging, 30 Hz interpolation and a start transition each beat a retrain.
- Lower offline error did not mean better closed-loop behaviour. We saw it repeatedly; offline metrics became gates only.
- Read the score weights early. Tasks 4–5 were two thirds of the score, and we started training on them too late.
- Freeze earlier. Our last two versions were still queued at the deadline.
From the official evaluation
Static cameras on top, wrist cameras below. Episodes from variants with 10/10 successes.