ARC-AGI-3

Solution

Our solution is an online-learning ARC-AGI agent built around a small set of VLM roles. Here, we abandoned explicit world and goal models. Agent X chooses an action, other VLM roles explain what changed, and updater P rewrites the agent’s context so the next decision can use what was learned.

Observed frames are converted into two aligned model inputs. We serialize the ARC grid into exact symbols, coordinates, components, and deterministic change evidence. Cropped PNG images provide matching visual context for frame-consuming VLM roles. The text representation is authoritative when text and image appearance disagree, while the image helps the VLM inspect spatial patterns.

Updater P is the online (context-level) learning mechanism in the current VLM architecture. During the game loop, the agent-game updater revises the game-context from the observed transition, change summary, recent action history, historizer summary, progress signals, and the previous agent context. Orchestration applies the returned context to the live working state before later model calls use it.

Public games and competition results

We ran this solution in parallel on the 25 public games. Overall, when running for an extended period (3–4 hours on a single GPU) it was able to solve 10–11 levels, mostly first levels of different games with one or two second levels. The corresponding score on these games was between 0.8 and 1.0. Since the system required heavy reasoning and several models, even after running for 3–4 hours it only got to about 120 average actions taken per game.

Our final score on the 55 (hidden) evaluation games on Kaggle was 0.7, which put us at 8th place on the leaderboard just before the June 30, 2026 milestone cutoff. We also ran a few experiments on the evaluation games by lowering or increasing the reasoning budget, and adjusting history length, but these produced worse results.

Limitations

The agent often returned to the same local behaviors even when history showed they were ineffective, such as moving back and forth, retrying no-op actions, or reusing failed strategies. The updater could add this evidence to the agent context, but the agent model did not always treat the revised context as binding, so context-level online learning influenced behavior without reliably forcing a clean change of plan.

Overall, the results are evidence that structured VLM agents can perform useful online adaptation in ARC-AGI-3, but not yet that they can robustly infer and exploit new game mechanics with human-like efficiency.

Future directions

A first direction is to improve the agent’s world representation by learning compact text or program-like world models that can simulate action effects without generating full frames.

A second direction is to restore the original learning-progress idea in a cheaper and more robust form. Instead of using curiosity only as a loose contextual signal, the agent should explicitly choose actions that maximize expected improvement in prediction or compression while being penalized for time, action count, memory use, and tool calls. This would align exploration with efficient problem solving rather than with novelty alone. The broader research hypothesis is that learning progress becomes more useful when combined with realistic constraints on sensing, acting, memory, compute, and self-maintenance.

A third direction is simulation and replay. After a level or failed trajectory, the system could use a learned or program-synthesized world model to replay alternative strategies, generate auxiliary tasks, and fine-tune lightweight adapters for better action efficiency. Frontier models could also be used offline as teachers: they can play, annotate transitions, propose abstractions, and generate training traces that are distilled into smaller models through LoRA or similar adaptation methods.

A fourth direction is to move beyond fixed language-reasoning loops toward flexible token-stream agents. Instead of forcing a model to follow a rigid observe → reason → act format, the agent should be able to interleave observation tokens, action tokens, language tokens, and silent deliberation tokens. This would let the model decide when to inspect more input, when to act, and when to spend computation on internal reasoning, with each emitted token carrying an explicit time or energy cost.

A fifth direction is cross-game improvement through generated environments. Instead of relying only on the 25 public games, a coding agent could generate large numbers of ARC-AGI-like environments with known latent mechanics, increasing difficulty, and controlled variation. This would allow continuous training on action efficiency, causal discovery, and transfer across related mechanics. A self-play or “hyperagent” setup could generate harder environments as the agent improves, producing an open-ended curriculum for fast learning rather than a fixed benchmark-tuning loop.

Finally, ARC-AGI-3 should be used as one evaluation signal inside a broader suite of constrained interactive environments. The next version of the research program should build partially observed environments with controllable bottlenecks, train agents under intrinsic learning-progress rewards, and test whether these agents become more human-like in their efficiency, exploration, and ability to generalize. Success would mean not only higher ARC-AGI-3 scores, but a clearer empirical account of which environmental constraints produce more robust, interpretable, and aligned problem-solving behavior.