Flexible token-stream agents

Implications for LLMs

All LLM-style (including VLMs and VLAs) models should have reasoning modules after having “learned” language. The purpose of a reasoning module, however, is not to produce a stream-of-words, but rather to allow for deliberation without action. Therefore when a model is post-trained for reasoning on various tasks (including language itself) it should be able to produce sequential hidden state updates without the use of language tokens. Such a model would have the flexibility to choose when to produce and when to avoid self-communication through language and do pure deliberation (without tokenization bottlenecks) instead. Additionally, deliberation can be expanded to any token modality, including vision and action tokens. This is analogous to visual or motor imagery in neuroscience.

Such a model might observe various types of input tokens and be able to produce and interleave these at any step. Importantly, during production there is no fixed order, similar to how a human might pause mid-sentence to do a tiny deliberation loop (with or without explicit language-thoughts). Allowing the model to flexibly start “thinking” while it has not finished a paragraph, or continue thinking while producing an output could help with error recovery and self-monitoring.

In the most general case we propose a modality-agnostic token taxonomy with three roles per modality $m\in\{V,S,T,A,\ldots\}$:

$$m_{i}\text{ (input)},\qquad m_{s}\text{ (private/self)},\qquad m_{o}\text{ (public/output)}.$$

This yields a unified interface where the policy can interleave incoming observations, external actions, and internal deliberation:

$$\underbrace{V_{i},S_{i},T_{i}}_{\text{incoming}}\;\;\Leftrightarrow\;\;\underbrace{H,V_{s},S_{s},T_{s},A_{s}}_{\text{deliberation / imagery / reasoning}}\;\;\Leftrightarrow\;\;\underbrace{V_{o},S_{o},T_{o},A_{o}}_{\text{communication / output}},$$

where V denotes vision, S denotes audio, T denotes text, and A denotes action tokens. H refers to the output hidden state. The key design question is not whether to allow private tokens, but when they are worth emitting under budget, and which modality is most effective for deliberation at any given step. This directly supports our compute-efficiency criterion.

It is an open and difficult question how to train such a flexible model given current LLM paradigms. One way to train it for hidden-state deliberation would be to choose blanks (i.e. reserved non-language tokens) whenever the confidence in the next text token is below a threshold. For example one can take a reasoning trace written by a human and run it through a base LLM to get confidence scores for each token. Any position whose score is below a threshold gets shifted to the right and a "blank" is inserted. Then the LLM can be fine-tuned on these augmented traces for a while, and subsequently use the fine-tuned model to regenerate scores for our augmented traces, inserting new blanks according to confidence scores, and so on. After each step the number of positions where blanks are included should decrease and perhaps there is an optimum where no more blanks are required. We should expect the final trace to have many blanks in the beginning and at conceptual jumps in the reasoning. This procedure is still somewhat artificial and not necessarily in line with the goal of using blanks for truly free deliberation. Ultimately, choosing between blanks and other types of imagery modalities (including text) should not be based on language modeling score.