Curiosity under constraints and adaptive interfaces

Curiosity, compression, and progressive capability expansion

We adopt a learning-progress view of curiosity: intrinsically valuable patterns are neither random nor trivial, but those for which the agent can currently improve compression or prediction (Schmidhuber, 2010; Oudeyer et al., 2007). This makes curiosity inherently capability-relative. An agent can only be curious about patterns it can (in some sense) sense, model, or act upon, and the difficulty of useful tasks should grow with the agent’s capabilities rather than being fixed in advance.

On this view, curiosity creates pressure for capability expansion. If learning progress is limited by sensing, actuation, memory, or compute, the agent should prefer interventions that relax whichever bottleneck most improves future learning progress, subject to cost. This links curiosity to empowerment, control, and communication: an agent may benefit from seeing more, doing more, delegating more, or coordinating with others and tools more effectively.

Learning-progress intrinsic reward

We model an embedded agent interacting with an environment as a partially observed controlled process. Let $X_{t}$ denote the (generally hidden) environment state, $S_{t}$ the agent internal state (memory + policy state ($\theta_{t}$) + latent world-model state), $O_{t}$ the observation, and $A_{t}:=(U_{t},V_{t})$ the action.

We adopt a Schmidhuber-style learning-progress signal, not raw novelty (Schmidhuber, 2010). Let $\mathcal{L}_{\text{pred}}$ be a predictive loss for future observations over horizon $H$. For any integers $a<b$, define the interaction segment

$$\mathcal{D}_{a:b}\;:=\;\big(O_{a},A_{a},O_{a+1},A_{a+1},\ldots,O_{b-1},A_{b-1},O_{b}\big),$$

where $A_{t}=(U_{t},V_{t})$ as in the formal setup.

Let $p_{\theta}$ denote the agent’s predictive model for observations.

We instantiate the horizon-$H$ predictive loss as the cumulative negative log-likelihood

$$\mathcal{L}_{\text{pred}}(\theta;\mathcal{D}_{t:t+H})\;:=\;\sum_{h=1}^{H}\ell\!\left(p_{\theta}(\cdot\mid O_{t:t+h-1},A_{t:t+h-1}),\,O_{t+h}\right),\qquad\ell(p,o):=-\log p(o).$$

A generic intrinsic reward is then

$$r_{t}\;:=\;\mathcal{L}_{\text{pred}}(\theta_{t-H-1};\mathcal{D}_{t-H:t})\;-\;\mathcal{L}_{\text{pred}}(\theta_{t-H};\mathcal{D}_{t-H:t}),\qquad t>H,$$

where $r_{t}:=0\text{ for }t\leq H$. This rewards improvement in predictive compression rather than complexity or surprise alone, consistent with learning-progress formulations and modern world-model practice (Schmidhuber, 2010; Ha and Schmidhuber, 2018; Hafner et al., 2023).

We combine intrinsic reward with costs:

$$J(\pi,p_{S})\;=\;\mathbb{E}\!\left[\sum_{t=1}^{T}\gamma^{t-1}\Big(r_{t}-\lambda_{O}C_{O}(t)-\lambda_{E}C_{E}(t)-\lambda_{C}C_{C}(t)-\lambda_{M}C_{M}(t)\Big)\right].$$

$C_{O}(t)$: observation-processing cost (e.g., number of observation tokens/bits ingested or sensor bandwidth used at time $t$),

$C_{E}(t)$: actuation/maintenance cost (e.g., environment-defined energy penalty for executing $U_{t}$),

$C_{C}(t)$: compute/deliberation/learning cost (e.g., FLOPs, number of deliberation tokens, or number of update steps executed at time $t$),

$C_{M}(t)$: a memory-maintenance cost (motivated by finite-state maintenance and thermodynamic viewpoints on information processing (Still et al., 2012; Parr et al., 2022)).

Unification as interface quality

We introduce unification as reduction of sensing/acting bottlenecks between agent and environment. We formalize unification as an interface-quality concept.

Let $b_{O},b_{A}$ denote observation and action bottleneck parameters (coarsening, noise, latency, cardinality constraints, etc.). These bottleneck parameters are simply an alternative parameterization of the capacity constraints defined earlier: $b_{O}$ indexes the observation channel family $p_{O}(\cdot\mid X;b_{O})$ with effective capacity $c^{O}(b_{O})$, and $b_{A}$ indexes the actuation/policy interface with effective capacity $c^{A}(b_{A})$. We likewise let $b_{L}$ parameterize a communication/self-communication interface whose cost is accounted for under the same budgeting terms already present in equation 12. Define a task-relative unification score

$\displaystyle\mathcal{U}_{t}\;:=\;w_{O}\,\underbrace{\Big(1-\frac{\mathcal{R}_{O}(b_{O})}{\mathcal{R}_{O}^{\max}}\Big)}_{\text{observation losslessness proxy}}$ $\displaystyle+w_{A}\,\underbrace{\Big(1-\frac{\mathcal{R}_{A}(b_{A})}{\mathcal{R}_{A}^{\max}}\Big)}_{\text{action authority proxy}}+w_{L}\,\underbrace{\Big(1-\frac{\mathcal{R}_{L}(b_{L})}{\mathcal{R}_{L}^{\max}}\Big)}_{\text{communication bottleneck proxy}},$

$\displaystyle w_{O}+w_{A}+w_{L}$ $\displaystyle=1,\qquad\mathcal{R}_{\cdot}\ \geq 0$

where $\mathcal{R}_{\cdot}$ are calibrated inefficiency measures induced by each bottleneck. The exact choice of $\mathcal{R}$ is environment-specific. This way we can test whether improved agents systematically spend resources to increase effective interface quality when allowed to do so.

One way to define the observation-interface inefficiency is equivocation:

$$\mathcal{R}_{O}\;:=\;H(X|O),\qquad\mathcal{R}^{\max}_{O}=H(X).$$

$\mathcal{R}_{A}$ and $\mathcal{R}_{L}$ can be similarly defined.

Adaptive interfaces

Discover what capability, information or interface it is missing, acquire it autonomously, and measurably improve future performance.

Experimental agenda

The first stage should use toy but diagnostic partially observed environments with known latent dynamics and controllable bottlenecks. Gridworld-like domains with sensor coarsening, latency, observation noise, and switchable action-set cardinality are particularly useful because they permit direct intervention on the variables that define AAP’s unification and cost definitions. This stage is primarily for calibrating the metrics, stress-testing H1–H3, and identifying degenerate regimes (e.g., environments where prediction is cheap but irrelevant, or control is high but uninformative).

H1: Pragmatic alignment of objectives

In resource-bounded embedded agents, interventions that increase learning progress on future observations also tend to increase useful control over task-relevant environmental degrees of freedom, and vice versa, over a substantial regime of tasks and constraints. Moreover, optimizing equation 12 with respect to agent-side observations implies similar learning progress (or control) over the true environment hidden state from which observations are drawn, when interfaces and bottlenecks are flexible enough to allow for losslessness (i.e. invertibility) in the limit.

This is weaker than strict equivalence between predictive compression, hidden-state compression, and control. It predicts alignment over a regime, not identity for all environments. It is compatible with counterexamples where prediction is easy but uncontrollable, or control is high but not informative.

H2: Boundary pressure toward unification

When an agent can invest resources to modify its sensing/acting/communication interfaces, optimization of equation 12 will allocate resources toward interface improvements that increase long-horizon learning progress and control, yielding monotonic gains in task-relative unification $\mathcal{U}$ until costs dominate. In finite environments and with favorable scaling of interface cost, a stronger boundary-collapse regime may emerge asymptotically. In an idealized limit where (i) environment complexity is finite and (ii) marginal interface-improvement costs scale favorably, the optimizer can drive $\mathcal{U}\to 1$ (maximal effective interface quality). This is a limit-case conjecture about the objective’s asymptotics, not a practical prediction for real systems.

H3: Constraint-induced predictive/control pressure

Under continued viability constraints (cannot stop interacting), coarse/noisy interfaces, and nonzero costs for action, memory maintenance, and frequent policy updates, agents are driven toward better prediction and selective control because these reduce costly plasticity and wasted computation. Therefore even without directly providing the intrinsic reward of equation 11, an agent may nevertheless be forced to optimize toward it in realistic environments.

This hypothesis aligns naturally with information-theoretic and thermodynamic arguments linking predictive state representations to energetic efficiency (Still et al., 2012; Friston, 2010; Parr et al., 2022; Ortega and Braun, 2013).

H4: Adaptive compute optimality

We further predict that agent performance under fixed budgets depends not only on model capacity, but on how computation is allocated over time. A meta-controller that dynamically allocates observation, action, and deliberation compute should outperform fixed schedules under the same total budget, especially when task difficulty and observability vary across timesteps. This connects to adaptive computation and metareasoning (Graves, 2016; Banino et al., 2021; Callaway et al., 2018; Lieder and Griffiths, 2020).

H5: Self-communication bottleneck

An explicit self-communication channel (e.g., text-like or symbol-like private tokens) can improve performance and sample efficiency on tasks requiring long-horizon credit assignment, compositional planning, or coordination, beyond what latent recurrence/planning alone achieves, provided the private channel is bandwidth-regularized. Importantly, the claim is not that verbalized traces are always optimal, but that selective self-communication may be an efficient option in a broader action space that also includes direct action and latent deliberation (Yao et al., 2023; Kim et al., 2025; Xie et al., 2025; Wang et al., 2026).