The problem
The field has no shared way to say how interactive a system is. As Thinking Machines' interaction-models work argues, existing interactivity-oriented benchmarks do not capture what real-time, proactive collaboration demands, and the field still lacks shared frameworks for assessing interactivity quality. It is a gap the field admits about itself: AlphaStar's authors note “no universally accepted notion of fairness” in real-time matches, and even simultaneous translation — the most developed latency-aware subfield — has competing latency metrics that rank systems inconsistently.
Most evaluations are turn-based, judging final traces without accounting for a world that changes on its own. Those that take latency seriously — simultaneous translation — trace a quality–latency curve in a single modality. The most recent full-duplex voice benchmarks go further, measuring response timing and even correct restraint (when not to respond) — but only within voice, with no shared unit across modalities and no known ground-truth demand to score against.
The idea: one cross-modal unit
To compare a voice assistant, a screen agent, and a robot with one number, measure the thing they all do: move task-relevant information through a model. Information comes in, the system decides, actions go back out. Each step carries some bits per second and takes some time — so the natural unit is task-relevant bits per second, paired with latency.
Accuracy-of-what does not cross modalities; bits do. EnvRate keeps the field's own criterion — a response is good when it is both correct and delivered within a timing window — and changes only the unit, with latency ℓ gridded in TML's 200ms micro-turns.
z_t drives every modality, so
R(D) — computed from the latent's law — is the same across audio, video, and
proprioception; only extraction difficulty differs. That ordering is what makes the unit generalize.
Rate–distortion theory defines
the minimum information rate a system must handle to act well enough. The distortion
D is how wrong the agent can be, weighted by consequence. The rate R(D)
is the minimum bits/s needed about the required-response stream — not raw pixels or audio. A 4K
feed carries a high raw rate but says nothing about what to do, so it does not count toward R(D).
Score a system by the interaction gap at allowed latency ℓ:
gap(ℓ) = R(D) − task-relevant throughput achieved within ℓ
A system “keeps up” when its frontier meets R(D) at low ℓ.
Why the demand is computable
On a real feed, R(D) can only be estimated, with nothing to check the estimate against. So we
generate the environment. A generator emits a small hidden latent z_t whose law we set — a
discrete Markov chain or a Gaussian process; the required response is a known function a* = f(z_t)
(the null action included); and each modality is a rendering of that latent, with realistic-but-irrelevant
detail (background texture, ambient sound) drawn independently so it carries no task-relevant bits.
Because we author the latent, the demanded rate is computed exactly — Blahut–Arimoto in the discrete
case, closed form in the Gaussian — and verified by agreement between Blahut–Arimoto on a discretized
Gaussian and the Gaussian closed form. The agent never sees z_t or f, only rendered
observations through a real bandwidth-and-delay channel, so the skill under test — recover task-relevant
information from a lossy stream in real time — is the real one. Inside the gym the bit rate is exact and
puts voice, screen, and robot agents on one axis; on real tasks the absolute rate is not recoverable, so what we
claim transfers is the ranking.
A worked example
A 3-state Markov latent ticks at 2 Hz; f maps states to {null, alert}. With every tick an
event, at D=0 the response entropy is ~1 bit/event → demanded rate ≈ 2 bits/s. A reactive agent with
300ms channel latency handles ~60% of events within bound; gap(300ms) ≈ 0.8 bits/s. The same latent
rendered as symbolic tokens or audio tones gives the same R(D), different extraction difficulty.
R(D); the audio-tone agent works harder and trails, leaving gap(300ms) ≈ 0.8 bits/s. The
ceiling is identical — only extraction difficulty differs.
Scoring restraint
From the output alone a correct silence looks like a miss. So after a no-action event we sometimes inject a
follow-up that depends on the earlier event but carries none of its content — answerable only if the event
was retained. A follow-up that re-supplied the information would let the agent answer without having retained
anything, so the probe adds no task-relevant bits of its own; because we author the latent, we can check that the
probe alone leaves a* undetermined. This separates correct silence from a miss. Acting when
a* was null is the over-actuation case of the distortion D, carrying its own severity
weight rather than counting as a flat failure.
Why existing systems are incomplete
Every modality already has a mature benchmark: each scores its own quantity in its own units, so none places a voice assistant, a screen agent, and a robot on one axis. The most developed reach latency and even restraint, yet stay locked to a single domain.
| Modality | Leading benchmark | What it scores | Latency | Restraint | Cross-modal |
|---|---|---|---|---|---|
| Text / chat | Chatbot Arena (LMArena) | human preference (Elo) | ✗ | ✗ | ✗ |
| Speech — streaming translation | SimulEval | quality vs. latency | ✓ | ✗ | ✗ |
| Speech — conversational voice | Full-Duplex-Bench / τ-Voice | turn-taking, selectivity | ✓ | ✓ | ✗ |
| Video — streaming | StreamingBench | streaming QA accuracy | ✗ | ✗ | ✗ |
| Embodied | Habitat / Meta-World | success rate (SPL) | ✗ | ✗ | ✗ |
| Real-time control | AlphaStar (StarCraft II) | human-relative skill | ✓ | ✗ | ✗ |
| Any of the above | EnvRate | task-relevant bits/s vs. latency | ✓ | ✓ | ✓ |
What it contributes
- A cross-modal unit task-relevant bits/s vs. latency — exact in-gym, with the agent ranking checkable against existing single-modality benchmarks — not per-modality scores in incomparable units.
- Computable demand synthetic latents where
R(D)is exact and cross-checked (Blahut–Arimoto on a discretized Gaussian vs. the Gaussian closed form), not estimated from real feeds. - Restraint scoring callback probes that separate correctly silent from missed, plus over-actuation as a scored failure mode.
- An open gym pip-installable generator, channel controls, scorer, and reference frontiers others can reproduce.
Where this comes from
EnvRate grew out of building real-time, multimodal systems where what matters is emotional salience spread across several signals at once. Owly is a companion robot meant to live with a child from infancy — catching the rare salient moment (a first word, a laugh, a cry) the instant it happens, and otherwise staying quiet. Its whole competence is real-time interaction under restraint, and its failures are not equally weighted: missing a first step costs far more than missing an ordinary second.
References
- Interaction models Thinking Machines (2026)
- Real-time control AlphaStar (Vinyals et al., 2019)
- Latency metrics Better Late Than Never (Polák et al., 2025), Differentiable Average Lagging (Cherry & Foster, 2019)
- Conversational voice Full-Duplex-Bench, Full-Duplex-Bench v1.5, τ-Voice
- Streaming translation / video SimulEval, StreamingBench
- Embodied Habitat, Meta-World
- Information theory / estimation Cover & Thomas, Elements of Information Theory (2006); MINE (Belghazi et al., 2018)
- Renderers Genesis, MuJoCo / MJX
- Preference eval Chatbot Arena (Chiang et al., 2024)