GPT-Live, OpenAI’s full-duplex speech model, is natively designed for real-time conversation. Rather than waiting for the user to complete a turn, it continuously processes the incoming audio stream and decides in real time what action to take.
This raises a simple question: can we teach an existing small language model to interact in the same way without training one from scratch?
Why text only?
GPT-Live is built for speech: it takes audio as input and produces audio as output. Our project, by contrast, is text-only.
The main reason is practical. Training a real-time speech model from scratch would require large-scale audio data and substantial compute—which we unfortunately don’t have.
Many of the questions we want to answer, however, are not specific to speech. How should a model behave while input is still arriving? When should it respond, wait, or act? Can these behaviors be learned through post-training? Although speech provides the clearest use case for real-time interaction, we can study the same questions through a text-based approximation, replacing the continuous audio stream with a continuous stream of text tokens.
Text also makes the system easier to inspect. Audio must be transcribed and segmented before it becomes human-readable, whereas every input snapshot, decision, and action in our setup is already recorded as text. This allows us to trace exactly what the model received and how its behavior changed as new input arrived.
Capabilities
Making real-time interaction part of the model enables capabilities that would otherwise need to be handled by the harness. The following demos show several behaviors we trained the model to perform.
Interjections. The model can respond when the context calls for it, rather than waiting for the user to finish typing.
Live translation. The model can translate in real time as the user types, rather than waiting for a completed message.
Dialog management. The model can infer whether the user is thinking, pausing, or inviting a response, allowing it to decide when to remain silent and when to speak.
Concurrent tool calls. The model can continue interacting with the user while spawning multiple tasks that run concurrently in the background, weaving their results back into the conversation as they complete.
Our approach
We represent evolving user input as timestamped events, train the model to choose one action for each event, and optimize inference so those decisions arrive in real time.
The first change is how the interaction is represented. In a turn-based system, the model waits for a completed user message before responding. Our setup replaces these completed turns with a continuous stream. As the user types, the system captures the current textbox state every 650 ms and adds it to the model’s history.
Turn based
Inputs and outputs are flattened into one ordered token sequence
Time-aligned micro turn-based
Continuous input and output streams split into micro-turns
→
A continuous stream preserves moments that turn-based interaction normally discards. A pause does not end the interaction, and an unchanged or empty textbox remains an observation. These observations help the model infer whether the user is still typing, thinking, revising, or waiting for a response.
Each observation becomes a timestamped stream event. These stream events are interleaved with the model’s previous actions, forming a history of what the model observed and how it responded.
<stream_event index="1" source="user" state="active" time="t+650ms">
Do you think I should
</stream_event>
<action>idle()</action>
<stream_event index="2" source="user" state="active" time="t+1300ms">
Do you think I should call it a night?
</stream_event>
<action>respond({"for":2,"message":"You should!"})</action>
<stream_event index="3" source="user" state="idle" time="t+1950ms">
</stream_event>
<PREDICT_THIS_ACTION>Given this event history, the model predicts exactly one action for the latest event. The model must decide not only what to do, but whether the current moment requires any action at all.
The model chooses from a closed grammar of six actions:
idle— remain silent; the default and most common correct action.respond— reply, acknowledge, translate, or fire a reminder.highlight— mark a specific occurrence of text.delegate— start an asynchronous background task.web_search— issue a search query and receive the results as a later event.translate_commit— append a stable unit of translation while the source text is still arriving.
This closed action grammar makes each prediction easy to validate and execute. Because the model selects the action itself, the runtime only needs to carry it out. The runtime does not decide whether the user has finished, whether the model should interrupt, or whether it should remain silent.
These timestamped events also give the model an explicit representation of
time. Rather than receiving time as a native input dimension, the model reads
timestamps such as t+1950ms, compares the latest event with the preceding
history, and decides whether the current moment calls for an action.
System overview
Our system follows the same broad two-model pattern used by Thinking Machines Lab’s interaction models and OpenAI’s GPT-Live: a real-time model remains in direct contact with the user, while heavier work is delegated asynchronously. However, our implementation differs in three design decisions.
The first design decision concerns delegation. Whereas the reference architecture presents a single background model, our system allows each delegation to spawn a separate background job. These jobs can run concurrently, allowing Text GPT-Live to continue interacting with the user while multiple intensive tasks are completed in parallel.
The second design decision concerns tool use. Whereas the reference architecture assigns tools primarily to the background model, our system gives tools to both Text GPT-Live and the spawned background jobs. Their toolsets reflect different roles: background jobs use compute-intensive tools for tasks such as generating interfaces, images, or code, while Text GPT-Live uses lightweight tools that directly support the live interaction, such as text highlighting and incremental translation. This division allows the live model to act on the shared interface without taking on work that would interrupt the interaction.
The third design decision concerns communication between the background jobs and Text GPT-Live. Whereas Thinking Machines Lab’s background model can send a substantive response back to the interaction model, our background jobs emit only structured status updates and a final result. These updates indicate whether a task has started, completed, or failed, allowing Text GPT-Live to track several jobs and weave their results back into the conversation as they arrive. This simpler protocol makes concurrent delegation easier to manage, but it also limits the kinds of tasks the system can support: some tasks require clarification, intermediate feedback, or an extended exchange between the live and background models, rather than only a status update and final result. Supporting those tasks would require a richer communication protocol.
Data
We train the model to make one decision at a time rather than complete an entire conversation. To create these decisions, we used LLM-based authoring agents to write full interaction scenarios under structured schemas. Each scenario specifies what the user types, where pauses or revisions occur, and where the model should take an action.
A deterministic compiler then expands each scenario into a continuous stream of textbox snapshots at 650 ms intervals. Every snapshot remains in the event history, together with all previous model actions. From this stream, we select the moments where the decision is most informative: immediately before an action becomes appropriate, when it becomes appropriate, and immediately after it has already been performed. We also include ordinary quiet moments and misleading cases that might tempt the model to act too early.
For example:
Do you think I should call it a night? → respond("You should!")
[tick 1] Do you think → idle() [tick 2] Do you think I should call it → idle() [tick 3] Do you think I should call it a night? → respond("You should!") [tick 4] Do you think I should call it a night? → idle()
History: [tick 1] Do you think → idle() [tick 2] Do you think I should call it → <PREDICT_THIS_ACTION> Target: idle()
History: [tick 1] Do you think → idle() [tick 2] Do you think I should call it → idle() [tick 3] Do you think I should call it a night? → <PREDICT_THIS_ACTION> Target: respond("You should!")
History: [tick 1] Do you think → idle() [tick 2] Do you think I should call it → idle() [tick 3] Do you think I should call it a night? → respond("You should!") [tick 4] Do you think I should call it a night? → <PREDICT_THIS_ACTION> Target: idle()
Selected ticks then become training examples. Each example contains the complete interaction history available at that moment, including previous model actions, while the expected next action becomes the target.
This structure makes timing part of the supervision. By sampling moments before, during, and after an action, the model learns to wait until an action is appropriate, act at the correct moment, and avoid repeating an action afterward.
We applied this methodology across three sequential SFT runs, adding new data
and capabilities at each stage. In the first run, approximately 68.5% of the
examples targeted idle(). This imbalance was intentional: because most
moments in a live interaction should not trigger an action, learning when to
remain silent was the model’s first and most important behavior.
To increase diversity, we used multiple authoring models with varied instructions, personas, domains, and writing styles. These generated scenarios then passed automatic checks for valid action syntax, grounded text references, and overlap before entering the dataset.
Training
Our natural first choice was supervised fine-tuning (SFT) because we were trying to teach the model a new desired behavior that was already known. For every typing tick, we could explicitly specify the correct action — such as remaining silent, speaking, searching, or translating — making the task naturally supervised. We therefore framed training as learning the correct action from demonstrations.
We used Qwen3.5-4B as the base model and trained it through three sequential stages of SFT, with each stage resuming from the previous checkpoint at a lower learning rate. The first stage taught the core interaction behaviors, the second introduced additional capabilities such as translation and web search, and the third repaired failures discovered during evaluation.
Those failures suggested that the remaining gaps reflected missing interaction capabilities rather than incorrect preferences. Additional supervised data was therefore a more appropriate next step than reinforcement learning or preference optimization methods.
Using Tinker, all three training stages cost approximately $50 in total.
Before training
After training
% of checks passed
Inference
Real-time interaction requires fast inference because the model must process every typing tick quickly.
Our naïve BF16 implementation was too slow. Because every tick reprocessed an increasingly long context, median latency reached 2.9 seconds and grew to over 60 seconds after 7.5 minutes of interaction.
This implementation also revealed that standard prefix caching was
ineffective. Each typing tick slightly changed the input, preventing the
inference engine from reusing previous computation. To overcome this
limitation, we quantized the model to 4-bit, replaced full-snapshot inference
with bounded per-session caches, prefilled the invariant prompt prefix, and
stopped decoding after the generated </action> token. Together, these
changes reduced median latency by 80% to 572 ms, within the 650 ms budget.
However, further testing showed that the 4-bit model did not preserve outputs reliably. We therefore faced a trade-off between speed and correctness: the 4-bit model met the latency target, while the more reliable 8-bit model did not. We ultimately chose the 8-bit model for the demos, accepting a higher median latency of 972 ms in exchange for reliable outputs.
Limitations
Synthetic data. Text interaction is inherently turn-based, so very little data captures real-time behavior. We therefore had to rely on synthetic data, and we found that the model often mimicked patterns in the dataset instead of making robust real-time decisions from context. For example, we generated training ticks by splitting full interactions using a fixed character range. Across 318,462 ticks, not a single one advanced by more than seven characters. This limit implicitly exposed the model to only a narrow range of typing speeds, so it performed much worse when users typed faster during testing. We plan to generate more diverse synthetic data spanning a wider range of typing speeds and interaction patterns.
Capability failures. We also trained two additional capabilities — reminders and grammar correction — but omitted both because they failed in live testing. For reminders, the model needed to act when a scheduled reminder became due. However, it remained silent even when the input explicitly indicated that a reminder was due, suggesting that the failure involved action triggering rather than time calculation. For grammar correction, the model needed to identify and correct errors while the user was typing. Unlike highlight, which only requires matching a span, this task also requires reasoning about the error and its correction. Although performance was strong on the validation set, it was unreliable in live testing. The small model size may have contributed to both failures, but we did not test this hypothesis. We plan to evaluate larger models and improve supervision for these capabilities.
Context growth. During streaming, the system appends a new snapshot of the user’s text at every tick. Because each snapshot repeats most of the previous text, the context quickly fills with duplicated prefixes and grows quadratically, or O(N²). As interactions become longer, memory use and inference latency increase, making real-time interaction harder to sustain. We plan to replace repeated snapshots with a more compact context representation.
Acknowledgements
This project was initially inspired by Rajan Agarwal’s Tiny Interaction Models, which introduced me to interaction models and demonstrated that the idea could be explored with a small, open model. The broader direction was also shaped by Thinking Machines Lab’s work on interaction models and OpenAI’s GPT-Live, both of which show how models can continuously perceive, respond, and make interaction decisions rather than waiting for fixed conversational turns.