A voice agent can pass every unit test and still feel unusable. It can also complete a polished browser demo while dropping the session, exposing a tool error, or ignoring a person who interrupts it. The usual mistake is not too little testing. It is calling four different kinds of evidence by the same name.
Use four proof planes
Structure asks whether the agent runtime makes the correct calls in the correct order. Script the model, speech-to-text, workflow, and text-to-speech layers. Assert call IDs, tool outputs, approvals, session state, and cleanup. This plane should be deterministic, fast, and free of provider calls.
Transport asks whether the browser or device can move audio through the real client path. Use fake microphone media, a real WebRTC or WebSocket client, permission transitions, connect/disconnect cycles, mute, barge-in, and reconnect. This proves the plumbing and UI contract. It does not prove that a person heard intelligible audio.
Heard asks what crossed a physical acoustic boundary. Capture speaker output through a physical or loopback device, transcribe it independently, and compare it with the expected meaning. Measure clipping, time to first audible response, interruption stop time, language switching, medication names, numbers, silence, and background noise.
Human asks whether the conversation worked for an actual person. Run a short, consented scenario with a listener. Record whether they understood the response, could interrupt naturally, knew when the agent was listening, and recovered after a mistake. A green synthetic waveform is not a human receipt.
The labels matter. A scripted voice test earns Structure. Fake browser media earns Transport. Only an independently captured acoustic path earns Heard, and only a real listener earns Human. Honest labels prevent a team from spending confidence it has not earned.
Make the text-and-tool runtime boring first
Voice adds timing and audio failure modes to the same agent loop used by text. Before exercising a microphone, cover the runtime with a reusable scripted model. The OpenAI Agents SDK now exposes public testing helpers through agents.testing, agents.realtime.testing, and agents.voice.testing. Use those public contracts instead of maintaining a private fake that drifts from the SDK.
The minimum deterministic regression packet should exercise seven paths:
- User → agent → function or MCP tool → final answer, with the tool output and caller identity preserved.
- Multiple tool calls, with unique call IDs and structured outputs kept in order.
- A tool failure followed by an explicit retry, including a safe final error if retry is exhausted.
- An agent or tool approval interruption, serialized as state and resumed without replaying an already completed side effect.
- Persisted session continuation, proving that application memory and conversation history survive a new runner invocation.
- A streamed response that completes, is cancelled, or errors while always cleaning up tasks, sockets, and audio playback.
- An MCP server that fails during startup or a call, remains isolated from healthy servers, backs off, and recovers without leaking its raw payload.
These tests are cheap enough to run on every change. They are also more diagnostic than a provider-backed conversation: when the fourth scripted response fails, you know which boundary moved.
Script audio, but do not pretend it is speech
For a deterministic voice test, create a short synthetic PCM buffer with a known sample rate. Feed it to a scripted speech-to-text model, return a fixed transcript, execute a scripted voice workflow, and generate a fixed text-to-speech buffer. Assert the transcript, workflow output, audio shape, and lifecycle events. The SDK’s voice pipeline and voice testing helpers provide the seams.
This pattern catches sample-format mistakes, missing end-of-stream signals, workflow exceptions, sensitive-tracing defaults, and cleanup bugs without paying for or waiting on a provider. It does not test accent recognition or natural audio. Those belong in the Heard and Human planes.
Realtime sessions need their own scripted event sequence. Drive session-created, speech-started, transcript, tool-call, response-done, error, and close events through the actual client state machine. Then assert the visible listening state, interruption behavior, transcript ordering, tool approval boundary, and teardown. The official Realtime guide is the protocol source; the scripted realtime model is the regression seam.
Test interruption as a state transition
Barge-in is not merely “audio stopped.” The runtime must stop playback quickly, cancel or truncate the correct response, keep any committed tool result, discard uncommitted speech, and leave the session ready for the next turn. Test at least these cases:
- Interruption before any tool call.
- Interruption while a read-only tool is running.
- Interruption after a tool result but before the spoken answer finishes.
- A false interruption caused by noise.
- A long pause that should not be mistaken for abandonment.
- Disconnect and reconnect while an approval is pending.
The recovery assertion is as important as the cancellation assertion: the person should have one obvious next safe action, and the agent should not replay a mutation because the audio path was interrupted.
Use a compact utterance matrix
A useful matrix is small enough to run and varied enough to expose boundary defects. Include a clean happy path, Hinglish or another expected language switch, medication-like names, doses and decimals, phone numbers and dates, silence, clipped speech, background noise, a long pause, user barge-in, and agent false-interruption recovery.
Score meaning rather than exact wording. For every case, record task correctness, completion, tool calls, model turns, tokens, latency, estimated cost, first audible response, interruption stop time, and recovery success. Keep a protected baseline so a faster model does not win by silently omitting an approval or a required detail.
Keep privacy receipts smaller than the event
Voice transcripts and tool payloads are unusually easy to leak through logs. Disable sensitive tracing for tests that contain realistic utterances. Log a small allowlisted receipt: event type, phase, latency, call ID, redacted-field count, and a safe action label. Never assert against a raw transcript, authorization header, tool exception, or provider response body merely because the test runner makes it convenient.
This is also a better test contract. A browser test should assert that a sanitized “workspace mode changed” receipt exists and that the UI visibly changed. It should not reach into a raw telemetry object whose shape and privacy policy are supposed to be hidden.
A field result, labeled honestly
In one recent agent-stack migration, the deterministic packet covered 32 runtime, MCP, realtime, and synthetic voice cases without real provider calls. A browser voice preset covered 11 Transport cases, including fake microphone media, interruption, error recovery, and sanitized observability. Both suites passed.
That evidence supports two claims: the agent structure behaved as scripted, and the browser transport handled the tested states. It does not support a claim about physical-device audibility or human comprehension. Those two gates remain separate by design.
The practical order
- Lock the seven deterministic text/tool contracts.
- Add scripted realtime and voice pipelines with synthetic PCM.
- Exercise the real browser transport with fake media and forced interruptions.
- Run a small provider-backed utterance matrix for recognition and synthesis quality.
- Capture and independently transcribe the physical audio path.
- Finish with short human scenarios and retain only privacy-safe receipts.
This order finds cheap structural defects before expensive acoustic ones. It also gives every failure a precise home. When a user says “the voice agent stopped listening,” you can ask whether Structure, Transport, Heard, or Human truth broke instead of rerunning the entire stack and hoping.