流式语音智能体管线开发
搭建语音识别到合成播放的流式代码管线,处理轮次判断、打断取消、延迟监测与故障测试。
完整提示词
<context>
A voice agent feels broken when the gap between the caller finishing and the agent speaking exceeds about a second, when it talks over the caller, or when it cannot be interrupted. Getting under that bar means streaming every stage: partial transcripts from speech-to-text, the LLM's first tokens starting text-to-speech on the first complete sentence, and audio playback beginning before the reply is finished. Turn detection is the hardest part: a fixed silence timeout either cuts people off mid-thought or adds dead air, so voice activity detection plus a semantic end-of-turn check works better. Barge-in means stopping playback and cancelling the in-flight LLM and TTS requests the moment the caller speaks, and recording what was actually heard so the history matches reality. Phone audio adds its own constraints: 8 kHz codecs, echo, and DTMF tones.
</context>
<task>
Build the voice pipeline for: {{use_case}}
Components: {{components}}
1. Read any existing code. If components are not chosen, list the criteria (streaming support, latency, languages, telephony support, data handling) and leave the choice to the user rather than picking vendors from memory.
2. Architecture: audio transport (WebRTC for browsers, the telephony provider's media stream for phone), a session process per call, and the stages wired as streams with cancellation: STT streaming, turn detector, LLM streaming, sentence chunker, TTS streaming, playback.
3. Latency budget: set targets per stage (end-of-turn detection, STT finalisation, LLM time to first token, TTS time to first audio, network) summing to the total target, and log each per turn.
4. Turn taking: voice activity detection with a tuned silence threshold, plus an end-of-turn check on the transcript where available; handle short backchannels ("uh-huh") without treating them as interruptions.
5. Barge-in: on detected caller speech during playback, stop audio, cancel the LLM and TTS requests, and truncate the assistant message in history to what was actually played.
6. Voice-specific prompting: short sentences, no Markdown, lists or URLs read aloud, numbers and dates in speakable form, and confirmation of critical details (amounts, addresses) by reading them back.
7. Failure handling: a spoken fallback when a stage times out, retry or switch for STT and TTS errors, a path to a human or a callback, and clean hang-up handling that releases resources.
8. Tests: unit tests for the sentence chunker and barge-in truncation, and a scripted audio or transcript replay test for turn taking. Run them and report. Describe a manual test script for real calls.
</task>
<constraints>
- Do not record or store call audio or transcripts unless the use case says it is allowed; note consent and disclosure duties vary by country and leave that decision to the owner.
- Use the real streaming APIs of the chosen components for the versions in the repository; if unsure, say so.
- Prefer the project's existing voice framework if one is in use rather than rewiring the stages by hand.
{{> guardrails/scope-discipline}}
{{> guardrails/verify-before-done}}
</constraints>
<output_format>
## Architecture
Stages, transport and cancellation path, in a short numbered list.
## Latency budget
Table: stage | target ms | how measured.
## Turn taking
Endpointing and barge-in rules with starting values to tune.
## Code
Code blocks with file paths.
## Tests
Code blocks with file paths, plus the manual call test script.
## Check
Commands run with real results.
## Risks
Latency, cost per minute, privacy and failure risks still open.
</output_format>可替换内容
- {{use_case}}
- {{components}}
- {{> guardrails/scope-discipline}}
- {{> guardrails/verify-before-done}}
分类与标签
编程开发、语音智能体、流式处理、打断控制、延迟优化、通话测试