Repository navigation
test(e2e): S103 judges readout integrity per response, not per energy burst - #1307
Conversation
… burst Both S103 attempts in Turbo S run 36599321227 failed "assistant audio started without a new input final or commentary (duplicate readout)". The rule treated every peer energy burst as a new assistant start and looked for a prompt since the previous burst's end entry. Two things were wrong: - A readout spoken line by line pauses longer than the peer's 600 ms end hysteresis between lines, so one readout is many bursts. Attempt 2 read the corrected brief as ten bursts of one response (peer `response` 5); attempt 1 resumed the interrupted readout after a 900 ms line pause in the same response 3. - The burst's `assistant_audio_end` entry is pushed only after the hysteresis, about 700 ms after the last audible window. The correction's input final closed 8 ms before that entry but 691 ms after the assistant was last audible, and was not counted. `unprompted_assistant_response_starts` groups bursts by the peer's `response` index, which advances only when a new user utterance starts, and opens each prompt window at the previous burst's `last_active_ms`. A response that repeats itself is the peer's own `duplicate_readout` fault on the output transcript, which the scenario already fails on. Unit tests replay both failing timelines and keep an unprompted new response failing. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 53539c59cd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let continues = response.is_some() && response == previous_response; | ||
| previous_response = response; | ||
| if continues || entry.t_ms < from_ms { |
There was a problem hiding this comment.
Derive response boundaries independently of user input
In a real peer timeline, response.index is not an assistant-response identifier: gpt-live-peer-e2e.mjs increments it in finishResponse, which during an active session is called when a new user transcript begins (lines 341-356 and 569-570). Consequently, an unsolicited second assistant response receives the same index as the preceding response and this branch skips it; unless it repeats a five-word sentence verbatim, the separate duplicate_readout detector also stays silent. The new negative unit test manually changes the index without an intervening input, a state the harness cannot produce, so this makes S103's unprompted-response gate ineffective for the regression it claims to detect.
Useful? React with 👍 / 👎.
A local S103 run flagged a readout as unprompted: the answer to the correction started as peer response 1, the late "Friday" delta advanced the peer's index to 2 while the assistant was still speaking, and the next line of the same readout started as response 2. A burst now continues the previous response when its index equals the previous burst's index at its start or at its end. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review of #1307: the burst rule groups bursts by the peer's response index, which advances only on new user speech, so an unprompted second readout with no user speech in between lands in the same response and is never flagged, and the peer's own duplicate_readout fault skips sentences under five words, which every kickoff-brief line is. The peer's response_end timeline entry now carries the response text, and `repeated_readout_lines` fails any response after the barge-in that speaks a line twice (lines split on newlines and sentence ends, three-word floor). The impossible-timeline negative test is replaced by a peer-producible one (the brief read twice in one response, invisible to the burst rule) and a single-readout positive. `format_timeline` now truncates on a char boundary (a curly apostrophe at byte 240 would have panicked). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…gative Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review of #1307: the peer pushes response_end only from finishResponse, which runs on a new user utterance, a disconnect or evidence stop. S103 reads its timeline right after wait_for_settled following the correction fixture, so the final corrected-brief readout (the Turbo S attempt-2 ten-burst response) had no response_end and repeated_readout_lines never saw it: a doubled final readout still passed. A new peer command, flush_response, closes the in-progress response (its response_end carries `flushed: true`) and returns the timeline; S103's final read uses it (`flush_response_timeline`). The negative unit test's final response_end is the flushed one the scenario now produces. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Why
Release Turbo S run 36599321227 (BuildBuddy 9de364bc) failed S103 on both attempts with, among others, "assistant audio started without a new input final or commentary at ms [...] (duplicate readout)". That check was wrong:
response5, the corrected brief read line by line). Attempt 1 flagged a readout resuming after a 900 ms line pause in the same response 3.assistant_audio_endentry. That entry is pushed only after the hysteresis, about 700 ms after the last audible window. The correction's input final closed 8 ms before that entry but 691 ms after the assistant was last audible, so it was not counted.What
unprompted_assistant_response_startsgroups bursts by the peer'sresponseindex, which advances only when a new user utterance starts. Each prompt window opens at the previous burst'slast_active_ms. A response that repeats itself is still caught by the peer's ownduplicate_readoutfault on the output transcript text, which the scenario already fails on. Unit tests replay both failing Turbo S timelines and check that an unprompted new response still fails.Review fixes
response_endtimeline entry now carries the response text. The burst rule can't see a second readout with no user speech between it and the first, because both share one response index. The newrepeated_readout_linescheck can: it fails any response after the barge-in that speaks a line twice (lines split on newlines and sentence ends, three-word floor, since kickoff-brief lines like "Client: Marigold account." fall under the peer's five-word sentence fault).format_timelinetruncates on a char boundary.Final-readout flush (ad3ece0)
The peer pushed
response_endonly from finishResponse, which runs on a new user utterance, a disconnect or evidence stop. S103 read its timeline right afterwait_for_settled, so the final corrected-brief readout had noresponse_end, and a doubled final readout still passed.A new peer command,
flush_response, closes the open response (itsresponse_endcarriesflushed: true) and returns the timeline. S103's final read now usesflush_response_timeline(), and the negative unit test's finalresponse_endis that flushed entry.BuildBuddy S103 x3 on this head (invocation 63e2c202-8f46-4133-a60f-641a0decd7e9):
config_tests: 14 passed.Evidence
cargo test --test gpt_live_public_e2e config_tests: 14 passed, 5 of them S103 oracle tests.S103 also needs #1308 for its monologue-overlap failure.
Combined live validation (2026-09-30, local, VM load < 120 at each run start)
One build combining #1307 (9837acb), #1308 (74e627a), #1309, #1310 (df32876), #1311 (f9a2f50), #1312 (2962983) and origin/main:
🤖 Generated with Claude Code