Skip to content

test(e2e): S103 judges readout integrity per response, not per energy burst - #1307

Merged
lukacf merged 6 commits into
mainfrom
fix/s103-readout-oracle
Sep 30, 2026
Merged

lukacf merged 6 commits into
mainfrom
fix/s103-readout-oracle

Conversation

@lukacf

@lukacf lukacf commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Why

Release Turbo S run 36599321227 (BuildBuddy 9de364bc) failed S103 on both attempts with, among others, "assistant audio started without a new input final or commentary at ms [...] (duplicate readout)". That check was wrong:

  • It treated every browser-peer energy burst as a new assistant start. A readout spoken line by line pauses longer than the peer's 600 ms end hysteresis between lines, so one readout is many bursts. Attempt 2 flagged ten bursts of one response (peer response 5, the corrected brief read line by line). Attempt 1 flagged a readout resuming after a 900 ms line pause in the same response 3.
  • It looked for a prompt after the burst's assistant_audio_end entry. That entry is pushed only after the hysteresis, about 700 ms after the last audible window. The correction's input final closed 8 ms before that entry but 691 ms after the assistant was last audible, so it was not counted.

What

unprompted_assistant_response_starts groups bursts by the peer's response index, which advances only when a new user utterance starts. Each prompt window opens at the previous burst's last_active_ms. A response that repeats itself is still caught by the peer's own duplicate_readout fault on the output transcript text, which the scenario already fails on. Unit tests replay both failing Turbo S timelines and check that an unprompted new response still fails.

Review fixes

  • Duplicate readout inside one response. The peer's response_end timeline entry now carries the response text. The burst rule can't see a second readout with no user speech between it and the first, because both share one response index. The new repeated_readout_lines check can: it fails any response after the barge-in that speaks a line twice (lines split on newlines and sentence ends, three-word floor, since kickoff-brief lines like "Client: Marigold account." fall under the peer's five-word sentence fault).
  • Tests. The impossible-timeline negative is replaced by a peer-producible one: the brief read twice in one response, with the burst rule silent and the line rule flagging all four lines. There is also a single-readout positive.
  • Burst rule. A burst continues the previous response when its index equals that burst's index at its start or at its end (a late user delta can advance the index mid-burst).
  • format_timeline truncates on a char boundary.

Final-readout flush (ad3ece0)

The peer pushed response_end only from finishResponse, which runs on a new user utterance, a disconnect or evidence stop. S103 read its timeline right after wait_for_settled, so the final corrected-brief readout had no response_end, and a doubled final readout still passed.

A new peer command, flush_response, closes the open response (its response_end carries flushed: true) and returns the timeline. S103's final read now uses flush_response_timeline(), and the negative unit test's final response_end is that flushed entry.

BuildBuddy S103 x3 on this head (invocation 63e2c202-8f46-4133-a60f-641a0decd7e9):

  • The readout checks passed in every run.
  • One first attempt failed only on a 500 ms monologue overlap: a "Go ahead" said into a natural pause. That is the backchannel case the classifier PR handles, stacked on this branch.
  • config_tests: 14 passed.

Evidence

S103 also needs #1308 for its monologue-overlap failure.

Combined live validation (2026-09-30, local, VM load < 120 at each run start)

One build combining #1307 (9837acb), #1308 (74e627a), #1309, #1310 (df32876), #1311 (f9a2f50), #1312 (2962983) and origin/main:

scenario result
S103 5/5 consecutive (monologue overlap 0 ms in every run; no repeated readout lines)
S104 5/5 consecutive (every reopen on the Late race path, no greeting)
S106 5/5 consecutive after #1310's native-answer wait fix; the one failure before it was that wait race (answer audio 16 ms ahead of its closing transcript delta)
S100 3/3
S105 0/3, failing only "no WorkGraph item title equals the user's final transcript", identical to Turbo S run 36599321227 attempt 1 on main (pre-existing, owned by the S105 work)

🤖 Generated with Claude Code

… burst

Both S103 attempts in Turbo S run 36599321227 failed "assistant audio
started without a new input final or commentary (duplicate readout)". The
rule treated every peer energy burst as a new assistant start and looked
for a prompt since the previous burst's end entry. Two things were wrong:

- A readout spoken line by line pauses longer than the peer's 600 ms end
  hysteresis between lines, so one readout is many bursts. Attempt 2 read
  the corrected brief as ten bursts of one response (peer `response` 5);
  attempt 1 resumed the interrupted readout after a 900 ms line pause in
  the same response 3.
- The burst's `assistant_audio_end` entry is pushed only after the
  hysteresis, about 700 ms after the last audible window. The correction's
  input final closed 8 ms before that entry but 691 ms after the assistant
  was last audible, and was not counted.

`unprompted_assistant_response_starts` groups bursts by the peer's
`response` index, which advances only when a new user utterance starts, and
opens each prompt window at the previous burst's `last_active_ms`. A
response that repeats itself is the peer's own `duplicate_readout` fault on
the output transcript, which the scenario already fails on. Unit tests
replay both failing timelines and keep an unprompted new response failing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-29T20:48:12.381915Z 53539c5 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 53539c59cd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +2947 to +2949
let continues = response.is_some() && response == previous_response;
previous_response = response;
if continues || entry.t_ms < from_ms {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Derive response boundaries independently of user input

In a real peer timeline, response.index is not an assistant-response identifier: gpt-live-peer-e2e.mjs increments it in finishResponse, which during an active session is called when a new user transcript begins (lines 341-356 and 569-570). Consequently, an unsolicited second assistant response receives the same index as the preceding response and this branch skips it; unless it repeats a five-word sentence verbatim, the separate duplicate_readout detector also stays silent. The new negative unit test manually changes the index without an intervening input, a state the harness cannot produce, so this makes S103's unprompted-response gate ineffective for the regression it claims to detect.

Useful? React with 👍 / 👎.

A local S103 run flagged a readout as unprompted: the answer to the
correction started as peer response 1, the late "Friday" delta advanced
the peer's index to 2 while the assistant was still speaking, and the next
line of the same readout started as response 2. A burst now continues the
previous response when its index equals the previous burst's index at its
start or at its end.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
lukacf and others added 2 commits September 30, 2026 01:39
Review of #1307: the burst rule groups bursts by the peer's response index,
which advances only on new user speech, so an unprompted second readout with
no user speech in between lands in the same response and is never flagged,
and the peer's own duplicate_readout fault skips sentences under five words,
which every kickoff-brief line is.

The peer's response_end timeline entry now carries the response text, and
`repeated_readout_lines` fails any response after the barge-in that speaks a
line twice (lines split on newlines and sentence ends, three-word floor).
The impossible-timeline negative test is replaced by a peer-producible one
(the brief read twice in one response, invisible to the burst rule) and a
single-readout positive. `format_timeline` now truncates on a char boundary
(a curly apostrophe at byte 240 would have panicked).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…gative

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
lukacf and others added 2 commits September 30, 2026 04:30
Review of #1307: the peer pushes response_end only from finishResponse,
which runs on a new user utterance, a disconnect or evidence stop. S103
reads its timeline right after wait_for_settled following the correction
fixture, so the final corrected-brief readout (the Turbo S attempt-2
ten-burst response) had no response_end and repeated_readout_lines never saw
it: a doubled final readout still passed.

A new peer command, flush_response, closes the in-progress response (its
response_end carries `flushed: true`) and returns the timeline; S103's final
read uses it (`flush_response_timeline`). The negative unit test's final
response_end is the flushed one the scenario now produces.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@lukacf
lukacf merged commit da511d8 into main Sep 30, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant