Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions drift-proposals/gemini-gemini-3.5-transcribe-live-new-family.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# New / unclassified model family: gemini-3.5-transcribe-live

Provider: gemini
Detected: 2026-08-27
Status: RESOLVED — decision recorded below and applied to the registry

This model family appeared in a live /models listing but matches no classification rule (include, exclude, -preview, gemma). drift-sync never silently classifies a new family.

## Decision

<!-- drift-sync never auto-classifies a new family. To approve adding it to
the registry, change the line below to `Decision: include` — the NEXT
drift-sync run will then apply the mechanical registry edit (still
zero-LLM: this is a human-authored decision, not generated code). -->

<!-- NOTE: the `Decision: include` marker documented above is drift-sync's
AUTOMATED path, and it writes EXCLUSIVELY into `includeFamilies`
(scripts/drift-sync.ts: addFamilyLiteralInSource(..., "includeFamilies", ...)).
There is no automated exclude path, so an EXCLUDE decision is recorded here
in prose and applied by hand — writing `include` would misclassify. -->

Decision: EXCLUDE (applied — excludeFamilies.gemini in
`src/__tests__/drift/model-registry.ts`, with the `excludeFamilies.gemini`
re-pin in `src/__tests__/drift/logic-pin.test.ts`).

Rationale: wrong modality — a bidirectional streaming SPEECH-TO-TEXT surface.
This is the Live-API half of the same GA line as `gemini-3.5-transcribe` (one
model, two surfaces, two distinct normalized families, so both are enumerated).
Google's release note calls it "Low-latency, bidirectional streaming
speech-to-text over WebSockets" with "interim and finalized transcription
events", and the model card gives it the same input `Audio` -> output `Text,
Word annotations` signature as its unary sibling.

TWO INDEPENDENT capability facts were observed live, not inferred from the id:

1. It DECLARES `bidiGenerateContent`. The Gemini Live leg's discovery
(`fetchLiveCapableModels` in `ws-gemini-live.drift.ts`) filters SOLELY on
that declared method and applies no name heuristic, and it selected this id
— so the declaration is present.

2. It CANNOT emit AUDIO. Google refused the resulting session out of band:
`code=1007 reason="The requested combination of response modalities (AUDIO)
is not supported by the model. models/gemini-3.5-transcribe-live"`
(drift run 33296393200, 2026-08-30).

So it is a bidi Live surface that emits TEXT and cannot emit AUDIO — a streaming
transcriber, not a native-audio conversational model. It belongs to the realtime
canary's domain alongside the already-excluded `gemini-live`, and can never be
text-generation drift. Mirrors the OpenAI `gpt-live-transcribe` decision (PR
#343).

The same 1007 is ALSO the diagnosis for the Gemini Live leg's own failure; that
half is fixed in `ws-gemini-live.drift.ts`, not here.
43 changes: 43 additions & 0 deletions drift-proposals/gemini-gemini-3.5-transcribe-new-family.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# New / unclassified model family: gemini-3.5-transcribe

Provider: gemini
Detected: 2026-08-27
Status: RESOLVED — decision recorded below and applied to the registry

This model family appeared in a live /models listing but matches no classification rule (include, exclude, -preview, gemma). drift-sync never silently classifies a new family.

## Decision

<!-- drift-sync never auto-classifies a new family. To approve adding it to
the registry, change the line below to `Decision: include` — the NEXT
drift-sync run will then apply the mechanical registry edit (still
zero-LLM: this is a human-authored decision, not generated code). -->

<!-- NOTE: the `Decision: include` marker documented above is drift-sync's
AUTOMATED path, and it writes EXCLUSIVELY into `includeFamilies`
(scripts/drift-sync.ts: addFamilyLiteralInSource(..., "includeFamilies", ...)).
There is no automated exclude path, so an EXCLUDE decision is recorded here
in prose and applied by hand — writing `include` would misclassify. -->

Decision: EXCLUDE (applied — excludeFamilies.gemini in
`src/__tests__/drift/model-registry.ts`, with the `excludeFamilies.gemini`
re-pin in `src/__tests__/drift/logic-pin.test.ts`).

Rationale: wrong modality — speech->text, not text generation. Google's model
card for the Gemini 3.5 Transcribe line (GA 2026-08-26) gives this id the
signature input `Audio (up to 1 hour)` -> output `Text, Word annotations`, and
the release note describes it as "High-accuracy, low-latency non-streaming
speech-to-text". It emits no chat completion of its own: the text it returns is
a transcript of the caller's audio, with speaker diarization and word-level
timestamps. That is the audio canary's domain, not this text check.

NOT rejected for the "transcribe" substring. The deciding evidence is the
provider-declared input/output signature, the same instrument used for
`gemini-3.7-flash-video-understanding-eap` (declared access status, not the
"video" in its name) and against which the native-audio misclassification was
caught. Mirrors the OpenAI `gpt-transcribe` / `gpt-live-transcribe` decisions
(PR #343).

Membership in `excludeFamilies` is a CLASSIFICATION for the `/models` listing
check in `models.drift.ts` and nothing more: it says the family is accounted
for, not which endpoints aimock implements for it.
44 changes: 44 additions & 0 deletions drift-proposals/gemini-gemini-omni-1.1-flash-new-family.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# New / unclassified model family: gemini-omni-1.1-flash

Provider: gemini
Detected: 2026-08-28
Status: RESOLVED — decision recorded below and applied to the registry

This model family appeared in a live /models listing but matches no classification rule (include, exclude, -preview, gemma). drift-sync never silently classifies a new family.

## Decision

<!-- drift-sync never auto-classifies a new family. To approve adding it to
the registry, change the line below to `Decision: include` — the NEXT
drift-sync run will then apply the mechanical registry edit (still
zero-LLM: this is a human-authored decision, not generated code). -->

<!-- NOTE: the `Decision: include` marker documented above is drift-sync's
AUTOMATED path, and it writes EXCLUSIVELY into `includeFamilies`
(scripts/drift-sync.ts: addFamilyLiteralInSource(..., "includeFamilies", ...)).
There is no automated exclude path, so an EXCLUDE decision is recorded here
in prose and applied by hand — writing `include` would misclassify. -->

Decision: EXCLUDE (applied — excludeFamilies.gemini in
`src/__tests__/drift/model-registry.ts`, with the `excludeFamilies.gemini`
re-pin in `src/__tests__/drift/logic-pin.test.ts`).

Rationale: non-text generative media. Google's 2026-08-27 release note announces
`gemini-omni-1.1-flash` as "the GA version of our fast, conversational video
generation and editing model", and its model-card signature is input `Video with
audio` -> output `Video with audio` (scene extension to 40s, keyframe
interpolation, 360p-4K upscale). It produces no text turn at all, so it cannot
be text-generation drift. Same category as the already-excluded `veo-*`,
`lyria-*` and `imagen-*` entries.

NOT classified from the "omni" substring — which would have argued the OPPOSITE.
"omni" names OpenAI's TEXT-capable omni line (`omni-moderation`, and the `gpt-4o`
"omni" lineage), so a name-shaped rule here would have read this as a
text-capable multimodal chat tier and INCLUDED it. The provider-declared
input/output signature is what decides it, exactly as with
`gemini-3.7-flash-video-understanding-eap`.

Its `-preview` sibling `gemini-omni-flash-preview` is already auto-excluded by
the PREVIEW_FAMILY rule. This GA id carries no trailing `-preview` token, so
that rule cannot reach it and it must be enumerated (mirrors the
`-preview-tts` / `-preview-customtools` entries).
2 changes: 1 addition & 1 deletion src/__tests__/drift/logic-pin.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -441,7 +441,7 @@ const DATA_FROZEN: Record<string, { members: () => string[]; pin: string }> = {
},
"excludeFamilies.gemini": {
members: () => [...excludeFamilies.gemini].sort(),
pin: "3c8fb0b7cee776619467afd34407bde69e30b063ae263da4871096e7a7afca1d",
pin: "8afbca596165c15f956e94fe0a358adf2144c8f3c5f93b6599ddb81e0eb2ecfa",
},
// The realtime canary's seed sets, previously pinned NOWHERE. An edit to
// either one was invisible to every guard in the repo: adding a family to
Expand Down
49 changes: 49 additions & 0 deletions src/__tests__/drift/model-registry.ts
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,55 @@ export const excludeFamilies: Record<Provider, Set<string>> = {
"gemini-2.0-flash-thinking-exp",
// Live/full-duplex voice — owned by the realtime canary, not this text check
"gemini-live",
// The Gemini 3.5 Transcribe line, GA 2026-08-26, first observed by the daily
// /models canary on 2026-08-27 (that later date is the `Detected:` stamp in
// the notes under drift-proposals/ — two dates for two events, not a
// disagreement). Two ids ship as ONE model on TWO surfaces, and they are two
// distinct normalized families, so both are enumerated.
//
// Classified on DECLARED CAPABILITY, not on the "transcribe" substring.
// Google's model card gives both the same signature — input `Audio (up to
// 1 hour)`, output `Text, Word annotations` — i.e. speech->text only. They
// emit no chat completion of their own; the text they return is a
// transcript of the caller's audio. That is the realtime/audio canary's
// domain (voice-models.ts), exactly like `gemini-live` above, and it can
// never be TEXT-GENERATION drift.
//
// `-live` carries a second, independently observed capability fact. The Live
// leg's discovery filters SOLELY on declared `bidiGenerateContent` (no name
// heuristic — see ws-gemini-live.drift.ts), and it selected this id, so the
// model does declare that method. Google then refused the session out of
// band: `code=1007 reason="The requested combination of response modalities
// (AUDIO) is not supported by the model. models/gemini-3.5-transcribe-live"`
// (run 33296393200). So it is a bidi Live surface that emits TEXT and cannot
// emit AUDIO — a streaming transcriber, not a native-audio model.
//
// NOTE membership here is a CLASSIFICATION for the `/models` listing check
// in models.drift.ts and nothing more: it says the family is accounted for,
// not which endpoints aimock implements for it. Mirrors the OpenAI
// `gpt-transcribe` / `gpt-live-transcribe` pair above.
//
// Decision: EXCLUDE, recorded in
// drift-proposals/gemini-gemini-3.5-transcribe-new-family.md and
// drift-proposals/gemini-gemini-3.5-transcribe-live-new-family.md.
"gemini-3.5-transcribe",
"gemini-3.5-transcribe-live",
// Gemini Omni Flash, GA 2026-08-27. Non-text generative media, in the same
// category as the veo-* / lyria-* / imagen-* entries.
//
// Classified on DECLARED CAPABILITY, not on the "omni" substring — "omni"
// names OpenAI's TEXT-capable omni line, so the substring would have argued
// the opposite. Google's release note calls it "our fast, conversational
// video generation and editing model", and its model-card signature is
// input `Video with audio` -> output `Video with audio` (extension,
// keyframe interpolation, 360p-4K). It produces no text turn, so it cannot
// be text-generation drift. Its `-preview` sibling `gemini-omni-flash-preview`
// is already auto-excluded by PREVIEW_FAMILY; this GA id carries no
// `-preview` token, so that rule cannot reach it and it must be enumerated.
//
// Decision: EXCLUDE, recorded in
// drift-proposals/gemini-gemini-omni-1.1-flash-new-family.md.
"gemini-omni-1.1-flash",
// Restricted EARLY-ACCESS (EAP) surface, not a GA tier. Google's own live
// /models entry declares `displayName` AND `description` as
// "[Confidential] Gemini 3.7 Flash Video Understanding EAP" — the only one
Expand Down
5 changes: 5 additions & 0 deletions src/__tests__/drift/models.drift.ts
Original file line number Diff line number Diff line change
Expand Up @@ -402,6 +402,11 @@ describe("full live /models wave is fully classified (2026-07-16 drift)", () =>
// image / audio / tts / video / music / robotics / embeddings
"gemini-2.5-flash-image",
"gemini-2.5-flash-native-audio-latest",
// 2026-08-26/27 GA line (run 33296393200): speech-to-text pair + the
// conversational video-generation tier — all explicit excludes
"gemini-3.5-transcribe",
"gemini-3.5-transcribe-live",
"gemini-omni-1.1-flash",
"gemini-2.5-flash-preview-tts", // explicit exclude (-preview-tts)
"gemini-2.5-pro-preview-tts", // explicit exclude (-preview-tts)
"gemini-3-pro-image",
Expand Down
70 changes: 67 additions & 3 deletions src/__tests__/drift/providers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -118,10 +118,23 @@ export interface LiveModelEntry {
* model.
*/
export function selectLiveModel(models: LiveModelEntry[], preferred: string[] = []): string | null {
const liveIds = models
const liveIds = liveModelIds(models);
return preferred.find((p) => liveIds.includes(p)) ?? liveIds[0] ?? null;
}

/**
* The usable (non-deprecated, non-empty) ids from a `/models` listing, in the
* provider's own order.
*
* Shared by {@link selectLiveModel} — which takes the first — and by
* {@link resolveLiveModelCandidates}, which keeps the whole ordered list for a
* leg that must try more than one because the LISTING alone cannot settle the
* capability it needs (see that function's note).
*/
function liveModelIds(models: LiveModelEntry[]): string[] {
return models
.filter((m) => typeof m.id === "string" && m.id.length > 0 && m.deprecated !== true)
.map((m) => m.id);
return preferred.find((p) => liveIds.includes(p)) ?? liveIds[0] ?? null;
}

/**
Expand Down Expand Up @@ -172,9 +185,60 @@ export function resolveLiveModel(
return promise;
}

/** Test-only: clear the {@link resolveLiveModel} memo cache between cases. */
/**
* Outcome of resolving the FULL ordered candidate list for a leg — the same
* three-way classification as {@link ResolvedModel}, carrying every usable id
* instead of just the first.
*/
export type ResolvedCandidates = { models: string[] } | { infra: number } | { unavailable: true };

/** Per-key memo for {@link resolveLiveModelCandidates}. */
const resolvedCandidatesCache = new Map<string, Promise<ResolvedCandidates>>();

/**
* Resolve EVERY usable model from a provider's listing, MEMOIZED per `key`,
* with the same infra/unavailable classification as {@link resolveLiveModel}.
*
* WHY A LIST. `resolveLiveModel` takes the first usable id, which is right
* whenever the listing filter already settles the capability the leg needs. It
* is NOT right when the listing cannot express that capability. Gemini Live is
* that case: `/models` declares `bidiGenerateContent` but says nothing about
* which RESPONSE MODALITY a model serves, and the two are genuinely
* independent — `gemini-3.5-transcribe-live` declares `bidiGenerateContent` and
* yet emits only TEXT, so a leg driving AUDIO must be able to move past it to
* the next candidate. The provider's own refusal is the only authority on that
* capability, so the leg learns it by asking (see `driveGeminiLiveAudio` in
* ws-providers.ts) — never by pattern-matching the model NAME.
*/
export function resolveLiveModelCandidates(
key: string,
fetchListing: () => Promise<{ status: number; models: LiveModelEntry[] }>,
): Promise<ResolvedCandidates> {
const cached = resolvedCandidatesCache.get(key);
if (cached) return cached;
const promise = (async (): Promise<ResolvedCandidates> => {
try {
const { status, models } = await fetchListing();
if (isInfraSkip(status)) return { infra: status };
if (status >= 400) return { unavailable: true };
const ids = liveModelIds(models);
return ids.length > 0 ? { models: ids } : { unavailable: true };
} catch (err) {
if (err instanceof InfraError) return { infra: err.status };
throw err;
}
})();
resolvedCandidatesCache.set(key, promise);
return promise;
}

/**
* Test-only: clear the {@link resolveLiveModel} and
* {@link resolveLiveModelCandidates} memo caches between cases.
*/
export function __resetResolveLiveModelCache(): void {
resolvedModelCache.clear();
resolvedCandidatesCache.clear();
}

// ---------------------------------------------------------------------------
Expand Down
83 changes: 83 additions & 0 deletions src/__tests__/drift/text-drift.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -127,3 +127,86 @@ describe("openai transcription line is classified as EXCLUDED (PR #343)", () =>
).toEqual([]);
});
});

// ---------------------------------------------------------------------------
// The 2026-08-26/27 Gemini transcription + omni-video line — BEHAVIOURAL
// coverage of the classification, in the same shape as the OpenAI block above.
//
// `gemini-3.5-transcribe`, `gemini-3.5-transcribe-live` and
// `gemini-omni-1.1-flash` were classified EXCLUDE in model-registry.ts on the
// provider's own DECLARED capabilities (see the rationale comment beside the
// entries, and drift-proposals/). Without the assertions below the only thing
// that would redden if an entry were dropped is the `excludeFamilies.gemini`
// membership CHECKSUM in logic-pin.test.ts — which says "the data moved" and
// nothing about what the classification MEANS.
//
// The `gemini-3.5-transcribe` / `gemini-3.5-transcribe-live` pair is asserted
// in BOTH directions on purpose: the first key is a strict PREFIX of the
// second, the same substring/prefix hazard the OpenAI `gpt-live` block exists
// for. They are DIFFERENT families and both must be classified on their own
// entry, not by one swallowing the other.
// ---------------------------------------------------------------------------

describe("gemini transcription + omni-video line is classified as EXCLUDED", () => {
it("gemini-3.5-transcribe is EXCLUDED in a /models-shaped payload", () => {
expect(isClassifiedFamily("gemini-3.5-transcribe", "gemini")).toBe(true);
expect(
unclassifiedFamilies(
[
"gemini-3.5-flash", // include, for a realistic mixed listing
"gemini-3.5-transcribe",
"gemini-3.5-transcribe-2026-08-26", // dated snapshot collapses onto the family
],
"gemini",
),
).toEqual([]);
});

it("gemini-3.5-transcribe-live is EXCLUDED in a /models-shaped payload", () => {
expect(isClassifiedFamily("gemini-3.5-transcribe-live", "gemini")).toBe(true);
expect(
unclassifiedFamilies(
[
"gemini-3.5-flash",
"gemini-3.5-transcribe-live",
"gemini-3.5-transcribe-live-2026-08-26",
"gemini-live", // the pre-existing full-duplex Live surface
],
"gemini",
),
).toEqual([]);
});

it("gemini-omni-1.1-flash is EXCLUDED in a /models-shaped payload", () => {
expect(isClassifiedFamily("gemini-omni-1.1-flash", "gemini")).toBe(true);
expect(
unclassifiedFamilies(
[
"gemini-3.5-flash",
"gemini-omni-1.1-flash",
"gemini-omni-1.1-flash-2026-08-27",
"gemini-omni-flash-preview", // sibling preview tier, excluded by pattern
],
"gemini",
),
).toEqual([]);
});

it("neither transcribe key classifies the other, nor an unrelated extension", () => {
// The two entries are distinct families; a `startsWith`-shaped classification
// bug would let the shorter key classify the longer one (or vice versa) and
// silently swallow a family the canary exists to report.
expect(normalizeModelFamily("gemini-3.5-transcribe-live", "gemini")).toBe(
"gemini-3.5-transcribe-live",
);
// NEGATIVE CONTROL: an id that merely EXTENDS an excluded key is still a new
// family and must be reported. Without this, `toEqual([])` above is also
// what a neutered `unclassifiedFamilies` would produce.
expect(unclassifiedFamilies(["gemini-3.5-transcribe-diarize"], "gemini")).toEqual([
"gemini-3.5-transcribe-diarize",
]);
expect(unclassifiedFamilies(["gemini-omni-1.1-pro"], "gemini")).toEqual([
"gemini-omni-1.1-pro",
]);
});
});
Loading
Loading