Repository navigation
async components bypass the plan executor concurrency limit — 429 storm (~36-42 parallel API calls) #225
Description
Symptom
Rendering templates/clients/formula-care-ep5.tsx (12 kling clips, each wrapped in an async DialogueClip that awaits vid.audio.speechRange()) instantly blows through the API rate window (60/min): attempt 2/6 with Retry-After=60s, retry storm from varg.ts submitJob backoff.
Root cause (analyzed, needs verification of exact numbers)
Async components run during resolveLazy, before executePlan — so the executor's concurrency: 3 never applies:
resolveLazy → Promise.all over 12 <DialogueClip> (no limit)
└ vid.audio.speechRange()
└ resolveAudioElement → parent video not resolved
└ resolveVideoElement → POST /v2/video (12 parallel)
└ resolveImageInput → resolveImageElement
└ POST /v2/image (~12-18 parallel)
└ transcribeAudio → POST /v2/transcription (12 parallel)
Total: ~36-42 simultaneous API calls from 20 "declared" generations. 429 retries then re-fill the window as it resets, keeping the storm alive.
ep2/ep3/ep4 never hit this because they don't await .audio inside async components; ep5 is the first one-pass auto-trim template.
Open question (why this issue is on hold)
Need to reconstruct the actual request timeline before fixing: 60/min window should absorb ~36-42 requests unless (a) the window was already partly full, (b) 429 retries multiply the count, or (c) standalone resolvers double-submit shared elements (a1/a2/gym cards are shared by 5 scenes each and the standalone resolve path has no in-flight dedup by cacheKey, unlike renderers' pendingFiles). Suspect (b)+(c) — dedup missing means shared cards submit N times.
Action: pull request logs / gateway logs for the failed run, count actual POSTs per second, confirm the multiplier.
Fix directions discussed (not decided)
- Shared semaphore in ResolveContext (preferred):
prepare()createspLimit(options.concurrency ?? 3), standalone resolvers wrap only the network calls (generateImage/Video/Speech/Music, transcription) — not whole resolvers (nested-dependency deadlock otherwise). 429 retries land inside the limited section, killing the retry storm too. - In-flight dedup by cacheKey in standalone resolve path (reuse
renderers/dedup.tswithDedup) — shared character cards currently double-submit. - NOT viable: "resolveAudioElement should wait for executePlan" —
DialogueClipcan't return its tree untilcutFromis known, which depends on generated content; executePlan only starts after the tree exists. Await-in-component must generate during resolveLazy. - Workaround tier: raise API rate limit (rejected as a fix, masks the bug).
Context
- Branch:
feature/ir-compile-audio(PR refactor: computation-graph IR, render lifecycle, renderers split #223) - Reproducer:
templates/templates/clients/formula-care-ep5.tsx - Related recent change: 429-retry added to
submitJobinsrc/ai-sdk/providers/varg.ts(uncommitted local change in templates/sdk checkout) — mitigates but also amplifies the storm
Activity
SecurityQQ commentedon Jul 21, 2026
Root cause found (production DB analysis) + fix landed
The numbers: 123 jobs in 7 minutes for ~8 unique requests — 52× the location card, 48× two character cards, ~5× each scene composite. All with
idempotency_key = NULL.Two compounding bugs:
resolveImageElement/resolveVideoElementetc. inresolve.tscallgenerateImage/Videodirectly, bypassing the renderers'withDedup/pendingFiles. Themeta?.filecheck only catches finished resolves — 12 concurrent async components racing throughspeechRange() → resolveAudioElement → resolveVideoElement → resolveImageElementtoward one sharedlocTrailelement all sawmeta === undefinedand fired parallel generations.makeThenablememoization didn't help because nested elements are resolved via direct function calls, not through the thenable.Idempotency-Keyon job creation. Every 429 retry insubmitJobcreated a brand-new job, refilling the rate window as it reset — the retry storm sustained itself.Fix (commit in PR #223):
memoizedElementResolveinresolve.ts— WeakMap-keyed by element identity, wraps all standalone resolvers (image/video/speech/music/talking-head/audio). Graph invariant: one element = one generation, regardless of how many paths lead to it. Meta write-back makes later paths (renderers, executePlan, compile) see the element as pre-resolved. Rejected promises stay memoized (no silent re-spend).computeIdempotencyKeyinvarg.ts— stable hash of (capability, canonicalized params) sent asIdempotency-Keyheader; 429/network retries reference the same job. Verified: the API stores it (be_varg_jobs.idempotency_keypopulated on live run).Verification (cheap, real API): reproduced the ep5 dependency shape with flux-schnell + nano-banana-pro/edit (3 async components sharing one nested image, concurrent via resolveLazy): exactly 4 jobs for 4 unique requests, shared image generated once, 0 rate-limit warnings, all jobs carry idempotency keys. Plus unit tests (
resolve-dedup.test.ts,varg-idempotency.test.ts).Still open (the original hold-question, now lower priority): a concurrency semaphore for the resolveLazy phase. With dedup, unique-call count for ep5-class templates (~20) fits the 60/min window fine; a semaphore only matters for graphs with >60 unique generations. Leaving this issue open for that follow-up, de-escalated.
API-side follow-up (separate repo): consider enforcing/deduping on
Idempotency-Keyserver-side if not already — client now always sends it.fix: caption timing in trimmed clips, in-flight resolve dedup, idempo…
SecurityQQ commentedon Jul 21, 2026
Refactor umbrella for the whole ep5 fix series (including the semaphore and API-side idempotency follow-ups tracked here): #227