Add durable-execution support to Lambda Test Tool v2 (on shared kernel)#2497
Draft
GarrettBeatty wants to merge 7 commits into
Draft
Add durable-execution support to Lambda Test Tool v2 (on shared kernel)#2497GarrettBeatty wants to merge 7 commits into
GarrettBeatty wants to merge 7 commits into
Conversation
Adds a local durable-execution service emulator to the Test Tool, enabled with --durable-execution. A durable function whose AWS_ENDPOINT_URL_LAMBDA is pointed at the Test Tool can now run a full workflow locally. Phase 1 (data plane): CheckpointDurableExecution and GetDurableExecutionState endpoints hosted alongside the Runtime API, backed by the operation store + checkpoint processor ported from Amazon.Lambda.DurableExecution.Testing and retyped to plain System.Text.Json request DTOs (the AWSSDK model types use ConstantClass and can't be STJ-deserialized). Response timestamps are emitted as unix seconds to match the AWSSDK rest-json unmarshaller. Phase 2 (control plane): invoking with X-Amz-Durable-Execution-Name starts an execution that DurableExecutionDriver drives to completion, re-invoking the function over the Runtime API across the replay cycle. Timers are time-skipped by default (--durable-time-skip). Chained durable invokes and external callbacks are not yet supported and fail fast with a clear message. Includes end-to-end tests exercising both the data plane and the full start-hook/drive-loop cycle through a real AmazonLambdaClient and Runtime API.
Callbacks: SendDurableExecutionCallback{Success,Failure,Heartbeat} endpoints
resolve a pending CALLBACK operation by its minted id and wake the parked
drive loop. The driver now parks on a pending callback (instead of failing)
and resumes when the callback is resolved; park/resume is lock-guarded so a
resolve that races ahead of the park is not lost.
Web UI: a new Durable Execution page lists local executions, renders each
one's operation timeline (id/type/status/detail), and can send a callback
from the browser. The page shows a friendly hint when --durable-execution is
not enabled.
Includes an end-to-end test covering the full start -> park-on-callback ->
SendDurableExecutionCallbackSuccess -> resume -> Succeeded cycle.
…ator (Phase 4) Chained durable invokes: a CHAINED_INVOKE operation is now resolved by starting the target function as a nested durable execution and stamping its result (or error) back onto the parent's operation, so ctx.InvokeAsync works locally. The sibling runs the same drive loop under its own function name. StopDurableExecution: the /stop endpoint cancels the execution's drive loop and marks the EXECUTION operation STOPPED, returning a StopTimestamp. Payload caps: checkpoint updates whose operation payload exceeds the service limits (256 KB general, 1 MB for chained invokes) are rejected with 413. Also fixes a latent bug: the drive loop awaited EventContainer.WaitForCompletion on a threadpool thread, a blocking 15-minute wait that ignored cancellation, so Stop could not interrupt an in-flight invocation. Replaced with a cancellation-aware status poll. The durable web-host tests now run in a non-parallel xUnit collection to avoid starving their Runtime-API poll loops under CPU pressure. Full Test Tool unit suite passes (207 passed, 1 pre-existing skip).
The callback-id and result inputs now bind on oninput (not the default onchange) so a value is committed as the user types, and the Send Success button is no longer gated on a disabled attribute that consumed the first click to commit the field. Verified end to end in a real browser (Playwright): starting a durable workflow, parking on a callback, clicking Send Callback to prefill the id, and clicking Send Success resumes the execution to Succeeded.
Adds a 'Start as durable execution' checkbox (with optional execution name) to the Function Tester page so users can start a durable workflow from the UI without manually setting the X-Amz-Durable-Execution-Name header. The checkbox only appears when the tool is started with --durable-execution. Fixes Re-Invoke for durable functions: the queued/executed events for a durable function are internal replay envelopes, so re-posting one was a no-op replay. Re-Invoke now detects a durable envelope, extracts the original user payload from the EXECUTION operation, and starts a fresh durable execution. Adds a durable-execution sample request and documents the feature in the README (Durable Execution Mode) and the in-app Documentation page. Verified end to end in a browser (Playwright): the checkbox starts an execution that appears on the Durable Execution page, and Re-Invoke creates a new execution.
Fixes --durable-time-skip effectively defaulting to false: as a plain bool Spectre.Console.Cli option it bound presence->true/absence->false regardless of the C# field initializer, so timers were never skipped even though the default was documented as true. Now uses [DefaultValue(true)] and accepts an explicit value (--durable-time-skip false). Adds a live 'Time-skip timers' toggle to the Durable Execution page. Time-skip is now read on each checkpoint (via a provider) rather than captured at store construction, so it can be flipped at runtime and applies to subsequent checkpoints across all executions. Fixes the timeline showing a completed WAIT as STARTED: when time-skip is off and a timer elapses, the SDK proceeds without emitting a SUCCEEDED checkpoint, leaving the op STARTED forever. The driver now stamps elapsed WAITs as SUCCEEDED before re-invoking (mirroring the service firing the timer), so the store and UI reflect that the wait completed. Verified end to end in a browser (Playwright): time-skip defaults on (execution parks in ~1s), the WAIT shows SUCCEEDED, the UI toggle flips interactively, and the checkbox-start and callback flows still work.
Replace the Test Tool durable emulator's hand-maintained fork of the checkpoint state machine and operation store with the shared Amazon.Lambda.DurableExecution.LocalEmulation package, so it cannot drift from the durable-execution testing package. - Delete the forked CheckpointProcessor and InMemoryOperationStore. - Add WireOperationUpdateMapper (STJ wire DTO -> OperationUpdateInput) and route DurableExecutionStore/DurableExecutionDriver through the kernel's processor, store, exec-0 seeding, and resume-delay helper. - Switch the durable SDK reference from a preview PackageReference to an in-repo source ProjectReference (matching how RuntimeSupport is referenced) so there is a single Amazon.Lambda.DurableExecution assembly identity across the SDK and the kernel. Stacked on the LocalEmulation kernel PR (and PR #2492); builds once both are on the target branch. No user-facing behavior change.
GarrettBeatty
force-pushed
the
feat/kernel-on-feature
branch
from
July 23, 2026 20:01
5711349 to
3d23038
Compare
GarrettBeatty
force-pushed
the
feat/durable-testtool-on-kernel
branch
from
July 23, 2026 20:01
3cd4238 to
72f7a6b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds durable-execution support to the Lambda Test Tool v2, built on the shared local-emulation kernel. Supersedes #2495 (that PR's durable commits are re-homed here on top of the kernel).
The 6 durable feature commits (emulator phases 1–4, callbacks + web UI, chained invokes/stop/payload caps, UI start + time-skip toggle) are unchanged. The final commit replaces the Test Tool's hand-maintained fork of the checkpoint state machine + operation store with the shared
Amazon.Lambda.DurableExecution.LocalEmulationpackage:CheckpointProcessorandInMemoryOperationStore.WireOperationUpdateMapper(STJ wire DTO →OperationUpdateInput) and routesDurableExecutionStore/DurableExecutionDriverthrough the kernel's processor, store,exec-0seeding, and resume-delay helper.PackageReferenceto an in-repo sourceProjectReference(matching how RuntimeSupport is referenced) so there is a singleAmazon.Lambda.DurableExecutionassembly identity across the SDK and the kernel.No user-facing behavior change from the fork removal.
Stack
PR 2 of 2. Base:
feat/kernel-on-feature(#2496), which must merge first. Both target thefeature/durable-testtoolintegration branch (=dev+ #2492's net10 fixes).Testing
~Durablefilter). Note:StartHook_DrivesWaitThenStepWorkflow_ToSucceededis a known flake under concurrent net8/net10 test load—it passes in isolation (<1s) and is unrelated to the kernel path.