Priority: P2 allocation/representation simplification. This is not a proposal to remove admission or structural validation, and no throughput/RSS improvement has been measured yet.
Evidence at #707 3d955ff998eef120cd398f163c613a54e3383ccb
crates/tracedecay-capture/src/parse.rs::normalize_prepared_observation_record_v1 receives a structurally validated native record, invokes its provider normalizer, and validates the resulting typed CanonicalObservationEnvelopeV1. It then:
- serializes the entire envelope into an owned
Vec<u8>;
- checks the canonical byte limit only after that allocation;
- parses those just-created bytes back into a new
serde_json::Value;
- walks the resulting Value for structural limits;
- discards the byte buffer and retains the Value in the parser token.
The conversion is on the real Codex project/profile admission path: sessions/.../codex/observation.rs::admit_codex_jsonl_page calls that function for accepted prepared frames before handing the parser token to durable admission. Reusing the native JSON decode therefore still leaves a payload-sized textual intermediate and a new parser pass per scoped canonical envelope.
The single-record convenience API parse_normalized_observation_record_v1 is documented as consuming the decoded native Value, but routes through prepared Arc<Value> and immediately calls native.clone() before its owning normalizer. Its freshly constructed prepared token is unique on that path; sharing it internally does not require copying the complete native tree. The multi-scope borrowed prepared API, by contrast, has a legitimate shared lifetime and must remain usable.
Deletion-oriented change
Keep one envelope-to-parser-token finishing boundary shared by owned and borrowed normalization. Remove JSON bytes as an internal transport between the typed envelope and the required Value representation. Prefer serde's existing structural conversion or retain the typed envelope to the next actual serialization boundary if that fits the existing consumer; do not introduce another parallel durable DTO family or custom JSON parser.
Preserve an exact bounded canonical-encoding-size check using the existing bounded/counting serialization mechanism, rejecting during serialization rather than allocating an oversized complete buffer. A counted limit must include escaping, keys, tags and all serialized fields; decoded string lengths are not an equivalent measure. If a required downstream boundary already produces those bytes, arrange the lifetime so they are not encoded solely to be reparsed and discarded here.
Keep the single-record native Value owned through its consuming normalizer, or recover unique ownership safely; do not clone it merely to route through the multi-scope helper. Do not force the shared prepared path to consume or mutate another scope's native tree. Reuse the same finishing validation rather than copy the whole normalization implementation into two functions.
Invariants that must remain
- Native raw digest and encoded length still describe the original raw frame, not a reserialized envelope.
- Ordering domain, source byte range, provider identity and canonical envelope validation stay exact.
- Canonical byte/depth/value limits and stricter subsequent policy verification remain effective. A smaller native record does not prove the normalized envelope is within limits.
- Scope-specific normalization stays independent; reuse of native input is not authorization to reuse another scope's canonical result.
- Preserve JSON numeric behavior, escaping, object/array content, nulls and downstream sanitized/canonical receipt identities. Direct serde conversion and serialize/parse are not assumed equivalent without these checks.
Acceptance
Compare old/new actual admission output on representative supported Codex/provider records, including nested tool payloads, large escaped strings, numeric edges and stricter policy limits. Canonical values, raw digests, source identities, sanitization outputs and durable receipts must agree for accepted input; oversized/invalid input must retain its typed refusal.
A controlled just-over-canonical-limit record rejects without retaining a full oversized canonical byte buffer. The single owning API no longer deep-clones its entire native Value solely for dispatch; multi-scope reuse remains immutable and correct under cancellation.
Measure allocations and decode/encode passes on the real normalization-to-admission path, separately from durable writes and projection work. Existing native-decode counters alone do not account for the canonical reparse. Do not use a source-string test asserting that from_slice disappeared or add a duplicate large provider fixture suite.
Keep #707 draft. This is residual work after prepared/native decode reuse, not a request to undo that sharing.
Priority: P2 allocation/representation simplification. This is not a proposal to remove admission or structural validation, and no throughput/RSS improvement has been measured yet.
Evidence at #707
3d955ff998eef120cd398f163c613a54e3383ccbcrates/tracedecay-capture/src/parse.rs::normalize_prepared_observation_record_v1receives a structurally validated native record, invokes its provider normalizer, and validates the resulting typedCanonicalObservationEnvelopeV1. It then:Vec<u8>;serde_json::Value;The conversion is on the real Codex project/profile admission path:
sessions/.../codex/observation.rs::admit_codex_jsonl_pagecalls that function for accepted prepared frames before handing the parser token to durable admission. Reusing the native JSON decode therefore still leaves a payload-sized textual intermediate and a new parser pass per scoped canonical envelope.The single-record convenience API
parse_normalized_observation_record_v1is documented as consuming the decoded native Value, but routes through preparedArc<Value>and immediately callsnative.clone()before its owning normalizer. Its freshly constructed prepared token is unique on that path; sharing it internally does not require copying the complete native tree. The multi-scope borrowed prepared API, by contrast, has a legitimate shared lifetime and must remain usable.Deletion-oriented change
Keep one envelope-to-parser-token finishing boundary shared by owned and borrowed normalization. Remove JSON bytes as an internal transport between the typed envelope and the required Value representation. Prefer serde's existing structural conversion or retain the typed envelope to the next actual serialization boundary if that fits the existing consumer; do not introduce another parallel durable DTO family or custom JSON parser.
Preserve an exact bounded canonical-encoding-size check using the existing bounded/counting serialization mechanism, rejecting during serialization rather than allocating an oversized complete buffer. A counted limit must include escaping, keys, tags and all serialized fields; decoded string lengths are not an equivalent measure. If a required downstream boundary already produces those bytes, arrange the lifetime so they are not encoded solely to be reparsed and discarded here.
Keep the single-record native Value owned through its consuming normalizer, or recover unique ownership safely; do not clone it merely to route through the multi-scope helper. Do not force the shared prepared path to consume or mutate another scope's native tree. Reuse the same finishing validation rather than copy the whole normalization implementation into two functions.
Invariants that must remain
Acceptance
Compare old/new actual admission output on representative supported Codex/provider records, including nested tool payloads, large escaped strings, numeric edges and stricter policy limits. Canonical values, raw digests, source identities, sanitization outputs and durable receipts must agree for accepted input; oversized/invalid input must retain its typed refusal.
A controlled just-over-canonical-limit record rejects without retaining a full oversized canonical byte buffer. The single owning API no longer deep-clones its entire native Value solely for dispatch; multi-scope reuse remains immutable and correct under cancellation.
Measure allocations and decode/encode passes on the real normalization-to-admission path, separately from durable writes and projection work. Existing native-decode counters alone do not account for the canonical reparse. Do not use a source-string test asserting that
from_slicedisappeared or add a duplicate large provider fixture suite.Keep #707 draft. This is residual work after prepared/native decode reuse, not a request to undo that sharing.