You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Codec\Igbinary::encode() is a bare igbinary_serialize(). Codec\Json goes through json_encode + json_decode(assoc: true), which flattens every object to an array. So the two codecs do not produce equivalent values, and the difference is invisible until a handler receives a type it has never been given.
That flattening is not incidental — it is the de-facto contract every handler has been written against, because JSON has been the only writer.
What it cost
Staging flipped _APP_QUEUE_CODEC to igbinary. worker-webhooks wedged within minutes:
Utopia\Database\Document::__construct(): Argument #1 ($input) must be of type array,
Utopia\Database\Document given
A webhooks payload carries a Document. Under JSON the handler had always received an array. 701 messages, each burning its full redelivery budget while holding a maxAckPending slot; the queue backed up to 3,434 and delivered nothing for hours. (#306 fixes the retry half of that. This issue is the half that would have stopped it happening at all.)
Why detection wasn't enough
The obvious check — sample the queue and compare — cannot work. The wire is JSON, so every object on it has already been flattened. A sweep of 513 envelopes across six queues reported clean, and webhooks was the seventh.
The only place the publisher's own value is visible is encode().
The design tension
An encode() that walks the value graph and refuses objects would catch this at the publisher, immediately, in the process that owns the payload. But a recursive PHP walk on every publish taxes exactly the hot path igbinary exists to make fast — igbinary_serialize is C, the walk would not be, and queue publishes sit on the request path.
Options, none obviously right:
Walk and throw on every encode. Correct and loud, unmeasured cost. Payloads are small and profiling (appwrite-labs/cloud#5895) puts encode well below broker round-trips in the per-message budget, so this may be cheap enough — but that is a guess until measured.
Walk only under a flag, as an audit run before a codec change. This is what ReportingCodec does in appwrite-labs/cloud#5965; it belongs in this package if it is the answer, so every consumer gets it rather than one.
Normalise instead of refusing — round-trip through array form so igbinary matches JSON exactly. Safe, and destroys the performance argument for igbinary entirely.
I lean 1 if a benchmark says the walk is noise against a publish, and 2 otherwise. What I am not prepared to do is ship 1 on the assumption.
What would settle it
Benchmark igbinary_serialize($payload) against walk($payload) + igbinary_serialize($payload) on representative envelopes. If the walk is within noise of the publish round-trip, take option 1.
Codec\Igbinary::encode()is a bareigbinary_serialize().Codec\Jsongoes throughjson_encode+json_decode(assoc: true), which flattens every object to an array. So the two codecs do not produce equivalent values, and the difference is invisible until a handler receives a type it has never been given.That flattening is not incidental — it is the de-facto contract every handler has been written against, because JSON has been the only writer.
What it cost
Staging flipped
_APP_QUEUE_CODECtoigbinary.worker-webhookswedged within minutes:A webhooks payload carries a
Document. Under JSON the handler had always received an array. 701 messages, each burning its full redelivery budget while holding amaxAckPendingslot; the queue backed up to 3,434 and delivered nothing for hours. (#306 fixes the retry half of that. This issue is the half that would have stopped it happening at all.)Why detection wasn't enough
The obvious check — sample the queue and compare — cannot work. The wire is JSON, so every object on it has already been flattened. A sweep of 513 envelopes across six queues reported clean, and webhooks was the seventh.
The only place the publisher's own value is visible is
encode().The design tension
An
encode()that walks the value graph and refuses objects would catch this at the publisher, immediately, in the process that owns the payload. But a recursive PHP walk on every publish taxes exactly the hot path igbinary exists to make fast —igbinary_serializeis C, the walk would not be, and queue publishes sit on the request path.Options, none obviously right:
ReportingCodecdoes in appwrite-labs/cloud#5965; it belongs in this package if it is the answer, so every consumer gets it rather than one.I lean 1 if a benchmark says the walk is noise against a publish, and 2 otherwise. What I am not prepared to do is ship 1 on the assumption.
What would settle it
Benchmark
igbinary_serialize($payload)againstwalk($payload) + igbinary_serialize($payload)on representative envelopes. If the walk is within noise of the publish round-trip, take option 1.