Filter across collections through declared joins

[ddeboer]

Summary

Give the query model cross-collection filtering by declaring joins on the edges the schema already describes. A reference field whose labelSource names a root type already asserts that its values are ids of documents in that type’s collection – exactly the fact a Typesense reference field needs. An explicit joinable flag turns that assertion into an engine-level reference, so “every object published by institution X” becomes one query instead of two round trips.

In limburg/lol, CreativeWork.dataset → Dataset.publisher → Publisher compiles to:

filter_by: $datasets($publishers(id:=X))

Decisions

Schema

1. joinable?: boolean on ReferenceField, default false, valid only alongside labelSource. It is a capability flag in the existing vocabulary of filterable / facetable / sortable – not a reinterpretation of every labelSource. Auto-deriving was rejected: Typesense silently refuses to index a mutual reference, so an existing schema would lose a field with no error.
2. At most one joinable field per (root type, label source), enforced by searchSchema. Typesense addresses a join by collection, not by field, so a second reference to the same collection is accepted, indexed, and then unreachable (see Constraints). publisher and creator both resolving to Organization is the ordinary case, so this must be a declaration error naming both fields.
3. joinGraph(schema) in @lde/search, built eagerly by searchSchema. One concept, two members – components and resolve(from, path). It hides edge derivation from joinable + labelSource, the uniqueness rule, cycle rejection, the depth cap, and the asymmetry that component membership is the undirected connected component while creation order is the directed topological sort. resolve returns a RootType, never a collection name: collection naming is engine- and deployment-specific and stays in the adapter.

Label resolution stays join-free even where a join exists. Only fields that need cross-collection filtering pay the rebuild coupling below; a labelSource added for display costs nothing it does not cost today.

Query

4. Criterion-level on: readonly string[] – a path of field names, resolved hop by hop:

   { on: ['dataset', 'publisher'], field: 'id', in: ['X'] }

   The criterion stays a leaf: on is a path, not boolean structure, so ADR 18’s “a criterion can never itself be a conjunction” holds verbatim and skip-own-filter (ADR 5) still scans one level, keyed by (on, field). Putting on on the Filter was rejected: it would scope a whole disjunction and make $publishers(country:=NL) || title:=x inexpressible.

5. Depth capped at 3, fixed, checked in validateQuery with a join-too-deep QueryIssue. Typesense imposes no limit of its own, and the existing graphql-armor max-depth counts selection sets, not input nesting. The cap lives in the IR so a later REST surface inherits it.

6. GraphQL emits one ‹Target›ReferenceFilter @oneOf { in: [String!], where: ‹Target›Where } per joinable target, shared by every field pointing at it. Non-joinable reference fields keep plain StringFilter, so the capability difference is visible in the schema rather than a runtime error. whereToFilters flattens nested where into an on path exactly as it already flattens and.

Index

7. Emitted reference fields carry async_reference: true and cascade_delete: false, and always target .id. All three are forced:

   • without async_reference, a document whose referent has not been indexed yet is rejected with a 400 – and importBatch runs throwOnFail: false, so those would be silently dropped documents. Documents stream per dataset (ADR 13), so out-of-order arrival is normal, not exceptional;
   • cascade_delete defaults to true, so a sweep removing a departed source’s Publisher documents would delete other sources’ CreativeWork documents with them. Disabling it requires async_reference: true, so the two travel together;
   • a reference match that hits more than one document is also a 400, and id is the only field the schema guarantees unique.

8. The join component is the unit of rebuild, in both index modes. searchIndexWriter already opens one run per root type in a single pipeline run, so this is an ordering and commit change, not a new orchestration: open in topological order (referenced first), and commit a component all-or-nothing. Locking needs no change – the existing single deterministic pass already takes every lock in a fixed order, which is what prevents lock-ordering deadlock.

   This weakens ADR 9: the unit of isolation becomes the component rather than the collection. Types with no joinable edge are singleton components and keep today’s behaviour exactly, so the weakening applies only where a schema author opted in.

9. collectionNameFor: (searchType) => string replaces name in the rebuild options, so a blue-green writer can derive a peer’s versioned collection name. No coordinator is needed: every writer in a run receives the same RunContext, so Date.parse(context.startedAt) is already identical across them.

10. InPlaceRebuild fails loudly when a collection exists but lacks a declared reference field, with a drop-and-rebuild message. It never alters. ensureCollectionExists only creates on a 404, so without this an existing deployment would index and commit successfully and then 400 on every join. Failing keeps the invariant that makes (8) sound: a component’s collections always come into existence with their references, and never acquire them later. Scoped to reference fields only – every other schema difference is self-correcting, and general drift detection is a separate feature.

Scope

In v1: everything above.

Out of v1:

• reverse joins – Typesense supports searching from either side of a reference, but the schema has no name for the inverse edge, so the GraphQL surface would have to invent one;
• facets on joined fields – leaves ADR 5 untouched;
• sort through joins – cheap and symmetric (Sort gains the same on), but the joined per-locale sort key and the many-to-one representative-value semantics need their own decision;
• free text through joins – not expressible: a join clause is filter_by only, with no q;
• shadow collections – one materialised join-target collection per edge would make joinable uniform, but aliases cannot provide it (a reference resolves an alias to the concrete collection at create time and discards it), so it means real duplicated storage and extra component members. Deferred, and unblocked by the API shape: it lives entirely in collection-definition.ts and the writer fan-out, so lifting the rule later changes nothing in the schema API, the IR or GraphQL.

Documented limitations

• Only one edge per (type, target type) can be joinable. Other edges keep labels, facets and id filtering; they just do not gain cross-collection filtering.
• Blue-green has a brief inconsistency window at the alias flip: Typesense stores the concrete collection name in a reference and re-resolves the alias at query time, and there is no atomic multi-alias swap, so a join query can see 400 Failed to join on … for one round trip. Because a component rebuilds both sides, we never hit the steady-state form of typesense#2827.
• A reference whose target is never indexed stays unresolved and its document is silently excluded from join filters, with no signal distinguishing that from “no value”. Normal in linked data, where an IRI may point outside the indexed corpus.

Constraints verified against Typesense 30.2

Checked against the docs, the v30.2 source, and a live typesense/typesense:30.2 container.

• Joins landed in v26, not 28. Our containers already run 30.0, so there is no version floor to raise.
• Altering a collection to add a reference field is supported in 30.x, despite the note still on the joins docs page saying otherwise (v30.0 release notes; CollectionJoinTest.AlterReferenceField passes at v30.2). We do not use it – see (10) – but the constraint as originally written here was wrong.
• A collection can only be joined through one reference field per target collection. Every join API resolves through a reverse map keyed by collection name (referenced_in, include/collection.h:476), populated with a non-overwriting emplace (src/collection.cpp:8540); the forward check discards which field matched (:8504). Reproduced live: with books.author_id and books.editor_id both referencing people.id, filter_by=$people(name:=Ann) returns the book, $people(name:=Bob) returns nothing though the book is edited by Bob, include_fields=$people(*) nests only the author, and $editor_id(…) fails with Referenced collection 'editor_id' not found. Filed upstream as typesense#3021, asking for either a field-qualified syntax or rejection at create time. The docs neither promise nor forbid it.
• Dropping a referenced collection is not blocked and leaves a dangling reference, which is why a component must commit all-or-nothing rather than per collection.

Deliverables

• ADR “Filter across collections through declared joins”, amending ADR 18 (criterion-level on) and ADR 9 (component-scoped isolation).
• docs/reference/ updates for search, search-typesense and search-api-graphql, including the limitations above.
• Integration test against a real Typesense on port 3010, covering the two-hop motivating query and – most importantly – reference back-fill when the referent is indexed after the referrer. That ordering is the assumption most likely to regress on a Typesense upgrade, and it is not ours to fix.