From 67b943039da12b156bb3e6136da3578ae683758f Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 31 Jul 2026 17:23:32 +0000 Subject: [PATCH 1/5] =?UTF-8?q?Batch=20the=20outbox=20drain=20=E2=80=94=20?= =?UTF-8?q?one=20request=20per=20table=20run,=20not=20per=20row=20(#54)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `SupabaseRemote.upsert` sent exactly one row and `drainOutbox` looped entry by entry, so a 500-row backlog was 500 sequential round trips: ~50s of pure latency at 100ms RTT, with the engine reporting `.syncing` the whole time. That is the common case — first sync after an import, a device back from a long offline stretch, `retryDeadLetters()` with no argument, any bulk operation. Consecutive entries sharing a table and an op now go out as one request: an array body for the upsert, `in.(…)` for the tombstone update, capped at 256 rows. Chunking is adjacency-only — entries are never reordered to make a bigger batch — so the FK-safe order `orderForUpload` produced survives intact. Per-entry failure isolation is the part worth keeping, and it is kept: a rejected batch names the request, not the row, so the chunk is re-run one entry at a time and exactly the offending entries are classified, retried and parked as before. The failed batch itself charges no attempts, or one poison row would spend the retry budget of every healthy row beside it. `SyncRemote` grows batched forms with default implementations that loop over the single-row calls, so an existing conformance keeps working and only misses the round-trip saving. Batch representations are matched back to their uploads by primary key rather than arrival order, so a reordered response can't write one row's server state onto another; the write-back for a whole batch now lands in one transaction. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015y9Y9kec3hj8S35oLQXxvQ --- .claude/skills/happysync/references/api.md | 15 +- Sources/HappySync/Outbox.swift | 23 ++ Sources/HappySync/Remote.swift | 61 ++++- Sources/HappySync/SyncEngine.swift | 197 +++++++++++--- .../InMemorySyncRemote.swift | 49 +++- Tests/HappySyncTests/BatchUploadTests.swift | 244 ++++++++++++++++++ docs/SYNC-CONTRACT.md | 8 + wiki/API-Reference.md | 8 +- wiki/Architecture.md | 5 + wiki/Testing.md | 7 +- 10 files changed, 564 insertions(+), 53 deletions(-) create mode 100644 Tests/HappySyncTests/BatchUploadTests.swift diff --git a/.claude/skills/happysync/references/api.md b/.claude/skills/happysync/references/api.md index 96e46c0..3d7d2e1 100644 --- a/.claude/skills/happysync/references/api.md +++ b/.claude/skills/happysync/references/api.md @@ -228,14 +228,16 @@ Everything but the network is production code — same drain, pull, retry classi ordering — so a test here asserts shipped behaviour. - `InMemorySyncRemote` (actor): `upsertCalls` / `deleteCalls` / `fetchCalls` / `lastScope` record - what the client sent; `seed(_:rows:)` publishes a change "another device" made; `representation:` - supplies the server's response (server-normalized values, a `conflictColumns` merge); - `onUpsert`/`onFetch` hooks suspend the *n*-th call so a test can race a local write against an - in-flight upload. + what the client sent; `upsertRequests` / `deleteRequests` record how many round trips it took, + which batching makes fewer than the per-row call counts (#54); `seed(_:rows:)` publishes a change + "another device" made; `representation:` supplies the server's response (server-normalized values, + a `conflictColumns` merge); `onUpsert`/`onFetch` hooks suspend the *n*-th call so a test can race a + local write against an in-flight upload. - Failure injection: `failUpserts` / `failFetches` counts with a `SimulatedFailure` — `.transient` (retried), `.permanent` (parks on the first attempt), or `.error(anError)` which applies the **production** classification to a real `HTTPError`/`PostgrestError`. Use `99` for - "always". + "always". `failUpserts` counts **requests**: a rejected batch spends one, and so does each row of + the per-row re-run that follows it. - `ManualDoorbell`: `fire()` rings the current subscription; `ringScopes` and `liveSubscriptions` prove an auth change re-subscribed and tore the old one down; `fire(subscription: 0)` rings a torn-down one. @@ -243,7 +245,8 @@ ordering — so a test here asserts shipped behaviour. never a fixed `Task.sleep` plus one assertion. - A bespoke double is three methods on `SyncRemote`. Signal permanence by conforming the thrown error to `ClassifiedSyncError`, or wrap a real one with `RemoteFailure(classifying:)`. Anything - unclassified is treated as transient. + unclassified is treated as transient. The batched `upsert(table:rows:)` / `delete(table:pks:)` the + drain prefers default to looping over the single-row calls, so implementing them is optional. The three tests worth writing first: a cascade delete queues every child's tombstone; status reaches `.degraded` when an upload parks; `stop()` has quiesced the engine before sign-out wipes the DB. diff --git a/Sources/HappySync/Outbox.swift b/Sources/HappySync/Outbox.swift index dbb5233..91b8c8c 100644 --- a/Sources/HappySync/Outbox.swift +++ b/Sources/HappySync/Outbox.swift @@ -90,6 +90,29 @@ func orderForUpload(_ entries: [OutboxEntry], tables: [SyncTable]) -> [OutboxEnt // `collapseOutbox`, which nets each key to one op before this FK ordering runs (APPS-472). } +/// Groups an already-ordered run of pending uploads into the chunks the drain sends as one request +/// each: **consecutive** entries sharing a table and an op, capped at `limit` (issue #54). +/// +/// Adjacency is the whole rule — entries are never reordered to make a bigger batch, so the FK-safe +/// order `orderForUpload` produced survives intact and a chunk is always one PostgREST call's worth: +/// one table, one op, one conflict target. The cap bounds both the request size and the blast radius +/// of the per-row fallback a rejected chunk falls back to. +func uploadChunks(_ entries: [OutboxEntry], limit: Int) -> [[OutboxEntry]] { + let limit = max(1, limit) + var chunks: [[OutboxEntry]] = [] + for entry in entries { + let extendsLast = chunks.last.map { chunk in + chunk.count < limit && chunk[0].tableName == entry.tableName && chunk[0].op == entry.op + } ?? false + if extendsLast { + chunks[chunks.count - 1].append(entry) + } else { + chunks.append([entry]) + } + } + return chunks +} + /// Exponential backoff between drain passes for an entry that has failed `attempts` times, with /// **±20% jitter** so a user's devices that failed together — a shared outage, one expired session — /// don't retry in lockstep and thundering-herd the server the moment it recovers (APPS-514). Capped diff --git a/Sources/HappySync/Remote.swift b/Sources/HappySync/Remote.swift index 58adccc..5338c07 100644 --- a/Sources/HappySync/Remote.swift +++ b/Sources/HappySync/Remote.swift @@ -123,8 +123,20 @@ public protocol SyncRemote: Sendable { /// is the comma-joined columns of a secondary unique constraint to use as the PostgREST conflict /// target, or nil to conflict on the primary key (the default). See APPS-478. func upsert(table: String, row: [String: AnyJSON], onConflict: String?) async throws -> [String: AnyJSON] + /// Upserts a **batch** of rows for one table in a single request, returning one representation + /// per row. Every row carries the same columns and the same `onConflict` target, so this is one + /// PostgREST call rather than `rows.count` of them (issue #54). + /// + /// The default implementation forwards to the single-row `upsert` in a loop, so an existing + /// conformance keeps working unchanged — override it to get the batch. Either way the drain + /// treats a thrown error as rejecting the **whole** batch and re-runs the rows one at a time, so + /// a single poison row parks alone instead of taking its neighbours with it. + func upsert(table: String, rows: [[String: AnyJSON]], onConflict: String?) async throws -> [[String: AnyJSON]] /// Propagates a delete for one row, keyed by primary key (soft delete — sets the tombstone). func delete(table: String, primaryKey: String, pk: String) async throws + /// Propagates a **batch** of deletes for one table in a single request. Defaults to looping over + /// the single-row `delete`; see the batch `upsert` for the failure contract. + func delete(table: String, primaryKey: String, pks: [String]) async throws /// Fetches up to `limit` rows changed since `cursor`, ordered by the `(cursorColumn, id)` tuple /// so the caller can resume exactly where it left off. Tombstoned rows (`deletedAt != null`) /// are included. When `scope` is set, only rows matching `scope.column = scope.value` are @@ -132,6 +144,26 @@ public protocol SyncRemote: Sendable { func fetch(table: String, cursorColumn: String, since cursor: SyncCursor?, primaryKey: String, scope: ScopeFilter?, limit: Int) async throws -> [[String: AnyJSON]] } +/// Batch defaults, so `SyncRemote` stays the same three methods to implement. A conformance that +/// can genuinely send one request per batch (`SupabaseRemote` does) overrides them; one that can't +/// loses nothing but the round trips. +public extension SyncRemote { + func upsert(table: String, rows: [[String: AnyJSON]], onConflict: String?) async throws -> [[String: AnyJSON]] { + var representations: [[String: AnyJSON]] = [] + representations.reserveCapacity(rows.count) + for row in rows { + representations.append(try await upsert(table: table, row: row, onConflict: onConflict)) + } + return representations + } + + func delete(table: String, primaryKey: String, pks: [String]) async throws { + for pk in pks { + try await delete(table: table, primaryKey: primaryKey, pk: pk) + } + } +} + /// `SyncRemote` over a Supabase PostgREST client. Idempotent by primary key, so the drain can /// safely retry. Auth token is fetched fresh per request. struct SupabaseRemote: SyncRemote { @@ -139,33 +171,54 @@ struct SupabaseRemote: SyncRemote { let auth: @Sendable () async -> String func upsert(table: String, row: [String: AnyJSON], onConflict: String?) async throws -> [String: AnyJSON] { + try await send(table: table, rows: [row], onConflict: onConflict).first ?? row + } + + /// One PostgREST request for the whole batch: an array body upserts against the same conflict + /// target, and the representation comes back one row per uploaded row, in request order (issue + /// #54). PostgREST requires every object in the body to carry the same keys — which holds here, + /// since a table's payloads are built from the same column list — and rejects the request + /// outright otherwise, which the drain reads as "re-run this chunk one row at a time". + func upsert(table: String, rows: [[String: AnyJSON]], onConflict: String?) async throws -> [[String: AnyJSON]] { + try await send(table: table, rows: rows, onConflict: onConflict) + } + + private func send(table: String, rows: [[String: AnyJSON]], onConflict: String?) async throws -> [[String: AnyJSON]] { + guard !rows.isEmpty else { return [] } // never spend a request (or send `[]`) on nothing let token = await auth() do { // onConflict nil → PostgREST conflicts on the primary key (its default); non-nil → the // named secondary unique constraint is the conflict target, merging onto the existing row. - let rows: [[String: AnyJSON]] = try await client + return try await client .from(table) - .upsert(row, onConflict: onConflict, returning: .representation) + .upsert(rows, onConflict: onConflict, returning: .representation) .setHeader(name: "Authorization", value: "Bearer \(token)") .execute() .value - return rows.first ?? row } catch { throw Self.classify(error) // tag permanence so the drain can dead-letter poison writes } } func delete(table: String, primaryKey: String, pk: String) async throws { + try await delete(table: table, primaryKey: primaryKey, pks: [pk]) + } + + func delete(table: String, primaryKey: String, pks: [String]) async throws { + guard !pks.isEmpty else { return } // an empty `in.()` is a filter no row matches — skip the trip let token = await auth() // Soft delete: set the tombstone so the deletion propagates on the next cursor pull (a hard // DELETE would simply vanish, never reaching other devices). The server's BEFORE UPDATE // trigger stamps updatedAt — advancing the cursor — and an AFTER trigger tombstones the row's // children. deletedAt is the marker; the server-stamped updatedAt is what ordering trusts. + // + // One request for the whole batch: `in.(pk, …)` tombstones every named row in a single + // UPDATE, the delete-side counterpart of the batched upsert (issue #54). do { try await client .from(table) .update(["deletedAt": AnyJSON.string(Self.nowISO8601())]) - .eq(primaryKey, value: pk) + .in(primaryKey, values: pks) .setHeader(name: "Authorization", value: "Bearer \(token)") .execute() } catch { diff --git a/Sources/HappySync/SyncEngine.swift b/Sources/HappySync/SyncEngine.swift index f449af6..6963b99 100644 --- a/Sources/HappySync/SyncEngine.swift +++ b/Sources/HappySync/SyncEngine.swift @@ -87,6 +87,9 @@ public actor SyncEngine { private let tables: [SyncTable] private let remote: any SyncRemote private let pageSize: Int + /// Most entries the drain sends in one upload request. Bounds the request body and, since a + /// rejected chunk is re-run row by row, the blast radius of that fallback (issue #54). + private let uploadChunkSize: Int private let doorbell: any SyncDoorbell private let pollInterval: TimeInterval private let debounceInterval: TimeInterval @@ -233,6 +236,8 @@ public actor SyncEngine { /// - debounceInterval: Seconds a doorbell burst coalesces over before one pull runs. /// - deadLetterAfter: Failed attempts before a transient entry parks. Lower it to reach the /// dead-letter path without driving eight retries. + /// - uploadChunkSize: Most entries per upload request. Shrink it (e.g. `2`) to force chunk + /// boundaries on a small outbox. public static func forTesting( db: any DatabaseWriter, remote: any SyncRemote, @@ -243,7 +248,8 @@ public actor SyncEngine { debounceInterval: TimeInterval = 0.3, scope: @escaping @Sendable () async -> String? = { nil }, deadLetterAfter: Int = 8, - maxOfflineGap: TimeInterval = 30 * 24 * 3600 + maxOfflineGap: TimeInterval = 30 * 24 * 3600, + uploadChunkSize: Int = SyncEngine.defaultUploadChunkSize ) throws -> SyncEngine { try SyncEngine( db: db, @@ -255,10 +261,16 @@ public actor SyncEngine { debounceInterval: debounceInterval, scope: scope, deadLetterAfter: deadLetterAfter, - maxOfflineGap: maxOfflineGap + maxOfflineGap: maxOfflineGap, + uploadChunkSize: uploadChunkSize ) } + /// Rows per upload request unless a caller says otherwise. Big enough that a realistic backlog is + /// a handful of round trips, small enough to keep a request body — and a rejected chunk's per-row + /// re-run — bounded (issue #54). + public static let defaultUploadChunkSize = 256 + /// Injects the `SyncRemote`/`SyncDoorbell` seams directly. Reached publicly through /// `forTesting(db:remote:tables:…)`, which documents the parameters. init( @@ -271,7 +283,8 @@ public actor SyncEngine { debounceInterval: TimeInterval = 0.3, scope: @escaping @Sendable () async -> String? = { nil }, deadLetterAfter: Int = 8, - maxOfflineGap: TimeInterval = 30 * 24 * 3600 + maxOfflineGap: TimeInterval = 30 * 24 * 3600, + uploadChunkSize: Int = SyncEngine.defaultUploadChunkSize ) throws { // The manifest is checked against the schema — and its `dependsOn` filled in from the schema's // foreign keys — before anything else reads it, so `self.tables` only ever holds specs the @@ -284,6 +297,7 @@ public actor SyncEngine { self.remote = remote self.tables = resolved self.pageSize = pageSize + self.uploadChunkSize = uploadChunkSize self.doorbell = doorbell self.pollInterval = pollInterval self.debounceInterval = debounceInterval @@ -1071,6 +1085,13 @@ public actor SyncEngine { /// bumped and is skipped until its per-entry backoff window elapses; one that fails permanently /// (4xx) or exhausts `deadLetterAfter` retries is dead-lettered (parked). One failure never /// blocks the others. + /// + /// Uploads go out **in batches**: consecutive same-table, same-op entries are sent as one request + /// (issue #54), so a 500-row backlog is a handful of round trips rather than 500 sequential ones. + /// A rejected batch says which chunk the server refused, not which row — so the chunk is re-run + /// one entry at a time, and exactly the offending entries are classified and parked the way they + /// would have been without batching. Batch speed on the healthy path; unchanged diagnostics on + /// the unhealthy one. @discardableResult func drainOutbox(now: Date = Date()) async throws -> DrainOutcome { let pending = try await db.read { db in @@ -1079,32 +1100,121 @@ public actor SyncEngine { // Net each (table, pk) to one op first (APPS-472), then FK-order the net ops. let collapsed = collapseOutbox(pending) let groupSeqs = Dictionary(uniqueKeysWithValues: collapsed.map { ($0.net.seq, $0.seqs) }) + // Per-entry exponential backoff: drop an entry still inside its retry window so a failing + // entry isn't re-attempted on every drain pass (and doorbell ring and poll) — and never rides + // into a batch that would attempt it early. Dropping preserves the FK-safe relative order, + // so what's left still chunks into single-table, single-op requests. + let ready = orderForUpload(collapsed.map(\.net), tables: tables).filter { entry in + guard let last = entry.lastAttemptAt else { return true } + return now.timeIntervalSince(last) >= backoffDelay(attempts: entry.attempts) + } var outcome = DrainOutcome(failed: 0, deadLettered: 0) - for entry in orderForUpload(collapsed.map(\.net), tables: tables) { - if stopping { throw StopRequested() } // teardown: bail between entries — each clear is transactional (APPS-513) - // Per-entry exponential backoff: skip an entry still inside its retry window so a failing - // entry isn't re-attempted on every drain pass (and doorbell ring and poll). - if let last = entry.lastAttemptAt, now.timeIntervalSince(last) < backoffDelay(attempts: entry.attempts) { - continue - } - let seqs = groupSeqs[entry.seq] ?? [entry.seq] - do { - try await process(entry, clearing: seqs) - } catch { - // Auth-shaped failures (expired/refreshing token) recover out-of-band, so retry them - // without charging the retry budget — a token-refresh stretch mustn't dead-letter - // healthy writes and strip their dirty-row protection (APPS-502). Genuine transient - // failures still count toward `deadLetterAfter`; permanent ones park at once. - let attempts = remoteErrorIsAuthTransient(error) ? entry.attempts : entry.attempts + 1 - // Permanent (4xx / constraint / RLS) → park now; transient → park once it exhausts the cap. - let park = remoteErrorIsPermanent(error) || attempts >= deadLetterAfter - try await recordFailure(entry, groupSeqs: seqs, attempts: attempts, now: now, park: park, error: error) - if park { outcome.deadLettered += 1 } else { outcome.failed += 1 } + for chunk in uploadChunks(ready, limit: uploadChunkSize) { + if stopping { throw StopRequested() } // teardown: bail between chunks — each clear is transactional (APPS-513) + // One request for the whole chunk; on any failure fall through to one request per entry, + // which is also the only path a single-entry chunk ever takes. + if chunk.count > 1, await uploadedAsBatch(chunk, groupSeqs: groupSeqs) { continue } + for entry in chunk { + if stopping { throw StopRequested() } + let seqs = groupSeqs[entry.seq] ?? [entry.seq] + do { + try await process(entry, clearing: seqs) + } catch { + // Auth-shaped failures (expired/refreshing token) recover out-of-band, so retry them + // without charging the retry budget — a token-refresh stretch mustn't dead-letter + // healthy writes and strip their dirty-row protection (APPS-502). Genuine transient + // failures still count toward `deadLetterAfter`; permanent ones park at once. + let attempts = remoteErrorIsAuthTransient(error) ? entry.attempts : entry.attempts + 1 + // Permanent (4xx / constraint / RLS) → park now; transient → park once it exhausts the cap. + let park = remoteErrorIsPermanent(error) || attempts >= deadLetterAfter + try await recordFailure(entry, groupSeqs: seqs, attempts: attempts, now: now, park: park, error: error) + if park { outcome.deadLettered += 1 } else { outcome.failed += 1 } + } } } return outcome } + /// Applies a whole chunk — same table, same op — as **one** request, returning false when it + /// didn't land so the caller re-runs the chunk entry by entry. + /// + /// Nothing is recorded against the entries on failure, deliberately: a rejected batch names the + /// request, not the row, so charging every entry an attempt would park up to `limit` healthy + /// writes for one poison neighbour. The per-entry re-run is what decides who was actually at + /// fault; the extra round trips it costs are paid only on the unhealthy path. + private func uploadedAsBatch(_ chunk: [OutboxEntry], groupSeqs: [Int64: [Int64]]) async -> Bool { + guard let spec = tables.first(where: { $0.name == chunk[0].tableName }) else { + return false // table no longer synced — let the per-entry path drop the stale entries + } + func seqs(of entry: OutboxEntry) -> [Int64] { groupSeqs[entry.seq] ?? [entry.seq] } + do { + switch chunk[0].op { + case .upsert: + // A row deleted locally before its upload went out has nothing to send: clear those + // entries and batch what's left, exactly as the per-entry path does one at a time. + var uploads: [OutboxEntry] = [] + var payloads: [[String: AnyJSON]] = [] + var vanished: [Int64] = [] + for entry in chunk { + if let payload = try await readPayload(spec: spec, pk: entry.pk) { + uploads.append(entry) + payloads.append(payload) + } else { + vanished.append(contentsOf: seqs(of: entry)) + } + } + try await clear(vanished) + guard !uploads.isEmpty else { return true } + let onConflict = spec.conflictColumns.isEmpty ? nil : spec.conflictColumns.joined(separator: ",") + let servers = try await remote.upsert(table: spec.name, rows: payloads, onConflict: onConflict) + let matched = Self.match(servers, to: uploads, primaryKey: spec.primaryKey) + try await stampAndClear( + zip(uploads, matched).map { ConfirmedUpload(entry: $0, server: $1, seqs: seqs(of: $0)) }, + spec: spec + ) + case .delete: + try await remote.delete(table: spec.name, primaryKey: spec.primaryKey, pks: chunk.map(\.pk)) + try await clear(chunk.flatMap(seqs(of:))) + } + return true + } catch { + return false + } + } + + /// Pairs a batch upsert's returned representations with the rows that produced them, one slot per + /// upload and `nil` where no representation could be identified. + /// + /// Primary key first: it's the only match that can't put one row's server state onto another, and + /// PostgREST returns the full row so the key is there. Request order is the documented fallback, + /// taken only when the counts agree — a short or unkeyed response leaves those uploads unstamped, + /// which costs a re-download on the next pull rather than corrupting a row. + private static func match( + _ servers: [[String: AnyJSON]], to uploads: [OutboxEntry], primaryKey: String + ) -> [[String: AnyJSON]?] { + let byPK = Dictionary( + servers.compactMap { row -> (String, [String: AnyJSON])? in + guard let key = row[primaryKey].flatMap(Self.wireKey) else { return nil } + return (key, row) + }, + uniquingKeysWith: { first, _ in first } + ) + let positional = servers.count == uploads.count + return uploads.enumerated().map { index, entry in + byPK[entry.pk] ?? (positional ? servers[index] : nil) + } + } + + /// A wire primary key as the outbox stores it. `_sync_outbox.pk` has TEXT affinity, so an integer + /// key is `"7"` there and must be compared as one; anything that isn't a scalar key is no key. + private static func wireKey(_ value: AnyJSON) -> String? { + switch value { + case .string(let string): string + case .integer(let int): String(int) + default: nil + } + } + /// Records a failed upload attempt: bumps `attempts`, stamps `last_attempt_at` (for the backoff /// window) and `last_error` (telemetry) on the net entry. When parking, the **whole collapsed /// group** is dead-lettered together — else a superseded older op (e.g. an orphaned delete) @@ -1173,7 +1283,7 @@ public actor SyncEngine { } let onConflict = spec.conflictColumns.isEmpty ? nil : spec.conflictColumns.joined(separator: ",") let server = try await remote.upsert(table: spec.name, row: payload, onConflict: onConflict) - try await stampAndClear(entry, spec: spec, server: server, seqs: seqs) + try await stampAndClear([ConfirmedUpload(entry: entry, server: server, seqs: seqs)], spec: spec) case .delete: try await remote.delete(table: spec.name, primaryKey: spec.primaryKey, pk: entry.pk) try await clear(seqs) @@ -1196,9 +1306,18 @@ public actor SyncEngine { } } - /// Writes the server's representation back to the local row and clears the collapsed group's - /// entries in one transaction, so the row is marked clean (its cursor won't re-pull it) only - /// after the server confirms. + /// One confirmed upload waiting to be written back: the entry the server accepted, its + /// representation of the row (nil when a batch response couldn't be matched to this upload — see + /// `match`), and every outbox `seq` the entry's collapsed group covers. + private struct ConfirmedUpload: Sendable { + let entry: OutboxEntry + let server: [String: AnyJSON]? + let seqs: [Int64] + } + + /// Writes the server's representation back to the local rows and clears the confirmed entries in + /// **one transaction**, so a row is marked clean (its cursor won't re-pull it) only after the + /// server confirms. A batch confirms as a unit, so it write-backs as one too. /// /// The representation carries what the client can't compute locally: column **defaults** the /// server filled in, **trigger-normalized** fields, recomputed `serverOwnedColumns`, and — for a @@ -1214,20 +1333,25 @@ public actor SyncEngine { /// holds that pending edit — the same dirty-check the pull's `lwwAllows` gate uses. Clobbering it /// would lose the edit; it uploads on the next drain instead. The confirmed group's entries clear /// either way. - private func stampAndClear(_ entry: OutboxEntry, spec: SyncTable, server: [String: AnyJSON], seqs: [Int64]) async throws { + private func stampAndClear(_ confirmed: [ConfirmedUpload], spec: SyncTable) async throws { + guard !confirmed.isEmpty else { return } try await db.write { db in - // The server's own representation of the row we just uploaded — suppressed, or writing it + // The server's own representation of the rows we just uploaded — suppressed, or writing it // back would queue the very upload it confirms and the drain would never settle (issue #48). try SyncTriggers.applyingRemoteState(db) { - if try Self.writeBackAllowed(db, table: spec.name, pk: entry.pk, excluding: seqs) { - try RowCoding.upsertLocalRow( - db, table: spec.name, primaryKey: spec.primaryKey, - columns: RowCoding.localColumns(from: server), - restrictingTo: try RowCoding.tableColumns(db, table: spec.name) - ) + let knownColumns = try RowCoding.tableColumns(db, table: spec.name) + for upload in confirmed { + guard let server = upload.server else { continue } + if try Self.writeBackAllowed(db, table: spec.name, pk: upload.entry.pk, excluding: upload.seqs) { + try RowCoding.upsertLocalRow( + db, table: spec.name, primaryKey: spec.primaryKey, + columns: RowCoding.localColumns(from: server), + restrictingTo: knownColumns + ) + } } } - try Self.deleteEntries(db, seqs: seqs) + try Self.deleteEntries(db, seqs: confirmed.flatMap(\.seqs)) } } @@ -1247,6 +1371,7 @@ public actor SyncEngine { } private func clear(_ seqs: [Int64]) async throws { + guard !seqs.isEmpty else { return } try await db.write { db in try Self.deleteEntries(db, seqs: seqs) } } diff --git a/Sources/HappySyncTestSupport/InMemorySyncRemote.swift b/Sources/HappySyncTestSupport/InMemorySyncRemote.swift index 77bb4d6..2c5d70d 100644 --- a/Sources/HappySyncTestSupport/InMemorySyncRemote.swift +++ b/Sources/HappySyncTestSupport/InMemorySyncRemote.swift @@ -76,11 +76,19 @@ public actor InMemorySyncRemote: SyncRemote { /// table cursors on something other than `updatedAt`. public nonisolated let cursorColumn: String - /// Every upsert the drain has issued, in order — table, uploaded payload, and the PostgREST - /// conflict target (nil = conflict on the primary key). + /// Every **row** the drain has upserted, in order — table, uploaded payload, and the PostgREST + /// conflict target (nil = conflict on the primary key). A batched upload records one entry per + /// row it carried, so this counts writes regardless of how they were packed; `upsertRequests` + /// counts the requests. public private(set) var upsertCalls: [(table: String, row: [String: AnyJSON], onConflict: String?)] = [] - /// Every tombstone the drain has propagated, in order. + /// How many upsert **requests** were issued — the number of network round trips, which batching + /// makes smaller than `upsertCalls.count` (issue #54). A rejected batch counts here too, then the + /// drain's per-row re-run adds one request each. + public private(set) var upsertRequests = 0 + /// Every tombstone the drain has propagated, in order — one entry per row, as for `upsertCalls`. public private(set) var deleteCalls: [(table: String, pk: String)] = [] + /// How many delete **requests** were issued. See `upsertRequests`. + public private(set) var deleteRequests = 0 /// How many download pages have been requested. One pull of a single-table manifest that fits in /// one page is one call, so this is the usual way to observe "a sync pass ran". public private(set) var fetchCalls = 0 @@ -100,7 +108,9 @@ public actor InMemorySyncRemote: SyncRemote { /// - Parameters: /// - dataset: Server rows per table, as PostgREST would return them. Downloads are served from /// here, filtered by scope and cursor and truncated to the page limit. - /// - failUpserts: How many upserts throw before one succeeds. `99` stands in for "always". + /// - failUpserts: How many upload **requests** throw before one succeeds — a rejected batch + /// spends one, as does each row of the per-row re-run that follows it. `99` stands in for + /// "always". /// - upsertFailure: What those failures are — the difference between a write that retries and /// one that parks. /// - failFetches: How many downloads throw before one succeeds. @@ -147,7 +157,7 @@ public actor InMemorySyncRemote: SyncRemote { dataset[table] ?? [] } - /// Makes the next `count` upserts throw. Replaces any run still outstanding. + /// Makes the next `count` upload requests throw. Replaces any run still outstanding. public func failNextUpserts(_ count: Int, with failure: SimulatedFailure = .transient) { remainingUpsertFailures = count upsertFailure = failure @@ -180,6 +190,29 @@ public actor InMemorySyncRemote: SyncRemote { // MARK: - SyncRemote public func upsert(table: String, row: [String: AnyJSON], onConflict: String?) async throws -> [String: AnyJSON] { + upsertRequests += 1 + return try await store(table: table, row: row, onConflict: onConflict) + } + + /// The batched upload the drain prefers (issue #54). One request, so an injected failure rejects + /// the **whole** batch before any of its rows is recorded — which is how PostgREST answers a + /// rejected array body, and what makes the drain fall back to one row at a time. + public func upsert(table: String, rows: [[String: AnyJSON]], onConflict: String?) async throws -> [[String: AnyJSON]] { + upsertRequests += 1 + if remainingUpsertFailures > 0 { + remainingUpsertFailures -= 1 + throw upsertFailure.makeError() + } + var representations: [[String: AnyJSON]] = [] + for row in rows { + representations.append(try await store(table: table, row: row, onConflict: onConflict)) + } + return representations + } + + /// Records one uploaded row and answers with the server's representation of it. Shared by the + /// single-row and batched entry points, which own the request accounting. + private func store(table: String, row: [String: AnyJSON], onConflict: String?) async throws -> [String: AnyJSON] { upsertCalls.append((table, row, onConflict)) if let upsertHook { await upsertHook(upsertCalls.count) } if remainingUpsertFailures > 0 { @@ -193,9 +226,15 @@ public actor InMemorySyncRemote: SyncRemote { } public func delete(table: String, primaryKey: String, pk: String) async throws { + deleteRequests += 1 deleteCalls.append((table, pk)) } + public func delete(table: String, primaryKey: String, pks: [String]) async throws { + deleteRequests += 1 + for pk in pks { deleteCalls.append((table, pk)) } + } + public func fetch( table: String, cursorColumn: String, diff --git a/Tests/HappySyncTests/BatchUploadTests.swift b/Tests/HappySyncTests/BatchUploadTests.swift new file mode 100644 index 0000000..294ad30 --- /dev/null +++ b/Tests/HappySyncTests/BatchUploadTests.swift @@ -0,0 +1,244 @@ +import Testing +import Foundation +import GRDB +import Supabase +import HappySyncTestSupport +@testable import HappySync + +// The outbox drain sends consecutive same-table, same-op entries as one request, and falls back to +// one request per entry when the server rejects the batch (issue #54). These tests hold both halves +// down: the request count on the healthy path, and per-entry failure isolation on the unhealthy one. + +// MARK: - Chunking + +@Test func chunksSplitOnTableOpAndCap() { + let entries = [ + entry(seq: 1, table: "recipes", pk: "r1", op: .upsert), + entry(seq: 2, table: "recipes", pk: "r2", op: .upsert), + entry(seq: 3, table: "recipes", pk: "r3", op: .upsert), + entry(seq: 4, table: "recipeSteps", pk: "s1", op: .upsert), // table boundary + entry(seq: 5, table: "recipeSteps", pk: "s2", op: .delete), // op boundary + ] + + let shape = uploadChunks(entries, limit: 2).map { $0.map(\.pk) } + #expect(shape == [["r1", "r2"], ["r3"], ["s1"], ["s2"]]) // cap splits the first run at two +} + +@Test func chunkingNeverReordersToMakeABiggerBatch() { + // Interleaved tables: batching only ever merges *adjacent* entries, so the FK-safe order + // `orderForUpload` produced survives even when regrouping would give fewer requests. + let entries = [ + entry(seq: 1, table: "recipes", pk: "r1", op: .upsert), + entry(seq: 2, table: "recipeSteps", pk: "s1", op: .upsert), + entry(seq: 3, table: "recipes", pk: "r2", op: .upsert), + ] + + let shape = uploadChunks(entries, limit: 256).map { $0.map(\.pk) } + #expect(shape == [["r1"], ["s1"], ["r2"]]) +} + +// MARK: - Healthy path + +@Test func healthyBacklogUploadsInOneRequest() async throws { + let db = try recipesDB() + let remote = InMemorySyncRemote() + let engine = try SyncEngine.forTesting(db: db, remote: remote, tables: [SyncTable(name: "recipes")]) + for i in 1...5 { + try await write(db, "INSERT INTO recipes (id, title) VALUES (?, ?)", ["r\(i)", "T\(i)"]) + } + + try await engine.drainOutbox() + + #expect(await remote.upsertRequests == 1) // five rows, one round trip + #expect(await remote.upsertCalls.count == 5) // …carrying every row + let (pending, stamped) = try await db.read { db -> (Int, Int) in + (try Int.fetchOne(db, sql: "SELECT COUNT(*) FROM _sync_outbox") ?? -1, + try Int.fetchOne(db, sql: "SELECT COUNT(*) FROM recipes WHERE updatedAt = ?", arguments: [remote.serverUpdatedAt]) ?? -1) + } + #expect(pending == 0) // every entry cleared + #expect(stamped == 5) // and every row carries the server's representation (#29 preserved) +} + +@Test func backlogIsSplitAtTheChunkCap() async throws { + let db = try recipesDB() + let remote = InMemorySyncRemote() + let engine = try SyncEngine.forTesting( + db: db, remote: remote, tables: [SyncTable(name: "recipes")], uploadChunkSize: 2 + ) + for i in 1...5 { + try await write(db, "INSERT INTO recipes (id, title) VALUES (?, ?)", ["r\(i)", "T\(i)"]) + } + + try await engine.drainOutbox() + + #expect(await remote.upsertRequests == 3) // ⌈5/2⌉ — the cap bounds the request, not the pass + #expect(await remote.upsertCalls.count == 5) + #expect(try await db.read { try Int.fetchOne($0, sql: "SELECT COUNT(*) FROM _sync_outbox") } == 0) +} + +@Test func batchedUploadKeepsParentsBeforeChildren() async throws { + let db = try recipeGraphDB() + let remote = InMemorySyncRemote() + let engine = try SyncEngine.forTesting(db: db, remote: remote, tables: recipeGraphTables) + try await write(db, "INSERT INTO recipes (id, title) VALUES ('r1', 'Soup')") + try await write(db, "INSERT INTO recipeIngredients (id, recipeId) VALUES ('i1', 'r1')") + try await write(db, "INSERT INTO recipeIngredients (id, recipeId) VALUES ('i2', 'r1')") + + try await engine.drainOutbox() + + // One request per table — the children batch together, but never ahead of their parent. + #expect(await remote.upsertRequests == 2) + #expect(await remote.upsertCalls.map(\.table) == ["recipes", "recipeIngredients", "recipeIngredients"]) +} + +@Test func deletesTombstoneInOneRequest() async throws { + let db = try recipesDB() + let remote = InMemorySyncRemote() + let engine = try SyncEngine.forTesting(db: db, remote: remote, tables: [SyncTable(name: "recipes")]) + for i in 1...3 { + try await write(db, "INSERT INTO recipes (id, title) VALUES (?, ?)", ["r\(i)", "T\(i)"]) + } + try await engine.drainOutbox() // upload them first, so the deletes below are the only pending ops + try await write(db, "DELETE FROM recipes") + + try await engine.drainOutbox() + + #expect(await remote.deleteRequests == 1) // one `in.(…)` tombstone update for all three + #expect(await remote.deleteCalls.map(\.pk).sorted() == ["r1", "r2", "r3"]) + #expect(try await db.read { try Int.fetchOne($0, sql: "SELECT COUNT(*) FROM _sync_outbox") } == 0) +} + +// MARK: - Failure isolation + +@Test func poisonRowInABatchParksAloneAndItsNeighboursUpload() async throws { + let db = try recipesDB() + // The server rejects r3 — and, like PostgREST, rejects any array body containing it rather than + // singling the row out. The drain has to find the culprit itself. + let remote = PoisonRowRemote(poison: "r3") + let engine = try SyncEngine.forTesting(db: db, remote: remote, tables: [SyncTable(name: "recipes")]) + for i in 1...4 { + try await write(db, "INSERT INTO recipes (id, title) VALUES (?, ?)", ["r\(i)", "T\(i)"]) + } + + let outcome = try await engine.drainOutbox() + + #expect(outcome.deadLettered == 1) + #expect(await remote.uploaded.sorted() == ["r1", "r2", "r4"]) // the healthy neighbours still landed + #expect(await remote.requests == 5) // the rejected batch, then one per row + let parked = try await engine.deadLetters() + #expect(parked.map(\.pk) == ["r3"]) + // Classified per row, exactly as an unbatched drain would — the batch failure told us nothing. + #expect(parked.first?.failure == .constraintViolation(code: "23505")) + let pending = try await db.read { try Int.fetchOne($0, sql: "SELECT COUNT(*) FROM _sync_outbox WHERE dead_lettered = 0") } + #expect(pending == 0) // nothing but the poison row is left in the outbox +} + +@Test func batchFailureChargesNoAttemptsToHealthyEntries() async throws { + let db = try recipesDB() + // Every request fails transiently. The batch attempt must not charge the retry budget on its own + // — otherwise one chunk-sized failure would spend two attempts per entry and park healthy writes + // at half the configured cap. + let remote = InMemorySyncRemote(failUpserts: 99) + let engine = try SyncEngine.forTesting( + db: db, remote: remote, tables: [SyncTable(name: "recipes")], deadLetterAfter: 3 + ) + for i in 1...3 { + try await write(db, "INSERT INTO recipes (id, title) VALUES (?, ?)", ["r\(i)", "T\(i)"]) + } + + let outcome = try await engine.drainOutbox() + + #expect(outcome.failed == 3) + #expect(outcome.deadLettered == 0) + let attempts = try await db.read { try Int.fetchAll($0, sql: "SELECT attempts FROM _sync_outbox ORDER BY seq") } + #expect(attempts == [1, 1, 1]) // one attempt each — the failed batch itself charged nothing +} + +@Test func representationsAreMatchedByPrimaryKeyNotArrivalOrder() async throws { + let db = try recipesDB() + // A server that answers a batch with the representations reversed. Matching them positionally + // would stamp each row with its neighbour's server state; matching by primary key can't. + let remote = ReversingRemote() + let engine = try SyncEngine.forTesting(db: db, remote: remote, tables: [SyncTable(name: "recipes")]) + for i in 1...3 { + try await write(db, "INSERT INTO recipes (id, title) VALUES (?, ?)", ["r\(i)", "T\(i)"]) + } + + try await engine.drainOutbox() + + let titles = try await db.read { + try String.fetchAll($0, sql: "SELECT title FROM recipes ORDER BY id") + } + #expect(titles == ["T1 (server)", "T2 (server)", "T3 (server)"]) // each row got *its own* representation +} + +// MARK: - Fixtures + +private func entry(seq: Int64, table: String, pk: String, op: SyncOp) -> OutboxEntry { + OutboxEntry(row: ["seq": seq, "table_name": table, "pk": pk, "op": op.rawValue, "attempts": 0]) +} + +/// A remote that rejects one primary key the way Postgres rejects a constraint violation — and +/// rejects any batch containing it, since PostgREST refuses the whole array body rather than the +/// offending row. Records what actually landed, and how many requests it took. +private actor PoisonRowRemote: SyncRemote { + let poison: String + private(set) var uploaded: [String] = [] + private(set) var requests = 0 + + init(poison: String) { self.poison = poison } + + func upsert(table: String, row: [String: AnyJSON], onConflict: String?) async throws -> [String: AnyJSON] { + try send([row])[0] + } + + func upsert(table: String, rows: [[String: AnyJSON]], onConflict: String?) async throws -> [[String: AnyJSON]] { + try send(rows) + } + + private func send(_ rows: [[String: AnyJSON]]) throws -> [[String: AnyJSON]] { + requests += 1 + guard !rows.contains(where: { $0["id"]?.stringValue == poison }) else { + throw RemoteFailure(classifying: PostgrestError(code: "23505", message: "unique_violation")) + } + uploaded.append(contentsOf: rows.compactMap { $0["id"]?.stringValue }) + return rows.map { row in + var server = row + server["updatedAt"] = .string("2026-06-30T12:00:00.000Z") + return server + } + } + + func delete(table: String, primaryKey: String, pk: String) async throws {} + + func fetch( + table: String, cursorColumn: String, since cursor: SyncCursor?, + primaryKey: String, scope: ScopeFilter?, limit: Int + ) async throws -> [[String: AnyJSON]] { [] } +} + +/// A remote that answers a batch with its representations in reverse order, each tagged so a +/// mis-matched write-back is visible in the row it lands on. +private struct ReversingRemote: SyncRemote { + func upsert(table: String, row: [String: AnyJSON], onConflict: String?) async throws -> [String: AnyJSON] { + Self.representation(of: row) + } + + func upsert(table: String, rows: [[String: AnyJSON]], onConflict: String?) async throws -> [[String: AnyJSON]] { + Array(rows.map(Self.representation).reversed()) + } + + private static func representation(of row: [String: AnyJSON]) -> [String: AnyJSON] { + var server = row + server["title"] = .string("\(row["title"]?.stringValue ?? "") (server)") + server["updatedAt"] = .string("2026-06-30T12:00:00.000Z") + return server + } + + func delete(table: String, primaryKey: String, pk: String) async throws {} + + func fetch( + table: String, cursorColumn: String, since cursor: SyncCursor?, + primaryKey: String, scope: ScopeFilter?, limit: Int + ) async throws -> [[String: AnyJSON]] { [] } +} diff --git a/docs/SYNC-CONTRACT.md b/docs/SYNC-CONTRACT.md index 51f5ff5..954a218 100644 --- a/docs/SYNC-CONTRACT.md +++ b/docs/SYNC-CONTRACT.md @@ -74,6 +74,14 @@ newer `now()` and wins; a plain PostgREST upsert is sufficient. - **PostgREST upsert** with `Prefer: return=representation` for `.upsert`; soft-delete for `.delete`. Both are **idempotent by primary key**, so retries are safe; back off exponentially **per entry** (`last_attempt_at` gates the window) and count `attempts`. +- **Uploads are batched, failures are not (#54).** Consecutive entries sharing a table and an op go + out as **one** request — an array body for the upsert, `in.(…)` for the tombstone update — capped + at 256 rows, so a backlog costs ⌈N/256⌉ round trips per run rather than N. Entries are never + reordered to make a bigger batch, so the FK order below survives. A rejected batch identifies the + request, not the row, so the chunk is **re-run one entry at a time** and only the entries that + genuinely fail are classified, retried and parked: batch speed on the healthy path, per-entry + diagnostics on the unhealthy one. The failed batch itself charges no attempts — otherwise one + poison row would spend the retry budget of every healthy row beside it. - **The full representation is written back locally (APPS-506).** On a successful upsert the server row — column defaults, trigger-normalized fields, recomputed `serverOwnedColumns`, and (for a `conflictColumns` upsert) the **merged** row re-keyed to the client's pk (APPS-478) — is applied to diff --git a/wiki/API-Reference.md b/wiki/API-Reference.md index de8c54f..1e0b00c 100644 --- a/wiki/API-Reference.md +++ b/wiki/API-Reference.md @@ -184,7 +184,8 @@ static func SyncEngine.forTesting( pageSize: Int = 500, doorbell: any SyncDoorbell = SilentDoorbell(), pollInterval: TimeInterval = 30, debounceInterval: TimeInterval = 0.3, scope: @Sendable () async -> String? = { nil }, - deadLetterAfter: Int = 8, maxOfflineGap: TimeInterval = 30 * 24 * 3600 + deadLetterAfter: Int = 8, maxOfflineGap: TimeInterval = 30 * 24 * 3600, + uploadChunkSize: Int = SyncEngine.defaultUploadChunkSize // rows per upload request ) throws -> SyncEngine protocol SyncRemote: Sendable { // the upload/download surface @@ -192,6 +193,11 @@ protocol SyncRemote: Sendable { // the upload/download surface func delete(table: String, primaryKey: String, pk: String) async throws func fetch(table: String, cursorColumn: String, since: SyncCursor?, primaryKey: String, scope: ScopeFilter?, limit: Int) async throws -> [[String: AnyJSON]] + + // Batched forms — the drain prefers these; both default to looping over the single-row calls, + // so implementing them is optional and only costs round trips (issue #54). + func upsert(table: String, rows: [[String: AnyJSON]], onConflict: String?) async throws -> [[String: AnyJSON]] + func delete(table: String, primaryKey: String, pks: [String]) async throws } protocol SyncDoorbell: Sendable { // the Realtime wake signal func ring(scope: String?) -> AsyncStream diff --git a/wiki/Architecture.md b/wiki/Architecture.md index 20d2709..4fbd7d5 100644 --- a/wiki/Architecture.md +++ b/wiki/Architecture.md @@ -50,6 +50,11 @@ optimistically. A background drain processes entries in `seq` order: server's, so LWW would skip the row forever). The write-back is skipped for a row that gained a newer outbox entry mid-flight — the pending edit wins. - **FK ordering:** upsert parents before children; tombstone children before parents. +- **Batched requests, per-entry failures.** Consecutive entries for the same table and op are sent as + one request (up to 256 rows), so a 500-row backlog is a handful of round trips instead of 500. + Entries are never reordered to make a bigger batch, so the FK order above holds. When the server + rejects a batch it hasn't said *which* row it objected to, so the chunk is re-run one entry at a + time — the poison row parks alone and its neighbours upload. - **Per-entry backoff** with jitter; failures are classified permanent vs transient (see [[Operations and Troubleshooting]]). diff --git a/wiki/Testing.md b/wiki/Testing.md index 28617c7..e807b62 100644 --- a/wiki/Testing.md +++ b/wiki/Testing.md @@ -64,8 +64,10 @@ let remote = InMemorySyncRemote(dataset: [ ]) // What the client sent. -await remote.upsertCalls // [(table, row, onConflict)] in order +await remote.upsertCalls // [(table, row, onConflict)] in order — one entry per *row* await remote.deleteCalls // [(table, pk)] — the tombstones +await remote.upsertRequests // round trips, which batching makes fewer than upsertCalls.count (#54) +await remote.deleteRequests // …and the same for tombstones await remote.fetchCalls // download pages requested; the usual "a pass ran" signal await remote.lastScope // the partition filter the engine resolved @@ -83,6 +85,9 @@ InMemorySyncRemote(failUpserts: 1, upsertFailure: .permanent) // dead-lettered InMemorySyncRemote(failUpserts: 1, upsertFailure: .error(HTTPError(…))) // real error, real classification ``` +`failUpserts` counts **requests**, not rows: the drain batches a run of same-table writes into one +call, so a rejected batch spends one — and the per-row re-run it triggers spends one per row. + `.error` wraps a genuine transport error (`HTTPError`, `PostgrestError`, `URLError`) with the classification the production remote applies — so *"an RLS reject parks, an expired token doesn't"* is asserted against the shipped table rather than a hand-picked answer. `failFetches` / From 1dc7ead130f32bca71647c3e5edb6cac2fc4eea0 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 31 Jul 2026 17:33:55 +0000 Subject: [PATCH 2/5] Ship a runnable example app (#58) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit There was no way to see HappySync working without wiring a Supabase project, writing a schema with the right triggers and RLS, and building an app. Every integration started from prose, and the only proof the API works was a private consumer nobody evaluating the library can read. `Examples/` is a separate SPM package — an app that consumes HappySync should look like one — with a two-table schema (`lists` → `items`, real FK) that is enough to show every behaviour that matters: a plain GRDB write uploading with no engine call, an expression update, a cascade delete tombstoning children before their parent, reads staying plain `ValueObservation`, `isHealthy` vs a green tick on `.idle`, a doorbell ring pulling a change from "device B", a refused write parking, and the dead-letter repair that unparks it — plus sign-out → stop → wipe → sign-in in the one order that's safe. It runs against `InMemorySyncRemote`, so there is no backend to stand up: clone and run. `supabase/migrations/` carries the server half of the contract for pointing it at a real project, since that's the half a fake can't check. The demo is split so the sync wiring lives in a SwiftUI-free `DemoCore`, and `Tests/DemoCoreTests` asserts the same behaviours headlessly — CI runs them, so the example can't rot into something that merely compiles. Also makes `SyncSchema.migrator()` / `register(into:)` public. Both have been documented as the way to share a migrator since the wiki was written, and neither was actually reachable from outside the module. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015y9Y9kec3hj8S35oLQXxvQ --- .github/workflows/ci.yml | 9 + Examples/Package.swift | 47 +++++ Examples/README.md | 50 +++++ Examples/Sources/DemoApp/ContentView.swift | 167 +++++++++++++++++ Examples/Sources/DemoApp/DemoApp.swift | 20 ++ Examples/Sources/DemoCore/DemoStore.swift | 152 ++++++++++++++++ Examples/Sources/DemoCore/DemoSync.swift | 172 ++++++++++++++++++ Examples/Sources/DemoCore/Schema.swift | 81 +++++++++ .../Tests/DemoCoreTests/DemoSyncTests.swift | 121 ++++++++++++ .../migrations/20260731000000_demo_schema.sql | 110 +++++++++++ README.md | 7 + Sources/HappySync/SyncEngine.swift | 12 +- Sources/HappySync/SyncSchema.swift | 13 +- 13 files changed, 951 insertions(+), 10 deletions(-) create mode 100644 Examples/Package.swift create mode 100644 Examples/README.md create mode 100644 Examples/Sources/DemoApp/ContentView.swift create mode 100644 Examples/Sources/DemoApp/DemoApp.swift create mode 100644 Examples/Sources/DemoCore/DemoStore.swift create mode 100644 Examples/Sources/DemoCore/DemoSync.swift create mode 100644 Examples/Sources/DemoCore/Schema.swift create mode 100644 Examples/Tests/DemoCoreTests/DemoSyncTests.swift create mode 100644 Examples/supabase/migrations/20260731000000_demo_schema.sql diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b163c5a..17bf6b9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -10,3 +10,12 @@ jobs: steps: - uses: actions/checkout@v4 - run: swift test + + example: + # The example is a separate package that consumes this one by path, so it catches an API change + # the library's own tests can't see — and its tests assert the behaviours the demo puts on + # screen, so the demo can't quietly rot (issue #58). + runs-on: macos-15 + steps: + - uses: actions/checkout@v4 + - run: swift test --package-path Examples diff --git a/Examples/Package.swift b/Examples/Package.swift new file mode 100644 index 0000000..2197da8 --- /dev/null +++ b/Examples/Package.swift @@ -0,0 +1,47 @@ +// swift-tools-version: 6.0 +import PackageDescription + +// A runnable HappySync demo (issue #58). Deliberately a *separate* package rooted here rather than +// extra targets in the library's manifest: an app that consumes HappySync should look like one, with +// its own dependency on the published package — and a consumer opening this folder builds only the +// demo, not the library's test suite. +let package = Package( + name: "HappySyncExamples", + platforms: [.iOS(.v16), .macOS(.v13)], + products: [ + // The demo's model, sync wiring, and view state — everything but the SwiftUI. Split out so + // the sync behaviour the app shows can be asserted headlessly in CI (`DemoCoreTests`). + .library(name: "DemoCore", targets: ["DemoCore"]), + ], + dependencies: [ + .package(path: ".."), + .package(url: "https://github.com/groue/GRDB.swift.git", from: "7.11.0"), + .package(url: "https://github.com/supabase-community/supabase-swift.git", from: "2.0.0"), + ], + targets: [ + .target( + name: "DemoCore", + dependencies: [ + .product(name: "HappySync", package: "HappySync"), + // The demo runs against the in-memory fake, so it needs no Supabase project at all — + // clone, `swift run`, watch two devices converge (issue #53 made this possible). + .product(name: "HappySyncTestSupport", package: "HappySync"), + .product(name: "GRDB", package: "GRDB.swift"), + .product(name: "Supabase", package: "supabase-swift"), + ] + ), + .executableTarget(name: "DemoApp", dependencies: ["DemoCore"]), + // The demo doubles as an integration test: the behaviours the app shows are asserted here, + // headlessly, against the same fake — so `swift test` in CI proves the example still works + // rather than merely still compiling. + .testTarget( + name: "DemoCoreTests", + dependencies: [ + "DemoCore", + .product(name: "HappySync", package: "HappySync"), + .product(name: "HappySyncTestSupport", package: "HappySync"), + .product(name: "GRDB", package: "GRDB.swift"), + ] + ), + ] +) diff --git a/Examples/README.md b/Examples/README.md new file mode 100644 index 0000000..6ba632c --- /dev/null +++ b/Examples/README.md @@ -0,0 +1,50 @@ +# HappySync example — a two-table app you can actually run + +A SwiftUI demo of the whole engine, wired to the **in-memory fake** from `HappySyncTestSupport`. No +Supabase project, no schema to stand up, no credentials: clone and run. + +```bash +open Examples/Package.swift # then run the DemoApp scheme (My Mac, or an iOS sim) +swift test --package-path Examples # the same behaviours, asserted headlessly +swift run --package-path Examples DemoApp # or straight from the terminal, on macOS +``` + +## What it shows + +Two tables — `lists` → `items`, with a real foreign key — is enough for every behaviour that matters: + +| In the app | What it demonstrates | +|---|---| +| Type a list or item name and hit Add | A **plain GRDB write**. No engine call: the table's capture trigger queues the upload in the same transaction. | +| Tick an item | An **expression update** (`SET done = NOT done`) — the kind of write a `save`-only sync API can't see. | +| **Delete list (cascades)** | One `DELETE`. SQLite removes the items too, and each removed item's own trigger queues its tombstone — children before parent, on the wire. | +| The status dot | `status.isHealthy`, **not** `phase == .idle`. The distinction is the point: a settled pass with writes still parked is `.degraded`. | +| **Change from device B** | Realtime as a doorbell: the ring triggers a pull, and the row arrives through the cursor fetch — never from the event payload. | +| **Make the server reject writes** | An RLS-shaped rejection: the write parks on its first attempt instead of retrying forever. | +| **Parked writes → Retry / Discard** | The dead-letter repair API. Fix the cause, then retry — or discard and take the server's version. | +| **Sign out → wipe → sign in** | The one order that's safe: `await stop()`, *then* replace the store. And replacing it, not `DELETE`-ing rows — a delete would fire the capture triggers and queue a tombstone for everything the user owns. | + +Reads are plain `ValueObservation`. HappySync never sits between the app and its data: a pulled row +lands in SQLite and the observation fires, exactly as it does for a local write. + +## Layout + +``` +Sources/DemoCore/ Schema, manifest, sync wiring, view state — no SwiftUI, so CI can drive it +Sources/DemoApp/ The SwiftUI app +Tests/DemoCoreTests/ The table above, as assertions +supabase/migrations/ The server half of the contract, for when you point it at a real project +``` + +`DemoCore` is where to look first: [`Schema.swift`](Sources/DemoCore/Schema.swift) is the entire +manifest, and [`DemoSync.swift`](Sources/DemoCore/DemoSync.swift) is the entire integration. + +## Pointing it at a real Supabase project + +One initializer changes — `SyncEngine(db:supabase:tables:auth:)` instead of +`SyncEngine.forTesting(db:remote:tables:)` (see the comment at the bottom of `DemoSync.swift`). The +server side is +[`supabase/migrations/`](supabase/migrations): `supabase start && supabase db reset` applies the +`updatedAt` trigger, the tombstone trigger, RLS, and the Realtime publication that the fake can't +check for you. The [Server Setup](https://github.com/happyface-studio/HappySync/wiki/Server-Setup) +wiki page explains why each of the four is non-negotiable. diff --git a/Examples/Sources/DemoApp/ContentView.swift b/Examples/Sources/DemoApp/ContentView.swift new file mode 100644 index 0000000..80312c2 --- /dev/null +++ b/Examples/Sources/DemoApp/ContentView.swift @@ -0,0 +1,167 @@ +import SwiftUI +import DemoCore + +struct ContentView: View { + @EnvironmentObject private var store: DemoStore + + var body: some View { + VStack(spacing: 0) { + SyncStatusBar() + Divider() + HStack(alignment: .top, spacing: 0) { + ListsPane() + Divider() + ItemsPane() + } + Divider() + DemoControls() + } + .frame(minWidth: 640, minHeight: 460) + } +} + +// MARK: - Lists + +private struct ListsPane: View { + @EnvironmentObject private var store: DemoStore + @State private var title = "" + + var body: some View { + VStack(alignment: .leading, spacing: 8) { + Text("Lists").font(.headline) + List(store.lists) { list in + Button { + store.selectedListID = list.id + } label: { + Text(list.title) + .foregroundStyle(list.id == store.selectedListID ? Color.accentColor : .primary) + .frame(maxWidth: .infinity, alignment: .leading) + } + .buttonStyle(.plain) + } + HStack { + TextField("New list", text: $title) + Button("Add") { add() }.disabled(title.isEmpty) + } + // One plain DELETE. SQLite cascades to the list's items and each removed item's own + // capture trigger queues its tombstone — watch the delete count in the controls below. + Button("Delete list (cascades)") { + Task { await store.deleteSelectedList() } + } + .disabled(store.selectedListID == nil) + } + .padding() + .frame(minWidth: 240) + } + + @MainActor private func add() { + let value = title + title = "" + Task { await store.addList(title: value) } + } +} + +// MARK: - Items + +private struct ItemsPane: View { + @EnvironmentObject private var store: DemoStore + @State private var title = "" + + var body: some View { + VStack(alignment: .leading, spacing: 8) { + Text("Items").font(.headline) + List(store.items(in: store.selectedListID)) { item in + Button { + Task { await store.toggle(item) } + } label: { + Label(item.title, systemImage: item.done ? "checkmark.circle.fill" : "circle") + } + .buttonStyle(.plain) + } + HStack { + TextField("New item", text: $title) + Button("Add") { add() }.disabled(title.isEmpty || store.selectedListID == nil) + } + } + .padding() + .frame(maxWidth: .infinity) + } + + @MainActor private func add() { + let value = title + title = "" + Task { await store.addItem(title: value) } + } +} + +// MARK: - The parts of sync a UI has to get right + +private struct SyncStatusBar: View { + @EnvironmentObject private var store: DemoStore + + var body: some View { + HStack { + // Ask `isHealthy`, never `phase == .idle` alone: a settled pass with writes still failing + // or parked is `.degraded`, and a green tick there is the bug this API exists to prevent. + Circle() + .fill(store.status.isHealthy ? Color.green : Color.orange) + .frame(width: 10, height: 10) + Text(store.healthLine) + Spacer() + if let lastSynced = store.status.lastSyncedAt { + Text("last sync \(lastSynced.formatted(date: .omitted, time: .standard))") + .foregroundStyle(.secondary) + } + } + .padding(.horizontal) + .padding(.vertical, 8) + } +} + +private struct DemoControls: View { + @EnvironmentObject private var store: DemoStore + + var body: some View { + VStack(alignment: .leading, spacing: 10) { + HStack { + Button("Change from device B") { Task { await store.publishFromAnotherDevice() } } + Button("Make the server reject writes") { Task { await store.sync.rejectWrites() } } + Button("Let the server accept again") { Task { await store.sync.acceptWrites() } } + Button("Sign out → wipe → sign in") { Task { await signOutAndBackIn() } } + } + DeadLetterInspector() + } + .padding() + } + + /// The order is the point: `stop()` awaits the in-flight pass, and only then is the store safe to + /// replace. The demo re-observes afterwards because the database object itself was swapped. + @MainActor private func signOutAndBackIn() async { + try? await store.sync.signOutAndBackIn() + store.observe() + } +} + +/// Parked writes, and the two ways out of them. Repair is an API, not a support ticket: a write that +/// the server refused permanently stops retrying and waits here until the cause is fixed. +private struct DeadLetterInspector: View { + @EnvironmentObject private var store: DemoStore + + var body: some View { + VStack(alignment: .leading, spacing: 6) { + HStack { + Text("Parked writes (\(store.deadLetters.count))").font(.headline) + Spacer() + Button("Retry") { Task { await store.retryDeadLetters() } } + Button("Discard") { Task { await store.discardDeadLetters() } } + } + .disabled(store.deadLetters.isEmpty) + + ForEach(store.deadLetters, id: \.seq) { letter in + Text("\(letter.op.rawValue) \(letter.table)/\(letter.pk) — \(letter.failure.description)") + .font(.caption) + .foregroundStyle(.secondary) + } + } + } +} diff --git a/Examples/Sources/DemoApp/DemoApp.swift b/Examples/Sources/DemoApp/DemoApp.swift new file mode 100644 index 0000000..fde445e --- /dev/null +++ b/Examples/Sources/DemoApp/DemoApp.swift @@ -0,0 +1,20 @@ +import SwiftUI +import DemoCore + +/// A HappySync demo you can run with no backend at all: the engine is wired to the in-memory fake +/// from `HappySyncTestSupport`, so "another device" is a button (issue #58). +/// +/// Everything the app does — writing, deleting with a cascade, reading through `ValueObservation`, +/// showing status, repairing parked writes — is the real engine. Only the transport is fake. +@main +struct HappySyncDemoApp: App { + @StateObject private var store = DemoStore.live() + + var body: some Scene { + WindowGroup { + ContentView() + .environmentObject(store) + .task { await store.start() } + } + } +} diff --git a/Examples/Sources/DemoCore/DemoStore.swift b/Examples/Sources/DemoCore/DemoStore.swift new file mode 100644 index 0000000..80f7606 --- /dev/null +++ b/Examples/Sources/DemoCore/DemoStore.swift @@ -0,0 +1,152 @@ +import Foundation +import Combine +import GRDB +import HappySync + +/// View state for the demo: observed reads, the live sync status, and the dead-letter inspector. +/// +/// The reads are the part worth noticing — they're **plain GRDB `ValueObservation`**. HappySync +/// never sits between the app and its data: a pulled row lands in SQLite and the observation fires, +/// exactly as it does for a local write. +@MainActor +public final class DemoStore: ObservableObject { + @Published public private(set) var lists: [TodoList] = [] + @Published public private(set) var items: [TodoItem] = [] + /// The engine's live status. Drive UI off `isHealthy`, not `phase == .idle` — see `healthLine`. + @Published public private(set) var status = SyncStatus() + /// Writes the server refused, parked so they stop retrying. Refreshed whenever the status says + /// the count changed. + @Published public private(set) var deadLetters: [DeadLetter] = [] + @Published public var selectedListID: String? + + public let sync: DemoSync + + private var readsTask: Task? + private var statusTask: Task? + + public init(sync: DemoSync) { + self.sync = sync + } + + /// Builds the whole stack. Trapping on failure is a demo's privilege: every error here is a + /// programming mistake (a manifest that disagrees with the schema, a migration that didn't run), + /// which is precisely what `SyncEngine.init` is designed to surface loudly at launch. + public static func live() -> DemoStore { + do { + return DemoStore(sync: try DemoSync()) + } catch { + fatalError("HappySync demo failed to start: \(error)") + } + } + + /// Starts sync and begins observing. Safe to call again after `signOutAndBackIn` replaced the + /// store — it re-points the observations at the new database. + public func start() async { + await sync.start() + observe() + } + + public func observe() { + readsTask?.cancel() + statusTask?.cancel() + + let database = sync.db + readsTask = Task { [weak self] in + let observation = ValueObservation.tracking { db -> ([TodoList], [TodoItem]) in + (try TodoList.order(Column("title")).fetchAll(db), + try TodoItem.order(Column("title")).fetchAll(db)) + } + do { + for try await (lists, items) in observation.values(in: database) { + guard let self else { return } + self.lists = lists + self.items = items + if self.selectedListID == nil || !lists.contains(where: { $0.id == self.selectedListID }) { + self.selectedListID = lists.first?.id + } + } + } catch { + // The store was replaced out from under the observation (sign-out). `observe()` runs + // again against the new one. + } + } + + let engine = sync.engine + statusTask = Task { [weak self] in + for await status in engine.status { + guard let self else { return } + let parkedCountChanged = status.deadLetters != self.status.deadLetters + self.status = status + if parkedCountChanged { await self.refreshDeadLetters() } + } + } + } + + public func items(in listID: String?) -> [TodoItem] { + guard let listID else { return [] } + return items.filter { $0.listId == listID } + } + + // MARK: - Actions + + public func addList(title: String) async { + try? await sync.addList(title: title) + } + + public func addItem(title: String) async { + guard let listID = selectedListID else { return } + try? await sync.addItem(title: title, to: listID) + } + + public func toggle(_ item: TodoItem) async { + try? await sync.toggle(itemID: item.id) + } + + public func deleteSelectedList() async { + guard let listID = selectedListID else { return } + try? await sync.deleteList(id: listID) + } + + public func publishFromAnotherDevice() async { + await sync.publishFromAnotherDevice(title: "From device B") + } + + // MARK: - Dead-letter repair + + public func refreshDeadLetters() async { + deadLetters = (try? await sync.engine.deadLetters()) ?? [] + } + + /// Re-queue parked writes. Do this *after* fixing the cause — an RLS policy, a migration, an app + /// update — or they park again on the next pass. + public func retryDeadLetters() async { + try? await sync.engine.retryDeadLetters() + await refreshDeadLetters() + } + + /// Abandon the local writes and take the server's version instead. Discard also re-pulls the + /// affected rows, so a row that diverged while parked converges rather than staying stuck. + public func discardDeadLetters() async { + try? await sync.engine.discardDeadLetters() + await refreshDeadLetters() + } + + // MARK: - Status, read the way a UI should read it + + /// `.idle` alone is the tempting check and the wrong one: it means "no pass is running", which is + /// also true while the user's writes sit in the outbox failing. Ask `isHealthy` first, then + /// branch — `.degraded` is a settled pass with writes outstanding, and `.failed` carries a + /// classified cause rather than prose to substring-match. + public var healthLine: String { + if status.isHealthy { return "Synced" } + switch status.phase { + case .syncing: return "Syncing…" + case .idle: return "Synced" + case .degraded: + return "\(status.failedUploads) retrying, \(status.deadLetters) parked" + case .failed(.authExpired): return "Session expired — sign in again" + case .failed(.network): return "Offline — will retry" + case .failed(let failure): return "Sync failed: \(failure)" + } + } +} diff --git a/Examples/Sources/DemoCore/DemoSync.swift b/Examples/Sources/DemoCore/DemoSync.swift new file mode 100644 index 0000000..d3a3531 --- /dev/null +++ b/Examples/Sources/DemoCore/DemoSync.swift @@ -0,0 +1,172 @@ +import Foundation +import GRDB +import Supabase +import HappySync +import HappySyncTestSupport + +/// The demo's sync stack: a local GRDB store, a HappySync engine, and — instead of a network — the +/// in-memory fake from `HappySyncTestSupport`. That's the point of the example: **no Supabase project +/// is needed to watch sync work**, and the same object drives both the SwiftUI app and the +/// integration tests in CI (issue #58). +/// +/// Everything except the transport is the shipping engine: the same capture triggers, the same +/// FK-ordered drain, the same cursor pull and LWW, the same dead-letter classification. Swapping in +/// the real backend is one initializer — see `productionEngine` at the bottom of this file. +@MainActor +public final class DemoSync { + /// The server, in-process. Seedable (a change "another device" made) and failable (a write the + /// server refuses), which is how the demo reaches the interesting states on command. + public let remote: InMemorySyncRemote + /// Stands in for the Supabase Realtime channel: `fire()` is a change event arriving. + public let doorbell: ManualDoorbell + + /// The local source of truth. Replaced wholesale on sign-out — see `signOutAndBackIn`. + public private(set) var db: DatabaseQueue + public private(set) var engine: SyncEngine + + /// Every row the fake server holds for `lists`. The fake serves downloads from what it was last + /// seeded with, so publishing a second remote change means re-seeding the accumulated set. + private var serverLists: [[String: AnyJSON]] = [] + + public init() throws { + remote = InMemorySyncRemote() + doorbell = ManualDoorbell() + db = try Self.makeDatabase() + engine = try Self.makeEngine(db: db, remote: remote, doorbell: doorbell) + } + + private static func makeDatabase() throws -> DatabaseQueue { + // In-memory, so every launch of the demo starts from nothing. A real app points GRDB at a + // file; nothing else about the wiring changes. + let db = try DatabaseQueue() + try DemoSchema.migrator().migrate(db) + return db + } + + private static func makeEngine( + db: DatabaseQueue, remote: InMemorySyncRemote, doorbell: ManualDoorbell + ) throws -> SyncEngine { + // The manifest is validated against the schema here, which is why the migrator ran first. + try SyncEngine.forTesting( + db: db, + remote: remote, + tables: DemoSchema.tables, + doorbell: doorbell, + pollInterval: 5, // short, so the demo converges while you watch it + debounceInterval: 0.2, + deadLetterAfter: 2 // reach the parked state in two failures rather than eight + ) + } + + public func start() async { + await engine.start() + } + + // MARK: - Writes (there is no write API — that's the point) + + @discardableResult + public func addList(title: String) async throws -> TodoList { + let list = TodoList(title: title) + try await db.write { try list.insert($0) } + // No engine call. The `lists` capture trigger queued the upload inside that transaction, and + // the engine's outbox observer wakes the drain. + return list + } + + @discardableResult + public func addItem(title: String, to listID: String) async throws -> TodoItem { + let item = TodoItem(listId: listID, title: title) + try await db.write { try item.insert($0) } + return item + } + + /// An expression update — the kind of write a `save`-only sync API can't see, and the capture + /// triggers do. + public func toggle(itemID: String) async throws { + try await db.write { db in + try db.execute( + sql: "UPDATE items SET done = NOT done WHERE id = ?", + arguments: [itemID] + ) + } + } + + /// One plain `DELETE`. SQLite cascades to the list's items, and each removed item's own trigger + /// queues its tombstone — the app never enumerates children. + public func deleteList(id: String) async throws { + try await db.write { db in + try db.execute(sql: "DELETE FROM lists WHERE id = ?", arguments: [id]) + } + } + + // MARK: - The other device + + /// Publishes a list "another device" created and rings the doorbell, exactly as Supabase Realtime + /// would. The engine debounces the ring into one `pullNow()`; the row arrives through the cursor + /// pull, never from the event payload. + public func publishFromAnotherDevice(title: String) async { + serverLists.append([ + "id": .string(UUID().uuidString), + "title": .string(title), + "updatedAt": .string(Self.serverTimestamp()), + "deletedAt": .null, + ]) + await remote.seed("lists", rows: serverLists) + doorbell.fire() + } + + // MARK: - The server saying no + + /// Makes the server reject every write from now on, the way an RLS policy does: permanently, so + /// the drain parks the entry instead of retrying it forever. + public func rejectWrites() async { + await remote.failNextUpserts(99, with: .permanent) + } + + /// Stops rejecting. Parked writes don't retry on their own — that's what `retryDeadLetters()` is + /// for, and it's the repair flow this demo exists to show. + public func acceptWrites() async { + await remote.failNextUpserts(0) + } + + // MARK: - Teardown, in the order that matters + + /// Sign-out → **stop** → wipe → sign-in. `stop()` awaits the in-flight pass, so after it returns + /// nothing is still writing to the store being thrown away. Doing this in any other order is how + /// an app uploads the previous user's rows with the new user's token. + /// + /// The wipe replaces the whole database rather than `DELETE`-ing rows: a delete would fire the + /// capture triggers and queue a tombstone for every row the user owns, which the next sign-in + /// would dutifully upload. + public func signOutAndBackIn() async throws { + await engine.stop() + db = try Self.makeDatabase() + engine = try Self.makeEngine(db: db, remote: remote, doorbell: doorbell) + await engine.start() + } + + /// ISO-8601 with fractional seconds — what a Postgres `timestamptz` serializes to, and what the + /// tuple cursor orders by. + private static func serverTimestamp() -> String { + let formatter = ISO8601DateFormatter() + formatter.formatOptions = [.withInternetDateTime, .withFractionalSeconds] + formatter.timeZone = TimeZone(identifier: "UTC") + return formatter.string(from: Date()) + } +} + +// MARK: - What this would be against a real backend + +// The demo's only concession to running offline is *which* initializer builds the engine. Against a +// real project it is this instead, with `Examples/supabase/migrations/` standing the schema up: +// +// let engine = try SyncEngine( +// db: databaseQueue, +// supabase: SupabaseClient(supabaseURL: url, supabaseKey: anonKey), +// tables: DemoSchema.tables, +// auth: { await session.accessToken }, +// scope: { await session.user?.id.uuidString } // only if RLS is broader than the partition +// ) +// +// Everything above this comment — the writes, the cascade, the status, the dead-letter repair — +// is unchanged. diff --git a/Examples/Sources/DemoCore/Schema.swift b/Examples/Sources/DemoCore/Schema.swift new file mode 100644 index 0000000..528b6a3 --- /dev/null +++ b/Examples/Sources/DemoCore/Schema.swift @@ -0,0 +1,81 @@ +import Foundation +import GRDB +import HappySync + +// The smallest schema that still shows every interesting HappySync behaviour: a parent, a child, and +// a real foreign key between them, so a parent delete cascades locally *and* queues a tombstone per +// removed child — the behaviour an app most often gets wrong by hand-deleting children. + +/// A list of things to do. The parent row. +public struct TodoList: Codable, Identifiable, Hashable, FetchableRecord, PersistableRecord { + public static let databaseTableName = "lists" + + public var id: String + public var title: String + + public init(id: String = UUID().uuidString, title: String) { + self.id = id + self.title = title + } +} + +/// One entry in a list. `listId` carries `ON DELETE CASCADE`, which is the whole trick: deleting a +/// list removes its items in the same transaction and each item's own capture trigger queues its +/// tombstone. +public struct TodoItem: Codable, Identifiable, Hashable, FetchableRecord, PersistableRecord { + public static let databaseTableName = "items" + + public var id: String + public var listId: String + public var title: String + public var done: Bool + + public init(id: String = UUID().uuidString, listId: String, title: String, done: Bool = false) { + self.id = id + self.listId = listId + self.title = title + self.done = done + } +} + +// Neither record declares `updatedAt` or `deletedAt`. They exist as columns — the sync contract +// requires them — but they belong to the *server*: it stamps `updatedAt`, the drain writes it back, +// and a domain type that carried them would write nulls over both on every `save`. + +public enum DemoSchema { + /// The demo's migrator, with HappySync's internal tables registered into it — the shape an app + /// that owns its own `DatabaseMigrator` should use, so both share one `grdb_migrations` table. + /// + /// The write-capture triggers are deliberately *not* here: they're derived from the manifest + /// below, which is source code, and the engine (re)installs them idempotently at every init. + public static func migrator() -> DatabaseMigrator { + var migrator = DatabaseMigrator() + SyncSchema.register(into: &migrator) + migrator.registerMigration("demo_v1") { db in + try db.create(table: TodoList.databaseTableName) { t in + t.column("id", .text).primaryKey() + t.column("title", .text).notNull() + t.column("updatedAt", .text) // server-stamped; never written by this app + t.column("deletedAt", .text) // tombstone marker, likewise + } + try db.create(table: TodoItem.databaseTableName) { t in + t.column("id", .text).primaryKey() + t.column("listId", .text) + .notNull() + .references(TodoList.databaseTableName, onDelete: .cascade) + t.column("title", .text).notNull() + t.column("done", .boolean).notNull().defaults(to: false) + t.column("updatedAt", .text) + t.column("deletedAt", .text) + } + } + return migrator + } + + /// The whole manifest. Declared in any order — the engine sorts it — and with no `dependsOn`: + /// the engine reads `items`' foreign key at init and derives that lists upload first (issue #49). + public static let tables: [SyncTable] = [ + SyncTable(name: TodoItem.databaseTableName), + SyncTable(name: TodoList.databaseTableName), + ] +} diff --git a/Examples/Tests/DemoCoreTests/DemoSyncTests.swift b/Examples/Tests/DemoCoreTests/DemoSyncTests.swift new file mode 100644 index 0000000..a4f0fcf --- /dev/null +++ b/Examples/Tests/DemoCoreTests/DemoSyncTests.swift @@ -0,0 +1,121 @@ +import Testing +import Foundation +import GRDB +import HappySync +import HappySyncTestSupport +import DemoCore + +// The example, asserted. Every one of these is a behaviour the SwiftUI app puts on screen — writes +// uploading with no engine call, a parent delete tombstoning its children, a change from "device B" +// arriving through the doorbell, a refused write parking and being repaired, and sign-out quiescing +// the engine before the store is replaced. Running in CI is what stops the demo from rotting (#58). + +@MainActor +@Test func aPlainWriteUploadsWithNoEngineCall() async throws { + let sync = try DemoSync() + let remote = sync.remote + await sync.start() + defer { Task { await sync.engine.stop() } } + + try await sync.addList(title: "Groceries") + + #expect(await eventually { await remote.upsertCalls.contains { $0.table == "lists" } }) +} + +@MainActor +@Test func deletingAListTombstonesItsItemsChildrenFirst() async throws { + let sync = try DemoSync() + let remote = sync.remote + await sync.start() + defer { Task { await sync.engine.stop() } } + + let list = try await sync.addList(title: "Trip") + try await sync.addItem(title: "Passport", to: list.id) + try await sync.addItem(title: "Charger", to: list.id) + #expect(await eventually { await remote.upsertCalls.count == 3 }) + + // One DELETE of the parent. SQLite cascades; each removed child's own capture trigger queues its + // tombstone, so the app never enumerates children. + try await sync.deleteList(id: list.id) + + #expect(await eventually { await remote.deleteCalls.count == 3 }) + let deletes = await remote.deleteCalls + #expect(deletes.filter { $0.table == "items" }.count == 2) + #expect(deletes.last?.table == "lists") // children before parent — the FK-safe delete order +} + +@MainActor +@Test func aChangeFromAnotherDeviceArrivesThroughTheDoorbell() async throws { + let sync = try DemoSync() + let database = sync.db + await sync.start() + defer { Task { await sync.engine.stop() } } + + // Realtime is a doorbell only: the ring triggers a pull, and the row arrives through the cursor + // fetch — never from the event payload. + await sync.publishFromAnotherDevice(title: "From device B") + + #expect(await eventually { + await count(in: database, "SELECT COUNT(*) FROM lists WHERE title = 'From device B'") == 1 + }) +} + +@MainActor +@Test func aRefusedWriteParksAndRetryRepairsIt() async throws { + let sync = try DemoSync() + let engine = sync.engine + await sync.start() + defer { Task { await engine.stop() } } + + // The server refuses the write the way an RLS policy does — permanently, so the drain parks the + // entry on its first attempt instead of retrying it forever. + await sync.rejectWrites() + try await sync.addList(title: "Doomed") + + #expect(await eventually { await parked(engine).count == 1 }) + let letter = try #require(await parked(engine).first) + #expect(letter.table == "lists") + #expect(letter.op == .upsert) + + // Repair: fix the cause first, then re-queue. Retrying a still-broken write just parks it again. + await sync.acceptWrites() + try await engine.retryDeadLetters() + + #expect(await eventually { await parked(engine).isEmpty }) + #expect(await eventually { await outboxCount(sync) == 0 }) // the write finally landed +} + +@MainActor +@Test func signOutQuiescesTheEngineBeforeTheStoreIsReplaced() async throws { + let sync = try DemoSync() + let remote = sync.remote + await sync.start() + try await sync.addList(title: "Before sign-out") + #expect(await eventually { await remote.upsertCalls.contains { $0.table == "lists" } }) + + // stop() → replace the store → start(). Wiping before the engine has quiesced is how an app + // writes into a database it is about to delete; deleting rows instead of replacing the store is + // how it queues a tombstone for everything the user owns. + try await sync.signOutAndBackIn() + defer { Task { await sync.engine.stop() } } + + #expect(await count(in: sync.db, "SELECT COUNT(*) FROM lists") == 0) + #expect(await count(in: sync.db, "SELECT COUNT(*) FROM _sync_outbox") == 0) +} + +// MARK: - Helpers + +/// A scalar count, or `-1` if the read failed — shaped for use inside `eventually`, which polls a +/// non-throwing condition. +private func count(in db: DatabaseQueue, _ sql: String) async -> Int { + (try? await db.read { db in try Int.fetchOne(db, sql: sql) ?? 0 }) ?? -1 +} + +private func parked(_ engine: SyncEngine) async -> [DeadLetter] { + (try? await engine.deadLetters()) ?? [] +} + +@MainActor +private func outboxCount(_ sync: DemoSync) async -> Int { + await count(in: sync.db, "SELECT COUNT(*) FROM _sync_outbox") +} diff --git a/Examples/supabase/migrations/20260731000000_demo_schema.sql b/Examples/supabase/migrations/20260731000000_demo_schema.sql new file mode 100644 index 0000000..3b59660 --- /dev/null +++ b/Examples/supabase/migrations/20260731000000_demo_schema.sql @@ -0,0 +1,110 @@ +-- The demo's server side, in one file: `supabase start && supabase db reset` stands it up. +-- +-- The app in Examples/ runs against an in-process fake and needs none of this. It's here because the +-- fake cannot check the half of the contract that lives on the server — the `updatedAt` trigger LWW +-- depends on, the tombstone propagation deletes depend on, RLS, and the Realtime publication. This +-- is what those four requirements look like for a two-table schema (see the Server Setup wiki page). + +-- --------------------------------------------------------------------------- +-- Tables. Column names are camelCase and identical to the local SQLite schema — +-- HappySync does no name mapping, by contract. +-- --------------------------------------------------------------------------- + +create table if not exists public.lists ( + id uuid primary key, + "userId" uuid not null default auth.uid() references auth.users (id) on delete cascade, + title text not null, + "updatedAt" timestamptz not null default now(), + "deletedAt" timestamptz +); + +create table if not exists public.items ( + id uuid primary key, + "listId" uuid not null references public.lists (id) on delete cascade, + "userId" uuid not null default auth.uid() references auth.users (id) on delete cascade, + title text not null, + done boolean not null default false, + "updatedAt" timestamptz not null default now(), + "deletedAt" timestamptz +); + +create index if not exists lists_cursor_idx on public.lists ("updatedAt", id); +create index if not exists items_cursor_idx on public.items ("updatedAt", id); + +-- --------------------------------------------------------------------------- +-- 1. updatedAt — server-stamped, always. This is what makes last-write-wins +-- correct: two devices' clocks never enter the ordering. A client-sent value +-- is ignored (and HappySync strips it from every upload anyway). +-- --------------------------------------------------------------------------- + +create or replace function public.stamp_updated_at() +returns trigger language plpgsql as $$ +begin + new."updatedAt" := now(); + return new; +end $$; + +drop trigger if exists stamp_updated_at on public.lists; +create trigger stamp_updated_at + before insert or update on public.lists + for each row execute function public.stamp_updated_at(); + +drop trigger if exists stamp_updated_at on public.items; +create trigger stamp_updated_at + before insert or update on public.items + for each row execute function public.stamp_updated_at(); + +-- --------------------------------------------------------------------------- +-- 2. deletedAt — deletes are soft, so they can propagate on the next cursor +-- pull. A parent's children are tombstoned by trigger, NOT by a hard +-- `ON DELETE CASCADE`: a hard cascade removes the rows with no tombstone +-- left for other devices to pull, and they keep the children forever. +-- --------------------------------------------------------------------------- + +create or replace function public.tombstone_list_items() +returns trigger language plpgsql as $$ +begin + if new."deletedAt" is not null and old."deletedAt" is null then + update public.items + set "deletedAt" = new."deletedAt" + where "listId" = new.id and "deletedAt" is null; + end if; + return new; +end $$; + +drop trigger if exists tombstone_list_items on public.lists; +create trigger tombstone_list_items + after update on public.lists + for each row execute function public.tombstone_list_items(); + +-- Purge tombstones on a schedule if you like — but the retention must exceed the engine's +-- `maxOfflineGap` (30 days by default), or a device offline longer than the retention never sees the +-- tombstone and resurrects the row: +-- +-- delete from public.items where "deletedAt" < now() - interval '90 days'; +-- delete from public.lists where "deletedAt" < now() - interval '90 days'; + +-- --------------------------------------------------------------------------- +-- 3. RLS — the security boundary. Here it equals the sync partition exactly, +-- so the demo's SyncTable declares no `scopeColumn`. Declare one when a +-- policy is deliberately broader than what a device should download. +-- --------------------------------------------------------------------------- + +alter table public.lists enable row level security; +alter table public.items enable row level security; + +drop policy if exists "own lists" on public.lists; +create policy "own lists" on public.lists + for all using ("userId" = auth.uid()) with check ("userId" = auth.uid()); + +drop policy if exists "own items" on public.items; +create policy "own items" on public.items + for all using ("userId" = auth.uid()) with check ("userId" = auth.uid()); + +-- --------------------------------------------------------------------------- +-- 4. Realtime — the doorbell. An event triggers a debounced pull; the payload +-- is never applied directly, so sync converges with or without this. +-- --------------------------------------------------------------------------- + +alter publication supabase_realtime add table public.lists; +alter publication supabase_realtime add table public.items; diff --git a/README.md b/README.md index 1540c63..bbe8811 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,11 @@ CRDTs needed. > APPS-415) are all live. Next is M2 server prep (server-side `updated_at` triggers + `deleted_at` > tombstones) before the M3 CookThis cutover. +**See it run first:** [`Examples/`](Examples) is a two-table SwiftUI app wired to the in-memory fake +— no Supabase project, no schema, no credentials. `swift run --package-path Examples DemoApp` shows +writes uploading with no engine call, a cascade delete tombstoning its children, a change arriving +from "device B", a refused write parking, and the repair that unparks it. + ## Model Local GRDB SQLite is the source of truth for reads (observed with `ValueObservation`). You write @@ -242,6 +247,8 @@ deliberately generic, but is pressure-tested against one real app before it's tr ## Documentation & Claude skill +- **[Example app](Examples)** — a runnable two-table SwiftUI demo against the in-memory fake, with + the Supabase migrations for when you point it at a real project. - **[Wiki](https://github.com/happyface-studio/HappySync/wiki)** — Getting Started, Server Setup, API Reference, Operations & Troubleshooting, and Architecture. (Sources live in [`wiki/`](wiki).) - **[Sync contract](docs/SYNC-CONTRACT.md)** — the language-neutral contract every client and the diff --git a/Sources/HappySync/SyncEngine.swift b/Sources/HappySync/SyncEngine.swift index 6963b99..d27e2c9 100644 --- a/Sources/HappySync/SyncEngine.swift +++ b/Sources/HappySync/SyncEngine.swift @@ -1169,12 +1169,14 @@ public actor SyncEngine { let servers = try await remote.upsert(table: spec.name, rows: payloads, onConflict: onConflict) let matched = Self.match(servers, to: uploads, primaryKey: spec.primaryKey) try await stampAndClear( - zip(uploads, matched).map { ConfirmedUpload(entry: $0, server: $1, seqs: seqs(of: $0)) }, + zip(uploads, matched).map { entry, server in + ConfirmedUpload(entry: entry, server: server, seqs: seqs(of: entry)) + }, spec: spec ) case .delete: try await remote.delete(table: spec.name, primaryKey: spec.primaryKey, pks: chunk.map(\.pk)) - try await clear(chunk.flatMap(seqs(of:))) + try await clear(chunk.flatMap { seqs(of: $0) }) } return true } catch { @@ -1209,9 +1211,9 @@ public actor SyncEngine { /// key is `"7"` there and must be compared as one; anything that isn't a scalar key is no key. private static func wireKey(_ value: AnyJSON) -> String? { switch value { - case .string(let string): string - case .integer(let int): String(int) - default: nil + case .string(let string): return string + case .integer(let int): return String(int) + default: return nil } } diff --git a/Sources/HappySync/SyncSchema.swift b/Sources/HappySync/SyncSchema.swift index b6631f7..4be13a7 100644 --- a/Sources/HappySync/SyncSchema.swift +++ b/Sources/HappySync/SyncSchema.swift @@ -1,7 +1,9 @@ import GRDB -/// Internal bookkeeping tables HappySync owns. The app never reads or writes these directly. -enum SyncSchema { +/// Internal bookkeeping tables HappySync owns. The app never reads or writes these directly — the +/// only public surface here is the migrator, for an app that runs its own `DatabaseMigrator` on the +/// same database and therefore has to share one with the engine. +public enum SyncSchema { /// Shared by every internal table below, so a schema scan can tell the engine's own tables from /// the app's without listing them (see `LocalSchema`). static let tablePrefix = "_sync_" @@ -16,8 +18,9 @@ enum SyncSchema { /// writes don't queue themselves straight back into the outbox (issue #48). static let controlTable = "_sync_control" - /// A standalone migrator for HappySync's internal tables. Used when the engine owns the DB. - static func migrator() -> DatabaseMigrator { + /// A standalone migrator for HappySync's internal tables. Used when the engine owns the DB — + /// which it runs itself at init, so an app only needs this to migrate ahead of time. + public static func migrator() -> DatabaseMigrator { var migrator = DatabaseMigrator() register(into: &migrator) return migrator @@ -30,7 +33,7 @@ enum SyncSchema { /// The per-table **write-capture triggers** (issue #48) are deliberately *not* here: they're /// derived from the `SyncTable` manifest, which is source code and ships without a schema version /// bump, so `SyncTriggers.install` (re)runs idempotently at every engine init instead. - static func register(into migrator: inout DatabaseMigrator) { + public static func register(into migrator: inout DatabaseMigrator) { migrator.registerMigration("happysync_v1") { db in try db.create(table: outboxTable) { t in t.autoIncrementedPrimaryKey("seq") From 2b8b06e1059d4758f31c692aad4f20f24f48134b Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 31 Jul 2026 17:37:58 +0000 Subject: [PATCH 3/5] Add a DocC catalog and make it the documentation (#57) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The source carries unusually good documentation — `SyncTable`'s field comments explain not just what each option does but when to reach for it and what breaks if you get it wrong — and none of it was reachable from a browser. There was no catalog, so Swift Package Index rendered nothing and Xcode Quick Help was the only way in. Meanwhile the wiki duplicated a subset of the same content by hand and had already started to drift. `Sources/HappySync/Documentation.docc/` now carries a landing page (the model in five sentences, plus curation so SyncEngine/SyncTable/SyncStatus/DeadLetter lead the symbol list) and the five articles, ported from the wiki so there is one copy: Getting Started, Server Setup, Operations & Troubleshooting, Testing, and Architecture. `.spi.yml` turns it into hosted documentation; the README links it. The wiki pages that held that content are now one-paragraph pointers to it. The sync contract stays in `docs/SYNC-CONTRACT.md` and is linked, not duplicated — it is language-neutral and belongs outside the Swift docs. Also adds the doc comments the two public symbols were missing (`SyncOp`'s cases and `SyncCursor`'s fields), and a CI job that builds the catalog, so an unresolvable symbol link surfaces on the PR rather than on Swift Package Index. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015y9Y9kec3hj8S35oLQXxvQ --- .github/workflows/ci.yml | 8 + .spi.yml | 7 + Package.swift | 3 + README.md | 8 +- .../Documentation.docc/Architecture.md | 124 +++++++++ .../Documentation.docc/GettingStarted.md | 188 ++++++++++++++ .../HappySync/Documentation.docc/HappySync.md | 92 +++++++ .../Documentation.docc/Operations.md | 142 +++++++++++ .../Documentation.docc/ServerSetup.md | 135 ++++++++++ .../TestingYourIntegration.md | 225 ++++++++++++++++ Sources/HappySync/Models.swift | 4 + Sources/HappySync/Outbox.swift | 3 + wiki/API-Reference.md | 240 +----------------- wiki/Architecture.md | 126 +-------- wiki/Getting-Started.md | 189 +------------- wiki/Home.md | 18 +- wiki/Operations-and-Troubleshooting.md | 141 +--------- wiki/README.md | 28 +- wiki/Server-Setup.md | 137 +--------- wiki/Testing.md | 227 +---------------- wiki/_Sidebar.md | 8 +- 21 files changed, 1015 insertions(+), 1038 deletions(-) create mode 100644 .spi.yml create mode 100644 Sources/HappySync/Documentation.docc/Architecture.md create mode 100644 Sources/HappySync/Documentation.docc/GettingStarted.md create mode 100644 Sources/HappySync/Documentation.docc/HappySync.md create mode 100644 Sources/HappySync/Documentation.docc/Operations.md create mode 100644 Sources/HappySync/Documentation.docc/ServerSetup.md create mode 100644 Sources/HappySync/Documentation.docc/TestingYourIntegration.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 17bf6b9..8339099 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -11,6 +11,14 @@ jobs: - uses: actions/checkout@v4 - run: swift test + docs: + # The DocC catalog is the documentation now, so a broken link or an unresolvable symbol + # reference should surface here rather than on Swift Package Index (issue #57). + runs-on: macos-15 + steps: + - uses: actions/checkout@v4 + - run: swift package --allow-writing-to-directory ./docs-build generate-documentation --target HappySync --output-path ./docs-build + example: # The example is a separate package that consumes this one by path, so it catches an API change # the library's own tests can't see — and its tests assert the behaviours the demo puts on diff --git a/.spi.yml b/.spi.yml new file mode 100644 index 0000000..0d382c5 --- /dev/null +++ b/.spi.yml @@ -0,0 +1,7 @@ +# Swift Package Index build settings. `documentation_targets` is what makes SPI build and host the +# DocC catalog in Sources/HappySync/Documentation.docc, so the doc comments are reachable from a +# browser instead of only from Xcode Quick Help (issue #57). +version: 1 +builder: + configs: + - documentation_targets: [HappySync] diff --git a/Package.swift b/Package.swift index 95b1258..2121db9 100644 --- a/Package.swift +++ b/Package.swift @@ -16,6 +16,9 @@ let package = Package( // supabase-community is the fork CookThis consumes; matching the URL (and SPM // identity) lets the app share one supabase-swift instead of resolving two. .package(url: "https://github.com/supabase-community/supabase-swift.git", from: "2.0.0"), + // Build-tools only: `swift package generate-documentation` / `preview-documentation` for the + // DocC catalog (issue #57). Swift Package Index builds the same catalog from `.spi.yml`. + .package(url: "https://github.com/apple/swift-docc-plugin", from: "1.3.0"), ], targets: [ .target( diff --git a/README.md b/README.md index bbe8811..7ad5717 100644 --- a/README.md +++ b/README.md @@ -247,10 +247,14 @@ deliberately generic, but is pressure-tested against one real app before it's tr ## Documentation & Claude skill +- **[Documentation](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync)** + — Getting Started, Server Setup, Operations & Troubleshooting, Testing, Architecture, and every + public symbol. Built from [`Sources/HappySync/Documentation.docc`](Sources/HappySync/Documentation.docc); + `swift package generate-documentation` builds it locally. - **[Example app](Examples)** — a runnable two-table SwiftUI demo against the in-memory fake, with the Supabase migrations for when you point it at a real project. -- **[Wiki](https://github.com/happyface-studio/HappySync/wiki)** — Getting Started, Server Setup, - API Reference, Operations & Troubleshooting, and Architecture. (Sources live in [`wiki/`](wiki).) +- **[Wiki](https://github.com/happyface-studio/HappySync/wiki)** — points at the documentation + above, and hosts the Claude skill page. (Sources live in [`wiki/`](wiki).) - **[Sync contract](docs/SYNC-CONTRACT.md)** — the language-neutral contract every client and the server must honor. - **Claude skill** — [`.claude/skills/happysync/`](.claude/skills/happysync) is an installable diff --git a/Sources/HappySync/Documentation.docc/Architecture.md b/Sources/HappySync/Documentation.docc/Architecture.md new file mode 100644 index 0000000..af18e74 --- /dev/null +++ b/Sources/HappySync/Documentation.docc/Architecture.md @@ -0,0 +1,124 @@ +# Architecture + +HappySync is deliberately small. Its correctness rests on one enabling constraint and a handful of +mechanisms that compose cleanly. + +## The enabling constraint + +Personal data is **single-user, multi-device — not collaborative**. One user edits their own rows, +occasionally from two devices. Because two devices rarely edit the *same* row at the *same* instant, +**last-write-wins (LWW) by server timestamp** is correct. No CRDTs, no operational transforms, no +conflict-resolution RPC. Revisit only if the data ever becomes collaborative. + +## The pieces + +``` + write read + │ ▲ + ▼ │ + ┌──────────────┐ ValueObservation │ + │ GRDB SQLite │───────────────────► UI + │ (local truth)│ + └──────┬───────┘ + │ same transaction + ▼ + ┌──────────────┐ drain (upsert) ┌───────────────┐ + │ _sync_outbox│───────────────────► │ Supabase │ + └──────────────┘ │ Postgres │ + ▲ cursor pull (LWW) │ (server truth│ + └───────────────────────────── │ for writes) │ + └──────┬────────┘ + Realtime doorbell │ (debounced pullNow) + ◄───────┘ +``` + +### Upload — outbox drain + +A write appends to `_sync_outbox` **in the same transaction** as the domain write, then returns +optimistically. A background drain processes entries in `seq` order: + +- **PostgREST upsert** with `Prefer: return=representation` for `.upsert`; soft-delete for + `.delete`. Both are **idempotent by primary key**, so retries are safe. +- **The cursor column never leaves the device.** It's server-stamped by contract, so the payload + drops it (along with any declared `serverOwnedColumns`) with nothing declared — a table whose + `updatedAt` trigger is missing can't quietly promote a device's clock to the LWW authority. + `deletedAt` still ships, as `null`, which is how a re-created row un-tombstones. +- **The full server representation is written back locally** on success — column defaults, + trigger-normalized fields, recomputed server-owned columns, and (for a `conflictColumns` upsert) + the merged row re-keyed to the client's pk. Otherwise the writing device would be the one device + that never sees the server's version of its own write (its local `updatedAt` would equal the + server's, so LWW would skip the row forever). The write-back is skipped for a row that gained a + newer outbox entry mid-flight — the pending edit wins. +- **FK ordering:** upsert parents before children; tombstone children before parents. +- **Batched requests, per-entry failures.** Consecutive entries for the same table and op are sent as + one request (up to 256 rows), so a 500-row backlog is a handful of round trips instead of 500. + Entries are never reordered to make a bigger batch, so the FK order above holds. When the server + rejects a batch it hasn't said *which* row it objected to, so the chunk is re-run one entry at a + time — the poison row parks alone and its neighbours upload. +- **Per-entry backoff** with jitter; failures are classified permanent vs transient (see + ). + +### Download — cursor pull with LWW + +Per table: `SELECT * WHERE updatedAt > :cursor [AND scopeColumn = :partition] ORDER BY (updatedAt, +id)`, RLS-scoped. + +- **Tuple cursor `(updatedAt, id)`** — not a bare timestamp — so rows sharing a millisecond at a + page boundary aren't dropped. +- **LWW apply:** apply a remote row only if `remote.updatedAt > local.updatedAt` **and** the local + row is not dirty (a pending local edit is never clobbered). The compare is at **microsecond** + precision — Postgres `now()` is µs-precise, so two writes inside the same millisecond don't tie. +- **Tombstones** (`deletedAt` set) arrive through the same pull and delete locally. +- **Schema-drift tolerance:** each wire row is intersected against the local table's columns, so a + server that migrated ahead doesn't brick an old client's pulls. + +### Realtime is a doorbell + +A Realtime change event triggers a **debounced `pullNow()`** — it never applies payloads directly. +All correctness lives in the idempotent cursor pull, so sync converges even if Realtime drops; the +doorbell just makes it feel instant. Foreground and periodic pulls converge on their own. + +### Writes and capture + +There is no write API to remember. Each declared table carries `AFTER INSERT` / `AFTER UPDATE` / +`AFTER DELETE` triggers that append to `_sync_outbox`, so any GRDB write — `save`, a partial update, +raw SQL, a whole multi-table transaction — is queued **in the caller's transaction**. The unit of +atomicity is the user's intent, not one row: "create a recipe + 12 ingredients" commits the rows and +their queued uploads together or not at all. + +Each trigger's `WHEN` clause reads a one-row `_sync_control.applying` flag that the engine raises +while it writes *server* state locally (a pulled page, a tombstone, an upload's write-back, a resync +reconcile, a discarded dead letter). Without it every downloaded row would queue itself straight back +for upload and the sync would never settle. + +The triggers come from the `tables` manifest, which is source code rather than a numbered migration, +so the engine (re)installs them idempotently at init and drops the ones for tables you've stopped +syncing. A declared table that doesn't exist locally throws at init — the manifest and the schema are +not allowed to drift silently. + +A GRDB `TransactionObserver` watches for **inserts into the outbox** and wakes the runner, so a plain +write uploads as promptly as an explicit call used to. Drain bookkeeping (updates and deletes on the +outbox) is deliberately not observed, so a drain can't feed itself another pass. + +### Deletes and cascades + +A plain `DELETE` removes the row locally and queues a tombstone. For a parent, the schema's +`ON DELETE CASCADE` does the graph walk and each removed child's own delete trigger queues its +tombstone — mirroring the server's child-tombstone trigger. Result: the local and server deleted sets +stay symmetric, with no orphan window and no round-trip. A table that only `dependsOn` a parent +without an FK action is not cascaded; `dependsOn` orders uploads, it doesn't delete rows. + +## What HappySync owns (and doesn't) + +**Owns:** the write-capture triggers on your declared tables, the outbox drain, cursor pull, +tombstones, FK ordering, the Realtime doorbell, status, and retry/backoff. + +**Does not own:** your reads (observe GRDB directly) or your schema (you declare tables; the server +holds the authoritative schema). + +## The contract + +Every mechanism here has a precise, language-neutral spec — server conventions, wire semantics, +field mapping, schema-evolution rules — in +[`docs/SYNC-CONTRACT.md`](https://github.com/happyface-studio/HappySync/blob/main/docs/SYNC-CONTRACT.md). +A future non-Swift client would implement that contract, not reuse this engine's code. diff --git a/Sources/HappySync/Documentation.docc/GettingStarted.md b/Sources/HappySync/Documentation.docc/GettingStarted.md new file mode 100644 index 0000000..b3cbe76 --- /dev/null +++ b/Sources/HappySync/Documentation.docc/GettingStarted.md @@ -0,0 +1,188 @@ +# Getting Started + +This walks you from zero to a working sync loop. Before your first sync completes, you also need the +server schema in **** — the engine cannot correct a server that lacks the +`updatedAt` trigger. + +## 1. Add the package + +In `Package.swift`: + +```swift +dependencies: [ + .package(url: "https://github.com/happyface-studio/HappySync.git", from: "0.4.0"), +], +targets: [ + .target(name: "MyApp", dependencies: ["HappySync"]), +] +``` + +Or in Xcode: **File → Add Package Dependencies →** `https://github.com/happyface-studio/HappySync`. + +## 2. Declare your tables + +Each synced table is a `SyncTable`, declared once, **in any order** — the engine sorts the manifest +by FK dependency itself. Only `name` is required. + +Every declared table must **already exist** in your local schema when the engine is constructed, so +run your migrations first. `SyncEngine.init` validates the whole manifest against that schema and +throws `SyncError.invalidManifest(table:reason:)` — naming the table, the field and the value — if +anything disagrees: a table that isn't there, or a `cursorColumn` / `scopeColumn` / `jsonColumns` / +`conflictColumns` / `serverOwnedColumns` entry naming a column the table doesn't have. Each of those +used to fail silently and late, so it's worth reading the message rather than working around it. + +You don't declare `dependsOn`: the engine reads your foreign keys at init and derives it. Pass one +explicitly only for a *logical* parent your schema has no FK for (see +[Cascade](#delete-with-cascade)). + +Declare `ON DELETE CASCADE` on child foreign keys if you want a parent delete to remove (and +tombstone) its children — see [Delete](#delete-with-cascade) below. + +```swift +import HappySync + +let tables = [ + SyncTable(name: "recipes", jsonColumns: ["nutrition"]), + SyncTable(name: "recipeIngredients"), // dependsOn derived from the FK + SyncTable(name: "recipeSteps", jsonColumns: ["temperature"]), +] +``` + +``SyncTable`` documents every field it takes — `scopeColumn`, `conflictColumns`, +`serverOwnedColumns`, `serverColumns`, `cursorColumn` — including when to reach for each and what +breaks if you get it wrong. + +## 3. Create and start the engine + +```swift +let engine = try SyncEngine( + db: databaseQueue, // any GRDB DatabaseWriter — your local source of truth + supabase: client, // your SupabaseClient + tables: tables, + auth: { await session.accessToken } // return a FRESH access token each call +) + +await engine.start() +``` + +`start()` is idempotent. It runs an immediate convergence sync, then keeps syncing from three +triggers — the Realtime doorbell (debounced), a periodic poll (converges even if Realtime drops), +and retry backoff — all funnelled through one serialized runner. + +### Sharing a migrator + +The engine's internal tables (`_sync_outbox`, `_sync_state`, `_sync_meta`, `_sync_control`) migrate +at init. If your app owns the GRDB `DatabaseMigrator`, register HappySync's migrations into it so +both share one `grdb_migrations` table: + +```swift +var migrator = DatabaseMigrator() +SyncSchema.register(into: &migrator) // happysync_v1..v4 +// … your app migrations … +try migrator.migrate(databaseQueue) +``` + +The **write-capture triggers** are not part of that migrator. They're derived from your `tables` +manifest — source code, which ships without a schema version bump — so the engine (re)installs them +idempotently at every init and drops the ones for tables you've stopped syncing. + +## 4. Write + +Write GRDB **normally**. Each declared table carries `AFTER INSERT` / `UPDATE` / `DELETE` triggers +that append to the outbox in your transaction, so there is no second API to call and nothing to +forget. The engine notices the queued write and uploads promptly, so there's **no `syncNow()` needed +after a write** — a burst debounces into one drain pass. + +```swift +try await db.write { try recipe.save($0) } // PersistableRecord + +try await db.write { db in // partial / expression updates + try db.execute(sql: "UPDATE recipes SET title = ? WHERE id = ?", arguments: [title, id]) + try db.execute(sql: "UPDATE recipes SET cookCount = cookCount + 1 WHERE id = ?", arguments: [id]) +} + +try await db.write { db in // one transaction = one user intent + try recipe.save(db) + for ingredient in ingredients { try ingredient.save(db) } +} +``` + +That last one matters: the rows and their queued uploads commit together, or neither does. A crash +part-way through leaves no half-recipe *and* no half-batch in the outbox. + +> `engine.enqueue(op:table:row:)` still works but is **deprecated**. It now just performs the write +> and lets the trigger queue it. Replace `enqueue(.upsert, table: t, row: r)` with +> `try await db.write { try r.save($0) }`, and `enqueue(.delete, …)` with a plain `DELETE`. + +### Delete (with cascade) + +A plain `DELETE` removes the row locally and queues a tombstone. For a **parent** row, declare +`ON DELETE CASCADE` on the child foreign keys — SQLite then walks the graph and each removed child's +own trigger queues its tombstone, so you never delete children by hand and the UI never shows +orphans. + +```swift +// In your migration: +try db.create(table: "recipeIngredients") { t in + t.column("id", .text).primaryKey() + t.column("recipeId", .text).references("recipes", onDelete: .cascade) + // … +} + +// Then: +try await db.write { try $0.execute(sql: "DELETE FROM recipes WHERE id = ?", arguments: [recipeID]) } +``` + +A table that only `dependsOn` a parent without an FK action is **not** cascaded — `dependsOn` orders +uploads, it doesn't delete rows. Its orphans reconcile on the next pull via the server tombstone. + +## 5. Pull + +Downloads happen automatically (doorbell + periodic). Call `pullNow()` for app-driven nudges — +returning to the foreground, pull-to-refresh: + +```swift +try await engine.pullNow() +``` + +## 6. Show sync status + +Ask `isHealthy` — never `phase == .idle` alone. A settled pass with writes still failing or parked +is `.degraded`, so switching on the phase forces you to confront it: + +```swift +for await status in engine.status { + guard !status.isHealthy else { return showSynced() } + switch status.phase { + case .idle, .syncing: break // .idle here means healthy; .syncing is in flight + case .degraded: showPending(status.failedUploads + status.deadLetters) + case .failed(.authExpired): promptReauth() // classified — no string matching + case .failed(.network): break // recovers on its own; show nothing + case .failed(let failure): showBanner(failure) + } +} +``` + +See **** for what `failedUploads` / `deadLetters` mean and how to +repair parked writes. + +## 7. Tear down on sign-out (order matters) + +`stop()` is **async** and awaits the in-flight pass. Always `await engine.stop()` **before** wiping +or replacing the database on sign-out / account switch — otherwise an in-flight pass could write to +the store you're about to delete, or upload the old user's rows with the new user's token. + +```swift +await engine.stop() // engine is quiesced here +try await wipeLocalDatabase() +``` + +Status subscriptions survive teardown: `stop()` broadcasts a settled `.idle` status but doesn't end +the streams, so a `for await status in engine.status` loop keeps delivering after the next `start()` +— no re-subscribing needed. + +## Next + +- **** — required before the first successful sync. +- **** — health, dead letters, schema evolution. +- **[Claude Skill](https://github.com/happyface-studio/HappySync/wiki/Claude-Skill)** — let your AI assistant wire this correctly for you. diff --git a/Sources/HappySync/Documentation.docc/HappySync.md b/Sources/HappySync/Documentation.docc/HappySync.md new file mode 100644 index 0000000..472fe85 --- /dev/null +++ b/Sources/HappySync/Documentation.docc/HappySync.md @@ -0,0 +1,92 @@ +# ``HappySync`` + +Sync one user's data between their devices, over GRDB and Supabase, without pretending it's a +collaborative editing problem. + +## Overview + +Personal data is not collaborative: one user edits their own rows, occasionally from two devices. +That single constraint is what makes **last-write-wins by a server timestamp** correct — no CRDTs, +no conflict-resolution RPC — and it's what the whole engine is built on. + +The model in five sentences: + +1. **Local GRDB SQLite is the source of truth for reads.** You keep observing it with + `ValueObservation`; HappySync never sits between your app and its data. +2. **You write GRDB normally** — `save`, a partial `UPDATE`, raw SQL, a whole multi-table + transaction — and per-table SQLite triggers record each change into an outbox *in that same + transaction*, so a write can't be half-committed and half-queued. +3. **A background drain uploads the outbox** via PostgREST upsert, parents before children, batched + per table, idempotent by primary key, retrying with per-entry backoff. +4. **A pull downloads rows changed since a per-table `(updatedAt, id)` cursor** and applies them + last-write-wins, never clobbering a local row with a pending write of its own. +5. **Supabase Realtime is a doorbell only** — an event triggers a debounced pull; payloads are never + applied directly, so all the correctness lives in the idempotent cursor pull. + +Server-authoritative `updatedAt` (a Postgres `BEFORE INSERT/UPDATE` trigger → `now()`) is required +for that to be correct, and deletes propagate as `deletedAt` tombstones rather than vanishing. See + — the engine cannot correct a server that skips it. + +```swift +let engine = try SyncEngine( + db: databaseQueue, + supabase: client, + tables: [SyncTable(name: "recipes", jsonColumns: ["nutrition"])], + auth: { await session.accessToken } +) +await engine.start() + +// That's the whole write API: there isn't one. +try await databaseQueue.write { try recipe.save($0) } +``` + +There is a **runnable example** — a two-table SwiftUI app wired to the in-memory fake, needing no +Supabase project at all — in [`Examples/`](https://github.com/happyface-studio/HappySync/tree/main/Examples). + +The full, language-neutral contract every client *and the server* must honor lives in +[`docs/SYNC-CONTRACT.md`](https://github.com/happyface-studio/HappySync/blob/main/docs/SYNC-CONTRACT.md). +These pages are the Swift practitioner's guide; the contract is the source of truth. + +## Topics + +### Essentials + +- +- +- ``SyncEngine`` +- ``SyncTable`` + +### Watching sync + +- ``SyncStatus`` +- ``SyncFailure`` + +### Repairing writes the server refused + +- +- ``DeadLetter`` + +### Testing your integration + +- +- ``SyncRemote`` +- ``SyncDoorbell`` +- ``SilentDoorbell`` +- ``ClassifiedSyncError`` +- ``RemoteFailure`` + +### How it works + +- +- ``SyncCursor`` +- ``ScopeFilter`` +- ``SyncOp`` + +### Configuration errors + +- ``SyncError`` +- ``ManifestProblem`` + +### Sharing a migrator + +- ``SyncSchema`` diff --git a/Sources/HappySync/Documentation.docc/Operations.md b/Sources/HappySync/Documentation.docc/Operations.md new file mode 100644 index 0000000..dcb2127 --- /dev/null +++ b/Sources/HappySync/Documentation.docc/Operations.md @@ -0,0 +1,142 @@ +# Operations and Troubleshooting + +Running a synced app: what the status means, how a failed write is classified, how to repair one the +server refused, and what to do when a symptom shows up in the field. + +## Sync-status UI + +Consume `engine.status` and ask `isHealthy`. `.idle` means settled **and** clean; a pass that +completed while writes are still failing or parked settles as `.degraded`. + +```swift +for await status in engine.status { + switch status.phase { + case .idle: showSynced() // settled and clean + case .syncing: showSpinner() + case .degraded: showPending(status.failedUploads + status.deadLetters) + case .failed(let failure): showBanner(failure) // classified — see below + } +} +``` + +- `failedUploads > 0` → some local changes haven't reached the server yet (degraded, still + retrying with backoff). +- `deadLetters > 0` → some writes parked and need inspection/repair. + +## Failure model + +- **Permanent** (constraint `23xxx`, RLS `42501`, undefined-column `42703`, other 4xx) → + dead-lettered immediately. +- **Transient** (network, 5xx, 408/429, transient Postgres `40001`/`40P01`/`53xxx`/`08xxx`, unknown + codes) → retried with per-entry exponential backoff (jittered) until a cap, then dead-lettered. +- **Auth-shaped** (401/403, `PGRST301`/`PGRST302`) → transient **and exempt from the retry budget**, + so an expired-token stretch never dead-letters the whole outbox; it recovers when the token + refreshes out-of-band. + +A dead-lettered entry stops retrying **and** stops counting as a dirty row, so it never permanently +blocks downloads for its key. + +### What the app sees + +That classification is surfaced as `SyncFailure`, on `Phase.failed` and on `DeadLetter.failure`. +It answers *what to show the user* — the retry decision is already made by then, and is visible as +`failedUploads` (still retrying) vs `deadLetters` (parked): + +| `SyncFailure` | Raised by | What the app should do | +|---|---|---| +| `.network` | `URLError`, Postgres `08xxx` | Nothing — it recovers | +| `.authExpired` | 401/403, `PGRST301`/`PGRST302` | Nothing at first; prompt re-auth if it persists | +| `.permissionDenied` | `42501` | Surface it — a policy bug or a real permissions problem | +| `.constraintViolation(code:)` | `23xxx` | Inspect the row; the write will never land as-is | +| `.schemaMismatch(column:)` | `42703`, `PGRST204` | Prompt an app update (or restore the server column) | +| `.server(status:)` | any other HTTP status, incl. 5xx | Nothing — retried; surface if it persists | +| `.other(String)` | anything unrecognised, transient Postgres states | Log the text | + +`.network` means the request got no answer at all; a server that answered with a 5xx is +`.server(status:)`, so the status code isn't thrown away. + +The kind and the retry decision agree everywhere except `PGRST204`: it classifies as +`.schemaMismatch` but is **retried** rather than parked immediately, because PostgREST also raises it +from a stale schema cache, which clears on its own. + +Every case renders via `CustomStringConvertible`, so `"\(failure)"` is safe to put in a banner. The +classified cases carry no server prose of their own — the full text lives in the engine's log, and on +`DeadLetter.lastError` for a parked write. + +## Repairing dead letters + +```swift +let parked = try await engine.deadLetters() +for letter in parked { + switch letter.failure { + case .permissionDenied: reportPolicyBug(letter) // fix RLS, then retryDeadLetters() + case .schemaMismatch(let column): promptAppUpdate(column) + case .constraintViolation(let code): inspect(letter, code) + default: log("\(letter.op) \(letter.table)/\(letter.pk): \(letter.lastError ?? "unknown")") + } +} + +// After fixing the cause (RLS/policy change, schema migration, app update): +try await engine.retryDeadLetters() // all, or pass [seq] for specific entries + +// Or abandon the local write and accept the server's version — discard drops the entries and +// re-pulls the affected rows so local converges to the server (including changes missed while parked): +try await engine.discardDeadLetters([badSeq]) // specific seqs, or omit for all +``` + +Both mutations refresh the status stream immediately, so `deadLetters` drops as soon as they return. + +## Teardown / account switch (order matters) + +`stop()` is async and awaits the in-flight pass. A consumer that wipes or replaces the database on +sign-out / account switch **must `await engine.stop()` before touching the DB file** — otherwise an +in-flight pass could write to the store you're about to delete, or (mid-account-switch) upload the +old user's rows with the new user's token. + +```swift +await engine.stop() // quiesced here +try await wipeLocalDatabase() +``` + +`stop()` settles the status stream (`.idle`, keeping the outbox counts) but leaves subscriptions +open, so the sync-status UI resumes updating on the next `start()` without being re-subscribed. + +## Stale-cursor full resync + +A device offline past `maxOfflineGap` (default 30 days) can't trust its cursors — a tombstone it +never saw may already be purged. Once `now − lastSyncedAt > maxOfflineGap` the engine clears +cursors, re-pulls, and reconciles purged deletes instead of resurrecting them. Keep `maxOfflineGap` +**≤ the server's tombstone-purge retention** (see ). For a `scopeColumn` table the +resync is deferred until a partition value resolves, so it never wipes an un-pulled table. + +## Schema evolution (staggered clients) + +Shipped clients and the server are **not lockstep** — App Store review + staggered updates mean old +builds run against a newer server for weeks. The engine tolerates drift both ways, but follow these +rules so any single migration is safe for every client in the field: + +- **Additive-only.** Never drop-and-recreate or repurpose a column in place. +- **Never rename in place.** Add the new name, dual-write server-side, wait for adoption, then + remove the old name by the removal rule. +- **Adding a column — server-first.** Deploy it (nullable/default) before any client reads/writes + it. Old clients drop the unknown column on download and don't send it on upload. +- **Removing a column — client-first.** Drop it server-side **only after no shipped client still + writes it**: (1) ship a build that stops sending it — optionally declaring `serverColumns` so the + payload is intersected immediately; (2) wait out the update-lag window; (3) then drop it. Skipping + step 2 dead-letters every write from clients still sending the column. + +## Troubleshooting + +| Symptom | Likely cause | Fix | +|---|---|---| +| Writes never reach the server; `deadLetters` climbing | RLS rejects (`42501`) or a constraint violation | `deadLetters()` → switch on `failure` (`lastError` for the raw text); fix the policy/data, then `retryDeadLetters()`. | +| A device never sees the server's version of its **own** write | Missing `updatedAt` trigger (the engine already strips the client's `updatedAt` from every upload, so a stale client value can't be the cause) | Add the server trigger. | +| Rows on an insert-only table never download to other devices | The table's `cursorColumn` has no server default — the engine never uploads it, so it lands null | `default now()` (or a trigger) on that column server-side. | +| Whole public catalog downloads to every device | RLS broader than the partition, no `scopeColumn` | Declare `scopeColumn` + supply the engine `scope:` closure. | +| A fresh-pk insert 409s forever | Row already exists under a secondary `UNIQUE` | Declare `conflictColumns` (leaf tables only). | +| Every write on a table dead-letters after a server migration | Dropped/renamed server column (`PGRST204`) | Restore the column until clients stop sending it; or declare `serverColumns`. | +| Deleted parent leaves orphaned children in the UI | Child FK not declared `ON DELETE CASCADE` locally, or missing server child-tombstone trigger | Declare `ON DELETE CASCADE` on the child FKs (`dependsOn` alone deletes nothing) and add the server child-tombstone trigger. | +| `SyncEngine.init` throws `invalidManifest` | A declared `SyncTable` disagrees with the local schema — a table or column that isn't there, a duplicate declaration, `dependsOn` naming an undeclared table, `conflictColumns` on a table something foreign-keys, or `serverColumns` without the primary key | Read the message: it names the table, the field and the value. Run your app's migrations **before** constructing the engine, then fix the manifest. It throws rather than silently syncing the wrong thing — every one of these used to be invisible until the damage showed up on another device. | +| Some writes sync, one code path never does | A write that bypasses the declared tables — writing a table not in the manifest, or a `DatabasePool` reader-side hack | Add the table to the manifest. Writes to declared tables are captured by triggers, so there is no way to "forget" to enqueue one. | +| Offline-long device resurrects deleted rows | `maxOfflineGap` > server tombstone retention | Set `maxOfflineGap` ≤ retention. | +| Idle status shows healthy but changes aren't syncing | Reading `phase == .idle` as health | Ask `status.isHealthy`. The engine now settles such a pass as `.degraded`, so a `switch` over the phase surfaces it. | diff --git a/Sources/HappySync/Documentation.docc/ServerSetup.md b/Sources/HappySync/Documentation.docc/ServerSetup.md new file mode 100644 index 0000000..ac86ce5 --- /dev/null +++ b/Sources/HappySync/Documentation.docc/ServerSetup.md @@ -0,0 +1,135 @@ +# Server Setup + +Every synced table needs four things on the Supabase side. These are **non-negotiable** — LWW +compares a *server* clock, and deletes, scoping, and live updates all depend on this contract. The +engine will not paper over a server that violates it. + +Run the SQL below per synced table (swap `recipes` for your table). + +## 1. `updatedAt` — server-stamped change time + +LWW compares `updatedAt`, and a *server* clock is what stops two devices' clock skew from silently +losing writes. **Never trust a client-sent `updatedAt`.** The engine holds up its half unasked: a +table's `cursorColumn` is stripped from every upload payload, so no HappySync client ever sends one +— you don't declare anything for this. The trigger below is still required (it's what advances the +column on update, and what the cursor pull depends on); the stripping just means a table that lost +its trigger can't quietly promote a device's clock to the ordering authority. + +```sql +alter table public.recipes + add column if not exists "updatedAt" timestamptz not null default now(); + +create or replace function public.stamp_updated_at() +returns trigger language plpgsql as $$ +begin + new."updatedAt" := now(); -- server clock, always; ignore any client value + return new; +end $$; + +drop trigger if exists stamp_updated_at on public.recipes; +create trigger stamp_updated_at + before insert or update on public.recipes + for each row execute function public.stamp_updated_at(); +``` + +> An **insert-only / immutable** table needs no update trigger. Give it a monotonic stamp column +> (e.g. `translatedAt`) and set that table's `cursorColumn` in its `SyncTable` descriptor. Because +> the engine never uploads the cursor column, that column must be `default now()` server-side — +> otherwise inserts land with a null stamp the cursor can't order. + +## 2. `deletedAt` — soft-delete tombstones + +Deletes are **soft**: set `deletedAt` instead of removing the row, so deletions propagate on the +next cursor pull. Deleting a parent must **tombstone its children too** — use a trigger, *not* +`ON DELETE CASCADE` (a hard cascade leaves no tombstone for other devices to pull). + +```sql +alter table public.recipes + add column if not exists "deletedAt" timestamptz; + +create or replace function public.tombstone_recipe_children() +returns trigger language plpgsql as $$ +begin + if new."deletedAt" is not null and old."deletedAt" is null then + update public."recipeIngredients" + set "deletedAt" = new."deletedAt" + where "recipeId" = new.id and "deletedAt" is null; + -- repeat per child table + end if; + return new; +end $$; + +drop trigger if exists tombstone_recipe_children on public.recipes; +create trigger tombstone_recipe_children + after update on public.recipes + for each row execute function public.tombstone_recipe_children(); +``` + +Locally, declare the child foreign keys `ON DELETE CASCADE` in your **SQLite** schema: deleting a +parent then removes its children in the same transaction and each child's capture trigger queues its +tombstone, keeping the local and server deleted sets symmetric with no orphan window. The server +trigger above is what makes *other* devices converge. + +### Tombstone purge and the `maxOfflineGap` invariant + +Purge old tombstones on a schedule, but respect this **invariant**: the purge retention must exceed +the engine's `maxOfflineGap` (default 30 days). A device offline longer than the retention would +never see a purged tombstone and would resurrect the row. Set `maxOfflineGap` **≤ your server's +retention**. + +```sql +-- e.g. a pg_cron job; retention (90d) > engine maxOfflineGap (30d default) +delete from public.recipes where "deletedAt" < now() - interval '90 days'; +``` + +## 3. RLS scoped to `auth.uid()` + +RLS is the **security boundary**. Enable it and add policies keyed on `auth.uid()`. + +```sql +alter table public.recipes enable row level security; + +create policy "own rows: select" on public.recipes + for select using ("userId" = auth.uid()); +create policy "own rows: write" on public.recipes + for all using ("userId" = auth.uid()) with check ("userId" = auth.uid()); +``` + +**RLS is not necessarily the sync partition.** If a policy is deliberately broader than what a +device should download — e.g. `isPublic = true OR userId = auth.uid()` — an unfiltered pull drags +the whole public catalog to every device. Declare a `scopeColumn` (e.g. `userId`) on the `SyncTable` +and supply the engine's `scope:` closure; the engine filters the pull **and** the doorbell to +`scopeColumn = `. Omit `scopeColumn` when RLS already equals the partition. + +## 4. Realtime publication (the doorbell) + +```sql +alter publication supabase_realtime add table public.recipes; +``` + +Realtime is a **doorbell only** — an event triggers a debounced `pullNow()`; payloads are never +applied directly. Sync converges without it; Realtime just makes it feel instant. + +## Field-mapping conventions + +| Concern | Convention | +|---|---| +| Column names | **camelCase, identical** in SQLite and Postgres — no snake_case mapping. | +| Dates | ISO-8601 with fractional seconds; the LWW gate compares instants at **microsecond** precision. | +| UUID | text locally ↔ `uuid` in Postgres. | +| Bool | integer `0/1` locally ↔ `boolean`. | +| Enum | `rawValue` string. | +| JSON columns | JSON **text** locally ↔ `json`/`jsonb`; declare in `jsonColumns`. | +| Server-owned | RPC-managed columns; declare in `serverOwnedColumns`, never in an upsert. The `cursorColumn` is server-owned too and the engine strips it with nothing declared. | + +## Checklist + +- [ ] `updatedAt` + `BEFORE INSERT/UPDATE` trigger on every table (`default now()` on an + insert-only table's stamp column — the client never sends it). +- [ ] `deletedAt` on every table + a child-tombstone trigger on every parent. +- [ ] Tombstone purge scheduled, retention **>** engine `maxOfflineGap`. +- [ ] RLS enabled + `auth.uid()` policies on every table. +- [ ] `scopeColumn` declared for any table whose RLS is broader than the partition. +- [ ] Every synced table in the `supabase_realtime` publication. + +See ** → Schema evolution** before you change a live schema. diff --git a/Sources/HappySync/Documentation.docc/TestingYourIntegration.md b/Sources/HappySync/Documentation.docc/TestingYourIntegration.md new file mode 100644 index 0000000..e807b62 --- /dev/null +++ b/Sources/HappySync/Documentation.docc/TestingYourIntegration.md @@ -0,0 +1,225 @@ +# Testing + +Sync is the part of an app where bugs are silent and data-losing, and it is usually the part with no +tests — because testing it seems to require a live Supabase project. + +It doesn't. `SyncRemote` and `SyncDoorbell` are the seams the engine's own 2,700 lines of tests run +on, and both are public. The `HappySyncTestSupport` product ships the fakes, so a test can answer +questions like *does deleting a recipe queue tombstones for all its children?* in milliseconds, with +no network. + +## Add the test-support product + +Add it to your **test** target only — it contains no production code: + +```swift +.testTarget( + name: "MyAppTests", + dependencies: [ + "MyApp", + .product(name: "HappySync", package: "HappySync"), + .product(name: "HappySyncTestSupport", package: "HappySync"), + ] +), +``` + +## Build an engine with no Supabase + +`SyncEngine.forTesting(db:remote:tables:…)` is the production engine with the network seams +injected. Everything else — drain, pull, retry classification, dead-lettering, cascade ordering — is +the code that ships, so what a test proves here is what the app does. + +```swift +import HappySync +import HappySyncTestSupport + +let db = try DatabaseQueue() +try MyApp.migrator.migrate(db) // your migrations, as at launch + +let remote = InMemorySyncRemote() +let engine = try SyncEngine.forTesting( + db: db, + remote: remote, + tables: MyApp.syncTables, // the real manifest — a typo in it fails here, too + pollInterval: 999, // only the passes the test triggers + debounceInterval: 0.02 +) +``` + +The parameters worth knowing: `pageSize` (shrink to `1` to force pagination), `pollInterval` (raise +to silence the background poll), `debounceInterval` (shrink so a doorbell ring converges promptly), +`deadLetterAfter` (lower to reach the parked state without driving eight retries), and `scope` (the +partition value, to test a user switch). + +## `InMemorySyncRemote` + +A server that lives in the test process. It records every upload, serves downloads from a seedable +per-table row store honouring the `(cursorColumn, primaryKey)` tuple cursor and the page limit, and +can be told to fail. + +```swift +// What the server holds — rows are [String: AnyJSON], exactly what PostgREST returns. +let remote = InMemorySyncRemote(dataset: [ + "recipes": [["id": "r1", "title": "Soup", "updatedAt": "2026-06-30T12:00:00.000Z"]], +]) + +// What the client sent. +await remote.upsertCalls // [(table, row, onConflict)] in order — one entry per *row* +await remote.deleteCalls // [(table, pk)] — the tombstones +await remote.upsertRequests // round trips, which batching makes fewer than upsertCalls.count (#54) +await remote.deleteRequests // …and the same for tombstones +await remote.fetchCalls // download pages requested; the usual "a pass ran" signal +await remote.lastScope // the partition filter the engine resolved + +// Publish a change "another device" made, between two pulls. +await remote.seed("recipes", rows: [updatedRow]) +``` + +### Failure injection + +The one knob that matters is whether the drain **retries** the write or **parks** it: + +```swift +InMemorySyncRemote(failUpserts: 1, upsertFailure: .transient) // retried, then succeeds +InMemorySyncRemote(failUpserts: 1, upsertFailure: .permanent) // dead-lettered on the first attempt +InMemorySyncRemote(failUpserts: 1, upsertFailure: .error(HTTPError(…))) // real error, real classification +``` + +`failUpserts` counts **requests**, not rows: the drain batches a run of same-table writes into one +call, so a rejected batch spends one — and the per-row re-run it triggers spends one per row. + +`.error` wraps a genuine transport error (`HTTPError`, `PostgrestError`, `URLError`) with the +classification the production remote applies — so *"an RLS reject parks, an expired token doesn't"* +is asserted against the shipped table rather than a hand-picked answer. `failFetches` / +`fetchFailure` do the same for downloads, which drives a whole pass to `.failed(SyncFailure)`. + +Use `99` for "always", and `failNextUpserts(_:with:)` / `failNextFetches(_:with:)` to change the +injection mid-test — e.g. fail, assert parked, then let the retry succeed. + +### Simulating server-side computation + +`representation` maps an uploaded row to the row the server returns. The drain writes the whole +representation back locally, so this is how you test server-normalized values, defaulted columns, or +a `conflictColumns` merge: + +```swift +let remote = InMemorySyncRemote(representation: { row in + var server = row + server["cookedCount"] = .integer(7) // the merged server total, not what this client sent + server["updatedAt"] = .string("2026-06-30T12:00:00.000Z") + return server +}) +``` + +### Holding a call in flight + +`onUpsert` / `onFetch` run at the start of each call with its 1-based number. Because the remote +suspends there, this is how a test races a local write against an in-flight upload, or stops the +engine mid-pull: + +```swift +let started = AsyncSignal(), gate = AsyncSignal() +await remote.onUpsert { _ in await started.fire(); await gate.wait() } + +let pass = Task { await engine.syncNow() } +await started.wait() // the upload is blocked in the remote +try await db.write { … } // a competing local edit lands behind it +await gate.fire() // let the upload finish +``` + +## `ManualDoorbell` + +The Realtime channel, made deterministic — it rings only when you say so. + +```swift +let doorbell = ManualDoorbell() +let engine = try SyncEngine.forTesting(db: db, remote: remote, tables: tables, doorbell: doorbell) +await engine.start() + +doorbell.fire() // a remote change lands +#expect(await eventually { await remote.fetchCalls >= 2 }) + +doorbell.ringScopes // [nil, "u1"] proves a sign-in re-scoped the subscription +doorbell.liveSubscriptions // settles to 1 — the old subscription was torn down +doorbell.fire(subscription: 0) // ring a torn-down one; it must not poke the engine +``` + +`SilentDoorbell` (the default) never rings, so convergence comes only from the periodic poll and the +passes you trigger. + +## Async helpers + +`eventually` polls until a condition holds or the timeout elapses. Reach for it any time the engine +converges in the background — a fixed sleep plus one assertion is the alternative, and it flakes +under a loaded CI runner: + +```swift +#expect(await eventually { await remote.fetchCalls >= 1 }) +``` + +For a *negative* assertion ("no further pull happened"), poll until the count stops moving, then +assert it doesn't move again. + +`AsyncSignal` is a one-shot rendezvous — `fire()` releases every `wait()`, now and later, so a wait +can't miss a signal that arrived first. + +## Three tests worth having + +These are the ones that catch real bugs, and none of them need a network. + +**A cascade delete queues every child's tombstone.** The app deletes one row; SQLite cascades and +each child's capture trigger queues its own tombstone. If a child table is missing from your +manifest, this test fails and the field doesn't. + +```swift +try await db.write { try $0.execute(sql: "DELETE FROM recipes WHERE id = 'r1'") } +#expect(await eventually { await remote.deleteCalls.count == 3 }) +``` + +**The status UI shows degraded when a write parks.** `.degraded` is the case everyone gets wrong: the +pass completed, so nothing *looks* broken, but the user's write never reached the server. Assert on +`isHealthy`, never on the phase alone. + +```swift +let remote = InMemorySyncRemote(failUpserts: 99, upsertFailure: .permanent) +… +#expect(await eventually { await latestStatus(engine)?.phase == .degraded }) +#expect(try await engine.deadLetters().count == 1) +``` + +**Sign-out quiesces the engine before the wipe.** After `await engine.stop()` returns, nothing may +still be in flight — a late pull writes into the store you're about to delete, and mid-account-switch +uploads the previous user's rows under the new user's token. + +```swift +await engine.stop() +let atStop = await remote.fetchCalls +doorbell.fire() // a Realtime event racing the sign-out +try await Task.sleep(for: .milliseconds(200)) +#expect(await remote.fetchCalls == atStop) // nothing restarted +``` + +A worked version of all three lives in +[`Tests/HappySyncTests/PublicSeamsTests.swift`](https://github.com/happyface-studio/HappySync/blob/main/Tests/HappySyncTests/PublicSeamsTests.swift), +which imports HappySync the way your app does — no `@testable` — so it stops compiling if these +seams ever regress. + +## Writing your own remote + +`SyncRemote` and `SyncDoorbell` are protocols, so a bespoke double is three methods. Signal whether a +failure is retryable by conforming your error to `ClassifiedSyncError`, or wrap a real transport +error with `RemoteFailure(classifying:)` to get the production classification: + +```swift +struct RejectingRemote: SyncRemote { + func upsert(table: String, row: [String: AnyJSON], onConflict: String?) async throws -> [String: AnyJSON] { + throw RemoteFailure(classifying: PostgrestError(code: "42501", message: "rls")) + } + func delete(table: String, primaryKey: String, pk: String) async throws {} + func fetch(table: String, cursorColumn: String, since cursor: SyncCursor?, primaryKey: String, + scope: ScopeFilter?, limit: Int) async throws -> [[String: AnyJSON]] { [] } +} +``` + +An error the engine can't classify is treated as **transient** — safer to retry than to silently drop +a user's write. diff --git a/Sources/HappySync/Models.swift b/Sources/HappySync/Models.swift index 5e46091..47f6dc7 100644 --- a/Sources/HappySync/Models.swift +++ b/Sources/HappySync/Models.swift @@ -122,7 +122,11 @@ public struct SyncTable: Sendable, Hashable { /// A pending sync operation recorded in the outbox. public enum SyncOp: String, Sendable, Codable { + /// The row exists locally and its current state should be uploaded — queued by an `INSERT` or an + /// `UPDATE`, and idempotent by primary key. case upsert + /// The row was removed locally and should be **soft**-deleted on the server (its tombstone set), + /// so the deletion propagates to other devices on their next pull. case delete } diff --git a/Sources/HappySync/Outbox.swift b/Sources/HappySync/Outbox.swift index 91b8c8c..12cdf7e 100644 --- a/Sources/HappySync/Outbox.swift +++ b/Sources/HappySync/Outbox.swift @@ -8,7 +8,10 @@ import GRDB /// Public because it appears in `SyncRemote.fetch` — a custom or fake remote has to be able to read /// the position it's being asked to resume from (issue #53). public struct SyncCursor: Sendable, Equatable { + /// The table's `cursorColumn` value of the last applied row — a server-stamped timestamp. public var updatedAt: String + /// That row's primary key, which breaks ties between rows sharing a `updatedAt` so a page + /// boundary can't skip one. public var id: String public init(updatedAt: String, id: String) { diff --git a/wiki/API-Reference.md b/wiki/API-Reference.md index 1e0b00c..c934360 100644 --- a/wiki/API-Reference.md +++ b/wiki/API-Reference.md @@ -1,239 +1,13 @@ # API Reference -All types are in the `HappySync` module. `SyncEngine` is an `actor`; its async methods are `await`ed -and safe to call from any task. +**The API reference is now generated from the source**, so a signature here can no longer drift from +the one you actually call: -## SyncEngine +### → [HappySync documentation](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync) -```swift -public init( - db: any DatabaseWriter, // GRDB writer; local source of truth - supabase: SupabaseClient, - tables: [SyncTable], // the manifest, declared once - auth: @escaping @Sendable () async -> String, // fresh access token per call - scope: @escaping @Sendable () async -> String? = { nil }, // partition value for scopeColumn tables - maxOfflineGap: TimeInterval = 30 * 24 * 3600 // stale-cursor full-resync threshold -) throws -``` +Every public type carries its documentation in the source it documents. Start with +[`SyncEngine`](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/syncengine) and [`SyncTable`](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/synctable). -| Method | Behavior | -|---|---| -| `func start()` | Begins background sync. Idempotent. Immediate convergence sync, then drives from the Realtime doorbell, a periodic poll, and retry backoff through one serialized runner. | -| `func stop() async` | **Async.** Awaits the in-flight pass, quiesces (no more DB writes / network), unsubscribes Realtime. Settles `status` to `.idle` but leaves subscriptions open. `start()` re-subscribes cleanly. Call **before** wiping/replacing the DB. | -| `func enqueue(_ op: SyncOp, table: String, row: some Encodable & Sendable) throws` | **Deprecated.** Performs the write; the table's capture trigger queues it. Write GRDB directly instead. Throws `SyncError`. | -| `func syncNow()` | Fire-and-forget nudge (foreground, pull-to-refresh). Not needed after a write — the engine wakes on queued writes itself. | -| `func pullNow() async throws` | Cursor pull now — the app-driven nudge (foreground return, pull-to-refresh). Returns nothing. | -| `var status: AsyncStream` | `nonisolated`. Live status for the UI. Each access is an independent stream replaying the latest snapshot; it survives `stop()`/`start()` and only ends when the engine is deallocated. | -| `func deadLetters() async throws -> [DeadLetter]` | Inspect parked entries. | -| `func retryDeadLetters(_ seqs: [Int64]? = nil) async throws` | Re-queue parked writes (specific `seq`s, or all). | -| `func discardDeadLetters(_ seqs: [Int64]? = nil) async throws` | Drop parked writes and re-pull affected rows so local converges to the server. | +--- -## SyncTable - -```swift -public init( - name: String, // identical local + remote table name - primaryKey: String = "id", - cursorColumn: String = "updatedAt", // change-time column the cursor orders/filters/advances by - // — server-stamped: never included in an upload payload - dependsOn: [String]? = nil, // nil = derive from the schema's FKs; drives sync + delete ordering - jsonColumns: [String] = [], // JSON text locally ↔ json/jsonb remote - serverOwnedColumns: [String] = [], // *extra* RPC-managed columns; stripped too, applied on download - scopeColumn: String? = nil, // partition column when RLS is broader than the partition - conflictColumns: [String] = [], // secondary-unique upsert target; LEAF tables only - serverColumns: [String] = [] // optional allow-list to survive a dropped/renamed column -) -``` - -| Field | Notes | -|---|---| -| `dependsOn` | FK dependencies. Parents upsert/download before children; children tombstone before parents. Leave it `nil` (the default) and the engine reads `PRAGMA foreign_key_list` at init and uses the table's real FK parents. Declare it only to **override** that for a *logical* parent with no FK constraint — an explicit list replaces the derived one rather than adding to it, and every name in it must be a declared `SyncTable`. A self-reference (a tree table) is dropped from either form. | -| `scopeColumn` | Requires the engine `scope:` closure. While no partition value is available (cold launch, signed out), the table's pull is skipped and the stale-resync check stays armed — never wipes an un-pulled table. | -| `conflictColumns` | Merges a fresh-pk insert onto an existing server row by a secondary `UNIQUE` (avoids a permanent 409). The merge **re-keys the server row to the client's pk** → **leaf tables only**, or you orphan children. | -| `serverColumns` | Opt-in backstop; a stale list silently stops uploading a real column. Prefer the client-first removal rule. Downloads need no equivalent (they intersect against the local schema). | -| `serverOwnedColumns` | Stripped from every upsert so a stale client value never clobbers the authoritative one; still arrive on download. Declare only the *extra* ones — the `cursorColumn` is always stripped. | -| `cursorColumn` | Server-stamped by contract (§1/§4), so the engine **never uploads it** — nothing to declare, and no way to opt out. Defence in depth for the one server-side requirement the engine can't verify: on a table whose trigger is missing, a client-sent value would silently make that device's clock the LWW authority. `deletedAt` is *not* stripped — an upsert carrying `deletedAt = null` is how a re-created row un-tombstones. | - -## SyncStatus - -```swift -public struct SyncStatus { - public enum Phase { - case idle // settled and clean — genuinely healthy - case syncing - case degraded // settled, but writes are still failing or parked - case failed(SyncFailure) - } - public var phase: Phase - public var lastSyncedAt: Date? // last successful pull/push, or nil - public var failedUploads: Int // entries retrying with backoff (attempts > 0, not parked) - public var deadLetters: Int // entries parked after a permanent 4xx or exhausted retries - public var isHealthy: Bool // phase == .idle && both counts zero -} -``` - -**Ask `status.isHealthy`.** `.idle` on a status the engine broadcast already implies it — the engine -derives `.idle` vs `.degraded` from the counts — but `isHealthy` is the one read that stays correct -on a `SyncStatus` you construct yourself, and it's the one that documents the intent. - -## SyncFailure - -The classified cause carried by `Phase.failed` and `DeadLetter.failure` — so an app can branch on -*why* instead of substring-matching PostgREST prose. - -```swift -public enum SyncFailure: Error, Sendable, Equatable { - case network // URLError, dropped connection, Postgres 08xxx - case authExpired // 401/403, PGRST301/302 — prompt re-auth - case permissionDenied // 42501 — RLS rejected the write - case constraintViolation(code: String) // 23xxx — unique, FK, not-null, check - case schemaMismatch(column: String?) // 42703 / PGRST204 — prompt an app update - case server(status: Int) // the server answered, including 5xx - case other(String) // unrecognised — original text preserved -} -``` - -The kind says *what to show*, not whether to retry: `failedUploads` vs `deadLetters` is what tells -you a write is still being retried or has been parked. - -## DeadLetter - -```swift -public struct DeadLetter { - public let seq: Int64 // pass to retryDeadLetters / discardDeadLetters - public let table: String - public let pk: String - public let op: SyncOp // .upsert | .delete - public let attempts: Int // upload attempts charged before parking - public let failure: SyncFailure // classified cause — branch on it to offer the right repair - public let lastError: String?// the raw breadcrumb behind `failure`, for logs and bug reports - public let queuedAt: Date -} -``` - -## SyncOp / SyncError - -```swift -public enum SyncOp: String, Codable { case upsert; case delete } - -public enum SyncError: Error, CustomStringConvertible { - case unknownTable(String) // deprecated enqueue on an undeclared table - case missingPrimaryKey(table: String, column: String) - case encoding(String) - case invalidManifest(table: String, reason: ManifestProblem) // the manifest doesn't match the schema (init) - case recursiveTriggersUnavailable // PRAGMA recursive_triggers could not be enabled -} - -public enum ManifestProblem: Equatable, CustomStringConvertible { - case duplicateDeclaration // the same table declared twice - case noSuchTable(didYouMean: String?) // no local table of that name - case noSuchColumn(field: String, column: String, didYouMean: String?) - case unknownDependency(String) // dependsOn names an undeclared SyncTable - case conflictColumnsOnParentTable(referencedBy: String) - case serverColumnsOmitPrimaryKey(String) -} -``` - -### Manifest validation - -`SyncEngine.init` checks every declared `SyncTable` against the local schema before it installs a -single trigger, and throws `SyncError.invalidManifest` naming the table, the field and the value. -Each of these used to fail silently and late — the wrong partition downloads, a column corrupts in -both directions, an FK-order violation surfaces days later as a dead letter on another device. - -- `name` must be a local table; `primaryKey`, `cursorColumn`, `scopeColumn`, `jsonColumns`, - `conflictColumns` and `serverOwnedColumns` must each name a column on it, **spelled exactly** as - the schema spells it. Case counts even though SQLite would resolve either spelling: these names go - to PostgREST verbatim, and a camelCase Postgres identifier is quoted and matched - case-sensitively — so `scopeColumn: "userID"` against a `userId` column works locally and 400s (or - silently filters on nothing) on the wire. A case-only mismatch is reported with a "did you mean". -- `serverColumns` is **not** checked against the local schema — it describes the *server's* schema, - which is allowed to differ. It may not omit the primary key, since the payload is intersected - against it. -- `conflictColumns` is rejected on a table anything else foreign-keys, enforcing the leaf-only rule - that was previously documentation alone. The scan covers unsynced children too. -- `dependsOn` entries must name declared tables. -- The same table may not be declared twice. - -`SyncError` and `ManifestProblem` both print a full sentence, so the message stands on its own in a -crash log. - -## Writing - -There is no write API. Each declared table gets `AFTER INSERT` / `AFTER UPDATE` / `AFTER DELETE` -triggers that append to `_sync_outbox` inside your own transaction, so any GRDB write syncs: - -```swift -try await db.write { try recipe.save($0) } -try await db.write { try $0.execute(sql: "UPDATE recipes SET title = ? WHERE id = ?", arguments: [t, id]) } -try await db.write { try $0.execute(sql: "DELETE FROM recipes WHERE id = ?", arguments: [id]) } -``` - -Declare `ON DELETE CASCADE` on child foreign keys to have a parent delete tombstone its children. -The engine enables `PRAGMA recursive_triggers` on the writer connection at init. - -## Testing seams - -The network seams are public, so a consumer can drive sync with fakes instead of a live Supabase -project. Full guide: **[[Testing]]**. - -```swift -// The engine, with the network injected. Otherwise identical to the production one. -static func SyncEngine.forTesting( - db: any DatabaseWriter, remote: any SyncRemote, tables: [SyncTable], - pageSize: Int = 500, doorbell: any SyncDoorbell = SilentDoorbell(), - pollInterval: TimeInterval = 30, debounceInterval: TimeInterval = 0.3, - scope: @Sendable () async -> String? = { nil }, - deadLetterAfter: Int = 8, maxOfflineGap: TimeInterval = 30 * 24 * 3600, - uploadChunkSize: Int = SyncEngine.defaultUploadChunkSize // rows per upload request -) throws -> SyncEngine - -protocol SyncRemote: Sendable { // the upload/download surface - func upsert(table: String, row: [String: AnyJSON], onConflict: String?) async throws -> [String: AnyJSON] - func delete(table: String, primaryKey: String, pk: String) async throws - func fetch(table: String, cursorColumn: String, since: SyncCursor?, primaryKey: String, - scope: ScopeFilter?, limit: Int) async throws -> [[String: AnyJSON]] - - // Batched forms — the drain prefers these; both default to looping over the single-row calls, - // so implementing them is optional and only costs round trips (issue #54). - func upsert(table: String, rows: [[String: AnyJSON]], onConflict: String?) async throws -> [[String: AnyJSON]] - func delete(table: String, primaryKey: String, pks: [String]) async throws -} -protocol SyncDoorbell: Sendable { // the Realtime wake signal - func ring(scope: String?) -> AsyncStream -} -struct SilentDoorbell: SyncDoorbell // never rings; the default -struct SyncCursor { var updatedAt: String; var id: String } // per-table download position -struct ScopeFilter { let column: String; let value: String } // resolved partition filter - -protocol ClassifiedSyncError: Error { var isPermanent: Bool { get } } // retryable, or park it? -struct RemoteFailure: ClassifiedSyncError { - init(isPermanent: Bool, underlying: Error) - init(classifying underlying: Error) // production classification of a real transport error -} -``` - -An error that conforms to neither is treated as **transient** — safer to retry than to silently drop -a user's write. - -### HappySyncTestSupport - -A separate product; add it to your test target only. - -```swift -actor InMemorySyncRemote: SyncRemote // seedable server, recorded calls, injectable failures -final class ManualDoorbell: SyncDoorbell // rings on fire(); records every re-subscription -actor AsyncSignal // one-shot rendezvous: fire() / wait() / isFired -func eventually(timeout: Duration = .seconds(5), _ condition: () async -> Bool) async -> Bool -enum SimulatedFailure { case transient, permanent, error(any Error) } -``` - -## SyncSchema (migrator sharing) - -```swift -static func migrator() -> DatabaseMigrator // standalone; used when the engine owns the DB -static func register(into migrator: inout DatabaseMigrator) // prefer this when the app owns the migrator -// The capture triggers are NOT migrations — the engine installs them idempotently at init. -``` - -Registers `happysync_v1..v3` — the `_sync_outbox`, `_sync_state`, and `_sync_meta` internal tables. +- [[Home]] — this wiki's index diff --git a/wiki/Architecture.md b/wiki/Architecture.md index 4fbd7d5..fd3918a 100644 --- a/wiki/Architecture.md +++ b/wiki/Architecture.md @@ -1,124 +1,14 @@ # Architecture -HappySync is deliberately small. Its correctness rests on one enabling constraint and a handful of -mechanisms that compose cleanly. +**This page has moved into the API documentation**, where it sits alongside the symbols it talks +about — and where there is one copy to maintain instead of two that drift apart: -## The enabling constraint +### → [Architecture](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/architecture) -Personal data is **single-user, multi-device — not collaborative**. One user edits their own rows, -occasionally from two devices. Because two devices rarely edit the *same* row at the *same* instant, -**last-write-wins (LWW) by server timestamp** is correct. No CRDTs, no operational transforms, no -conflict-resolution RPC. Revisit only if the data ever becomes collaborative. +The source is [`Sources/HappySync/Documentation.docc/Architecture.md`](https://github.com/happyface-studio/HappySync/blob/main/Sources/HappySync/Documentation.docc/Architecture.md); the page above is +what Swift Package Index builds from it. Everything that used to be here is there, unabridged. -## The pieces +--- -``` - write read - │ ▲ - ▼ │ - ┌──────────────┐ ValueObservation │ - │ GRDB SQLite │───────────────────► UI - │ (local truth)│ - └──────┬───────┘ - │ same transaction - ▼ - ┌──────────────┐ drain (upsert) ┌───────────────┐ - │ _sync_outbox│───────────────────► │ Supabase │ - └──────────────┘ │ Postgres │ - ▲ cursor pull (LWW) │ (server truth│ - └───────────────────────────── │ for writes) │ - └──────┬────────┘ - Realtime doorbell │ (debounced pullNow) - ◄───────┘ -``` - -### Upload — outbox drain - -A write appends to `_sync_outbox` **in the same transaction** as the domain write, then returns -optimistically. A background drain processes entries in `seq` order: - -- **PostgREST upsert** with `Prefer: return=representation` for `.upsert`; soft-delete for - `.delete`. Both are **idempotent by primary key**, so retries are safe. -- **The cursor column never leaves the device.** It's server-stamped by contract, so the payload - drops it (along with any declared `serverOwnedColumns`) with nothing declared — a table whose - `updatedAt` trigger is missing can't quietly promote a device's clock to the LWW authority. - `deletedAt` still ships, as `null`, which is how a re-created row un-tombstones. -- **The full server representation is written back locally** on success — column defaults, - trigger-normalized fields, recomputed server-owned columns, and (for a `conflictColumns` upsert) - the merged row re-keyed to the client's pk. Otherwise the writing device would be the one device - that never sees the server's version of its own write (its local `updatedAt` would equal the - server's, so LWW would skip the row forever). The write-back is skipped for a row that gained a - newer outbox entry mid-flight — the pending edit wins. -- **FK ordering:** upsert parents before children; tombstone children before parents. -- **Batched requests, per-entry failures.** Consecutive entries for the same table and op are sent as - one request (up to 256 rows), so a 500-row backlog is a handful of round trips instead of 500. - Entries are never reordered to make a bigger batch, so the FK order above holds. When the server - rejects a batch it hasn't said *which* row it objected to, so the chunk is re-run one entry at a - time — the poison row parks alone and its neighbours upload. -- **Per-entry backoff** with jitter; failures are classified permanent vs transient (see - [[Operations and Troubleshooting]]). - -### Download — cursor pull with LWW - -Per table: `SELECT * WHERE updatedAt > :cursor [AND scopeColumn = :partition] ORDER BY (updatedAt, -id)`, RLS-scoped. - -- **Tuple cursor `(updatedAt, id)`** — not a bare timestamp — so rows sharing a millisecond at a - page boundary aren't dropped. -- **LWW apply:** apply a remote row only if `remote.updatedAt > local.updatedAt` **and** the local - row is not dirty (a pending local edit is never clobbered). The compare is at **microsecond** - precision — Postgres `now()` is µs-precise, so two writes inside the same millisecond don't tie. -- **Tombstones** (`deletedAt` set) arrive through the same pull and delete locally. -- **Schema-drift tolerance:** each wire row is intersected against the local table's columns, so a - server that migrated ahead doesn't brick an old client's pulls. - -### Realtime is a doorbell - -A Realtime change event triggers a **debounced `pullNow()`** — it never applies payloads directly. -All correctness lives in the idempotent cursor pull, so sync converges even if Realtime drops; the -doorbell just makes it feel instant. Foreground and periodic pulls converge on their own. - -### Writes and capture - -There is no write API to remember. Each declared table carries `AFTER INSERT` / `AFTER UPDATE` / -`AFTER DELETE` triggers that append to `_sync_outbox`, so any GRDB write — `save`, a partial update, -raw SQL, a whole multi-table transaction — is queued **in the caller's transaction**. The unit of -atomicity is the user's intent, not one row: "create a recipe + 12 ingredients" commits the rows and -their queued uploads together or not at all. - -Each trigger's `WHEN` clause reads a one-row `_sync_control.applying` flag that the engine raises -while it writes *server* state locally (a pulled page, a tombstone, an upload's write-back, a resync -reconcile, a discarded dead letter). Without it every downloaded row would queue itself straight back -for upload and the sync would never settle. - -The triggers come from the `tables` manifest, which is source code rather than a numbered migration, -so the engine (re)installs them idempotently at init and drops the ones for tables you've stopped -syncing. A declared table that doesn't exist locally throws at init — the manifest and the schema are -not allowed to drift silently. - -A GRDB `TransactionObserver` watches for **inserts into the outbox** and wakes the runner, so a plain -write uploads as promptly as an explicit call used to. Drain bookkeeping (updates and deletes on the -outbox) is deliberately not observed, so a drain can't feed itself another pass. - -### Deletes and cascades - -A plain `DELETE` removes the row locally and queues a tombstone. For a parent, the schema's -`ON DELETE CASCADE` does the graph walk and each removed child's own delete trigger queues its -tombstone — mirroring the server's child-tombstone trigger. Result: the local and server deleted sets -stay symmetric, with no orphan window and no round-trip. A table that only `dependsOn` a parent -without an FK action is not cascaded; `dependsOn` orders uploads, it doesn't delete rows. - -## What HappySync owns (and doesn't) - -**Owns:** the write-capture triggers on your declared tables, the outbox drain, cursor pull, -tombstones, FK ordering, the Realtime doorbell, status, and retry/backoff. - -**Does not own:** your reads (observe GRDB directly) or your schema (you declare tables; the server -holds the authoritative schema). - -## The contract - -Every mechanism here has a precise, language-neutral spec — server conventions, wire semantics, -field mapping, schema-evolution rules — in -[`docs/SYNC-CONTRACT.md`](https://github.com/happyface-studio/HappySync/blob/main/docs/SYNC-CONTRACT.md). -A future non-Swift client would implement that contract, not reuse this engine's code. +- [Documentation home](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync) — the landing page, and the full symbol reference +- [[Home]] — this wiki's index diff --git a/wiki/Getting-Started.md b/wiki/Getting-Started.md index eb931f3..1d8fc3f 100644 --- a/wiki/Getting-Started.md +++ b/wiki/Getting-Started.md @@ -1,187 +1,14 @@ # Getting Started -This walks you from zero to a working sync loop. Before your first sync completes, you also need the -server schema in **[[Server Setup]]** — the engine cannot correct a server that lacks the -`updatedAt` trigger. +**This page has moved into the API documentation**, where it sits alongside the symbols it talks +about — and where there is one copy to maintain instead of two that drift apart: -## 1. Add the package +### → [Getting Started](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/gettingstarted) -In `Package.swift`: +The source is [`Sources/HappySync/Documentation.docc/GettingStarted.md`](https://github.com/happyface-studio/HappySync/blob/main/Sources/HappySync/Documentation.docc/GettingStarted.md); the page above is +what Swift Package Index builds from it. Everything that used to be here is there, unabridged. -```swift -dependencies: [ - .package(url: "https://github.com/happyface-studio/HappySync.git", from: "0.4.0"), -], -targets: [ - .target(name: "MyApp", dependencies: ["HappySync"]), -] -``` +--- -Or in Xcode: **File → Add Package Dependencies →** `https://github.com/happyface-studio/HappySync`. - -## 2. Declare your tables - -Each synced table is a `SyncTable`, declared once, **in any order** — the engine sorts the manifest -by FK dependency itself. Only `name` is required. - -Every declared table must **already exist** in your local schema when the engine is constructed, so -run your migrations first. `SyncEngine.init` validates the whole manifest against that schema and -throws `SyncError.invalidManifest(table:reason:)` — naming the table, the field and the value — if -anything disagrees: a table that isn't there, or a `cursorColumn` / `scopeColumn` / `jsonColumns` / -`conflictColumns` / `serverOwnedColumns` entry naming a column the table doesn't have. Each of those -used to fail silently and late, so it's worth reading the message rather than working around it. - -You don't declare `dependsOn`: the engine reads your foreign keys at init and derives it. Pass one -explicitly only for a *logical* parent your schema has no FK for (see -[Cascade](#delete-with-cascade)). - -Declare `ON DELETE CASCADE` on child foreign keys if you want a parent delete to remove (and -tombstone) its children — see [Delete](#delete-with-cascade) below. - -```swift -import HappySync - -let tables = [ - SyncTable(name: "recipes", jsonColumns: ["nutrition"]), - SyncTable(name: "recipeIngredients"), // dependsOn derived from the FK - SyncTable(name: "recipeSteps", jsonColumns: ["temperature"]), -] -``` - -See **[[API Reference]]** for every `SyncTable` field (`scopeColumn`, `conflictColumns`, -`serverOwnedColumns`, `serverColumns`, `cursorColumn`). - -## 3. Create and start the engine - -```swift -let engine = try SyncEngine( - db: databaseQueue, // any GRDB DatabaseWriter — your local source of truth - supabase: client, // your SupabaseClient - tables: tables, - auth: { await session.accessToken } // return a FRESH access token each call -) - -await engine.start() -``` - -`start()` is idempotent. It runs an immediate convergence sync, then keeps syncing from three -triggers — the Realtime doorbell (debounced), a periodic poll (converges even if Realtime drops), -and retry backoff — all funnelled through one serialized runner. - -### Sharing a migrator - -The engine's internal tables (`_sync_outbox`, `_sync_state`, `_sync_meta`, `_sync_control`) migrate -at init. If your app owns the GRDB `DatabaseMigrator`, register HappySync's migrations into it so -both share one `grdb_migrations` table: - -```swift -var migrator = DatabaseMigrator() -SyncSchema.register(into: &migrator) // happysync_v1..v4 -// … your app migrations … -try migrator.migrate(databaseQueue) -``` - -The **write-capture triggers** are not part of that migrator. They're derived from your `tables` -manifest — source code, which ships without a schema version bump — so the engine (re)installs them -idempotently at every init and drops the ones for tables you've stopped syncing. - -## 4. Write - -Write GRDB **normally**. Each declared table carries `AFTER INSERT` / `UPDATE` / `DELETE` triggers -that append to the outbox in your transaction, so there is no second API to call and nothing to -forget. The engine notices the queued write and uploads promptly, so there's **no `syncNow()` needed -after a write** — a burst debounces into one drain pass. - -```swift -try await db.write { try recipe.save($0) } // PersistableRecord - -try await db.write { db in // partial / expression updates - try db.execute(sql: "UPDATE recipes SET title = ? WHERE id = ?", arguments: [title, id]) - try db.execute(sql: "UPDATE recipes SET cookCount = cookCount + 1 WHERE id = ?", arguments: [id]) -} - -try await db.write { db in // one transaction = one user intent - try recipe.save(db) - for ingredient in ingredients { try ingredient.save(db) } -} -``` - -That last one matters: the rows and their queued uploads commit together, or neither does. A crash -part-way through leaves no half-recipe *and* no half-batch in the outbox. - -> `engine.enqueue(op:table:row:)` still works but is **deprecated**. It now just performs the write -> and lets the trigger queue it. Replace `enqueue(.upsert, table: t, row: r)` with -> `try await db.write { try r.save($0) }`, and `enqueue(.delete, …)` with a plain `DELETE`. - -### Delete (with cascade) - -A plain `DELETE` removes the row locally and queues a tombstone. For a **parent** row, declare -`ON DELETE CASCADE` on the child foreign keys — SQLite then walks the graph and each removed child's -own trigger queues its tombstone, so you never delete children by hand and the UI never shows -orphans. - -```swift -// In your migration: -try db.create(table: "recipeIngredients") { t in - t.column("id", .text).primaryKey() - t.column("recipeId", .text).references("recipes", onDelete: .cascade) - // … -} - -// Then: -try await db.write { try $0.execute(sql: "DELETE FROM recipes WHERE id = ?", arguments: [recipeID]) } -``` - -A table that only `dependsOn` a parent without an FK action is **not** cascaded — `dependsOn` orders -uploads, it doesn't delete rows. Its orphans reconcile on the next pull via the server tombstone. - -## 5. Pull - -Downloads happen automatically (doorbell + periodic). Call `pullNow()` for app-driven nudges — -returning to the foreground, pull-to-refresh: - -```swift -try await engine.pullNow() -``` - -## 6. Show sync status - -Ask `isHealthy` — never `phase == .idle` alone. A settled pass with writes still failing or parked -is `.degraded`, so switching on the phase forces you to confront it: - -```swift -for await status in engine.status { - guard !status.isHealthy else { return showSynced() } - switch status.phase { - case .idle, .syncing: break // .idle here means healthy; .syncing is in flight - case .degraded: showPending(status.failedUploads + status.deadLetters) - case .failed(.authExpired): promptReauth() // classified — no string matching - case .failed(.network): break // recovers on its own; show nothing - case .failed(let failure): showBanner(failure) - } -} -``` - -See **[[Operations and Troubleshooting]]** for what `failedUploads` / `deadLetters` mean and how to -repair parked writes. - -## 7. Tear down on sign-out (order matters) - -`stop()` is **async** and awaits the in-flight pass. Always `await engine.stop()` **before** wiping -or replacing the database on sign-out / account switch — otherwise an in-flight pass could write to -the store you're about to delete, or upload the old user's rows with the new user's token. - -```swift -await engine.stop() // engine is quiesced here -try await wipeLocalDatabase() -``` - -Status subscriptions survive teardown: `stop()` broadcasts a settled `.idle` status but doesn't end -the streams, so a `for await status in engine.status` loop keeps delivering after the next `start()` -— no re-subscribing needed. - -## Next - -- **[[Server Setup]]** — required before the first successful sync. -- **[[Operations and Troubleshooting]]** — health, dead letters, schema evolution. -- **[[Claude Skill]]** — let your AI assistant wire this correctly for you. +- [Documentation home](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync) — the landing page, and the full symbol reference +- [[Home]] — this wiki's index diff --git a/wiki/Home.md b/wiki/Home.md index bc55f99..3c019a9 100644 --- a/wiki/Home.md +++ b/wiki/Home.md @@ -23,16 +23,22 @@ status, and retry/backoff. It does **not** own reads or schema. ## Start here +The developer documentation is **[built from the source](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync)** — one copy, next to the symbols it +describes. These wiki pages point at it: + | Page | What's on it | |---|---| -| **[[Getting Started]]** | Add the package, wire a `SyncEngine`, first write and pull. | -| **[[Server Setup]]** | The required Supabase schema: `updatedAt` trigger, `deletedAt` tombstones, RLS, Realtime. | -| **[[API Reference]]** | Every public type and method with signatures and field semantics. | -| **[[Operations and Troubleshooting]]** | Status UI, dead-letter repair, teardown, schema evolution, a troubleshooting table. | -| **[[Testing]]** | Test your sync integration offline with the shipped fakes — no Supabase project. | -| **[[Architecture]]** | How upload, download, LWW, the doorbell, and cascades fit together. | +| **[Getting Started](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/gettingstarted)** | Add the package, wire a `SyncEngine`, first write and pull. | +| **[Server Setup](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/serversetup)** | The required Supabase schema: `updatedAt` trigger, `deletedAt` tombstones, RLS, Realtime. | +| **[API Reference](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync)** | Every public type and method, generated from the source. | +| **[Operations and Troubleshooting](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/operations)** | Status UI, dead-letter repair, teardown, schema evolution, a troubleshooting table. | +| **[Testing](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/testingyourintegration)** | Test your sync integration offline with the shipped fakes — no Supabase project. | +| **[Architecture](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/architecture)** | How upload, download, LWW, the doorbell, and cascades fit together. | | **[[Claude Skill]]** | Install the HappySync Claude skill so your AI assistant integrates it correctly. | +Want to *see* it work first? [`Examples/`](https://github.com/happyface-studio/HappySync/tree/main/Examples) +is a two-table SwiftUI app wired to the in-memory fake — no Supabase project needed. + ## Requirements - Swift 6, iOS 16+ / macOS 13+ diff --git a/wiki/Operations-and-Troubleshooting.md b/wiki/Operations-and-Troubleshooting.md index c57f5db..e7f3cb2 100644 --- a/wiki/Operations-and-Troubleshooting.md +++ b/wiki/Operations-and-Troubleshooting.md @@ -1,139 +1,14 @@ # Operations and Troubleshooting -## Sync-status UI +**This page has moved into the API documentation**, where it sits alongside the symbols it talks +about — and where there is one copy to maintain instead of two that drift apart: -Consume `engine.status` and ask `isHealthy`. `.idle` means settled **and** clean; a pass that -completed while writes are still failing or parked settles as `.degraded`. +### → [Operations and Troubleshooting](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/operations) -```swift -for await status in engine.status { - switch status.phase { - case .idle: showSynced() // settled and clean - case .syncing: showSpinner() - case .degraded: showPending(status.failedUploads + status.deadLetters) - case .failed(let failure): showBanner(failure) // classified — see below - } -} -``` +The source is [`Sources/HappySync/Documentation.docc/Operations.md`](https://github.com/happyface-studio/HappySync/blob/main/Sources/HappySync/Documentation.docc/Operations.md); the page above is +what Swift Package Index builds from it. Everything that used to be here is there, unabridged. -- `failedUploads > 0` → some local changes haven't reached the server yet (degraded, still - retrying with backoff). -- `deadLetters > 0` → some writes parked and need inspection/repair. +--- -## Failure model - -- **Permanent** (constraint `23xxx`, RLS `42501`, undefined-column `42703`, other 4xx) → - dead-lettered immediately. -- **Transient** (network, 5xx, 408/429, transient Postgres `40001`/`40P01`/`53xxx`/`08xxx`, unknown - codes) → retried with per-entry exponential backoff (jittered) until a cap, then dead-lettered. -- **Auth-shaped** (401/403, `PGRST301`/`PGRST302`) → transient **and exempt from the retry budget**, - so an expired-token stretch never dead-letters the whole outbox; it recovers when the token - refreshes out-of-band. - -A dead-lettered entry stops retrying **and** stops counting as a dirty row, so it never permanently -blocks downloads for its key. - -### What the app sees - -That classification is surfaced as `SyncFailure`, on `Phase.failed` and on `DeadLetter.failure`. -It answers *what to show the user* — the retry decision is already made by then, and is visible as -`failedUploads` (still retrying) vs `deadLetters` (parked): - -| `SyncFailure` | Raised by | What the app should do | -|---|---|---| -| `.network` | `URLError`, Postgres `08xxx` | Nothing — it recovers | -| `.authExpired` | 401/403, `PGRST301`/`PGRST302` | Nothing at first; prompt re-auth if it persists | -| `.permissionDenied` | `42501` | Surface it — a policy bug or a real permissions problem | -| `.constraintViolation(code:)` | `23xxx` | Inspect the row; the write will never land as-is | -| `.schemaMismatch(column:)` | `42703`, `PGRST204` | Prompt an app update (or restore the server column) | -| `.server(status:)` | any other HTTP status, incl. 5xx | Nothing — retried; surface if it persists | -| `.other(String)` | anything unrecognised, transient Postgres states | Log the text | - -`.network` means the request got no answer at all; a server that answered with a 5xx is -`.server(status:)`, so the status code isn't thrown away. - -The kind and the retry decision agree everywhere except `PGRST204`: it classifies as -`.schemaMismatch` but is **retried** rather than parked immediately, because PostgREST also raises it -from a stale schema cache, which clears on its own. - -Every case renders via `CustomStringConvertible`, so `"\(failure)"` is safe to put in a banner. The -classified cases carry no server prose of their own — the full text lives in the engine's log, and on -`DeadLetter.lastError` for a parked write. - -## Repairing dead letters - -```swift -let parked = try await engine.deadLetters() -for letter in parked { - switch letter.failure { - case .permissionDenied: reportPolicyBug(letter) // fix RLS, then retryDeadLetters() - case .schemaMismatch(let column): promptAppUpdate(column) - case .constraintViolation(let code): inspect(letter, code) - default: log("\(letter.op) \(letter.table)/\(letter.pk): \(letter.lastError ?? "unknown")") - } -} - -// After fixing the cause (RLS/policy change, schema migration, app update): -try await engine.retryDeadLetters() // all, or pass [seq] for specific entries - -// Or abandon the local write and accept the server's version — discard drops the entries and -// re-pulls the affected rows so local converges to the server (including changes missed while parked): -try await engine.discardDeadLetters([badSeq]) // specific seqs, or omit for all -``` - -Both mutations refresh the status stream immediately, so `deadLetters` drops as soon as they return. - -## Teardown / account switch (order matters) - -`stop()` is async and awaits the in-flight pass. A consumer that wipes or replaces the database on -sign-out / account switch **must `await engine.stop()` before touching the DB file** — otherwise an -in-flight pass could write to the store you're about to delete, or (mid-account-switch) upload the -old user's rows with the new user's token. - -```swift -await engine.stop() // quiesced here -try await wipeLocalDatabase() -``` - -`stop()` settles the status stream (`.idle`, keeping the outbox counts) but leaves subscriptions -open, so the sync-status UI resumes updating on the next `start()` without being re-subscribed. - -## Stale-cursor full resync - -A device offline past `maxOfflineGap` (default 30 days) can't trust its cursors — a tombstone it -never saw may already be purged. Once `now − lastSyncedAt > maxOfflineGap` the engine clears -cursors, re-pulls, and reconciles purged deletes instead of resurrecting them. Keep `maxOfflineGap` -**≤ the server's tombstone-purge retention** (see [[Server Setup]]). For a `scopeColumn` table the -resync is deferred until a partition value resolves, so it never wipes an un-pulled table. - -## Schema evolution (staggered clients) - -Shipped clients and the server are **not lockstep** — App Store review + staggered updates mean old -builds run against a newer server for weeks. The engine tolerates drift both ways, but follow these -rules so any single migration is safe for every client in the field: - -- **Additive-only.** Never drop-and-recreate or repurpose a column in place. -- **Never rename in place.** Add the new name, dual-write server-side, wait for adoption, then - remove the old name by the removal rule. -- **Adding a column — server-first.** Deploy it (nullable/default) before any client reads/writes - it. Old clients drop the unknown column on download and don't send it on upload. -- **Removing a column — client-first.** Drop it server-side **only after no shipped client still - writes it**: (1) ship a build that stops sending it — optionally declaring `serverColumns` so the - payload is intersected immediately; (2) wait out the update-lag window; (3) then drop it. Skipping - step 2 dead-letters every write from clients still sending the column. - -## Troubleshooting - -| Symptom | Likely cause | Fix | -|---|---|---| -| Writes never reach the server; `deadLetters` climbing | RLS rejects (`42501`) or a constraint violation | `deadLetters()` → switch on `failure` (`lastError` for the raw text); fix the policy/data, then `retryDeadLetters()`. | -| A device never sees the server's version of its **own** write | Missing `updatedAt` trigger (the engine already strips the client's `updatedAt` from every upload, so a stale client value can't be the cause) | Add the server trigger. | -| Rows on an insert-only table never download to other devices | The table's `cursorColumn` has no server default — the engine never uploads it, so it lands null | `default now()` (or a trigger) on that column server-side. | -| Whole public catalog downloads to every device | RLS broader than the partition, no `scopeColumn` | Declare `scopeColumn` + supply the engine `scope:` closure. | -| A fresh-pk insert 409s forever | Row already exists under a secondary `UNIQUE` | Declare `conflictColumns` (leaf tables only). | -| Every write on a table dead-letters after a server migration | Dropped/renamed server column (`PGRST204`) | Restore the column until clients stop sending it; or declare `serverColumns`. | -| Deleted parent leaves orphaned children in the UI | Child FK not declared `ON DELETE CASCADE` locally, or missing server child-tombstone trigger | Declare `ON DELETE CASCADE` on the child FKs (`dependsOn` alone deletes nothing) and add the server child-tombstone trigger. | -| `SyncEngine.init` throws `invalidManifest` | A declared `SyncTable` disagrees with the local schema — a table or column that isn't there, a duplicate declaration, `dependsOn` naming an undeclared table, `conflictColumns` on a table something foreign-keys, or `serverColumns` without the primary key | Read the message: it names the table, the field and the value. Run your app's migrations **before** constructing the engine, then fix the manifest. It throws rather than silently syncing the wrong thing — every one of these used to be invisible until the damage showed up on another device. | -| Some writes sync, one code path never does | A write that bypasses the declared tables — writing a table not in the manifest, or a `DatabasePool` reader-side hack | Add the table to the manifest. Writes to declared tables are captured by triggers, so there is no way to "forget" to enqueue one. | -| Offline-long device resurrects deleted rows | `maxOfflineGap` > server tombstone retention | Set `maxOfflineGap` ≤ retention. | -| Idle status shows healthy but changes aren't syncing | Reading `phase == .idle` as health | Ask `status.isHealthy`. The engine now settles such a pass as `.degraded`, so a `switch` over the phase surfaces it. | +- [Documentation home](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync) — the landing page, and the full symbol reference +- [[Home]] — this wiki's index diff --git a/wiki/README.md b/wiki/README.md index 12a003a..04e5eaa 100644 --- a/wiki/README.md +++ b/wiki/README.md @@ -7,17 +7,23 @@ GitHub resolves `Getting Started` → `Getting-Started.md`, so keep filenames hy ## Pages -| File | Wiki page | -|---|---| -| `Home.md` | Landing page | -| `Getting-Started.md` | Getting Started | -| `Server-Setup.md` | Server Setup | -| `API-Reference.md` | API Reference | -| `Operations-and-Troubleshooting.md` | Operations and Troubleshooting | -| `Testing.md` | Testing | -| `Architecture.md` | Architecture | -| `Claude-Skill.md` | Claude Skill | -| `_Sidebar.md` | Sidebar nav (all pages) | +**The developer documentation lives in DocC**, not here — `Sources/HappySync/Documentation.docc/`, +published by [Swift Package Index](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync) +and readable in Xcode. It sits next to the symbols it documents, which is the only arrangement that +keeps the two from drifting (issue #57). The pages below that used to hold that content are now +one-paragraph pointers to it; edit the DocC article instead. + +| File | Wiki page | | +|---|---|---| +| `Home.md` | Landing page | index | +| `Claude-Skill.md` | Claude Skill | the only full article left here | +| `_Sidebar.md` | Sidebar nav | index | +| `Getting-Started.md` | Getting Started | → DocC | +| `Server-Setup.md` | Server Setup | → DocC | +| `API-Reference.md` | API Reference | → DocC | +| `Operations-and-Troubleshooting.md` | Operations and Troubleshooting | → DocC | +| `Testing.md` | Testing | → DocC | +| `Architecture.md` | Architecture | → DocC | ## Publish diff --git a/wiki/Server-Setup.md b/wiki/Server-Setup.md index 86cdb9e..ce1c313 100644 --- a/wiki/Server-Setup.md +++ b/wiki/Server-Setup.md @@ -1,135 +1,14 @@ # Server Setup -Every synced table needs four things on the Supabase side. These are **non-negotiable** — LWW -compares a *server* clock, and deletes, scoping, and live updates all depend on this contract. The -engine will not paper over a server that violates it. +**This page has moved into the API documentation**, where it sits alongside the symbols it talks +about — and where there is one copy to maintain instead of two that drift apart: -Run the SQL below per synced table (swap `recipes` for your table). +### → [Server Setup](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/serversetup) -## 1. `updatedAt` — server-stamped change time +The source is [`Sources/HappySync/Documentation.docc/ServerSetup.md`](https://github.com/happyface-studio/HappySync/blob/main/Sources/HappySync/Documentation.docc/ServerSetup.md); the page above is +what Swift Package Index builds from it. Everything that used to be here is there, unabridged. -LWW compares `updatedAt`, and a *server* clock is what stops two devices' clock skew from silently -losing writes. **Never trust a client-sent `updatedAt`.** The engine holds up its half unasked: a -table's `cursorColumn` is stripped from every upload payload, so no HappySync client ever sends one -— you don't declare anything for this. The trigger below is still required (it's what advances the -column on update, and what the cursor pull depends on); the stripping just means a table that lost -its trigger can't quietly promote a device's clock to the ordering authority. +--- -```sql -alter table public.recipes - add column if not exists "updatedAt" timestamptz not null default now(); - -create or replace function public.stamp_updated_at() -returns trigger language plpgsql as $$ -begin - new."updatedAt" := now(); -- server clock, always; ignore any client value - return new; -end $$; - -drop trigger if exists stamp_updated_at on public.recipes; -create trigger stamp_updated_at - before insert or update on public.recipes - for each row execute function public.stamp_updated_at(); -``` - -> An **insert-only / immutable** table needs no update trigger. Give it a monotonic stamp column -> (e.g. `translatedAt`) and set that table's `cursorColumn` in its `SyncTable` descriptor. Because -> the engine never uploads the cursor column, that column must be `default now()` server-side — -> otherwise inserts land with a null stamp the cursor can't order. - -## 2. `deletedAt` — soft-delete tombstones - -Deletes are **soft**: set `deletedAt` instead of removing the row, so deletions propagate on the -next cursor pull. Deleting a parent must **tombstone its children too** — use a trigger, *not* -`ON DELETE CASCADE` (a hard cascade leaves no tombstone for other devices to pull). - -```sql -alter table public.recipes - add column if not exists "deletedAt" timestamptz; - -create or replace function public.tombstone_recipe_children() -returns trigger language plpgsql as $$ -begin - if new."deletedAt" is not null and old."deletedAt" is null then - update public."recipeIngredients" - set "deletedAt" = new."deletedAt" - where "recipeId" = new.id and "deletedAt" is null; - -- repeat per child table - end if; - return new; -end $$; - -drop trigger if exists tombstone_recipe_children on public.recipes; -create trigger tombstone_recipe_children - after update on public.recipes - for each row execute function public.tombstone_recipe_children(); -``` - -Locally, declare the child foreign keys `ON DELETE CASCADE` in your **SQLite** schema: deleting a -parent then removes its children in the same transaction and each child's capture trigger queues its -tombstone, keeping the local and server deleted sets symmetric with no orphan window. The server -trigger above is what makes *other* devices converge. - -### Tombstone purge and the `maxOfflineGap` invariant - -Purge old tombstones on a schedule, but respect this **invariant**: the purge retention must exceed -the engine's `maxOfflineGap` (default 30 days). A device offline longer than the retention would -never see a purged tombstone and would resurrect the row. Set `maxOfflineGap` **≤ your server's -retention**. - -```sql --- e.g. a pg_cron job; retention (90d) > engine maxOfflineGap (30d default) -delete from public.recipes where "deletedAt" < now() - interval '90 days'; -``` - -## 3. RLS scoped to `auth.uid()` - -RLS is the **security boundary**. Enable it and add policies keyed on `auth.uid()`. - -```sql -alter table public.recipes enable row level security; - -create policy "own rows: select" on public.recipes - for select using ("userId" = auth.uid()); -create policy "own rows: write" on public.recipes - for all using ("userId" = auth.uid()) with check ("userId" = auth.uid()); -``` - -**RLS is not necessarily the sync partition.** If a policy is deliberately broader than what a -device should download — e.g. `isPublic = true OR userId = auth.uid()` — an unfiltered pull drags -the whole public catalog to every device. Declare a `scopeColumn` (e.g. `userId`) on the `SyncTable` -and supply the engine's `scope:` closure; the engine filters the pull **and** the doorbell to -`scopeColumn = `. Omit `scopeColumn` when RLS already equals the partition. - -## 4. Realtime publication (the doorbell) - -```sql -alter publication supabase_realtime add table public.recipes; -``` - -Realtime is a **doorbell only** — an event triggers a debounced `pullNow()`; payloads are never -applied directly. Sync converges without it; Realtime just makes it feel instant. - -## Field-mapping conventions - -| Concern | Convention | -|---|---| -| Column names | **camelCase, identical** in SQLite and Postgres — no snake_case mapping. | -| Dates | ISO-8601 with fractional seconds; the LWW gate compares instants at **microsecond** precision. | -| UUID | text locally ↔ `uuid` in Postgres. | -| Bool | integer `0/1` locally ↔ `boolean`. | -| Enum | `rawValue` string. | -| JSON columns | JSON **text** locally ↔ `json`/`jsonb`; declare in `jsonColumns`. | -| Server-owned | RPC-managed columns; declare in `serverOwnedColumns`, never in an upsert. The `cursorColumn` is server-owned too and the engine strips it with nothing declared. | - -## Checklist - -- [ ] `updatedAt` + `BEFORE INSERT/UPDATE` trigger on every table (`default now()` on an - insert-only table's stamp column — the client never sends it). -- [ ] `deletedAt` on every table + a child-tombstone trigger on every parent. -- [ ] Tombstone purge scheduled, retention **>** engine `maxOfflineGap`. -- [ ] RLS enabled + `auth.uid()` policies on every table. -- [ ] `scopeColumn` declared for any table whose RLS is broader than the partition. -- [ ] Every synced table in the `supabase_realtime` publication. - -See **[[Operations and Troubleshooting]] → Schema evolution** before you change a live schema. +- [Documentation home](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync) — the landing page, and the full symbol reference +- [[Home]] — this wiki's index diff --git a/wiki/Testing.md b/wiki/Testing.md index e807b62..b2f743f 100644 --- a/wiki/Testing.md +++ b/wiki/Testing.md @@ -1,225 +1,14 @@ # Testing -Sync is the part of an app where bugs are silent and data-losing, and it is usually the part with no -tests — because testing it seems to require a live Supabase project. +**This page has moved into the API documentation**, where it sits alongside the symbols it talks +about — and where there is one copy to maintain instead of two that drift apart: -It doesn't. `SyncRemote` and `SyncDoorbell` are the seams the engine's own 2,700 lines of tests run -on, and both are public. The `HappySyncTestSupport` product ships the fakes, so a test can answer -questions like *does deleting a recipe queue tombstones for all its children?* in milliseconds, with -no network. +### → [Testing](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/testingyourintegration) -## Add the test-support product +The source is [`Sources/HappySync/Documentation.docc/TestingYourIntegration.md`](https://github.com/happyface-studio/HappySync/blob/main/Sources/HappySync/Documentation.docc/TestingYourIntegration.md); the page above is +what Swift Package Index builds from it. Everything that used to be here is there, unabridged. -Add it to your **test** target only — it contains no production code: +--- -```swift -.testTarget( - name: "MyAppTests", - dependencies: [ - "MyApp", - .product(name: "HappySync", package: "HappySync"), - .product(name: "HappySyncTestSupport", package: "HappySync"), - ] -), -``` - -## Build an engine with no Supabase - -`SyncEngine.forTesting(db:remote:tables:…)` is the production engine with the network seams -injected. Everything else — drain, pull, retry classification, dead-lettering, cascade ordering — is -the code that ships, so what a test proves here is what the app does. - -```swift -import HappySync -import HappySyncTestSupport - -let db = try DatabaseQueue() -try MyApp.migrator.migrate(db) // your migrations, as at launch - -let remote = InMemorySyncRemote() -let engine = try SyncEngine.forTesting( - db: db, - remote: remote, - tables: MyApp.syncTables, // the real manifest — a typo in it fails here, too - pollInterval: 999, // only the passes the test triggers - debounceInterval: 0.02 -) -``` - -The parameters worth knowing: `pageSize` (shrink to `1` to force pagination), `pollInterval` (raise -to silence the background poll), `debounceInterval` (shrink so a doorbell ring converges promptly), -`deadLetterAfter` (lower to reach the parked state without driving eight retries), and `scope` (the -partition value, to test a user switch). - -## `InMemorySyncRemote` - -A server that lives in the test process. It records every upload, serves downloads from a seedable -per-table row store honouring the `(cursorColumn, primaryKey)` tuple cursor and the page limit, and -can be told to fail. - -```swift -// What the server holds — rows are [String: AnyJSON], exactly what PostgREST returns. -let remote = InMemorySyncRemote(dataset: [ - "recipes": [["id": "r1", "title": "Soup", "updatedAt": "2026-06-30T12:00:00.000Z"]], -]) - -// What the client sent. -await remote.upsertCalls // [(table, row, onConflict)] in order — one entry per *row* -await remote.deleteCalls // [(table, pk)] — the tombstones -await remote.upsertRequests // round trips, which batching makes fewer than upsertCalls.count (#54) -await remote.deleteRequests // …and the same for tombstones -await remote.fetchCalls // download pages requested; the usual "a pass ran" signal -await remote.lastScope // the partition filter the engine resolved - -// Publish a change "another device" made, between two pulls. -await remote.seed("recipes", rows: [updatedRow]) -``` - -### Failure injection - -The one knob that matters is whether the drain **retries** the write or **parks** it: - -```swift -InMemorySyncRemote(failUpserts: 1, upsertFailure: .transient) // retried, then succeeds -InMemorySyncRemote(failUpserts: 1, upsertFailure: .permanent) // dead-lettered on the first attempt -InMemorySyncRemote(failUpserts: 1, upsertFailure: .error(HTTPError(…))) // real error, real classification -``` - -`failUpserts` counts **requests**, not rows: the drain batches a run of same-table writes into one -call, so a rejected batch spends one — and the per-row re-run it triggers spends one per row. - -`.error` wraps a genuine transport error (`HTTPError`, `PostgrestError`, `URLError`) with the -classification the production remote applies — so *"an RLS reject parks, an expired token doesn't"* -is asserted against the shipped table rather than a hand-picked answer. `failFetches` / -`fetchFailure` do the same for downloads, which drives a whole pass to `.failed(SyncFailure)`. - -Use `99` for "always", and `failNextUpserts(_:with:)` / `failNextFetches(_:with:)` to change the -injection mid-test — e.g. fail, assert parked, then let the retry succeed. - -### Simulating server-side computation - -`representation` maps an uploaded row to the row the server returns. The drain writes the whole -representation back locally, so this is how you test server-normalized values, defaulted columns, or -a `conflictColumns` merge: - -```swift -let remote = InMemorySyncRemote(representation: { row in - var server = row - server["cookedCount"] = .integer(7) // the merged server total, not what this client sent - server["updatedAt"] = .string("2026-06-30T12:00:00.000Z") - return server -}) -``` - -### Holding a call in flight - -`onUpsert` / `onFetch` run at the start of each call with its 1-based number. Because the remote -suspends there, this is how a test races a local write against an in-flight upload, or stops the -engine mid-pull: - -```swift -let started = AsyncSignal(), gate = AsyncSignal() -await remote.onUpsert { _ in await started.fire(); await gate.wait() } - -let pass = Task { await engine.syncNow() } -await started.wait() // the upload is blocked in the remote -try await db.write { … } // a competing local edit lands behind it -await gate.fire() // let the upload finish -``` - -## `ManualDoorbell` - -The Realtime channel, made deterministic — it rings only when you say so. - -```swift -let doorbell = ManualDoorbell() -let engine = try SyncEngine.forTesting(db: db, remote: remote, tables: tables, doorbell: doorbell) -await engine.start() - -doorbell.fire() // a remote change lands -#expect(await eventually { await remote.fetchCalls >= 2 }) - -doorbell.ringScopes // [nil, "u1"] proves a sign-in re-scoped the subscription -doorbell.liveSubscriptions // settles to 1 — the old subscription was torn down -doorbell.fire(subscription: 0) // ring a torn-down one; it must not poke the engine -``` - -`SilentDoorbell` (the default) never rings, so convergence comes only from the periodic poll and the -passes you trigger. - -## Async helpers - -`eventually` polls until a condition holds or the timeout elapses. Reach for it any time the engine -converges in the background — a fixed sleep plus one assertion is the alternative, and it flakes -under a loaded CI runner: - -```swift -#expect(await eventually { await remote.fetchCalls >= 1 }) -``` - -For a *negative* assertion ("no further pull happened"), poll until the count stops moving, then -assert it doesn't move again. - -`AsyncSignal` is a one-shot rendezvous — `fire()` releases every `wait()`, now and later, so a wait -can't miss a signal that arrived first. - -## Three tests worth having - -These are the ones that catch real bugs, and none of them need a network. - -**A cascade delete queues every child's tombstone.** The app deletes one row; SQLite cascades and -each child's capture trigger queues its own tombstone. If a child table is missing from your -manifest, this test fails and the field doesn't. - -```swift -try await db.write { try $0.execute(sql: "DELETE FROM recipes WHERE id = 'r1'") } -#expect(await eventually { await remote.deleteCalls.count == 3 }) -``` - -**The status UI shows degraded when a write parks.** `.degraded` is the case everyone gets wrong: the -pass completed, so nothing *looks* broken, but the user's write never reached the server. Assert on -`isHealthy`, never on the phase alone. - -```swift -let remote = InMemorySyncRemote(failUpserts: 99, upsertFailure: .permanent) -… -#expect(await eventually { await latestStatus(engine)?.phase == .degraded }) -#expect(try await engine.deadLetters().count == 1) -``` - -**Sign-out quiesces the engine before the wipe.** After `await engine.stop()` returns, nothing may -still be in flight — a late pull writes into the store you're about to delete, and mid-account-switch -uploads the previous user's rows under the new user's token. - -```swift -await engine.stop() -let atStop = await remote.fetchCalls -doorbell.fire() // a Realtime event racing the sign-out -try await Task.sleep(for: .milliseconds(200)) -#expect(await remote.fetchCalls == atStop) // nothing restarted -``` - -A worked version of all three lives in -[`Tests/HappySyncTests/PublicSeamsTests.swift`](https://github.com/happyface-studio/HappySync/blob/main/Tests/HappySyncTests/PublicSeamsTests.swift), -which imports HappySync the way your app does — no `@testable` — so it stops compiling if these -seams ever regress. - -## Writing your own remote - -`SyncRemote` and `SyncDoorbell` are protocols, so a bespoke double is three methods. Signal whether a -failure is retryable by conforming your error to `ClassifiedSyncError`, or wrap a real transport -error with `RemoteFailure(classifying:)` to get the production classification: - -```swift -struct RejectingRemote: SyncRemote { - func upsert(table: String, row: [String: AnyJSON], onConflict: String?) async throws -> [String: AnyJSON] { - throw RemoteFailure(classifying: PostgrestError(code: "42501", message: "rls")) - } - func delete(table: String, primaryKey: String, pk: String) async throws {} - func fetch(table: String, cursorColumn: String, since cursor: SyncCursor?, primaryKey: String, - scope: ScopeFilter?, limit: Int) async throws -> [[String: AnyJSON]] { [] } -} -``` - -An error the engine can't classify is treated as **transient** — safer to retry than to silently drop -a user's write. +- [Documentation home](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync) — the landing page, and the full symbol reference +- [[Home]] — this wiki's index diff --git a/wiki/_Sidebar.md b/wiki/_Sidebar.md index aa29143..eb59999 100644 --- a/wiki/_Sidebar.md +++ b/wiki/_Sidebar.md @@ -1,16 +1,12 @@ ### HappySync - [[Home]] -- [[Getting Started]] -- [[Server Setup]] -- [[API Reference]] -- [[Operations and Troubleshooting]] -- [[Testing]] -- [[Architecture]] +- [Documentation](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync) — Getting Started, Server Setup, API, Operations, Testing, Architecture - [[Claude Skill]] --- +- [Example app](https://github.com/happyface-studio/HappySync/tree/main/Examples) - [Releases](https://github.com/happyface-studio/HappySync/releases) - [Sync contract](https://github.com/happyface-studio/HappySync/blob/main/docs/SYNC-CONTRACT.md) - [Source](https://github.com/happyface-studio/HappySync) From 959b780fda6645f9d454a635f38099ab4bcfb910 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 31 Jul 2026 17:40:30 +0000 Subject: [PATCH 4/5] Declare the manifest from GRDB record types (#50) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `SyncTable` was eight stringly-typed parameters, so a rename in a Swift record or a GRDB migration left the manifest stale with no build error. #49 catches that at launch; this catches it at compile time, which is the difference between a crash on a tester's device and a red squiggle. `SyncTable.table(Recipe.self, …)` reads the table name off the record's `databaseTableName`, and `dependsOn:` takes record types rather than names. It returns the same `SyncTable` the string initializer does — validation, trigger install and the drain are one code path, so this is a spelling choice rather than a mode, and a manifest can mix both forms. The string form stays for tables with no Swift record type. Columns stay as checked as the record makes them: GRDB's conventional `enum Columns` ties each name to a single declaration, while a bare `Column("nutrition")` is exactly as unchecked as the string was. Key paths can't give column names without a `CodingKeys` bridge the app would have to write, so how much checking to buy stays the consumer's choice — and either way the launch check against the live schema still runs, since a record can be ahead of the migration that creates its table. Deliberately not included: `engine.save(_:)` / `engine.delete(_:)`. Since #48 made writes plain GRDB, a typed write API would re-introduce the parallel write path the trigger capture removed — and #56 dropped `delete(table:pk:)` for the same reason. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015y9Y9kec3hj8S35oLQXxvQ --- .claude/skills/happysync/references/api.md | 13 +++ README.md | 14 +++ .../Documentation.docc/GettingStarted.md | 19 +++ Sources/HappySync/TypedManifest.swift | 64 ++++++++++ Tests/HappySyncTests/TypedManifestTests.swift | 110 ++++++++++++++++++ 5 files changed, 220 insertions(+) create mode 100644 Sources/HappySync/TypedManifest.swift create mode 100644 Tests/HappySyncTests/TypedManifestTests.swift diff --git a/.claude/skills/happysync/references/api.md b/.claude/skills/happysync/references/api.md index 3d7d2e1..5fd6095 100644 --- a/.claude/skills/happysync/references/api.md +++ b/.claude/skills/happysync/references/api.md @@ -61,6 +61,19 @@ public init( ) ``` +Declared from a GRDB record type instead, so a rename is a build error rather than a table that +quietly stops syncing (#50). Same `SyncTable`, so a manifest can mix both forms — keep the string +form for a table with no Swift record type: + +```swift +SyncTable.table(Recipe.self, jsonColumns: [Recipe.Columns.nutrition]) // name from databaseTableName +SyncTable.table(RecipeStep.self, dependsOn: [Recipe.self]) // dependency as a type +``` + +Columns are as checked as the record makes them: GRDB's conventional `enum Columns` ties each name +to one declaration; a bare `Column("nutrition")` is exactly as unchecked as the string was. Either +way `SyncEngine.init` still validates every name against the live schema. + Field gotchas: - **`dependsOn`** is normally left alone: `nil` (the default) means the engine reads diff --git a/README.md b/README.md index 7ad5717..6ce9311 100644 --- a/README.md +++ b/README.md @@ -74,6 +74,20 @@ for await status in engine.status { } ``` +If your tables have GRDB record types, declare the manifest from those and the table names stop being +strings that can go stale — a rename becomes a build error rather than a table that quietly stops +syncing: + +```swift +tables: [ + SyncTable.table(Recipe.self, jsonColumns: [Recipe.Columns.nutrition]), + SyncTable.table(RecipeIngredient.self), // dependsOn still derived from the schema's FKs +] +``` + +Both forms produce the same `SyncTable`, so a manifest can mix them — keep the string form for a +table with no Swift record type. + For a table whose RLS is broader than the sync partition (e.g. a shared `recipes` table readable as `isPublic OR userId = auth.uid()`), declare a `scopeColumn` and supply the partition value so the engine downloads only the user's rows instead of the whole catalog: diff --git a/Sources/HappySync/Documentation.docc/GettingStarted.md b/Sources/HappySync/Documentation.docc/GettingStarted.md index b3cbe76..c7d3b4c 100644 --- a/Sources/HappySync/Documentation.docc/GettingStarted.md +++ b/Sources/HappySync/Documentation.docc/GettingStarted.md @@ -52,6 +52,25 @@ let tables = [ `serverOwnedColumns`, `serverColumns`, `cursorColumn` — including when to reach for each and what breaks if you get it wrong. +### Declaring them from your record types + +If your tables have GRDB record types, declare the manifest from those instead and the table names +stop being strings you can mistype or forget to update: + +```swift +let tables = [ + SyncTable.table(Recipe.self, jsonColumns: [Recipe.Columns.nutrition]), + SyncTable.table(RecipeIngredient.self), // dependsOn still derived from the FK + SyncTable.table(RecipeStep.self, dependsOn: [Recipe.self]), +] +``` + +Renaming a record's `databaseTableName` is now a build error at the manifest rather than a table +that quietly stops syncing. Column names are as checked as your record makes them: GRDB's +conventional `enum Columns` ties each one to a single declaration, while a bare `Column("nutrition")` +is exactly as unchecked as the string was. Both forms produce the same ``SyncTable``, so mixing them +in one manifest is fine — reach for the string form for a table with no Swift record type. + ## 3. Create and start the engine ```swift diff --git a/Sources/HappySync/TypedManifest.swift b/Sources/HappySync/TypedManifest.swift new file mode 100644 index 0000000..28491a8 --- /dev/null +++ b/Sources/HappySync/TypedManifest.swift @@ -0,0 +1,64 @@ +import Foundation +import GRDB + +// Issue #50: the manifest was eight stringly-typed parameters, so a rename in a GRDB migration or a +// Swift record left it stale with no build error. `SyncEngine.init` catches that at launch (#49); +// this catches it at compile time, which is the difference between a crash on the tester's device +// and a red squiggle. + +public extension SyncTable { + /// Declares a synced table from its **GRDB record type**, reading the table name off the record + /// instead of restating it as a string. + /// + /// ```swift + /// let tables = [ + /// SyncTable.table(Recipe.self, jsonColumns: [Recipe.Columns.nutrition]), + /// SyncTable.table(RecipeIngredient.self), // dependsOn derived from the FK (issue #49) + /// ] + /// ``` + /// + /// Renaming the record's table — in the record or in the migration that created it — is now a + /// build error at the manifest rather than a table that silently stops syncing. The same holds + /// for `dependsOn`, which takes record types rather than names. + /// + /// **Columns are as checked as your record makes them.** Passing GRDB's conventional + /// `enum Columns` (`Recipe.Columns.nutrition`) ties each name to a declaration you rename in one + /// place; passing a bare `Column("nutrition")` is exactly as unchecked as the string form was, and + /// still validated against the schema at init. Column names can't be derived from key paths + /// without a `CodingKeys` bridge the app would have to write anyway, so the choice of how much + /// checking to buy stays with the consumer. + /// + /// Every parameter means what it means on ``SyncTable/init(name:primaryKey:cursorColumn:dependsOn:jsonColumns:serverOwnedColumns:scopeColumn:conflictColumns:serverColumns:)`` + /// — this is that initializer with the strings replaced, not a second set of semantics. The + /// string form stays for tables with no Swift record type (a join table written only in SQL, or + /// one owned by another module). + /// + /// - Parameters: + /// - record: The GRDB record whose `databaseTableName` names the table. + /// - dependsOn: Records this one references. `nil` (the default) derives the dependency from + /// the schema's own foreign keys — prefer it; pass a list only for a *logical* parent the + /// schema has no FK for. + static func table( + _ record: Record.Type, + primaryKey: any ColumnExpression = Column("id"), + cursorColumn: any ColumnExpression = Column("updatedAt"), + dependsOn: [any TableRecord.Type]? = nil, + jsonColumns: [any ColumnExpression] = [], + serverOwnedColumns: [any ColumnExpression] = [], + scopeColumn: (any ColumnExpression)? = nil, + conflictColumns: [any ColumnExpression] = [], + serverColumns: [any ColumnExpression] = [] + ) -> SyncTable { + SyncTable( + name: Record.databaseTableName, + primaryKey: primaryKey.name, + cursorColumn: cursorColumn.name, + dependsOn: dependsOn.map { records in records.map { $0.databaseTableName } }, + jsonColumns: jsonColumns.map { $0.name }, + serverOwnedColumns: serverOwnedColumns.map { $0.name }, + scopeColumn: scopeColumn.map { $0.name }, + conflictColumns: conflictColumns.map { $0.name }, + serverColumns: serverColumns.map { $0.name } + ) + } +} diff --git a/Tests/HappySyncTests/TypedManifestTests.swift b/Tests/HappySyncTests/TypedManifestTests.swift new file mode 100644 index 0000000..06e4c7c --- /dev/null +++ b/Tests/HappySyncTests/TypedManifestTests.swift @@ -0,0 +1,110 @@ +import Testing +import Foundation +import GRDB +import HappySyncTestSupport +@testable import HappySync + +// Issue #50: `SyncTable.table(Record.self, …)` reads the table name off the GRDB record, so a rename +// is a build error at the manifest rather than a table that quietly stops syncing. It produces the +// same `SyncTable` the string form does — these tests hold that equivalence down, because the moment +// the two diverge the typed form becomes a second set of semantics to learn. + +// MARK: - Fixtures + +/// A record with GRDB's conventional `Columns` namespace — passing those instead of bare `Column`s is +/// what buys the column names their compile-time check. +private struct Recipe: TableRecord { + static let databaseTableName = "recipes" + + enum Columns { + static let id = Column("id") + static let nutrition = Column("nutrition") + static let userId = Column("userId") + } +} + +private struct RecipeIngredient: TableRecord { + static let databaseTableName = "recipeIngredients" + + enum Columns { + static let amount = Column("amount") + static let userId = Column("userId") + } +} + +/// A record whose table the schema doesn't have — the case compile-time checking can't catch. +private struct Ghost: TableRecord { + static let databaseTableName = "ghost" +} + +// MARK: - Tests + +@Test func typedTableReadsItsNameFromTheRecord() { + let table = SyncTable.table(Recipe.self) + + #expect(table.name == "recipes") + #expect(table.primaryKey == "id") + #expect(table.cursorColumn == "updatedAt") + #expect(table.dependsOn == nil) // nil still means "derive it from the schema's FKs" (issue #49) +} + +@Test func typedTableTakesColumnsAndDependenciesAsDeclarations() { + let table = SyncTable.table( + RecipeIngredient.self, + dependsOn: [Recipe.self], + jsonColumns: [RecipeIngredient.Columns.amount], + scopeColumn: RecipeIngredient.Columns.userId + ) + + #expect(table.name == "recipeIngredients") + #expect(table.dependsOn == ["recipes"]) // a record type, not a string that can rot + #expect(table.jsonColumns == ["amount"]) + #expect(table.scopeColumn == "userId") +} + +@Test func typedAndStringManifestsProduceTheSameTable() { + let typed = SyncTable.table( + Recipe.self, + primaryKey: Recipe.Columns.id, + dependsOn: [RecipeIngredient.self], + jsonColumns: [Recipe.Columns.nutrition], + scopeColumn: Recipe.Columns.userId + ) + let strings = SyncTable( + name: "recipes", + primaryKey: "id", + dependsOn: ["recipeIngredients"], + jsonColumns: ["nutrition"], + scopeColumn: "userId" + ) + + #expect(typed == strings) // one manifest type, two ways to spell it +} + +@Test func aTypedManifestDrivesTheEngine() async throws { + // The point of the equivalence: everything downstream — validation, trigger install, the drain — + // is the same code path, so a typed manifest is a spelling choice rather than a mode. + let db = try recipesDB() + let remote = InMemorySyncRemote() + let engine = try SyncEngine.forTesting( + db: db, + remote: remote, + tables: [SyncTable.table(Recipe.self, jsonColumns: [Recipe.Columns.nutrition])] + ) + + try await write(db, "INSERT INTO recipes (id, title) VALUES ('r1', 'Soup')") + try await engine.drainOutbox() + + #expect(await remote.upsertCalls.map(\.table) == ["recipes"]) + #expect(try await db.read { try Int.fetchOne($0, sql: "SELECT COUNT(*) FROM _sync_outbox") } == 0) +} + +@Test func aTypedManifestIsStillCheckedAgainstTheSchema() throws { + // Compile-time checking is not a substitute for the launch check: the record's table name is + // checked against the *record*, and the record can be ahead of the migration that creates it. + let db = try recipesDB() + + #expect(throws: SyncError.self) { + _ = try SyncEngine.forTesting(db: db, remote: InMemorySyncRemote(), tables: [.table(Ghost.self)]) + } +} From df13a3e674c2a09caffecd3ba413522c21b0d36f Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 31 Jul 2026 17:41:53 +0000 Subject: [PATCH 5/5] Wait for the doorbell subscription before ringing it in the example test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The engine subscribes during its first sync pass, so a ring fired immediately after start() can land before anything is listening — the periodic poll would still converge, but not inside the assertion's window. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015y9Y9kec3hj8S35oLQXxvQ --- Examples/Tests/DemoCoreTests/DemoSyncTests.swift | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/Examples/Tests/DemoCoreTests/DemoSyncTests.swift b/Examples/Tests/DemoCoreTests/DemoSyncTests.swift index a4f0fcf..93afbf1 100644 --- a/Examples/Tests/DemoCoreTests/DemoSyncTests.swift +++ b/Examples/Tests/DemoCoreTests/DemoSyncTests.swift @@ -48,8 +48,12 @@ import DemoCore @Test func aChangeFromAnotherDeviceArrivesThroughTheDoorbell() async throws { let sync = try DemoSync() let database = sync.db + let doorbell = sync.doorbell await sync.start() defer { Task { await sync.engine.stop() } } + // The engine subscribes during its first pass; ringing before that is a ring into the void (the + // periodic poll would still converge, just not in time to assert on). + #expect(await eventually { doorbell.liveSubscriptions >= 1 }) // Realtime is a doorbell only: the ring triggers a pull, and the row arrives through the cursor // fetch — never from the event payload.