diff --git a/.claude/skills/happysync/references/api.md b/.claude/skills/happysync/references/api.md index 96e46c0..5fd6095 100644 --- a/.claude/skills/happysync/references/api.md +++ b/.claude/skills/happysync/references/api.md @@ -61,6 +61,19 @@ public init( ) ``` +Declared from a GRDB record type instead, so a rename is a build error rather than a table that +quietly stops syncing (#50). Same `SyncTable`, so a manifest can mix both forms — keep the string +form for a table with no Swift record type: + +```swift +SyncTable.table(Recipe.self, jsonColumns: [Recipe.Columns.nutrition]) // name from databaseTableName +SyncTable.table(RecipeStep.self, dependsOn: [Recipe.self]) // dependency as a type +``` + +Columns are as checked as the record makes them: GRDB's conventional `enum Columns` ties each name +to one declaration; a bare `Column("nutrition")` is exactly as unchecked as the string was. Either +way `SyncEngine.init` still validates every name against the live schema. + Field gotchas: - **`dependsOn`** is normally left alone: `nil` (the default) means the engine reads @@ -228,14 +241,16 @@ Everything but the network is production code — same drain, pull, retry classi ordering — so a test here asserts shipped behaviour. - `InMemorySyncRemote` (actor): `upsertCalls` / `deleteCalls` / `fetchCalls` / `lastScope` record - what the client sent; `seed(_:rows:)` publishes a change "another device" made; `representation:` - supplies the server's response (server-normalized values, a `conflictColumns` merge); - `onUpsert`/`onFetch` hooks suspend the *n*-th call so a test can race a local write against an - in-flight upload. + what the client sent; `upsertRequests` / `deleteRequests` record how many round trips it took, + which batching makes fewer than the per-row call counts (#54); `seed(_:rows:)` publishes a change + "another device" made; `representation:` supplies the server's response (server-normalized values, + a `conflictColumns` merge); `onUpsert`/`onFetch` hooks suspend the *n*-th call so a test can race a + local write against an in-flight upload. - Failure injection: `failUpserts` / `failFetches` counts with a `SimulatedFailure` — `.transient` (retried), `.permanent` (parks on the first attempt), or `.error(anError)` which applies the **production** classification to a real `HTTPError`/`PostgrestError`. Use `99` for - "always". + "always". `failUpserts` counts **requests**: a rejected batch spends one, and so does each row of + the per-row re-run that follows it. - `ManualDoorbell`: `fire()` rings the current subscription; `ringScopes` and `liveSubscriptions` prove an auth change re-subscribed and tore the old one down; `fire(subscription: 0)` rings a torn-down one. @@ -243,7 +258,8 @@ ordering — so a test here asserts shipped behaviour. never a fixed `Task.sleep` plus one assertion. - A bespoke double is three methods on `SyncRemote`. Signal permanence by conforming the thrown error to `ClassifiedSyncError`, or wrap a real one with `RemoteFailure(classifying:)`. Anything - unclassified is treated as transient. + unclassified is treated as transient. The batched `upsert(table:rows:)` / `delete(table:pks:)` the + drain prefers default to looping over the single-row calls, so implementing them is optional. The three tests worth writing first: a cascade delete queues every child's tombstone; status reaches `.degraded` when an upload parks; `stop()` has quiesced the engine before sign-out wipes the DB. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b163c5a..8339099 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -10,3 +10,20 @@ jobs: steps: - uses: actions/checkout@v4 - run: swift test + + docs: + # The DocC catalog is the documentation now, so a broken link or an unresolvable symbol + # reference should surface here rather than on Swift Package Index (issue #57). + runs-on: macos-15 + steps: + - uses: actions/checkout@v4 + - run: swift package --allow-writing-to-directory ./docs-build generate-documentation --target HappySync --output-path ./docs-build + + example: + # The example is a separate package that consumes this one by path, so it catches an API change + # the library's own tests can't see — and its tests assert the behaviours the demo puts on + # screen, so the demo can't quietly rot (issue #58). + runs-on: macos-15 + steps: + - uses: actions/checkout@v4 + - run: swift test --package-path Examples diff --git a/.spi.yml b/.spi.yml new file mode 100644 index 0000000..0d382c5 --- /dev/null +++ b/.spi.yml @@ -0,0 +1,7 @@ +# Swift Package Index build settings. `documentation_targets` is what makes SPI build and host the +# DocC catalog in Sources/HappySync/Documentation.docc, so the doc comments are reachable from a +# browser instead of only from Xcode Quick Help (issue #57). +version: 1 +builder: + configs: + - documentation_targets: [HappySync] diff --git a/Examples/Package.swift b/Examples/Package.swift new file mode 100644 index 0000000..2197da8 --- /dev/null +++ b/Examples/Package.swift @@ -0,0 +1,47 @@ +// swift-tools-version: 6.0 +import PackageDescription + +// A runnable HappySync demo (issue #58). Deliberately a *separate* package rooted here rather than +// extra targets in the library's manifest: an app that consumes HappySync should look like one, with +// its own dependency on the published package — and a consumer opening this folder builds only the +// demo, not the library's test suite. +let package = Package( + name: "HappySyncExamples", + platforms: [.iOS(.v16), .macOS(.v13)], + products: [ + // The demo's model, sync wiring, and view state — everything but the SwiftUI. Split out so + // the sync behaviour the app shows can be asserted headlessly in CI (`DemoCoreTests`). + .library(name: "DemoCore", targets: ["DemoCore"]), + ], + dependencies: [ + .package(path: ".."), + .package(url: "https://github.com/groue/GRDB.swift.git", from: "7.11.0"), + .package(url: "https://github.com/supabase-community/supabase-swift.git", from: "2.0.0"), + ], + targets: [ + .target( + name: "DemoCore", + dependencies: [ + .product(name: "HappySync", package: "HappySync"), + // The demo runs against the in-memory fake, so it needs no Supabase project at all — + // clone, `swift run`, watch two devices converge (issue #53 made this possible). + .product(name: "HappySyncTestSupport", package: "HappySync"), + .product(name: "GRDB", package: "GRDB.swift"), + .product(name: "Supabase", package: "supabase-swift"), + ] + ), + .executableTarget(name: "DemoApp", dependencies: ["DemoCore"]), + // The demo doubles as an integration test: the behaviours the app shows are asserted here, + // headlessly, against the same fake — so `swift test` in CI proves the example still works + // rather than merely still compiling. + .testTarget( + name: "DemoCoreTests", + dependencies: [ + "DemoCore", + .product(name: "HappySync", package: "HappySync"), + .product(name: "HappySyncTestSupport", package: "HappySync"), + .product(name: "GRDB", package: "GRDB.swift"), + ] + ), + ] +) diff --git a/Examples/README.md b/Examples/README.md new file mode 100644 index 0000000..6ba632c --- /dev/null +++ b/Examples/README.md @@ -0,0 +1,50 @@ +# HappySync example — a two-table app you can actually run + +A SwiftUI demo of the whole engine, wired to the **in-memory fake** from `HappySyncTestSupport`. No +Supabase project, no schema to stand up, no credentials: clone and run. + +```bash +open Examples/Package.swift # then run the DemoApp scheme (My Mac, or an iOS sim) +swift test --package-path Examples # the same behaviours, asserted headlessly +swift run --package-path Examples DemoApp # or straight from the terminal, on macOS +``` + +## What it shows + +Two tables — `lists` → `items`, with a real foreign key — is enough for every behaviour that matters: + +| In the app | What it demonstrates | +|---|---| +| Type a list or item name and hit Add | A **plain GRDB write**. No engine call: the table's capture trigger queues the upload in the same transaction. | +| Tick an item | An **expression update** (`SET done = NOT done`) — the kind of write a `save`-only sync API can't see. | +| **Delete list (cascades)** | One `DELETE`. SQLite removes the items too, and each removed item's own trigger queues its tombstone — children before parent, on the wire. | +| The status dot | `status.isHealthy`, **not** `phase == .idle`. The distinction is the point: a settled pass with writes still parked is `.degraded`. | +| **Change from device B** | Realtime as a doorbell: the ring triggers a pull, and the row arrives through the cursor fetch — never from the event payload. | +| **Make the server reject writes** | An RLS-shaped rejection: the write parks on its first attempt instead of retrying forever. | +| **Parked writes → Retry / Discard** | The dead-letter repair API. Fix the cause, then retry — or discard and take the server's version. | +| **Sign out → wipe → sign in** | The one order that's safe: `await stop()`, *then* replace the store. And replacing it, not `DELETE`-ing rows — a delete would fire the capture triggers and queue a tombstone for everything the user owns. | + +Reads are plain `ValueObservation`. HappySync never sits between the app and its data: a pulled row +lands in SQLite and the observation fires, exactly as it does for a local write. + +## Layout + +``` +Sources/DemoCore/ Schema, manifest, sync wiring, view state — no SwiftUI, so CI can drive it +Sources/DemoApp/ The SwiftUI app +Tests/DemoCoreTests/ The table above, as assertions +supabase/migrations/ The server half of the contract, for when you point it at a real project +``` + +`DemoCore` is where to look first: [`Schema.swift`](Sources/DemoCore/Schema.swift) is the entire +manifest, and [`DemoSync.swift`](Sources/DemoCore/DemoSync.swift) is the entire integration. + +## Pointing it at a real Supabase project + +One initializer changes — `SyncEngine(db:supabase:tables:auth:)` instead of +`SyncEngine.forTesting(db:remote:tables:)` (see the comment at the bottom of `DemoSync.swift`). The +server side is +[`supabase/migrations/`](supabase/migrations): `supabase start && supabase db reset` applies the +`updatedAt` trigger, the tombstone trigger, RLS, and the Realtime publication that the fake can't +check for you. The [Server Setup](https://github.com/happyface-studio/HappySync/wiki/Server-Setup) +wiki page explains why each of the four is non-negotiable. diff --git a/Examples/Sources/DemoApp/ContentView.swift b/Examples/Sources/DemoApp/ContentView.swift new file mode 100644 index 0000000..80312c2 --- /dev/null +++ b/Examples/Sources/DemoApp/ContentView.swift @@ -0,0 +1,167 @@ +import SwiftUI +import DemoCore + +struct ContentView: View { + @EnvironmentObject private var store: DemoStore + + var body: some View { + VStack(spacing: 0) { + SyncStatusBar() + Divider() + HStack(alignment: .top, spacing: 0) { + ListsPane() + Divider() + ItemsPane() + } + Divider() + DemoControls() + } + .frame(minWidth: 640, minHeight: 460) + } +} + +// MARK: - Lists + +private struct ListsPane: View { + @EnvironmentObject private var store: DemoStore + @State private var title = "" + + var body: some View { + VStack(alignment: .leading, spacing: 8) { + Text("Lists").font(.headline) + List(store.lists) { list in + Button { + store.selectedListID = list.id + } label: { + Text(list.title) + .foregroundStyle(list.id == store.selectedListID ? Color.accentColor : .primary) + .frame(maxWidth: .infinity, alignment: .leading) + } + .buttonStyle(.plain) + } + HStack { + TextField("New list", text: $title) + Button("Add") { add() }.disabled(title.isEmpty) + } + // One plain DELETE. SQLite cascades to the list's items and each removed item's own + // capture trigger queues its tombstone — watch the delete count in the controls below. + Button("Delete list (cascades)") { + Task { await store.deleteSelectedList() } + } + .disabled(store.selectedListID == nil) + } + .padding() + .frame(minWidth: 240) + } + + @MainActor private func add() { + let value = title + title = "" + Task { await store.addList(title: value) } + } +} + +// MARK: - Items + +private struct ItemsPane: View { + @EnvironmentObject private var store: DemoStore + @State private var title = "" + + var body: some View { + VStack(alignment: .leading, spacing: 8) { + Text("Items").font(.headline) + List(store.items(in: store.selectedListID)) { item in + Button { + Task { await store.toggle(item) } + } label: { + Label(item.title, systemImage: item.done ? "checkmark.circle.fill" : "circle") + } + .buttonStyle(.plain) + } + HStack { + TextField("New item", text: $title) + Button("Add") { add() }.disabled(title.isEmpty || store.selectedListID == nil) + } + } + .padding() + .frame(maxWidth: .infinity) + } + + @MainActor private func add() { + let value = title + title = "" + Task { await store.addItem(title: value) } + } +} + +// MARK: - The parts of sync a UI has to get right + +private struct SyncStatusBar: View { + @EnvironmentObject private var store: DemoStore + + var body: some View { + HStack { + // Ask `isHealthy`, never `phase == .idle` alone: a settled pass with writes still failing + // or parked is `.degraded`, and a green tick there is the bug this API exists to prevent. + Circle() + .fill(store.status.isHealthy ? Color.green : Color.orange) + .frame(width: 10, height: 10) + Text(store.healthLine) + Spacer() + if let lastSynced = store.status.lastSyncedAt { + Text("last sync \(lastSynced.formatted(date: .omitted, time: .standard))") + .foregroundStyle(.secondary) + } + } + .padding(.horizontal) + .padding(.vertical, 8) + } +} + +private struct DemoControls: View { + @EnvironmentObject private var store: DemoStore + + var body: some View { + VStack(alignment: .leading, spacing: 10) { + HStack { + Button("Change from device B") { Task { await store.publishFromAnotherDevice() } } + Button("Make the server reject writes") { Task { await store.sync.rejectWrites() } } + Button("Let the server accept again") { Task { await store.sync.acceptWrites() } } + Button("Sign out → wipe → sign in") { Task { await signOutAndBackIn() } } + } + DeadLetterInspector() + } + .padding() + } + + /// The order is the point: `stop()` awaits the in-flight pass, and only then is the store safe to + /// replace. The demo re-observes afterwards because the database object itself was swapped. + @MainActor private func signOutAndBackIn() async { + try? await store.sync.signOutAndBackIn() + store.observe() + } +} + +/// Parked writes, and the two ways out of them. Repair is an API, not a support ticket: a write that +/// the server refused permanently stops retrying and waits here until the cause is fixed. +private struct DeadLetterInspector: View { + @EnvironmentObject private var store: DemoStore + + var body: some View { + VStack(alignment: .leading, spacing: 6) { + HStack { + Text("Parked writes (\(store.deadLetters.count))").font(.headline) + Spacer() + Button("Retry") { Task { await store.retryDeadLetters() } } + Button("Discard") { Task { await store.discardDeadLetters() } } + } + .disabled(store.deadLetters.isEmpty) + + ForEach(store.deadLetters, id: \.seq) { letter in + Text("\(letter.op.rawValue) \(letter.table)/\(letter.pk) — \(letter.failure.description)") + .font(.caption) + .foregroundStyle(.secondary) + } + } + } +} diff --git a/Examples/Sources/DemoApp/DemoApp.swift b/Examples/Sources/DemoApp/DemoApp.swift new file mode 100644 index 0000000..fde445e --- /dev/null +++ b/Examples/Sources/DemoApp/DemoApp.swift @@ -0,0 +1,20 @@ +import SwiftUI +import DemoCore + +/// A HappySync demo you can run with no backend at all: the engine is wired to the in-memory fake +/// from `HappySyncTestSupport`, so "another device" is a button (issue #58). +/// +/// Everything the app does — writing, deleting with a cascade, reading through `ValueObservation`, +/// showing status, repairing parked writes — is the real engine. Only the transport is fake. +@main +struct HappySyncDemoApp: App { + @StateObject private var store = DemoStore.live() + + var body: some Scene { + WindowGroup { + ContentView() + .environmentObject(store) + .task { await store.start() } + } + } +} diff --git a/Examples/Sources/DemoCore/DemoStore.swift b/Examples/Sources/DemoCore/DemoStore.swift new file mode 100644 index 0000000..80f7606 --- /dev/null +++ b/Examples/Sources/DemoCore/DemoStore.swift @@ -0,0 +1,152 @@ +import Foundation +import Combine +import GRDB +import HappySync + +/// View state for the demo: observed reads, the live sync status, and the dead-letter inspector. +/// +/// The reads are the part worth noticing — they're **plain GRDB `ValueObservation`**. HappySync +/// never sits between the app and its data: a pulled row lands in SQLite and the observation fires, +/// exactly as it does for a local write. +@MainActor +public final class DemoStore: ObservableObject { + @Published public private(set) var lists: [TodoList] = [] + @Published public private(set) var items: [TodoItem] = [] + /// The engine's live status. Drive UI off `isHealthy`, not `phase == .idle` — see `healthLine`. + @Published public private(set) var status = SyncStatus() + /// Writes the server refused, parked so they stop retrying. Refreshed whenever the status says + /// the count changed. + @Published public private(set) var deadLetters: [DeadLetter] = [] + @Published public var selectedListID: String? + + public let sync: DemoSync + + private var readsTask: Task? + private var statusTask: Task? + + public init(sync: DemoSync) { + self.sync = sync + } + + /// Builds the whole stack. Trapping on failure is a demo's privilege: every error here is a + /// programming mistake (a manifest that disagrees with the schema, a migration that didn't run), + /// which is precisely what `SyncEngine.init` is designed to surface loudly at launch. + public static func live() -> DemoStore { + do { + return DemoStore(sync: try DemoSync()) + } catch { + fatalError("HappySync demo failed to start: \(error)") + } + } + + /// Starts sync and begins observing. Safe to call again after `signOutAndBackIn` replaced the + /// store — it re-points the observations at the new database. + public func start() async { + await sync.start() + observe() + } + + public func observe() { + readsTask?.cancel() + statusTask?.cancel() + + let database = sync.db + readsTask = Task { [weak self] in + let observation = ValueObservation.tracking { db -> ([TodoList], [TodoItem]) in + (try TodoList.order(Column("title")).fetchAll(db), + try TodoItem.order(Column("title")).fetchAll(db)) + } + do { + for try await (lists, items) in observation.values(in: database) { + guard let self else { return } + self.lists = lists + self.items = items + if self.selectedListID == nil || !lists.contains(where: { $0.id == self.selectedListID }) { + self.selectedListID = lists.first?.id + } + } + } catch { + // The store was replaced out from under the observation (sign-out). `observe()` runs + // again against the new one. + } + } + + let engine = sync.engine + statusTask = Task { [weak self] in + for await status in engine.status { + guard let self else { return } + let parkedCountChanged = status.deadLetters != self.status.deadLetters + self.status = status + if parkedCountChanged { await self.refreshDeadLetters() } + } + } + } + + public func items(in listID: String?) -> [TodoItem] { + guard let listID else { return [] } + return items.filter { $0.listId == listID } + } + + // MARK: - Actions + + public func addList(title: String) async { + try? await sync.addList(title: title) + } + + public func addItem(title: String) async { + guard let listID = selectedListID else { return } + try? await sync.addItem(title: title, to: listID) + } + + public func toggle(_ item: TodoItem) async { + try? await sync.toggle(itemID: item.id) + } + + public func deleteSelectedList() async { + guard let listID = selectedListID else { return } + try? await sync.deleteList(id: listID) + } + + public func publishFromAnotherDevice() async { + await sync.publishFromAnotherDevice(title: "From device B") + } + + // MARK: - Dead-letter repair + + public func refreshDeadLetters() async { + deadLetters = (try? await sync.engine.deadLetters()) ?? [] + } + + /// Re-queue parked writes. Do this *after* fixing the cause — an RLS policy, a migration, an app + /// update — or they park again on the next pass. + public func retryDeadLetters() async { + try? await sync.engine.retryDeadLetters() + await refreshDeadLetters() + } + + /// Abandon the local writes and take the server's version instead. Discard also re-pulls the + /// affected rows, so a row that diverged while parked converges rather than staying stuck. + public func discardDeadLetters() async { + try? await sync.engine.discardDeadLetters() + await refreshDeadLetters() + } + + // MARK: - Status, read the way a UI should read it + + /// `.idle` alone is the tempting check and the wrong one: it means "no pass is running", which is + /// also true while the user's writes sit in the outbox failing. Ask `isHealthy` first, then + /// branch — `.degraded` is a settled pass with writes outstanding, and `.failed` carries a + /// classified cause rather than prose to substring-match. + public var healthLine: String { + if status.isHealthy { return "Synced" } + switch status.phase { + case .syncing: return "Syncing…" + case .idle: return "Synced" + case .degraded: + return "\(status.failedUploads) retrying, \(status.deadLetters) parked" + case .failed(.authExpired): return "Session expired — sign in again" + case .failed(.network): return "Offline — will retry" + case .failed(let failure): return "Sync failed: \(failure)" + } + } +} diff --git a/Examples/Sources/DemoCore/DemoSync.swift b/Examples/Sources/DemoCore/DemoSync.swift new file mode 100644 index 0000000..d3a3531 --- /dev/null +++ b/Examples/Sources/DemoCore/DemoSync.swift @@ -0,0 +1,172 @@ +import Foundation +import GRDB +import Supabase +import HappySync +import HappySyncTestSupport + +/// The demo's sync stack: a local GRDB store, a HappySync engine, and — instead of a network — the +/// in-memory fake from `HappySyncTestSupport`. That's the point of the example: **no Supabase project +/// is needed to watch sync work**, and the same object drives both the SwiftUI app and the +/// integration tests in CI (issue #58). +/// +/// Everything except the transport is the shipping engine: the same capture triggers, the same +/// FK-ordered drain, the same cursor pull and LWW, the same dead-letter classification. Swapping in +/// the real backend is one initializer — see `productionEngine` at the bottom of this file. +@MainActor +public final class DemoSync { + /// The server, in-process. Seedable (a change "another device" made) and failable (a write the + /// server refuses), which is how the demo reaches the interesting states on command. + public let remote: InMemorySyncRemote + /// Stands in for the Supabase Realtime channel: `fire()` is a change event arriving. + public let doorbell: ManualDoorbell + + /// The local source of truth. Replaced wholesale on sign-out — see `signOutAndBackIn`. + public private(set) var db: DatabaseQueue + public private(set) var engine: SyncEngine + + /// Every row the fake server holds for `lists`. The fake serves downloads from what it was last + /// seeded with, so publishing a second remote change means re-seeding the accumulated set. + private var serverLists: [[String: AnyJSON]] = [] + + public init() throws { + remote = InMemorySyncRemote() + doorbell = ManualDoorbell() + db = try Self.makeDatabase() + engine = try Self.makeEngine(db: db, remote: remote, doorbell: doorbell) + } + + private static func makeDatabase() throws -> DatabaseQueue { + // In-memory, so every launch of the demo starts from nothing. A real app points GRDB at a + // file; nothing else about the wiring changes. + let db = try DatabaseQueue() + try DemoSchema.migrator().migrate(db) + return db + } + + private static func makeEngine( + db: DatabaseQueue, remote: InMemorySyncRemote, doorbell: ManualDoorbell + ) throws -> SyncEngine { + // The manifest is validated against the schema here, which is why the migrator ran first. + try SyncEngine.forTesting( + db: db, + remote: remote, + tables: DemoSchema.tables, + doorbell: doorbell, + pollInterval: 5, // short, so the demo converges while you watch it + debounceInterval: 0.2, + deadLetterAfter: 2 // reach the parked state in two failures rather than eight + ) + } + + public func start() async { + await engine.start() + } + + // MARK: - Writes (there is no write API — that's the point) + + @discardableResult + public func addList(title: String) async throws -> TodoList { + let list = TodoList(title: title) + try await db.write { try list.insert($0) } + // No engine call. The `lists` capture trigger queued the upload inside that transaction, and + // the engine's outbox observer wakes the drain. + return list + } + + @discardableResult + public func addItem(title: String, to listID: String) async throws -> TodoItem { + let item = TodoItem(listId: listID, title: title) + try await db.write { try item.insert($0) } + return item + } + + /// An expression update — the kind of write a `save`-only sync API can't see, and the capture + /// triggers do. + public func toggle(itemID: String) async throws { + try await db.write { db in + try db.execute( + sql: "UPDATE items SET done = NOT done WHERE id = ?", + arguments: [itemID] + ) + } + } + + /// One plain `DELETE`. SQLite cascades to the list's items, and each removed item's own trigger + /// queues its tombstone — the app never enumerates children. + public func deleteList(id: String) async throws { + try await db.write { db in + try db.execute(sql: "DELETE FROM lists WHERE id = ?", arguments: [id]) + } + } + + // MARK: - The other device + + /// Publishes a list "another device" created and rings the doorbell, exactly as Supabase Realtime + /// would. The engine debounces the ring into one `pullNow()`; the row arrives through the cursor + /// pull, never from the event payload. + public func publishFromAnotherDevice(title: String) async { + serverLists.append([ + "id": .string(UUID().uuidString), + "title": .string(title), + "updatedAt": .string(Self.serverTimestamp()), + "deletedAt": .null, + ]) + await remote.seed("lists", rows: serverLists) + doorbell.fire() + } + + // MARK: - The server saying no + + /// Makes the server reject every write from now on, the way an RLS policy does: permanently, so + /// the drain parks the entry instead of retrying it forever. + public func rejectWrites() async { + await remote.failNextUpserts(99, with: .permanent) + } + + /// Stops rejecting. Parked writes don't retry on their own — that's what `retryDeadLetters()` is + /// for, and it's the repair flow this demo exists to show. + public func acceptWrites() async { + await remote.failNextUpserts(0) + } + + // MARK: - Teardown, in the order that matters + + /// Sign-out → **stop** → wipe → sign-in. `stop()` awaits the in-flight pass, so after it returns + /// nothing is still writing to the store being thrown away. Doing this in any other order is how + /// an app uploads the previous user's rows with the new user's token. + /// + /// The wipe replaces the whole database rather than `DELETE`-ing rows: a delete would fire the + /// capture triggers and queue a tombstone for every row the user owns, which the next sign-in + /// would dutifully upload. + public func signOutAndBackIn() async throws { + await engine.stop() + db = try Self.makeDatabase() + engine = try Self.makeEngine(db: db, remote: remote, doorbell: doorbell) + await engine.start() + } + + /// ISO-8601 with fractional seconds — what a Postgres `timestamptz` serializes to, and what the + /// tuple cursor orders by. + private static func serverTimestamp() -> String { + let formatter = ISO8601DateFormatter() + formatter.formatOptions = [.withInternetDateTime, .withFractionalSeconds] + formatter.timeZone = TimeZone(identifier: "UTC") + return formatter.string(from: Date()) + } +} + +// MARK: - What this would be against a real backend + +// The demo's only concession to running offline is *which* initializer builds the engine. Against a +// real project it is this instead, with `Examples/supabase/migrations/` standing the schema up: +// +// let engine = try SyncEngine( +// db: databaseQueue, +// supabase: SupabaseClient(supabaseURL: url, supabaseKey: anonKey), +// tables: DemoSchema.tables, +// auth: { await session.accessToken }, +// scope: { await session.user?.id.uuidString } // only if RLS is broader than the partition +// ) +// +// Everything above this comment — the writes, the cascade, the status, the dead-letter repair — +// is unchanged. diff --git a/Examples/Sources/DemoCore/Schema.swift b/Examples/Sources/DemoCore/Schema.swift new file mode 100644 index 0000000..528b6a3 --- /dev/null +++ b/Examples/Sources/DemoCore/Schema.swift @@ -0,0 +1,81 @@ +import Foundation +import GRDB +import HappySync + +// The smallest schema that still shows every interesting HappySync behaviour: a parent, a child, and +// a real foreign key between them, so a parent delete cascades locally *and* queues a tombstone per +// removed child — the behaviour an app most often gets wrong by hand-deleting children. + +/// A list of things to do. The parent row. +public struct TodoList: Codable, Identifiable, Hashable, FetchableRecord, PersistableRecord { + public static let databaseTableName = "lists" + + public var id: String + public var title: String + + public init(id: String = UUID().uuidString, title: String) { + self.id = id + self.title = title + } +} + +/// One entry in a list. `listId` carries `ON DELETE CASCADE`, which is the whole trick: deleting a +/// list removes its items in the same transaction and each item's own capture trigger queues its +/// tombstone. +public struct TodoItem: Codable, Identifiable, Hashable, FetchableRecord, PersistableRecord { + public static let databaseTableName = "items" + + public var id: String + public var listId: String + public var title: String + public var done: Bool + + public init(id: String = UUID().uuidString, listId: String, title: String, done: Bool = false) { + self.id = id + self.listId = listId + self.title = title + self.done = done + } +} + +// Neither record declares `updatedAt` or `deletedAt`. They exist as columns — the sync contract +// requires them — but they belong to the *server*: it stamps `updatedAt`, the drain writes it back, +// and a domain type that carried them would write nulls over both on every `save`. + +public enum DemoSchema { + /// The demo's migrator, with HappySync's internal tables registered into it — the shape an app + /// that owns its own `DatabaseMigrator` should use, so both share one `grdb_migrations` table. + /// + /// The write-capture triggers are deliberately *not* here: they're derived from the manifest + /// below, which is source code, and the engine (re)installs them idempotently at every init. + public static func migrator() -> DatabaseMigrator { + var migrator = DatabaseMigrator() + SyncSchema.register(into: &migrator) + migrator.registerMigration("demo_v1") { db in + try db.create(table: TodoList.databaseTableName) { t in + t.column("id", .text).primaryKey() + t.column("title", .text).notNull() + t.column("updatedAt", .text) // server-stamped; never written by this app + t.column("deletedAt", .text) // tombstone marker, likewise + } + try db.create(table: TodoItem.databaseTableName) { t in + t.column("id", .text).primaryKey() + t.column("listId", .text) + .notNull() + .references(TodoList.databaseTableName, onDelete: .cascade) + t.column("title", .text).notNull() + t.column("done", .boolean).notNull().defaults(to: false) + t.column("updatedAt", .text) + t.column("deletedAt", .text) + } + } + return migrator + } + + /// The whole manifest. Declared in any order — the engine sorts it — and with no `dependsOn`: + /// the engine reads `items`' foreign key at init and derives that lists upload first (issue #49). + public static let tables: [SyncTable] = [ + SyncTable(name: TodoItem.databaseTableName), + SyncTable(name: TodoList.databaseTableName), + ] +} diff --git a/Examples/Tests/DemoCoreTests/DemoSyncTests.swift b/Examples/Tests/DemoCoreTests/DemoSyncTests.swift new file mode 100644 index 0000000..93afbf1 --- /dev/null +++ b/Examples/Tests/DemoCoreTests/DemoSyncTests.swift @@ -0,0 +1,125 @@ +import Testing +import Foundation +import GRDB +import HappySync +import HappySyncTestSupport +import DemoCore + +// The example, asserted. Every one of these is a behaviour the SwiftUI app puts on screen — writes +// uploading with no engine call, a parent delete tombstoning its children, a change from "device B" +// arriving through the doorbell, a refused write parking and being repaired, and sign-out quiescing +// the engine before the store is replaced. Running in CI is what stops the demo from rotting (#58). + +@MainActor +@Test func aPlainWriteUploadsWithNoEngineCall() async throws { + let sync = try DemoSync() + let remote = sync.remote + await sync.start() + defer { Task { await sync.engine.stop() } } + + try await sync.addList(title: "Groceries") + + #expect(await eventually { await remote.upsertCalls.contains { $0.table == "lists" } }) +} + +@MainActor +@Test func deletingAListTombstonesItsItemsChildrenFirst() async throws { + let sync = try DemoSync() + let remote = sync.remote + await sync.start() + defer { Task { await sync.engine.stop() } } + + let list = try await sync.addList(title: "Trip") + try await sync.addItem(title: "Passport", to: list.id) + try await sync.addItem(title: "Charger", to: list.id) + #expect(await eventually { await remote.upsertCalls.count == 3 }) + + // One DELETE of the parent. SQLite cascades; each removed child's own capture trigger queues its + // tombstone, so the app never enumerates children. + try await sync.deleteList(id: list.id) + + #expect(await eventually { await remote.deleteCalls.count == 3 }) + let deletes = await remote.deleteCalls + #expect(deletes.filter { $0.table == "items" }.count == 2) + #expect(deletes.last?.table == "lists") // children before parent — the FK-safe delete order +} + +@MainActor +@Test func aChangeFromAnotherDeviceArrivesThroughTheDoorbell() async throws { + let sync = try DemoSync() + let database = sync.db + let doorbell = sync.doorbell + await sync.start() + defer { Task { await sync.engine.stop() } } + // The engine subscribes during its first pass; ringing before that is a ring into the void (the + // periodic poll would still converge, just not in time to assert on). + #expect(await eventually { doorbell.liveSubscriptions >= 1 }) + + // Realtime is a doorbell only: the ring triggers a pull, and the row arrives through the cursor + // fetch — never from the event payload. + await sync.publishFromAnotherDevice(title: "From device B") + + #expect(await eventually { + await count(in: database, "SELECT COUNT(*) FROM lists WHERE title = 'From device B'") == 1 + }) +} + +@MainActor +@Test func aRefusedWriteParksAndRetryRepairsIt() async throws { + let sync = try DemoSync() + let engine = sync.engine + await sync.start() + defer { Task { await engine.stop() } } + + // The server refuses the write the way an RLS policy does — permanently, so the drain parks the + // entry on its first attempt instead of retrying it forever. + await sync.rejectWrites() + try await sync.addList(title: "Doomed") + + #expect(await eventually { await parked(engine).count == 1 }) + let letter = try #require(await parked(engine).first) + #expect(letter.table == "lists") + #expect(letter.op == .upsert) + + // Repair: fix the cause first, then re-queue. Retrying a still-broken write just parks it again. + await sync.acceptWrites() + try await engine.retryDeadLetters() + + #expect(await eventually { await parked(engine).isEmpty }) + #expect(await eventually { await outboxCount(sync) == 0 }) // the write finally landed +} + +@MainActor +@Test func signOutQuiescesTheEngineBeforeTheStoreIsReplaced() async throws { + let sync = try DemoSync() + let remote = sync.remote + await sync.start() + try await sync.addList(title: "Before sign-out") + #expect(await eventually { await remote.upsertCalls.contains { $0.table == "lists" } }) + + // stop() → replace the store → start(). Wiping before the engine has quiesced is how an app + // writes into a database it is about to delete; deleting rows instead of replacing the store is + // how it queues a tombstone for everything the user owns. + try await sync.signOutAndBackIn() + defer { Task { await sync.engine.stop() } } + + #expect(await count(in: sync.db, "SELECT COUNT(*) FROM lists") == 0) + #expect(await count(in: sync.db, "SELECT COUNT(*) FROM _sync_outbox") == 0) +} + +// MARK: - Helpers + +/// A scalar count, or `-1` if the read failed — shaped for use inside `eventually`, which polls a +/// non-throwing condition. +private func count(in db: DatabaseQueue, _ sql: String) async -> Int { + (try? await db.read { db in try Int.fetchOne(db, sql: sql) ?? 0 }) ?? -1 +} + +private func parked(_ engine: SyncEngine) async -> [DeadLetter] { + (try? await engine.deadLetters()) ?? [] +} + +@MainActor +private func outboxCount(_ sync: DemoSync) async -> Int { + await count(in: sync.db, "SELECT COUNT(*) FROM _sync_outbox") +} diff --git a/Examples/supabase/migrations/20260731000000_demo_schema.sql b/Examples/supabase/migrations/20260731000000_demo_schema.sql new file mode 100644 index 0000000..3b59660 --- /dev/null +++ b/Examples/supabase/migrations/20260731000000_demo_schema.sql @@ -0,0 +1,110 @@ +-- The demo's server side, in one file: `supabase start && supabase db reset` stands it up. +-- +-- The app in Examples/ runs against an in-process fake and needs none of this. It's here because the +-- fake cannot check the half of the contract that lives on the server — the `updatedAt` trigger LWW +-- depends on, the tombstone propagation deletes depend on, RLS, and the Realtime publication. This +-- is what those four requirements look like for a two-table schema (see the Server Setup wiki page). + +-- --------------------------------------------------------------------------- +-- Tables. Column names are camelCase and identical to the local SQLite schema — +-- HappySync does no name mapping, by contract. +-- --------------------------------------------------------------------------- + +create table if not exists public.lists ( + id uuid primary key, + "userId" uuid not null default auth.uid() references auth.users (id) on delete cascade, + title text not null, + "updatedAt" timestamptz not null default now(), + "deletedAt" timestamptz +); + +create table if not exists public.items ( + id uuid primary key, + "listId" uuid not null references public.lists (id) on delete cascade, + "userId" uuid not null default auth.uid() references auth.users (id) on delete cascade, + title text not null, + done boolean not null default false, + "updatedAt" timestamptz not null default now(), + "deletedAt" timestamptz +); + +create index if not exists lists_cursor_idx on public.lists ("updatedAt", id); +create index if not exists items_cursor_idx on public.items ("updatedAt", id); + +-- --------------------------------------------------------------------------- +-- 1. updatedAt — server-stamped, always. This is what makes last-write-wins +-- correct: two devices' clocks never enter the ordering. A client-sent value +-- is ignored (and HappySync strips it from every upload anyway). +-- --------------------------------------------------------------------------- + +create or replace function public.stamp_updated_at() +returns trigger language plpgsql as $$ +begin + new."updatedAt" := now(); + return new; +end $$; + +drop trigger if exists stamp_updated_at on public.lists; +create trigger stamp_updated_at + before insert or update on public.lists + for each row execute function public.stamp_updated_at(); + +drop trigger if exists stamp_updated_at on public.items; +create trigger stamp_updated_at + before insert or update on public.items + for each row execute function public.stamp_updated_at(); + +-- --------------------------------------------------------------------------- +-- 2. deletedAt — deletes are soft, so they can propagate on the next cursor +-- pull. A parent's children are tombstoned by trigger, NOT by a hard +-- `ON DELETE CASCADE`: a hard cascade removes the rows with no tombstone +-- left for other devices to pull, and they keep the children forever. +-- --------------------------------------------------------------------------- + +create or replace function public.tombstone_list_items() +returns trigger language plpgsql as $$ +begin + if new."deletedAt" is not null and old."deletedAt" is null then + update public.items + set "deletedAt" = new."deletedAt" + where "listId" = new.id and "deletedAt" is null; + end if; + return new; +end $$; + +drop trigger if exists tombstone_list_items on public.lists; +create trigger tombstone_list_items + after update on public.lists + for each row execute function public.tombstone_list_items(); + +-- Purge tombstones on a schedule if you like — but the retention must exceed the engine's +-- `maxOfflineGap` (30 days by default), or a device offline longer than the retention never sees the +-- tombstone and resurrects the row: +-- +-- delete from public.items where "deletedAt" < now() - interval '90 days'; +-- delete from public.lists where "deletedAt" < now() - interval '90 days'; + +-- --------------------------------------------------------------------------- +-- 3. RLS — the security boundary. Here it equals the sync partition exactly, +-- so the demo's SyncTable declares no `scopeColumn`. Declare one when a +-- policy is deliberately broader than what a device should download. +-- --------------------------------------------------------------------------- + +alter table public.lists enable row level security; +alter table public.items enable row level security; + +drop policy if exists "own lists" on public.lists; +create policy "own lists" on public.lists + for all using ("userId" = auth.uid()) with check ("userId" = auth.uid()); + +drop policy if exists "own items" on public.items; +create policy "own items" on public.items + for all using ("userId" = auth.uid()) with check ("userId" = auth.uid()); + +-- --------------------------------------------------------------------------- +-- 4. Realtime — the doorbell. An event triggers a debounced pull; the payload +-- is never applied directly, so sync converges with or without this. +-- --------------------------------------------------------------------------- + +alter publication supabase_realtime add table public.lists; +alter publication supabase_realtime add table public.items; diff --git a/Package.swift b/Package.swift index 95b1258..2121db9 100644 --- a/Package.swift +++ b/Package.swift @@ -16,6 +16,9 @@ let package = Package( // supabase-community is the fork CookThis consumes; matching the URL (and SPM // identity) lets the app share one supabase-swift instead of resolving two. .package(url: "https://github.com/supabase-community/supabase-swift.git", from: "2.0.0"), + // Build-tools only: `swift package generate-documentation` / `preview-documentation` for the + // DocC catalog (issue #57). Swift Package Index builds the same catalog from `.spi.yml`. + .package(url: "https://github.com/apple/swift-docc-plugin", from: "1.3.0"), ], targets: [ .target( diff --git a/README.md b/README.md index 1540c63..6ce9311 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,11 @@ CRDTs needed. > APPS-415) are all live. Next is M2 server prep (server-side `updated_at` triggers + `deleted_at` > tombstones) before the M3 CookThis cutover. +**See it run first:** [`Examples/`](Examples) is a two-table SwiftUI app wired to the in-memory fake +— no Supabase project, no schema, no credentials. `swift run --package-path Examples DemoApp` shows +writes uploading with no engine call, a cascade delete tombstoning its children, a change arriving +from "device B", a refused write parking, and the repair that unparks it. + ## Model Local GRDB SQLite is the source of truth for reads (observed with `ValueObservation`). You write @@ -69,6 +74,20 @@ for await status in engine.status { } ``` +If your tables have GRDB record types, declare the manifest from those and the table names stop being +strings that can go stale — a rename becomes a build error rather than a table that quietly stops +syncing: + +```swift +tables: [ + SyncTable.table(Recipe.self, jsonColumns: [Recipe.Columns.nutrition]), + SyncTable.table(RecipeIngredient.self), // dependsOn still derived from the schema's FKs +] +``` + +Both forms produce the same `SyncTable`, so a manifest can mix them — keep the string form for a +table with no Swift record type. + For a table whose RLS is broader than the sync partition (e.g. a shared `recipes` table readable as `isPublic OR userId = auth.uid()`), declare a `scopeColumn` and supply the partition value so the engine downloads only the user's rows instead of the whole catalog: @@ -242,8 +261,14 @@ deliberately generic, but is pressure-tested against one real app before it's tr ## Documentation & Claude skill -- **[Wiki](https://github.com/happyface-studio/HappySync/wiki)** — Getting Started, Server Setup, - API Reference, Operations & Troubleshooting, and Architecture. (Sources live in [`wiki/`](wiki).) +- **[Documentation](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync)** + — Getting Started, Server Setup, Operations & Troubleshooting, Testing, Architecture, and every + public symbol. Built from [`Sources/HappySync/Documentation.docc`](Sources/HappySync/Documentation.docc); + `swift package generate-documentation` builds it locally. +- **[Example app](Examples)** — a runnable two-table SwiftUI demo against the in-memory fake, with + the Supabase migrations for when you point it at a real project. +- **[Wiki](https://github.com/happyface-studio/HappySync/wiki)** — points at the documentation + above, and hosts the Claude skill page. (Sources live in [`wiki/`](wiki).) - **[Sync contract](docs/SYNC-CONTRACT.md)** — the language-neutral contract every client and the server must honor. - **Claude skill** — [`.claude/skills/happysync/`](.claude/skills/happysync) is an installable diff --git a/Sources/HappySync/Documentation.docc/Architecture.md b/Sources/HappySync/Documentation.docc/Architecture.md new file mode 100644 index 0000000..af18e74 --- /dev/null +++ b/Sources/HappySync/Documentation.docc/Architecture.md @@ -0,0 +1,124 @@ +# Architecture + +HappySync is deliberately small. Its correctness rests on one enabling constraint and a handful of +mechanisms that compose cleanly. + +## The enabling constraint + +Personal data is **single-user, multi-device — not collaborative**. One user edits their own rows, +occasionally from two devices. Because two devices rarely edit the *same* row at the *same* instant, +**last-write-wins (LWW) by server timestamp** is correct. No CRDTs, no operational transforms, no +conflict-resolution RPC. Revisit only if the data ever becomes collaborative. + +## The pieces + +``` + write read + │ ▲ + ▼ │ + ┌──────────────┐ ValueObservation │ + │ GRDB SQLite │───────────────────► UI + │ (local truth)│ + └──────┬───────┘ + │ same transaction + ▼ + ┌──────────────┐ drain (upsert) ┌───────────────┐ + │ _sync_outbox│───────────────────► │ Supabase │ + └──────────────┘ │ Postgres │ + ▲ cursor pull (LWW) │ (server truth│ + └───────────────────────────── │ for writes) │ + └──────┬────────┘ + Realtime doorbell │ (debounced pullNow) + ◄───────┘ +``` + +### Upload — outbox drain + +A write appends to `_sync_outbox` **in the same transaction** as the domain write, then returns +optimistically. A background drain processes entries in `seq` order: + +- **PostgREST upsert** with `Prefer: return=representation` for `.upsert`; soft-delete for + `.delete`. Both are **idempotent by primary key**, so retries are safe. +- **The cursor column never leaves the device.** It's server-stamped by contract, so the payload + drops it (along with any declared `serverOwnedColumns`) with nothing declared — a table whose + `updatedAt` trigger is missing can't quietly promote a device's clock to the LWW authority. + `deletedAt` still ships, as `null`, which is how a re-created row un-tombstones. +- **The full server representation is written back locally** on success — column defaults, + trigger-normalized fields, recomputed server-owned columns, and (for a `conflictColumns` upsert) + the merged row re-keyed to the client's pk. Otherwise the writing device would be the one device + that never sees the server's version of its own write (its local `updatedAt` would equal the + server's, so LWW would skip the row forever). The write-back is skipped for a row that gained a + newer outbox entry mid-flight — the pending edit wins. +- **FK ordering:** upsert parents before children; tombstone children before parents. +- **Batched requests, per-entry failures.** Consecutive entries for the same table and op are sent as + one request (up to 256 rows), so a 500-row backlog is a handful of round trips instead of 500. + Entries are never reordered to make a bigger batch, so the FK order above holds. When the server + rejects a batch it hasn't said *which* row it objected to, so the chunk is re-run one entry at a + time — the poison row parks alone and its neighbours upload. +- **Per-entry backoff** with jitter; failures are classified permanent vs transient (see + ). + +### Download — cursor pull with LWW + +Per table: `SELECT * WHERE updatedAt > :cursor [AND scopeColumn = :partition] ORDER BY (updatedAt, +id)`, RLS-scoped. + +- **Tuple cursor `(updatedAt, id)`** — not a bare timestamp — so rows sharing a millisecond at a + page boundary aren't dropped. +- **LWW apply:** apply a remote row only if `remote.updatedAt > local.updatedAt` **and** the local + row is not dirty (a pending local edit is never clobbered). The compare is at **microsecond** + precision — Postgres `now()` is µs-precise, so two writes inside the same millisecond don't tie. +- **Tombstones** (`deletedAt` set) arrive through the same pull and delete locally. +- **Schema-drift tolerance:** each wire row is intersected against the local table's columns, so a + server that migrated ahead doesn't brick an old client's pulls. + +### Realtime is a doorbell + +A Realtime change event triggers a **debounced `pullNow()`** — it never applies payloads directly. +All correctness lives in the idempotent cursor pull, so sync converges even if Realtime drops; the +doorbell just makes it feel instant. Foreground and periodic pulls converge on their own. + +### Writes and capture + +There is no write API to remember. Each declared table carries `AFTER INSERT` / `AFTER UPDATE` / +`AFTER DELETE` triggers that append to `_sync_outbox`, so any GRDB write — `save`, a partial update, +raw SQL, a whole multi-table transaction — is queued **in the caller's transaction**. The unit of +atomicity is the user's intent, not one row: "create a recipe + 12 ingredients" commits the rows and +their queued uploads together or not at all. + +Each trigger's `WHEN` clause reads a one-row `_sync_control.applying` flag that the engine raises +while it writes *server* state locally (a pulled page, a tombstone, an upload's write-back, a resync +reconcile, a discarded dead letter). Without it every downloaded row would queue itself straight back +for upload and the sync would never settle. + +The triggers come from the `tables` manifest, which is source code rather than a numbered migration, +so the engine (re)installs them idempotently at init and drops the ones for tables you've stopped +syncing. A declared table that doesn't exist locally throws at init — the manifest and the schema are +not allowed to drift silently. + +A GRDB `TransactionObserver` watches for **inserts into the outbox** and wakes the runner, so a plain +write uploads as promptly as an explicit call used to. Drain bookkeeping (updates and deletes on the +outbox) is deliberately not observed, so a drain can't feed itself another pass. + +### Deletes and cascades + +A plain `DELETE` removes the row locally and queues a tombstone. For a parent, the schema's +`ON DELETE CASCADE` does the graph walk and each removed child's own delete trigger queues its +tombstone — mirroring the server's child-tombstone trigger. Result: the local and server deleted sets +stay symmetric, with no orphan window and no round-trip. A table that only `dependsOn` a parent +without an FK action is not cascaded; `dependsOn` orders uploads, it doesn't delete rows. + +## What HappySync owns (and doesn't) + +**Owns:** the write-capture triggers on your declared tables, the outbox drain, cursor pull, +tombstones, FK ordering, the Realtime doorbell, status, and retry/backoff. + +**Does not own:** your reads (observe GRDB directly) or your schema (you declare tables; the server +holds the authoritative schema). + +## The contract + +Every mechanism here has a precise, language-neutral spec — server conventions, wire semantics, +field mapping, schema-evolution rules — in +[`docs/SYNC-CONTRACT.md`](https://github.com/happyface-studio/HappySync/blob/main/docs/SYNC-CONTRACT.md). +A future non-Swift client would implement that contract, not reuse this engine's code. diff --git a/Sources/HappySync/Documentation.docc/GettingStarted.md b/Sources/HappySync/Documentation.docc/GettingStarted.md new file mode 100644 index 0000000..c7d3b4c --- /dev/null +++ b/Sources/HappySync/Documentation.docc/GettingStarted.md @@ -0,0 +1,207 @@ +# Getting Started + +This walks you from zero to a working sync loop. Before your first sync completes, you also need the +server schema in **** — the engine cannot correct a server that lacks the +`updatedAt` trigger. + +## 1. Add the package + +In `Package.swift`: + +```swift +dependencies: [ + .package(url: "https://github.com/happyface-studio/HappySync.git", from: "0.4.0"), +], +targets: [ + .target(name: "MyApp", dependencies: ["HappySync"]), +] +``` + +Or in Xcode: **File → Add Package Dependencies →** `https://github.com/happyface-studio/HappySync`. + +## 2. Declare your tables + +Each synced table is a `SyncTable`, declared once, **in any order** — the engine sorts the manifest +by FK dependency itself. Only `name` is required. + +Every declared table must **already exist** in your local schema when the engine is constructed, so +run your migrations first. `SyncEngine.init` validates the whole manifest against that schema and +throws `SyncError.invalidManifest(table:reason:)` — naming the table, the field and the value — if +anything disagrees: a table that isn't there, or a `cursorColumn` / `scopeColumn` / `jsonColumns` / +`conflictColumns` / `serverOwnedColumns` entry naming a column the table doesn't have. Each of those +used to fail silently and late, so it's worth reading the message rather than working around it. + +You don't declare `dependsOn`: the engine reads your foreign keys at init and derives it. Pass one +explicitly only for a *logical* parent your schema has no FK for (see +[Cascade](#delete-with-cascade)). + +Declare `ON DELETE CASCADE` on child foreign keys if you want a parent delete to remove (and +tombstone) its children — see [Delete](#delete-with-cascade) below. + +```swift +import HappySync + +let tables = [ + SyncTable(name: "recipes", jsonColumns: ["nutrition"]), + SyncTable(name: "recipeIngredients"), // dependsOn derived from the FK + SyncTable(name: "recipeSteps", jsonColumns: ["temperature"]), +] +``` + +``SyncTable`` documents every field it takes — `scopeColumn`, `conflictColumns`, +`serverOwnedColumns`, `serverColumns`, `cursorColumn` — including when to reach for each and what +breaks if you get it wrong. + +### Declaring them from your record types + +If your tables have GRDB record types, declare the manifest from those instead and the table names +stop being strings you can mistype or forget to update: + +```swift +let tables = [ + SyncTable.table(Recipe.self, jsonColumns: [Recipe.Columns.nutrition]), + SyncTable.table(RecipeIngredient.self), // dependsOn still derived from the FK + SyncTable.table(RecipeStep.self, dependsOn: [Recipe.self]), +] +``` + +Renaming a record's `databaseTableName` is now a build error at the manifest rather than a table +that quietly stops syncing. Column names are as checked as your record makes them: GRDB's +conventional `enum Columns` ties each one to a single declaration, while a bare `Column("nutrition")` +is exactly as unchecked as the string was. Both forms produce the same ``SyncTable``, so mixing them +in one manifest is fine — reach for the string form for a table with no Swift record type. + +## 3. Create and start the engine + +```swift +let engine = try SyncEngine( + db: databaseQueue, // any GRDB DatabaseWriter — your local source of truth + supabase: client, // your SupabaseClient + tables: tables, + auth: { await session.accessToken } // return a FRESH access token each call +) + +await engine.start() +``` + +`start()` is idempotent. It runs an immediate convergence sync, then keeps syncing from three +triggers — the Realtime doorbell (debounced), a periodic poll (converges even if Realtime drops), +and retry backoff — all funnelled through one serialized runner. + +### Sharing a migrator + +The engine's internal tables (`_sync_outbox`, `_sync_state`, `_sync_meta`, `_sync_control`) migrate +at init. If your app owns the GRDB `DatabaseMigrator`, register HappySync's migrations into it so +both share one `grdb_migrations` table: + +```swift +var migrator = DatabaseMigrator() +SyncSchema.register(into: &migrator) // happysync_v1..v4 +// … your app migrations … +try migrator.migrate(databaseQueue) +``` + +The **write-capture triggers** are not part of that migrator. They're derived from your `tables` +manifest — source code, which ships without a schema version bump — so the engine (re)installs them +idempotently at every init and drops the ones for tables you've stopped syncing. + +## 4. Write + +Write GRDB **normally**. Each declared table carries `AFTER INSERT` / `UPDATE` / `DELETE` triggers +that append to the outbox in your transaction, so there is no second API to call and nothing to +forget. The engine notices the queued write and uploads promptly, so there's **no `syncNow()` needed +after a write** — a burst debounces into one drain pass. + +```swift +try await db.write { try recipe.save($0) } // PersistableRecord + +try await db.write { db in // partial / expression updates + try db.execute(sql: "UPDATE recipes SET title = ? WHERE id = ?", arguments: [title, id]) + try db.execute(sql: "UPDATE recipes SET cookCount = cookCount + 1 WHERE id = ?", arguments: [id]) +} + +try await db.write { db in // one transaction = one user intent + try recipe.save(db) + for ingredient in ingredients { try ingredient.save(db) } +} +``` + +That last one matters: the rows and their queued uploads commit together, or neither does. A crash +part-way through leaves no half-recipe *and* no half-batch in the outbox. + +> `engine.enqueue(op:table:row:)` still works but is **deprecated**. It now just performs the write +> and lets the trigger queue it. Replace `enqueue(.upsert, table: t, row: r)` with +> `try await db.write { try r.save($0) }`, and `enqueue(.delete, …)` with a plain `DELETE`. + +### Delete (with cascade) + +A plain `DELETE` removes the row locally and queues a tombstone. For a **parent** row, declare +`ON DELETE CASCADE` on the child foreign keys — SQLite then walks the graph and each removed child's +own trigger queues its tombstone, so you never delete children by hand and the UI never shows +orphans. + +```swift +// In your migration: +try db.create(table: "recipeIngredients") { t in + t.column("id", .text).primaryKey() + t.column("recipeId", .text).references("recipes", onDelete: .cascade) + // … +} + +// Then: +try await db.write { try $0.execute(sql: "DELETE FROM recipes WHERE id = ?", arguments: [recipeID]) } +``` + +A table that only `dependsOn` a parent without an FK action is **not** cascaded — `dependsOn` orders +uploads, it doesn't delete rows. Its orphans reconcile on the next pull via the server tombstone. + +## 5. Pull + +Downloads happen automatically (doorbell + periodic). Call `pullNow()` for app-driven nudges — +returning to the foreground, pull-to-refresh: + +```swift +try await engine.pullNow() +``` + +## 6. Show sync status + +Ask `isHealthy` — never `phase == .idle` alone. A settled pass with writes still failing or parked +is `.degraded`, so switching on the phase forces you to confront it: + +```swift +for await status in engine.status { + guard !status.isHealthy else { return showSynced() } + switch status.phase { + case .idle, .syncing: break // .idle here means healthy; .syncing is in flight + case .degraded: showPending(status.failedUploads + status.deadLetters) + case .failed(.authExpired): promptReauth() // classified — no string matching + case .failed(.network): break // recovers on its own; show nothing + case .failed(let failure): showBanner(failure) + } +} +``` + +See **** for what `failedUploads` / `deadLetters` mean and how to +repair parked writes. + +## 7. Tear down on sign-out (order matters) + +`stop()` is **async** and awaits the in-flight pass. Always `await engine.stop()` **before** wiping +or replacing the database on sign-out / account switch — otherwise an in-flight pass could write to +the store you're about to delete, or upload the old user's rows with the new user's token. + +```swift +await engine.stop() // engine is quiesced here +try await wipeLocalDatabase() +``` + +Status subscriptions survive teardown: `stop()` broadcasts a settled `.idle` status but doesn't end +the streams, so a `for await status in engine.status` loop keeps delivering after the next `start()` +— no re-subscribing needed. + +## Next + +- **** — required before the first successful sync. +- **** — health, dead letters, schema evolution. +- **[Claude Skill](https://github.com/happyface-studio/HappySync/wiki/Claude-Skill)** — let your AI assistant wire this correctly for you. diff --git a/Sources/HappySync/Documentation.docc/HappySync.md b/Sources/HappySync/Documentation.docc/HappySync.md new file mode 100644 index 0000000..472fe85 --- /dev/null +++ b/Sources/HappySync/Documentation.docc/HappySync.md @@ -0,0 +1,92 @@ +# ``HappySync`` + +Sync one user's data between their devices, over GRDB and Supabase, without pretending it's a +collaborative editing problem. + +## Overview + +Personal data is not collaborative: one user edits their own rows, occasionally from two devices. +That single constraint is what makes **last-write-wins by a server timestamp** correct — no CRDTs, +no conflict-resolution RPC — and it's what the whole engine is built on. + +The model in five sentences: + +1. **Local GRDB SQLite is the source of truth for reads.** You keep observing it with + `ValueObservation`; HappySync never sits between your app and its data. +2. **You write GRDB normally** — `save`, a partial `UPDATE`, raw SQL, a whole multi-table + transaction — and per-table SQLite triggers record each change into an outbox *in that same + transaction*, so a write can't be half-committed and half-queued. +3. **A background drain uploads the outbox** via PostgREST upsert, parents before children, batched + per table, idempotent by primary key, retrying with per-entry backoff. +4. **A pull downloads rows changed since a per-table `(updatedAt, id)` cursor** and applies them + last-write-wins, never clobbering a local row with a pending write of its own. +5. **Supabase Realtime is a doorbell only** — an event triggers a debounced pull; payloads are never + applied directly, so all the correctness lives in the idempotent cursor pull. + +Server-authoritative `updatedAt` (a Postgres `BEFORE INSERT/UPDATE` trigger → `now()`) is required +for that to be correct, and deletes propagate as `deletedAt` tombstones rather than vanishing. See + — the engine cannot correct a server that skips it. + +```swift +let engine = try SyncEngine( + db: databaseQueue, + supabase: client, + tables: [SyncTable(name: "recipes", jsonColumns: ["nutrition"])], + auth: { await session.accessToken } +) +await engine.start() + +// That's the whole write API: there isn't one. +try await databaseQueue.write { try recipe.save($0) } +``` + +There is a **runnable example** — a two-table SwiftUI app wired to the in-memory fake, needing no +Supabase project at all — in [`Examples/`](https://github.com/happyface-studio/HappySync/tree/main/Examples). + +The full, language-neutral contract every client *and the server* must honor lives in +[`docs/SYNC-CONTRACT.md`](https://github.com/happyface-studio/HappySync/blob/main/docs/SYNC-CONTRACT.md). +These pages are the Swift practitioner's guide; the contract is the source of truth. + +## Topics + +### Essentials + +- +- +- ``SyncEngine`` +- ``SyncTable`` + +### Watching sync + +- ``SyncStatus`` +- ``SyncFailure`` + +### Repairing writes the server refused + +- +- ``DeadLetter`` + +### Testing your integration + +- +- ``SyncRemote`` +- ``SyncDoorbell`` +- ``SilentDoorbell`` +- ``ClassifiedSyncError`` +- ``RemoteFailure`` + +### How it works + +- +- ``SyncCursor`` +- ``ScopeFilter`` +- ``SyncOp`` + +### Configuration errors + +- ``SyncError`` +- ``ManifestProblem`` + +### Sharing a migrator + +- ``SyncSchema`` diff --git a/Sources/HappySync/Documentation.docc/Operations.md b/Sources/HappySync/Documentation.docc/Operations.md new file mode 100644 index 0000000..dcb2127 --- /dev/null +++ b/Sources/HappySync/Documentation.docc/Operations.md @@ -0,0 +1,142 @@ +# Operations and Troubleshooting + +Running a synced app: what the status means, how a failed write is classified, how to repair one the +server refused, and what to do when a symptom shows up in the field. + +## Sync-status UI + +Consume `engine.status` and ask `isHealthy`. `.idle` means settled **and** clean; a pass that +completed while writes are still failing or parked settles as `.degraded`. + +```swift +for await status in engine.status { + switch status.phase { + case .idle: showSynced() // settled and clean + case .syncing: showSpinner() + case .degraded: showPending(status.failedUploads + status.deadLetters) + case .failed(let failure): showBanner(failure) // classified — see below + } +} +``` + +- `failedUploads > 0` → some local changes haven't reached the server yet (degraded, still + retrying with backoff). +- `deadLetters > 0` → some writes parked and need inspection/repair. + +## Failure model + +- **Permanent** (constraint `23xxx`, RLS `42501`, undefined-column `42703`, other 4xx) → + dead-lettered immediately. +- **Transient** (network, 5xx, 408/429, transient Postgres `40001`/`40P01`/`53xxx`/`08xxx`, unknown + codes) → retried with per-entry exponential backoff (jittered) until a cap, then dead-lettered. +- **Auth-shaped** (401/403, `PGRST301`/`PGRST302`) → transient **and exempt from the retry budget**, + so an expired-token stretch never dead-letters the whole outbox; it recovers when the token + refreshes out-of-band. + +A dead-lettered entry stops retrying **and** stops counting as a dirty row, so it never permanently +blocks downloads for its key. + +### What the app sees + +That classification is surfaced as `SyncFailure`, on `Phase.failed` and on `DeadLetter.failure`. +It answers *what to show the user* — the retry decision is already made by then, and is visible as +`failedUploads` (still retrying) vs `deadLetters` (parked): + +| `SyncFailure` | Raised by | What the app should do | +|---|---|---| +| `.network` | `URLError`, Postgres `08xxx` | Nothing — it recovers | +| `.authExpired` | 401/403, `PGRST301`/`PGRST302` | Nothing at first; prompt re-auth if it persists | +| `.permissionDenied` | `42501` | Surface it — a policy bug or a real permissions problem | +| `.constraintViolation(code:)` | `23xxx` | Inspect the row; the write will never land as-is | +| `.schemaMismatch(column:)` | `42703`, `PGRST204` | Prompt an app update (or restore the server column) | +| `.server(status:)` | any other HTTP status, incl. 5xx | Nothing — retried; surface if it persists | +| `.other(String)` | anything unrecognised, transient Postgres states | Log the text | + +`.network` means the request got no answer at all; a server that answered with a 5xx is +`.server(status:)`, so the status code isn't thrown away. + +The kind and the retry decision agree everywhere except `PGRST204`: it classifies as +`.schemaMismatch` but is **retried** rather than parked immediately, because PostgREST also raises it +from a stale schema cache, which clears on its own. + +Every case renders via `CustomStringConvertible`, so `"\(failure)"` is safe to put in a banner. The +classified cases carry no server prose of their own — the full text lives in the engine's log, and on +`DeadLetter.lastError` for a parked write. + +## Repairing dead letters + +```swift +let parked = try await engine.deadLetters() +for letter in parked { + switch letter.failure { + case .permissionDenied: reportPolicyBug(letter) // fix RLS, then retryDeadLetters() + case .schemaMismatch(let column): promptAppUpdate(column) + case .constraintViolation(let code): inspect(letter, code) + default: log("\(letter.op) \(letter.table)/\(letter.pk): \(letter.lastError ?? "unknown")") + } +} + +// After fixing the cause (RLS/policy change, schema migration, app update): +try await engine.retryDeadLetters() // all, or pass [seq] for specific entries + +// Or abandon the local write and accept the server's version — discard drops the entries and +// re-pulls the affected rows so local converges to the server (including changes missed while parked): +try await engine.discardDeadLetters([badSeq]) // specific seqs, or omit for all +``` + +Both mutations refresh the status stream immediately, so `deadLetters` drops as soon as they return. + +## Teardown / account switch (order matters) + +`stop()` is async and awaits the in-flight pass. A consumer that wipes or replaces the database on +sign-out / account switch **must `await engine.stop()` before touching the DB file** — otherwise an +in-flight pass could write to the store you're about to delete, or (mid-account-switch) upload the +old user's rows with the new user's token. + +```swift +await engine.stop() // quiesced here +try await wipeLocalDatabase() +``` + +`stop()` settles the status stream (`.idle`, keeping the outbox counts) but leaves subscriptions +open, so the sync-status UI resumes updating on the next `start()` without being re-subscribed. + +## Stale-cursor full resync + +A device offline past `maxOfflineGap` (default 30 days) can't trust its cursors — a tombstone it +never saw may already be purged. Once `now − lastSyncedAt > maxOfflineGap` the engine clears +cursors, re-pulls, and reconciles purged deletes instead of resurrecting them. Keep `maxOfflineGap` +**≤ the server's tombstone-purge retention** (see ). For a `scopeColumn` table the +resync is deferred until a partition value resolves, so it never wipes an un-pulled table. + +## Schema evolution (staggered clients) + +Shipped clients and the server are **not lockstep** — App Store review + staggered updates mean old +builds run against a newer server for weeks. The engine tolerates drift both ways, but follow these +rules so any single migration is safe for every client in the field: + +- **Additive-only.** Never drop-and-recreate or repurpose a column in place. +- **Never rename in place.** Add the new name, dual-write server-side, wait for adoption, then + remove the old name by the removal rule. +- **Adding a column — server-first.** Deploy it (nullable/default) before any client reads/writes + it. Old clients drop the unknown column on download and don't send it on upload. +- **Removing a column — client-first.** Drop it server-side **only after no shipped client still + writes it**: (1) ship a build that stops sending it — optionally declaring `serverColumns` so the + payload is intersected immediately; (2) wait out the update-lag window; (3) then drop it. Skipping + step 2 dead-letters every write from clients still sending the column. + +## Troubleshooting + +| Symptom | Likely cause | Fix | +|---|---|---| +| Writes never reach the server; `deadLetters` climbing | RLS rejects (`42501`) or a constraint violation | `deadLetters()` → switch on `failure` (`lastError` for the raw text); fix the policy/data, then `retryDeadLetters()`. | +| A device never sees the server's version of its **own** write | Missing `updatedAt` trigger (the engine already strips the client's `updatedAt` from every upload, so a stale client value can't be the cause) | Add the server trigger. | +| Rows on an insert-only table never download to other devices | The table's `cursorColumn` has no server default — the engine never uploads it, so it lands null | `default now()` (or a trigger) on that column server-side. | +| Whole public catalog downloads to every device | RLS broader than the partition, no `scopeColumn` | Declare `scopeColumn` + supply the engine `scope:` closure. | +| A fresh-pk insert 409s forever | Row already exists under a secondary `UNIQUE` | Declare `conflictColumns` (leaf tables only). | +| Every write on a table dead-letters after a server migration | Dropped/renamed server column (`PGRST204`) | Restore the column until clients stop sending it; or declare `serverColumns`. | +| Deleted parent leaves orphaned children in the UI | Child FK not declared `ON DELETE CASCADE` locally, or missing server child-tombstone trigger | Declare `ON DELETE CASCADE` on the child FKs (`dependsOn` alone deletes nothing) and add the server child-tombstone trigger. | +| `SyncEngine.init` throws `invalidManifest` | A declared `SyncTable` disagrees with the local schema — a table or column that isn't there, a duplicate declaration, `dependsOn` naming an undeclared table, `conflictColumns` on a table something foreign-keys, or `serverColumns` without the primary key | Read the message: it names the table, the field and the value. Run your app's migrations **before** constructing the engine, then fix the manifest. It throws rather than silently syncing the wrong thing — every one of these used to be invisible until the damage showed up on another device. | +| Some writes sync, one code path never does | A write that bypasses the declared tables — writing a table not in the manifest, or a `DatabasePool` reader-side hack | Add the table to the manifest. Writes to declared tables are captured by triggers, so there is no way to "forget" to enqueue one. | +| Offline-long device resurrects deleted rows | `maxOfflineGap` > server tombstone retention | Set `maxOfflineGap` ≤ retention. | +| Idle status shows healthy but changes aren't syncing | Reading `phase == .idle` as health | Ask `status.isHealthy`. The engine now settles such a pass as `.degraded`, so a `switch` over the phase surfaces it. | diff --git a/Sources/HappySync/Documentation.docc/ServerSetup.md b/Sources/HappySync/Documentation.docc/ServerSetup.md new file mode 100644 index 0000000..ac86ce5 --- /dev/null +++ b/Sources/HappySync/Documentation.docc/ServerSetup.md @@ -0,0 +1,135 @@ +# Server Setup + +Every synced table needs four things on the Supabase side. These are **non-negotiable** — LWW +compares a *server* clock, and deletes, scoping, and live updates all depend on this contract. The +engine will not paper over a server that violates it. + +Run the SQL below per synced table (swap `recipes` for your table). + +## 1. `updatedAt` — server-stamped change time + +LWW compares `updatedAt`, and a *server* clock is what stops two devices' clock skew from silently +losing writes. **Never trust a client-sent `updatedAt`.** The engine holds up its half unasked: a +table's `cursorColumn` is stripped from every upload payload, so no HappySync client ever sends one +— you don't declare anything for this. The trigger below is still required (it's what advances the +column on update, and what the cursor pull depends on); the stripping just means a table that lost +its trigger can't quietly promote a device's clock to the ordering authority. + +```sql +alter table public.recipes + add column if not exists "updatedAt" timestamptz not null default now(); + +create or replace function public.stamp_updated_at() +returns trigger language plpgsql as $$ +begin + new."updatedAt" := now(); -- server clock, always; ignore any client value + return new; +end $$; + +drop trigger if exists stamp_updated_at on public.recipes; +create trigger stamp_updated_at + before insert or update on public.recipes + for each row execute function public.stamp_updated_at(); +``` + +> An **insert-only / immutable** table needs no update trigger. Give it a monotonic stamp column +> (e.g. `translatedAt`) and set that table's `cursorColumn` in its `SyncTable` descriptor. Because +> the engine never uploads the cursor column, that column must be `default now()` server-side — +> otherwise inserts land with a null stamp the cursor can't order. + +## 2. `deletedAt` — soft-delete tombstones + +Deletes are **soft**: set `deletedAt` instead of removing the row, so deletions propagate on the +next cursor pull. Deleting a parent must **tombstone its children too** — use a trigger, *not* +`ON DELETE CASCADE` (a hard cascade leaves no tombstone for other devices to pull). + +```sql +alter table public.recipes + add column if not exists "deletedAt" timestamptz; + +create or replace function public.tombstone_recipe_children() +returns trigger language plpgsql as $$ +begin + if new."deletedAt" is not null and old."deletedAt" is null then + update public."recipeIngredients" + set "deletedAt" = new."deletedAt" + where "recipeId" = new.id and "deletedAt" is null; + -- repeat per child table + end if; + return new; +end $$; + +drop trigger if exists tombstone_recipe_children on public.recipes; +create trigger tombstone_recipe_children + after update on public.recipes + for each row execute function public.tombstone_recipe_children(); +``` + +Locally, declare the child foreign keys `ON DELETE CASCADE` in your **SQLite** schema: deleting a +parent then removes its children in the same transaction and each child's capture trigger queues its +tombstone, keeping the local and server deleted sets symmetric with no orphan window. The server +trigger above is what makes *other* devices converge. + +### Tombstone purge and the `maxOfflineGap` invariant + +Purge old tombstones on a schedule, but respect this **invariant**: the purge retention must exceed +the engine's `maxOfflineGap` (default 30 days). A device offline longer than the retention would +never see a purged tombstone and would resurrect the row. Set `maxOfflineGap` **≤ your server's +retention**. + +```sql +-- e.g. a pg_cron job; retention (90d) > engine maxOfflineGap (30d default) +delete from public.recipes where "deletedAt" < now() - interval '90 days'; +``` + +## 3. RLS scoped to `auth.uid()` + +RLS is the **security boundary**. Enable it and add policies keyed on `auth.uid()`. + +```sql +alter table public.recipes enable row level security; + +create policy "own rows: select" on public.recipes + for select using ("userId" = auth.uid()); +create policy "own rows: write" on public.recipes + for all using ("userId" = auth.uid()) with check ("userId" = auth.uid()); +``` + +**RLS is not necessarily the sync partition.** If a policy is deliberately broader than what a +device should download — e.g. `isPublic = true OR userId = auth.uid()` — an unfiltered pull drags +the whole public catalog to every device. Declare a `scopeColumn` (e.g. `userId`) on the `SyncTable` +and supply the engine's `scope:` closure; the engine filters the pull **and** the doorbell to +`scopeColumn = `. Omit `scopeColumn` when RLS already equals the partition. + +## 4. Realtime publication (the doorbell) + +```sql +alter publication supabase_realtime add table public.recipes; +``` + +Realtime is a **doorbell only** — an event triggers a debounced `pullNow()`; payloads are never +applied directly. Sync converges without it; Realtime just makes it feel instant. + +## Field-mapping conventions + +| Concern | Convention | +|---|---| +| Column names | **camelCase, identical** in SQLite and Postgres — no snake_case mapping. | +| Dates | ISO-8601 with fractional seconds; the LWW gate compares instants at **microsecond** precision. | +| UUID | text locally ↔ `uuid` in Postgres. | +| Bool | integer `0/1` locally ↔ `boolean`. | +| Enum | `rawValue` string. | +| JSON columns | JSON **text** locally ↔ `json`/`jsonb`; declare in `jsonColumns`. | +| Server-owned | RPC-managed columns; declare in `serverOwnedColumns`, never in an upsert. The `cursorColumn` is server-owned too and the engine strips it with nothing declared. | + +## Checklist + +- [ ] `updatedAt` + `BEFORE INSERT/UPDATE` trigger on every table (`default now()` on an + insert-only table's stamp column — the client never sends it). +- [ ] `deletedAt` on every table + a child-tombstone trigger on every parent. +- [ ] Tombstone purge scheduled, retention **>** engine `maxOfflineGap`. +- [ ] RLS enabled + `auth.uid()` policies on every table. +- [ ] `scopeColumn` declared for any table whose RLS is broader than the partition. +- [ ] Every synced table in the `supabase_realtime` publication. + +See ** → Schema evolution** before you change a live schema. diff --git a/Sources/HappySync/Documentation.docc/TestingYourIntegration.md b/Sources/HappySync/Documentation.docc/TestingYourIntegration.md new file mode 100644 index 0000000..e807b62 --- /dev/null +++ b/Sources/HappySync/Documentation.docc/TestingYourIntegration.md @@ -0,0 +1,225 @@ +# Testing + +Sync is the part of an app where bugs are silent and data-losing, and it is usually the part with no +tests — because testing it seems to require a live Supabase project. + +It doesn't. `SyncRemote` and `SyncDoorbell` are the seams the engine's own 2,700 lines of tests run +on, and both are public. The `HappySyncTestSupport` product ships the fakes, so a test can answer +questions like *does deleting a recipe queue tombstones for all its children?* in milliseconds, with +no network. + +## Add the test-support product + +Add it to your **test** target only — it contains no production code: + +```swift +.testTarget( + name: "MyAppTests", + dependencies: [ + "MyApp", + .product(name: "HappySync", package: "HappySync"), + .product(name: "HappySyncTestSupport", package: "HappySync"), + ] +), +``` + +## Build an engine with no Supabase + +`SyncEngine.forTesting(db:remote:tables:…)` is the production engine with the network seams +injected. Everything else — drain, pull, retry classification, dead-lettering, cascade ordering — is +the code that ships, so what a test proves here is what the app does. + +```swift +import HappySync +import HappySyncTestSupport + +let db = try DatabaseQueue() +try MyApp.migrator.migrate(db) // your migrations, as at launch + +let remote = InMemorySyncRemote() +let engine = try SyncEngine.forTesting( + db: db, + remote: remote, + tables: MyApp.syncTables, // the real manifest — a typo in it fails here, too + pollInterval: 999, // only the passes the test triggers + debounceInterval: 0.02 +) +``` + +The parameters worth knowing: `pageSize` (shrink to `1` to force pagination), `pollInterval` (raise +to silence the background poll), `debounceInterval` (shrink so a doorbell ring converges promptly), +`deadLetterAfter` (lower to reach the parked state without driving eight retries), and `scope` (the +partition value, to test a user switch). + +## `InMemorySyncRemote` + +A server that lives in the test process. It records every upload, serves downloads from a seedable +per-table row store honouring the `(cursorColumn, primaryKey)` tuple cursor and the page limit, and +can be told to fail. + +```swift +// What the server holds — rows are [String: AnyJSON], exactly what PostgREST returns. +let remote = InMemorySyncRemote(dataset: [ + "recipes": [["id": "r1", "title": "Soup", "updatedAt": "2026-06-30T12:00:00.000Z"]], +]) + +// What the client sent. +await remote.upsertCalls // [(table, row, onConflict)] in order — one entry per *row* +await remote.deleteCalls // [(table, pk)] — the tombstones +await remote.upsertRequests // round trips, which batching makes fewer than upsertCalls.count (#54) +await remote.deleteRequests // …and the same for tombstones +await remote.fetchCalls // download pages requested; the usual "a pass ran" signal +await remote.lastScope // the partition filter the engine resolved + +// Publish a change "another device" made, between two pulls. +await remote.seed("recipes", rows: [updatedRow]) +``` + +### Failure injection + +The one knob that matters is whether the drain **retries** the write or **parks** it: + +```swift +InMemorySyncRemote(failUpserts: 1, upsertFailure: .transient) // retried, then succeeds +InMemorySyncRemote(failUpserts: 1, upsertFailure: .permanent) // dead-lettered on the first attempt +InMemorySyncRemote(failUpserts: 1, upsertFailure: .error(HTTPError(…))) // real error, real classification +``` + +`failUpserts` counts **requests**, not rows: the drain batches a run of same-table writes into one +call, so a rejected batch spends one — and the per-row re-run it triggers spends one per row. + +`.error` wraps a genuine transport error (`HTTPError`, `PostgrestError`, `URLError`) with the +classification the production remote applies — so *"an RLS reject parks, an expired token doesn't"* +is asserted against the shipped table rather than a hand-picked answer. `failFetches` / +`fetchFailure` do the same for downloads, which drives a whole pass to `.failed(SyncFailure)`. + +Use `99` for "always", and `failNextUpserts(_:with:)` / `failNextFetches(_:with:)` to change the +injection mid-test — e.g. fail, assert parked, then let the retry succeed. + +### Simulating server-side computation + +`representation` maps an uploaded row to the row the server returns. The drain writes the whole +representation back locally, so this is how you test server-normalized values, defaulted columns, or +a `conflictColumns` merge: + +```swift +let remote = InMemorySyncRemote(representation: { row in + var server = row + server["cookedCount"] = .integer(7) // the merged server total, not what this client sent + server["updatedAt"] = .string("2026-06-30T12:00:00.000Z") + return server +}) +``` + +### Holding a call in flight + +`onUpsert` / `onFetch` run at the start of each call with its 1-based number. Because the remote +suspends there, this is how a test races a local write against an in-flight upload, or stops the +engine mid-pull: + +```swift +let started = AsyncSignal(), gate = AsyncSignal() +await remote.onUpsert { _ in await started.fire(); await gate.wait() } + +let pass = Task { await engine.syncNow() } +await started.wait() // the upload is blocked in the remote +try await db.write { … } // a competing local edit lands behind it +await gate.fire() // let the upload finish +``` + +## `ManualDoorbell` + +The Realtime channel, made deterministic — it rings only when you say so. + +```swift +let doorbell = ManualDoorbell() +let engine = try SyncEngine.forTesting(db: db, remote: remote, tables: tables, doorbell: doorbell) +await engine.start() + +doorbell.fire() // a remote change lands +#expect(await eventually { await remote.fetchCalls >= 2 }) + +doorbell.ringScopes // [nil, "u1"] proves a sign-in re-scoped the subscription +doorbell.liveSubscriptions // settles to 1 — the old subscription was torn down +doorbell.fire(subscription: 0) // ring a torn-down one; it must not poke the engine +``` + +`SilentDoorbell` (the default) never rings, so convergence comes only from the periodic poll and the +passes you trigger. + +## Async helpers + +`eventually` polls until a condition holds or the timeout elapses. Reach for it any time the engine +converges in the background — a fixed sleep plus one assertion is the alternative, and it flakes +under a loaded CI runner: + +```swift +#expect(await eventually { await remote.fetchCalls >= 1 }) +``` + +For a *negative* assertion ("no further pull happened"), poll until the count stops moving, then +assert it doesn't move again. + +`AsyncSignal` is a one-shot rendezvous — `fire()` releases every `wait()`, now and later, so a wait +can't miss a signal that arrived first. + +## Three tests worth having + +These are the ones that catch real bugs, and none of them need a network. + +**A cascade delete queues every child's tombstone.** The app deletes one row; SQLite cascades and +each child's capture trigger queues its own tombstone. If a child table is missing from your +manifest, this test fails and the field doesn't. + +```swift +try await db.write { try $0.execute(sql: "DELETE FROM recipes WHERE id = 'r1'") } +#expect(await eventually { await remote.deleteCalls.count == 3 }) +``` + +**The status UI shows degraded when a write parks.** `.degraded` is the case everyone gets wrong: the +pass completed, so nothing *looks* broken, but the user's write never reached the server. Assert on +`isHealthy`, never on the phase alone. + +```swift +let remote = InMemorySyncRemote(failUpserts: 99, upsertFailure: .permanent) +… +#expect(await eventually { await latestStatus(engine)?.phase == .degraded }) +#expect(try await engine.deadLetters().count == 1) +``` + +**Sign-out quiesces the engine before the wipe.** After `await engine.stop()` returns, nothing may +still be in flight — a late pull writes into the store you're about to delete, and mid-account-switch +uploads the previous user's rows under the new user's token. + +```swift +await engine.stop() +let atStop = await remote.fetchCalls +doorbell.fire() // a Realtime event racing the sign-out +try await Task.sleep(for: .milliseconds(200)) +#expect(await remote.fetchCalls == atStop) // nothing restarted +``` + +A worked version of all three lives in +[`Tests/HappySyncTests/PublicSeamsTests.swift`](https://github.com/happyface-studio/HappySync/blob/main/Tests/HappySyncTests/PublicSeamsTests.swift), +which imports HappySync the way your app does — no `@testable` — so it stops compiling if these +seams ever regress. + +## Writing your own remote + +`SyncRemote` and `SyncDoorbell` are protocols, so a bespoke double is three methods. Signal whether a +failure is retryable by conforming your error to `ClassifiedSyncError`, or wrap a real transport +error with `RemoteFailure(classifying:)` to get the production classification: + +```swift +struct RejectingRemote: SyncRemote { + func upsert(table: String, row: [String: AnyJSON], onConflict: String?) async throws -> [String: AnyJSON] { + throw RemoteFailure(classifying: PostgrestError(code: "42501", message: "rls")) + } + func delete(table: String, primaryKey: String, pk: String) async throws {} + func fetch(table: String, cursorColumn: String, since cursor: SyncCursor?, primaryKey: String, + scope: ScopeFilter?, limit: Int) async throws -> [[String: AnyJSON]] { [] } +} +``` + +An error the engine can't classify is treated as **transient** — safer to retry than to silently drop +a user's write. diff --git a/Sources/HappySync/Models.swift b/Sources/HappySync/Models.swift index 5e46091..47f6dc7 100644 --- a/Sources/HappySync/Models.swift +++ b/Sources/HappySync/Models.swift @@ -122,7 +122,11 @@ public struct SyncTable: Sendable, Hashable { /// A pending sync operation recorded in the outbox. public enum SyncOp: String, Sendable, Codable { + /// The row exists locally and its current state should be uploaded — queued by an `INSERT` or an + /// `UPDATE`, and idempotent by primary key. case upsert + /// The row was removed locally and should be **soft**-deleted on the server (its tombstone set), + /// so the deletion propagates to other devices on their next pull. case delete } diff --git a/Sources/HappySync/Outbox.swift b/Sources/HappySync/Outbox.swift index dbb5233..12cdf7e 100644 --- a/Sources/HappySync/Outbox.swift +++ b/Sources/HappySync/Outbox.swift @@ -8,7 +8,10 @@ import GRDB /// Public because it appears in `SyncRemote.fetch` — a custom or fake remote has to be able to read /// the position it's being asked to resume from (issue #53). public struct SyncCursor: Sendable, Equatable { + /// The table's `cursorColumn` value of the last applied row — a server-stamped timestamp. public var updatedAt: String + /// That row's primary key, which breaks ties between rows sharing a `updatedAt` so a page + /// boundary can't skip one. public var id: String public init(updatedAt: String, id: String) { @@ -90,6 +93,29 @@ func orderForUpload(_ entries: [OutboxEntry], tables: [SyncTable]) -> [OutboxEnt // `collapseOutbox`, which nets each key to one op before this FK ordering runs (APPS-472). } +/// Groups an already-ordered run of pending uploads into the chunks the drain sends as one request +/// each: **consecutive** entries sharing a table and an op, capped at `limit` (issue #54). +/// +/// Adjacency is the whole rule — entries are never reordered to make a bigger batch, so the FK-safe +/// order `orderForUpload` produced survives intact and a chunk is always one PostgREST call's worth: +/// one table, one op, one conflict target. The cap bounds both the request size and the blast radius +/// of the per-row fallback a rejected chunk falls back to. +func uploadChunks(_ entries: [OutboxEntry], limit: Int) -> [[OutboxEntry]] { + let limit = max(1, limit) + var chunks: [[OutboxEntry]] = [] + for entry in entries { + let extendsLast = chunks.last.map { chunk in + chunk.count < limit && chunk[0].tableName == entry.tableName && chunk[0].op == entry.op + } ?? false + if extendsLast { + chunks[chunks.count - 1].append(entry) + } else { + chunks.append([entry]) + } + } + return chunks +} + /// Exponential backoff between drain passes for an entry that has failed `attempts` times, with /// **±20% jitter** so a user's devices that failed together — a shared outage, one expired session — /// don't retry in lockstep and thundering-herd the server the moment it recovers (APPS-514). Capped diff --git a/Sources/HappySync/Remote.swift b/Sources/HappySync/Remote.swift index 58adccc..5338c07 100644 --- a/Sources/HappySync/Remote.swift +++ b/Sources/HappySync/Remote.swift @@ -123,8 +123,20 @@ public protocol SyncRemote: Sendable { /// is the comma-joined columns of a secondary unique constraint to use as the PostgREST conflict /// target, or nil to conflict on the primary key (the default). See APPS-478. func upsert(table: String, row: [String: AnyJSON], onConflict: String?) async throws -> [String: AnyJSON] + /// Upserts a **batch** of rows for one table in a single request, returning one representation + /// per row. Every row carries the same columns and the same `onConflict` target, so this is one + /// PostgREST call rather than `rows.count` of them (issue #54). + /// + /// The default implementation forwards to the single-row `upsert` in a loop, so an existing + /// conformance keeps working unchanged — override it to get the batch. Either way the drain + /// treats a thrown error as rejecting the **whole** batch and re-runs the rows one at a time, so + /// a single poison row parks alone instead of taking its neighbours with it. + func upsert(table: String, rows: [[String: AnyJSON]], onConflict: String?) async throws -> [[String: AnyJSON]] /// Propagates a delete for one row, keyed by primary key (soft delete — sets the tombstone). func delete(table: String, primaryKey: String, pk: String) async throws + /// Propagates a **batch** of deletes for one table in a single request. Defaults to looping over + /// the single-row `delete`; see the batch `upsert` for the failure contract. + func delete(table: String, primaryKey: String, pks: [String]) async throws /// Fetches up to `limit` rows changed since `cursor`, ordered by the `(cursorColumn, id)` tuple /// so the caller can resume exactly where it left off. Tombstoned rows (`deletedAt != null`) /// are included. When `scope` is set, only rows matching `scope.column = scope.value` are @@ -132,6 +144,26 @@ public protocol SyncRemote: Sendable { func fetch(table: String, cursorColumn: String, since cursor: SyncCursor?, primaryKey: String, scope: ScopeFilter?, limit: Int) async throws -> [[String: AnyJSON]] } +/// Batch defaults, so `SyncRemote` stays the same three methods to implement. A conformance that +/// can genuinely send one request per batch (`SupabaseRemote` does) overrides them; one that can't +/// loses nothing but the round trips. +public extension SyncRemote { + func upsert(table: String, rows: [[String: AnyJSON]], onConflict: String?) async throws -> [[String: AnyJSON]] { + var representations: [[String: AnyJSON]] = [] + representations.reserveCapacity(rows.count) + for row in rows { + representations.append(try await upsert(table: table, row: row, onConflict: onConflict)) + } + return representations + } + + func delete(table: String, primaryKey: String, pks: [String]) async throws { + for pk in pks { + try await delete(table: table, primaryKey: primaryKey, pk: pk) + } + } +} + /// `SyncRemote` over a Supabase PostgREST client. Idempotent by primary key, so the drain can /// safely retry. Auth token is fetched fresh per request. struct SupabaseRemote: SyncRemote { @@ -139,33 +171,54 @@ struct SupabaseRemote: SyncRemote { let auth: @Sendable () async -> String func upsert(table: String, row: [String: AnyJSON], onConflict: String?) async throws -> [String: AnyJSON] { + try await send(table: table, rows: [row], onConflict: onConflict).first ?? row + } + + /// One PostgREST request for the whole batch: an array body upserts against the same conflict + /// target, and the representation comes back one row per uploaded row, in request order (issue + /// #54). PostgREST requires every object in the body to carry the same keys — which holds here, + /// since a table's payloads are built from the same column list — and rejects the request + /// outright otherwise, which the drain reads as "re-run this chunk one row at a time". + func upsert(table: String, rows: [[String: AnyJSON]], onConflict: String?) async throws -> [[String: AnyJSON]] { + try await send(table: table, rows: rows, onConflict: onConflict) + } + + private func send(table: String, rows: [[String: AnyJSON]], onConflict: String?) async throws -> [[String: AnyJSON]] { + guard !rows.isEmpty else { return [] } // never spend a request (or send `[]`) on nothing let token = await auth() do { // onConflict nil → PostgREST conflicts on the primary key (its default); non-nil → the // named secondary unique constraint is the conflict target, merging onto the existing row. - let rows: [[String: AnyJSON]] = try await client + return try await client .from(table) - .upsert(row, onConflict: onConflict, returning: .representation) + .upsert(rows, onConflict: onConflict, returning: .representation) .setHeader(name: "Authorization", value: "Bearer \(token)") .execute() .value - return rows.first ?? row } catch { throw Self.classify(error) // tag permanence so the drain can dead-letter poison writes } } func delete(table: String, primaryKey: String, pk: String) async throws { + try await delete(table: table, primaryKey: primaryKey, pks: [pk]) + } + + func delete(table: String, primaryKey: String, pks: [String]) async throws { + guard !pks.isEmpty else { return } // an empty `in.()` is a filter no row matches — skip the trip let token = await auth() // Soft delete: set the tombstone so the deletion propagates on the next cursor pull (a hard // DELETE would simply vanish, never reaching other devices). The server's BEFORE UPDATE // trigger stamps updatedAt — advancing the cursor — and an AFTER trigger tombstones the row's // children. deletedAt is the marker; the server-stamped updatedAt is what ordering trusts. + // + // One request for the whole batch: `in.(pk, …)` tombstones every named row in a single + // UPDATE, the delete-side counterpart of the batched upsert (issue #54). do { try await client .from(table) .update(["deletedAt": AnyJSON.string(Self.nowISO8601())]) - .eq(primaryKey, value: pk) + .in(primaryKey, values: pks) .setHeader(name: "Authorization", value: "Bearer \(token)") .execute() } catch { diff --git a/Sources/HappySync/SyncEngine.swift b/Sources/HappySync/SyncEngine.swift index f449af6..d27e2c9 100644 --- a/Sources/HappySync/SyncEngine.swift +++ b/Sources/HappySync/SyncEngine.swift @@ -87,6 +87,9 @@ public actor SyncEngine { private let tables: [SyncTable] private let remote: any SyncRemote private let pageSize: Int + /// Most entries the drain sends in one upload request. Bounds the request body and, since a + /// rejected chunk is re-run row by row, the blast radius of that fallback (issue #54). + private let uploadChunkSize: Int private let doorbell: any SyncDoorbell private let pollInterval: TimeInterval private let debounceInterval: TimeInterval @@ -233,6 +236,8 @@ public actor SyncEngine { /// - debounceInterval: Seconds a doorbell burst coalesces over before one pull runs. /// - deadLetterAfter: Failed attempts before a transient entry parks. Lower it to reach the /// dead-letter path without driving eight retries. + /// - uploadChunkSize: Most entries per upload request. Shrink it (e.g. `2`) to force chunk + /// boundaries on a small outbox. public static func forTesting( db: any DatabaseWriter, remote: any SyncRemote, @@ -243,7 +248,8 @@ public actor SyncEngine { debounceInterval: TimeInterval = 0.3, scope: @escaping @Sendable () async -> String? = { nil }, deadLetterAfter: Int = 8, - maxOfflineGap: TimeInterval = 30 * 24 * 3600 + maxOfflineGap: TimeInterval = 30 * 24 * 3600, + uploadChunkSize: Int = SyncEngine.defaultUploadChunkSize ) throws -> SyncEngine { try SyncEngine( db: db, @@ -255,10 +261,16 @@ public actor SyncEngine { debounceInterval: debounceInterval, scope: scope, deadLetterAfter: deadLetterAfter, - maxOfflineGap: maxOfflineGap + maxOfflineGap: maxOfflineGap, + uploadChunkSize: uploadChunkSize ) } + /// Rows per upload request unless a caller says otherwise. Big enough that a realistic backlog is + /// a handful of round trips, small enough to keep a request body — and a rejected chunk's per-row + /// re-run — bounded (issue #54). + public static let defaultUploadChunkSize = 256 + /// Injects the `SyncRemote`/`SyncDoorbell` seams directly. Reached publicly through /// `forTesting(db:remote:tables:…)`, which documents the parameters. init( @@ -271,7 +283,8 @@ public actor SyncEngine { debounceInterval: TimeInterval = 0.3, scope: @escaping @Sendable () async -> String? = { nil }, deadLetterAfter: Int = 8, - maxOfflineGap: TimeInterval = 30 * 24 * 3600 + maxOfflineGap: TimeInterval = 30 * 24 * 3600, + uploadChunkSize: Int = SyncEngine.defaultUploadChunkSize ) throws { // The manifest is checked against the schema — and its `dependsOn` filled in from the schema's // foreign keys — before anything else reads it, so `self.tables` only ever holds specs the @@ -284,6 +297,7 @@ public actor SyncEngine { self.remote = remote self.tables = resolved self.pageSize = pageSize + self.uploadChunkSize = uploadChunkSize self.doorbell = doorbell self.pollInterval = pollInterval self.debounceInterval = debounceInterval @@ -1071,6 +1085,13 @@ public actor SyncEngine { /// bumped and is skipped until its per-entry backoff window elapses; one that fails permanently /// (4xx) or exhausts `deadLetterAfter` retries is dead-lettered (parked). One failure never /// blocks the others. + /// + /// Uploads go out **in batches**: consecutive same-table, same-op entries are sent as one request + /// (issue #54), so a 500-row backlog is a handful of round trips rather than 500 sequential ones. + /// A rejected batch says which chunk the server refused, not which row — so the chunk is re-run + /// one entry at a time, and exactly the offending entries are classified and parked the way they + /// would have been without batching. Batch speed on the healthy path; unchanged diagnostics on + /// the unhealthy one. @discardableResult func drainOutbox(now: Date = Date()) async throws -> DrainOutcome { let pending = try await db.read { db in @@ -1079,32 +1100,123 @@ public actor SyncEngine { // Net each (table, pk) to one op first (APPS-472), then FK-order the net ops. let collapsed = collapseOutbox(pending) let groupSeqs = Dictionary(uniqueKeysWithValues: collapsed.map { ($0.net.seq, $0.seqs) }) + // Per-entry exponential backoff: drop an entry still inside its retry window so a failing + // entry isn't re-attempted on every drain pass (and doorbell ring and poll) — and never rides + // into a batch that would attempt it early. Dropping preserves the FK-safe relative order, + // so what's left still chunks into single-table, single-op requests. + let ready = orderForUpload(collapsed.map(\.net), tables: tables).filter { entry in + guard let last = entry.lastAttemptAt else { return true } + return now.timeIntervalSince(last) >= backoffDelay(attempts: entry.attempts) + } var outcome = DrainOutcome(failed: 0, deadLettered: 0) - for entry in orderForUpload(collapsed.map(\.net), tables: tables) { - if stopping { throw StopRequested() } // teardown: bail between entries — each clear is transactional (APPS-513) - // Per-entry exponential backoff: skip an entry still inside its retry window so a failing - // entry isn't re-attempted on every drain pass (and doorbell ring and poll). - if let last = entry.lastAttemptAt, now.timeIntervalSince(last) < backoffDelay(attempts: entry.attempts) { - continue - } - let seqs = groupSeqs[entry.seq] ?? [entry.seq] - do { - try await process(entry, clearing: seqs) - } catch { - // Auth-shaped failures (expired/refreshing token) recover out-of-band, so retry them - // without charging the retry budget — a token-refresh stretch mustn't dead-letter - // healthy writes and strip their dirty-row protection (APPS-502). Genuine transient - // failures still count toward `deadLetterAfter`; permanent ones park at once. - let attempts = remoteErrorIsAuthTransient(error) ? entry.attempts : entry.attempts + 1 - // Permanent (4xx / constraint / RLS) → park now; transient → park once it exhausts the cap. - let park = remoteErrorIsPermanent(error) || attempts >= deadLetterAfter - try await recordFailure(entry, groupSeqs: seqs, attempts: attempts, now: now, park: park, error: error) - if park { outcome.deadLettered += 1 } else { outcome.failed += 1 } + for chunk in uploadChunks(ready, limit: uploadChunkSize) { + if stopping { throw StopRequested() } // teardown: bail between chunks — each clear is transactional (APPS-513) + // One request for the whole chunk; on any failure fall through to one request per entry, + // which is also the only path a single-entry chunk ever takes. + if chunk.count > 1, await uploadedAsBatch(chunk, groupSeqs: groupSeqs) { continue } + for entry in chunk { + if stopping { throw StopRequested() } + let seqs = groupSeqs[entry.seq] ?? [entry.seq] + do { + try await process(entry, clearing: seqs) + } catch { + // Auth-shaped failures (expired/refreshing token) recover out-of-band, so retry them + // without charging the retry budget — a token-refresh stretch mustn't dead-letter + // healthy writes and strip their dirty-row protection (APPS-502). Genuine transient + // failures still count toward `deadLetterAfter`; permanent ones park at once. + let attempts = remoteErrorIsAuthTransient(error) ? entry.attempts : entry.attempts + 1 + // Permanent (4xx / constraint / RLS) → park now; transient → park once it exhausts the cap. + let park = remoteErrorIsPermanent(error) || attempts >= deadLetterAfter + try await recordFailure(entry, groupSeqs: seqs, attempts: attempts, now: now, park: park, error: error) + if park { outcome.deadLettered += 1 } else { outcome.failed += 1 } + } } } return outcome } + /// Applies a whole chunk — same table, same op — as **one** request, returning false when it + /// didn't land so the caller re-runs the chunk entry by entry. + /// + /// Nothing is recorded against the entries on failure, deliberately: a rejected batch names the + /// request, not the row, so charging every entry an attempt would park up to `limit` healthy + /// writes for one poison neighbour. The per-entry re-run is what decides who was actually at + /// fault; the extra round trips it costs are paid only on the unhealthy path. + private func uploadedAsBatch(_ chunk: [OutboxEntry], groupSeqs: [Int64: [Int64]]) async -> Bool { + guard let spec = tables.first(where: { $0.name == chunk[0].tableName }) else { + return false // table no longer synced — let the per-entry path drop the stale entries + } + func seqs(of entry: OutboxEntry) -> [Int64] { groupSeqs[entry.seq] ?? [entry.seq] } + do { + switch chunk[0].op { + case .upsert: + // A row deleted locally before its upload went out has nothing to send: clear those + // entries and batch what's left, exactly as the per-entry path does one at a time. + var uploads: [OutboxEntry] = [] + var payloads: [[String: AnyJSON]] = [] + var vanished: [Int64] = [] + for entry in chunk { + if let payload = try await readPayload(spec: spec, pk: entry.pk) { + uploads.append(entry) + payloads.append(payload) + } else { + vanished.append(contentsOf: seqs(of: entry)) + } + } + try await clear(vanished) + guard !uploads.isEmpty else { return true } + let onConflict = spec.conflictColumns.isEmpty ? nil : spec.conflictColumns.joined(separator: ",") + let servers = try await remote.upsert(table: spec.name, rows: payloads, onConflict: onConflict) + let matched = Self.match(servers, to: uploads, primaryKey: spec.primaryKey) + try await stampAndClear( + zip(uploads, matched).map { entry, server in + ConfirmedUpload(entry: entry, server: server, seqs: seqs(of: entry)) + }, + spec: spec + ) + case .delete: + try await remote.delete(table: spec.name, primaryKey: spec.primaryKey, pks: chunk.map(\.pk)) + try await clear(chunk.flatMap { seqs(of: $0) }) + } + return true + } catch { + return false + } + } + + /// Pairs a batch upsert's returned representations with the rows that produced them, one slot per + /// upload and `nil` where no representation could be identified. + /// + /// Primary key first: it's the only match that can't put one row's server state onto another, and + /// PostgREST returns the full row so the key is there. Request order is the documented fallback, + /// taken only when the counts agree — a short or unkeyed response leaves those uploads unstamped, + /// which costs a re-download on the next pull rather than corrupting a row. + private static func match( + _ servers: [[String: AnyJSON]], to uploads: [OutboxEntry], primaryKey: String + ) -> [[String: AnyJSON]?] { + let byPK = Dictionary( + servers.compactMap { row -> (String, [String: AnyJSON])? in + guard let key = row[primaryKey].flatMap(Self.wireKey) else { return nil } + return (key, row) + }, + uniquingKeysWith: { first, _ in first } + ) + let positional = servers.count == uploads.count + return uploads.enumerated().map { index, entry in + byPK[entry.pk] ?? (positional ? servers[index] : nil) + } + } + + /// A wire primary key as the outbox stores it. `_sync_outbox.pk` has TEXT affinity, so an integer + /// key is `"7"` there and must be compared as one; anything that isn't a scalar key is no key. + private static func wireKey(_ value: AnyJSON) -> String? { + switch value { + case .string(let string): return string + case .integer(let int): return String(int) + default: return nil + } + } + /// Records a failed upload attempt: bumps `attempts`, stamps `last_attempt_at` (for the backoff /// window) and `last_error` (telemetry) on the net entry. When parking, the **whole collapsed /// group** is dead-lettered together — else a superseded older op (e.g. an orphaned delete) @@ -1173,7 +1285,7 @@ public actor SyncEngine { } let onConflict = spec.conflictColumns.isEmpty ? nil : spec.conflictColumns.joined(separator: ",") let server = try await remote.upsert(table: spec.name, row: payload, onConflict: onConflict) - try await stampAndClear(entry, spec: spec, server: server, seqs: seqs) + try await stampAndClear([ConfirmedUpload(entry: entry, server: server, seqs: seqs)], spec: spec) case .delete: try await remote.delete(table: spec.name, primaryKey: spec.primaryKey, pk: entry.pk) try await clear(seqs) @@ -1196,9 +1308,18 @@ public actor SyncEngine { } } - /// Writes the server's representation back to the local row and clears the collapsed group's - /// entries in one transaction, so the row is marked clean (its cursor won't re-pull it) only - /// after the server confirms. + /// One confirmed upload waiting to be written back: the entry the server accepted, its + /// representation of the row (nil when a batch response couldn't be matched to this upload — see + /// `match`), and every outbox `seq` the entry's collapsed group covers. + private struct ConfirmedUpload: Sendable { + let entry: OutboxEntry + let server: [String: AnyJSON]? + let seqs: [Int64] + } + + /// Writes the server's representation back to the local rows and clears the confirmed entries in + /// **one transaction**, so a row is marked clean (its cursor won't re-pull it) only after the + /// server confirms. A batch confirms as a unit, so it write-backs as one too. /// /// The representation carries what the client can't compute locally: column **defaults** the /// server filled in, **trigger-normalized** fields, recomputed `serverOwnedColumns`, and — for a @@ -1214,20 +1335,25 @@ public actor SyncEngine { /// holds that pending edit — the same dirty-check the pull's `lwwAllows` gate uses. Clobbering it /// would lose the edit; it uploads on the next drain instead. The confirmed group's entries clear /// either way. - private func stampAndClear(_ entry: OutboxEntry, spec: SyncTable, server: [String: AnyJSON], seqs: [Int64]) async throws { + private func stampAndClear(_ confirmed: [ConfirmedUpload], spec: SyncTable) async throws { + guard !confirmed.isEmpty else { return } try await db.write { db in - // The server's own representation of the row we just uploaded — suppressed, or writing it + // The server's own representation of the rows we just uploaded — suppressed, or writing it // back would queue the very upload it confirms and the drain would never settle (issue #48). try SyncTriggers.applyingRemoteState(db) { - if try Self.writeBackAllowed(db, table: spec.name, pk: entry.pk, excluding: seqs) { - try RowCoding.upsertLocalRow( - db, table: spec.name, primaryKey: spec.primaryKey, - columns: RowCoding.localColumns(from: server), - restrictingTo: try RowCoding.tableColumns(db, table: spec.name) - ) + let knownColumns = try RowCoding.tableColumns(db, table: spec.name) + for upload in confirmed { + guard let server = upload.server else { continue } + if try Self.writeBackAllowed(db, table: spec.name, pk: upload.entry.pk, excluding: upload.seqs) { + try RowCoding.upsertLocalRow( + db, table: spec.name, primaryKey: spec.primaryKey, + columns: RowCoding.localColumns(from: server), + restrictingTo: knownColumns + ) + } } } - try Self.deleteEntries(db, seqs: seqs) + try Self.deleteEntries(db, seqs: confirmed.flatMap(\.seqs)) } } @@ -1247,6 +1373,7 @@ public actor SyncEngine { } private func clear(_ seqs: [Int64]) async throws { + guard !seqs.isEmpty else { return } try await db.write { db in try Self.deleteEntries(db, seqs: seqs) } } diff --git a/Sources/HappySync/SyncSchema.swift b/Sources/HappySync/SyncSchema.swift index b6631f7..4be13a7 100644 --- a/Sources/HappySync/SyncSchema.swift +++ b/Sources/HappySync/SyncSchema.swift @@ -1,7 +1,9 @@ import GRDB -/// Internal bookkeeping tables HappySync owns. The app never reads or writes these directly. -enum SyncSchema { +/// Internal bookkeeping tables HappySync owns. The app never reads or writes these directly — the +/// only public surface here is the migrator, for an app that runs its own `DatabaseMigrator` on the +/// same database and therefore has to share one with the engine. +public enum SyncSchema { /// Shared by every internal table below, so a schema scan can tell the engine's own tables from /// the app's without listing them (see `LocalSchema`). static let tablePrefix = "_sync_" @@ -16,8 +18,9 @@ enum SyncSchema { /// writes don't queue themselves straight back into the outbox (issue #48). static let controlTable = "_sync_control" - /// A standalone migrator for HappySync's internal tables. Used when the engine owns the DB. - static func migrator() -> DatabaseMigrator { + /// A standalone migrator for HappySync's internal tables. Used when the engine owns the DB — + /// which it runs itself at init, so an app only needs this to migrate ahead of time. + public static func migrator() -> DatabaseMigrator { var migrator = DatabaseMigrator() register(into: &migrator) return migrator @@ -30,7 +33,7 @@ enum SyncSchema { /// The per-table **write-capture triggers** (issue #48) are deliberately *not* here: they're /// derived from the `SyncTable` manifest, which is source code and ships without a schema version /// bump, so `SyncTriggers.install` (re)runs idempotently at every engine init instead. - static func register(into migrator: inout DatabaseMigrator) { + public static func register(into migrator: inout DatabaseMigrator) { migrator.registerMigration("happysync_v1") { db in try db.create(table: outboxTable) { t in t.autoIncrementedPrimaryKey("seq") diff --git a/Sources/HappySync/TypedManifest.swift b/Sources/HappySync/TypedManifest.swift new file mode 100644 index 0000000..28491a8 --- /dev/null +++ b/Sources/HappySync/TypedManifest.swift @@ -0,0 +1,64 @@ +import Foundation +import GRDB + +// Issue #50: the manifest was eight stringly-typed parameters, so a rename in a GRDB migration or a +// Swift record left it stale with no build error. `SyncEngine.init` catches that at launch (#49); +// this catches it at compile time, which is the difference between a crash on the tester's device +// and a red squiggle. + +public extension SyncTable { + /// Declares a synced table from its **GRDB record type**, reading the table name off the record + /// instead of restating it as a string. + /// + /// ```swift + /// let tables = [ + /// SyncTable.table(Recipe.self, jsonColumns: [Recipe.Columns.nutrition]), + /// SyncTable.table(RecipeIngredient.self), // dependsOn derived from the FK (issue #49) + /// ] + /// ``` + /// + /// Renaming the record's table — in the record or in the migration that created it — is now a + /// build error at the manifest rather than a table that silently stops syncing. The same holds + /// for `dependsOn`, which takes record types rather than names. + /// + /// **Columns are as checked as your record makes them.** Passing GRDB's conventional + /// `enum Columns` (`Recipe.Columns.nutrition`) ties each name to a declaration you rename in one + /// place; passing a bare `Column("nutrition")` is exactly as unchecked as the string form was, and + /// still validated against the schema at init. Column names can't be derived from key paths + /// without a `CodingKeys` bridge the app would have to write anyway, so the choice of how much + /// checking to buy stays with the consumer. + /// + /// Every parameter means what it means on ``SyncTable/init(name:primaryKey:cursorColumn:dependsOn:jsonColumns:serverOwnedColumns:scopeColumn:conflictColumns:serverColumns:)`` + /// — this is that initializer with the strings replaced, not a second set of semantics. The + /// string form stays for tables with no Swift record type (a join table written only in SQL, or + /// one owned by another module). + /// + /// - Parameters: + /// - record: The GRDB record whose `databaseTableName` names the table. + /// - dependsOn: Records this one references. `nil` (the default) derives the dependency from + /// the schema's own foreign keys — prefer it; pass a list only for a *logical* parent the + /// schema has no FK for. + static func table( + _ record: Record.Type, + primaryKey: any ColumnExpression = Column("id"), + cursorColumn: any ColumnExpression = Column("updatedAt"), + dependsOn: [any TableRecord.Type]? = nil, + jsonColumns: [any ColumnExpression] = [], + serverOwnedColumns: [any ColumnExpression] = [], + scopeColumn: (any ColumnExpression)? = nil, + conflictColumns: [any ColumnExpression] = [], + serverColumns: [any ColumnExpression] = [] + ) -> SyncTable { + SyncTable( + name: Record.databaseTableName, + primaryKey: primaryKey.name, + cursorColumn: cursorColumn.name, + dependsOn: dependsOn.map { records in records.map { $0.databaseTableName } }, + jsonColumns: jsonColumns.map { $0.name }, + serverOwnedColumns: serverOwnedColumns.map { $0.name }, + scopeColumn: scopeColumn.map { $0.name }, + conflictColumns: conflictColumns.map { $0.name }, + serverColumns: serverColumns.map { $0.name } + ) + } +} diff --git a/Sources/HappySyncTestSupport/InMemorySyncRemote.swift b/Sources/HappySyncTestSupport/InMemorySyncRemote.swift index 77bb4d6..2c5d70d 100644 --- a/Sources/HappySyncTestSupport/InMemorySyncRemote.swift +++ b/Sources/HappySyncTestSupport/InMemorySyncRemote.swift @@ -76,11 +76,19 @@ public actor InMemorySyncRemote: SyncRemote { /// table cursors on something other than `updatedAt`. public nonisolated let cursorColumn: String - /// Every upsert the drain has issued, in order — table, uploaded payload, and the PostgREST - /// conflict target (nil = conflict on the primary key). + /// Every **row** the drain has upserted, in order — table, uploaded payload, and the PostgREST + /// conflict target (nil = conflict on the primary key). A batched upload records one entry per + /// row it carried, so this counts writes regardless of how they were packed; `upsertRequests` + /// counts the requests. public private(set) var upsertCalls: [(table: String, row: [String: AnyJSON], onConflict: String?)] = [] - /// Every tombstone the drain has propagated, in order. + /// How many upsert **requests** were issued — the number of network round trips, which batching + /// makes smaller than `upsertCalls.count` (issue #54). A rejected batch counts here too, then the + /// drain's per-row re-run adds one request each. + public private(set) var upsertRequests = 0 + /// Every tombstone the drain has propagated, in order — one entry per row, as for `upsertCalls`. public private(set) var deleteCalls: [(table: String, pk: String)] = [] + /// How many delete **requests** were issued. See `upsertRequests`. + public private(set) var deleteRequests = 0 /// How many download pages have been requested. One pull of a single-table manifest that fits in /// one page is one call, so this is the usual way to observe "a sync pass ran". public private(set) var fetchCalls = 0 @@ -100,7 +108,9 @@ public actor InMemorySyncRemote: SyncRemote { /// - Parameters: /// - dataset: Server rows per table, as PostgREST would return them. Downloads are served from /// here, filtered by scope and cursor and truncated to the page limit. - /// - failUpserts: How many upserts throw before one succeeds. `99` stands in for "always". + /// - failUpserts: How many upload **requests** throw before one succeeds — a rejected batch + /// spends one, as does each row of the per-row re-run that follows it. `99` stands in for + /// "always". /// - upsertFailure: What those failures are — the difference between a write that retries and /// one that parks. /// - failFetches: How many downloads throw before one succeeds. @@ -147,7 +157,7 @@ public actor InMemorySyncRemote: SyncRemote { dataset[table] ?? [] } - /// Makes the next `count` upserts throw. Replaces any run still outstanding. + /// Makes the next `count` upload requests throw. Replaces any run still outstanding. public func failNextUpserts(_ count: Int, with failure: SimulatedFailure = .transient) { remainingUpsertFailures = count upsertFailure = failure @@ -180,6 +190,29 @@ public actor InMemorySyncRemote: SyncRemote { // MARK: - SyncRemote public func upsert(table: String, row: [String: AnyJSON], onConflict: String?) async throws -> [String: AnyJSON] { + upsertRequests += 1 + return try await store(table: table, row: row, onConflict: onConflict) + } + + /// The batched upload the drain prefers (issue #54). One request, so an injected failure rejects + /// the **whole** batch before any of its rows is recorded — which is how PostgREST answers a + /// rejected array body, and what makes the drain fall back to one row at a time. + public func upsert(table: String, rows: [[String: AnyJSON]], onConflict: String?) async throws -> [[String: AnyJSON]] { + upsertRequests += 1 + if remainingUpsertFailures > 0 { + remainingUpsertFailures -= 1 + throw upsertFailure.makeError() + } + var representations: [[String: AnyJSON]] = [] + for row in rows { + representations.append(try await store(table: table, row: row, onConflict: onConflict)) + } + return representations + } + + /// Records one uploaded row and answers with the server's representation of it. Shared by the + /// single-row and batched entry points, which own the request accounting. + private func store(table: String, row: [String: AnyJSON], onConflict: String?) async throws -> [String: AnyJSON] { upsertCalls.append((table, row, onConflict)) if let upsertHook { await upsertHook(upsertCalls.count) } if remainingUpsertFailures > 0 { @@ -193,9 +226,15 @@ public actor InMemorySyncRemote: SyncRemote { } public func delete(table: String, primaryKey: String, pk: String) async throws { + deleteRequests += 1 deleteCalls.append((table, pk)) } + public func delete(table: String, primaryKey: String, pks: [String]) async throws { + deleteRequests += 1 + for pk in pks { deleteCalls.append((table, pk)) } + } + public func fetch( table: String, cursorColumn: String, diff --git a/Tests/HappySyncTests/BatchUploadTests.swift b/Tests/HappySyncTests/BatchUploadTests.swift new file mode 100644 index 0000000..294ad30 --- /dev/null +++ b/Tests/HappySyncTests/BatchUploadTests.swift @@ -0,0 +1,244 @@ +import Testing +import Foundation +import GRDB +import Supabase +import HappySyncTestSupport +@testable import HappySync + +// The outbox drain sends consecutive same-table, same-op entries as one request, and falls back to +// one request per entry when the server rejects the batch (issue #54). These tests hold both halves +// down: the request count on the healthy path, and per-entry failure isolation on the unhealthy one. + +// MARK: - Chunking + +@Test func chunksSplitOnTableOpAndCap() { + let entries = [ + entry(seq: 1, table: "recipes", pk: "r1", op: .upsert), + entry(seq: 2, table: "recipes", pk: "r2", op: .upsert), + entry(seq: 3, table: "recipes", pk: "r3", op: .upsert), + entry(seq: 4, table: "recipeSteps", pk: "s1", op: .upsert), // table boundary + entry(seq: 5, table: "recipeSteps", pk: "s2", op: .delete), // op boundary + ] + + let shape = uploadChunks(entries, limit: 2).map { $0.map(\.pk) } + #expect(shape == [["r1", "r2"], ["r3"], ["s1"], ["s2"]]) // cap splits the first run at two +} + +@Test func chunkingNeverReordersToMakeABiggerBatch() { + // Interleaved tables: batching only ever merges *adjacent* entries, so the FK-safe order + // `orderForUpload` produced survives even when regrouping would give fewer requests. + let entries = [ + entry(seq: 1, table: "recipes", pk: "r1", op: .upsert), + entry(seq: 2, table: "recipeSteps", pk: "s1", op: .upsert), + entry(seq: 3, table: "recipes", pk: "r2", op: .upsert), + ] + + let shape = uploadChunks(entries, limit: 256).map { $0.map(\.pk) } + #expect(shape == [["r1"], ["s1"], ["r2"]]) +} + +// MARK: - Healthy path + +@Test func healthyBacklogUploadsInOneRequest() async throws { + let db = try recipesDB() + let remote = InMemorySyncRemote() + let engine = try SyncEngine.forTesting(db: db, remote: remote, tables: [SyncTable(name: "recipes")]) + for i in 1...5 { + try await write(db, "INSERT INTO recipes (id, title) VALUES (?, ?)", ["r\(i)", "T\(i)"]) + } + + try await engine.drainOutbox() + + #expect(await remote.upsertRequests == 1) // five rows, one round trip + #expect(await remote.upsertCalls.count == 5) // …carrying every row + let (pending, stamped) = try await db.read { db -> (Int, Int) in + (try Int.fetchOne(db, sql: "SELECT COUNT(*) FROM _sync_outbox") ?? -1, + try Int.fetchOne(db, sql: "SELECT COUNT(*) FROM recipes WHERE updatedAt = ?", arguments: [remote.serverUpdatedAt]) ?? -1) + } + #expect(pending == 0) // every entry cleared + #expect(stamped == 5) // and every row carries the server's representation (#29 preserved) +} + +@Test func backlogIsSplitAtTheChunkCap() async throws { + let db = try recipesDB() + let remote = InMemorySyncRemote() + let engine = try SyncEngine.forTesting( + db: db, remote: remote, tables: [SyncTable(name: "recipes")], uploadChunkSize: 2 + ) + for i in 1...5 { + try await write(db, "INSERT INTO recipes (id, title) VALUES (?, ?)", ["r\(i)", "T\(i)"]) + } + + try await engine.drainOutbox() + + #expect(await remote.upsertRequests == 3) // ⌈5/2⌉ — the cap bounds the request, not the pass + #expect(await remote.upsertCalls.count == 5) + #expect(try await db.read { try Int.fetchOne($0, sql: "SELECT COUNT(*) FROM _sync_outbox") } == 0) +} + +@Test func batchedUploadKeepsParentsBeforeChildren() async throws { + let db = try recipeGraphDB() + let remote = InMemorySyncRemote() + let engine = try SyncEngine.forTesting(db: db, remote: remote, tables: recipeGraphTables) + try await write(db, "INSERT INTO recipes (id, title) VALUES ('r1', 'Soup')") + try await write(db, "INSERT INTO recipeIngredients (id, recipeId) VALUES ('i1', 'r1')") + try await write(db, "INSERT INTO recipeIngredients (id, recipeId) VALUES ('i2', 'r1')") + + try await engine.drainOutbox() + + // One request per table — the children batch together, but never ahead of their parent. + #expect(await remote.upsertRequests == 2) + #expect(await remote.upsertCalls.map(\.table) == ["recipes", "recipeIngredients", "recipeIngredients"]) +} + +@Test func deletesTombstoneInOneRequest() async throws { + let db = try recipesDB() + let remote = InMemorySyncRemote() + let engine = try SyncEngine.forTesting(db: db, remote: remote, tables: [SyncTable(name: "recipes")]) + for i in 1...3 { + try await write(db, "INSERT INTO recipes (id, title) VALUES (?, ?)", ["r\(i)", "T\(i)"]) + } + try await engine.drainOutbox() // upload them first, so the deletes below are the only pending ops + try await write(db, "DELETE FROM recipes") + + try await engine.drainOutbox() + + #expect(await remote.deleteRequests == 1) // one `in.(…)` tombstone update for all three + #expect(await remote.deleteCalls.map(\.pk).sorted() == ["r1", "r2", "r3"]) + #expect(try await db.read { try Int.fetchOne($0, sql: "SELECT COUNT(*) FROM _sync_outbox") } == 0) +} + +// MARK: - Failure isolation + +@Test func poisonRowInABatchParksAloneAndItsNeighboursUpload() async throws { + let db = try recipesDB() + // The server rejects r3 — and, like PostgREST, rejects any array body containing it rather than + // singling the row out. The drain has to find the culprit itself. + let remote = PoisonRowRemote(poison: "r3") + let engine = try SyncEngine.forTesting(db: db, remote: remote, tables: [SyncTable(name: "recipes")]) + for i in 1...4 { + try await write(db, "INSERT INTO recipes (id, title) VALUES (?, ?)", ["r\(i)", "T\(i)"]) + } + + let outcome = try await engine.drainOutbox() + + #expect(outcome.deadLettered == 1) + #expect(await remote.uploaded.sorted() == ["r1", "r2", "r4"]) // the healthy neighbours still landed + #expect(await remote.requests == 5) // the rejected batch, then one per row + let parked = try await engine.deadLetters() + #expect(parked.map(\.pk) == ["r3"]) + // Classified per row, exactly as an unbatched drain would — the batch failure told us nothing. + #expect(parked.first?.failure == .constraintViolation(code: "23505")) + let pending = try await db.read { try Int.fetchOne($0, sql: "SELECT COUNT(*) FROM _sync_outbox WHERE dead_lettered = 0") } + #expect(pending == 0) // nothing but the poison row is left in the outbox +} + +@Test func batchFailureChargesNoAttemptsToHealthyEntries() async throws { + let db = try recipesDB() + // Every request fails transiently. The batch attempt must not charge the retry budget on its own + // — otherwise one chunk-sized failure would spend two attempts per entry and park healthy writes + // at half the configured cap. + let remote = InMemorySyncRemote(failUpserts: 99) + let engine = try SyncEngine.forTesting( + db: db, remote: remote, tables: [SyncTable(name: "recipes")], deadLetterAfter: 3 + ) + for i in 1...3 { + try await write(db, "INSERT INTO recipes (id, title) VALUES (?, ?)", ["r\(i)", "T\(i)"]) + } + + let outcome = try await engine.drainOutbox() + + #expect(outcome.failed == 3) + #expect(outcome.deadLettered == 0) + let attempts = try await db.read { try Int.fetchAll($0, sql: "SELECT attempts FROM _sync_outbox ORDER BY seq") } + #expect(attempts == [1, 1, 1]) // one attempt each — the failed batch itself charged nothing +} + +@Test func representationsAreMatchedByPrimaryKeyNotArrivalOrder() async throws { + let db = try recipesDB() + // A server that answers a batch with the representations reversed. Matching them positionally + // would stamp each row with its neighbour's server state; matching by primary key can't. + let remote = ReversingRemote() + let engine = try SyncEngine.forTesting(db: db, remote: remote, tables: [SyncTable(name: "recipes")]) + for i in 1...3 { + try await write(db, "INSERT INTO recipes (id, title) VALUES (?, ?)", ["r\(i)", "T\(i)"]) + } + + try await engine.drainOutbox() + + let titles = try await db.read { + try String.fetchAll($0, sql: "SELECT title FROM recipes ORDER BY id") + } + #expect(titles == ["T1 (server)", "T2 (server)", "T3 (server)"]) // each row got *its own* representation +} + +// MARK: - Fixtures + +private func entry(seq: Int64, table: String, pk: String, op: SyncOp) -> OutboxEntry { + OutboxEntry(row: ["seq": seq, "table_name": table, "pk": pk, "op": op.rawValue, "attempts": 0]) +} + +/// A remote that rejects one primary key the way Postgres rejects a constraint violation — and +/// rejects any batch containing it, since PostgREST refuses the whole array body rather than the +/// offending row. Records what actually landed, and how many requests it took. +private actor PoisonRowRemote: SyncRemote { + let poison: String + private(set) var uploaded: [String] = [] + private(set) var requests = 0 + + init(poison: String) { self.poison = poison } + + func upsert(table: String, row: [String: AnyJSON], onConflict: String?) async throws -> [String: AnyJSON] { + try send([row])[0] + } + + func upsert(table: String, rows: [[String: AnyJSON]], onConflict: String?) async throws -> [[String: AnyJSON]] { + try send(rows) + } + + private func send(_ rows: [[String: AnyJSON]]) throws -> [[String: AnyJSON]] { + requests += 1 + guard !rows.contains(where: { $0["id"]?.stringValue == poison }) else { + throw RemoteFailure(classifying: PostgrestError(code: "23505", message: "unique_violation")) + } + uploaded.append(contentsOf: rows.compactMap { $0["id"]?.stringValue }) + return rows.map { row in + var server = row + server["updatedAt"] = .string("2026-06-30T12:00:00.000Z") + return server + } + } + + func delete(table: String, primaryKey: String, pk: String) async throws {} + + func fetch( + table: String, cursorColumn: String, since cursor: SyncCursor?, + primaryKey: String, scope: ScopeFilter?, limit: Int + ) async throws -> [[String: AnyJSON]] { [] } +} + +/// A remote that answers a batch with its representations in reverse order, each tagged so a +/// mis-matched write-back is visible in the row it lands on. +private struct ReversingRemote: SyncRemote { + func upsert(table: String, row: [String: AnyJSON], onConflict: String?) async throws -> [String: AnyJSON] { + Self.representation(of: row) + } + + func upsert(table: String, rows: [[String: AnyJSON]], onConflict: String?) async throws -> [[String: AnyJSON]] { + Array(rows.map(Self.representation).reversed()) + } + + private static func representation(of row: [String: AnyJSON]) -> [String: AnyJSON] { + var server = row + server["title"] = .string("\(row["title"]?.stringValue ?? "") (server)") + server["updatedAt"] = .string("2026-06-30T12:00:00.000Z") + return server + } + + func delete(table: String, primaryKey: String, pk: String) async throws {} + + func fetch( + table: String, cursorColumn: String, since cursor: SyncCursor?, + primaryKey: String, scope: ScopeFilter?, limit: Int + ) async throws -> [[String: AnyJSON]] { [] } +} diff --git a/Tests/HappySyncTests/TypedManifestTests.swift b/Tests/HappySyncTests/TypedManifestTests.swift new file mode 100644 index 0000000..06e4c7c --- /dev/null +++ b/Tests/HappySyncTests/TypedManifestTests.swift @@ -0,0 +1,110 @@ +import Testing +import Foundation +import GRDB +import HappySyncTestSupport +@testable import HappySync + +// Issue #50: `SyncTable.table(Record.self, …)` reads the table name off the GRDB record, so a rename +// is a build error at the manifest rather than a table that quietly stops syncing. It produces the +// same `SyncTable` the string form does — these tests hold that equivalence down, because the moment +// the two diverge the typed form becomes a second set of semantics to learn. + +// MARK: - Fixtures + +/// A record with GRDB's conventional `Columns` namespace — passing those instead of bare `Column`s is +/// what buys the column names their compile-time check. +private struct Recipe: TableRecord { + static let databaseTableName = "recipes" + + enum Columns { + static let id = Column("id") + static let nutrition = Column("nutrition") + static let userId = Column("userId") + } +} + +private struct RecipeIngredient: TableRecord { + static let databaseTableName = "recipeIngredients" + + enum Columns { + static let amount = Column("amount") + static let userId = Column("userId") + } +} + +/// A record whose table the schema doesn't have — the case compile-time checking can't catch. +private struct Ghost: TableRecord { + static let databaseTableName = "ghost" +} + +// MARK: - Tests + +@Test func typedTableReadsItsNameFromTheRecord() { + let table = SyncTable.table(Recipe.self) + + #expect(table.name == "recipes") + #expect(table.primaryKey == "id") + #expect(table.cursorColumn == "updatedAt") + #expect(table.dependsOn == nil) // nil still means "derive it from the schema's FKs" (issue #49) +} + +@Test func typedTableTakesColumnsAndDependenciesAsDeclarations() { + let table = SyncTable.table( + RecipeIngredient.self, + dependsOn: [Recipe.self], + jsonColumns: [RecipeIngredient.Columns.amount], + scopeColumn: RecipeIngredient.Columns.userId + ) + + #expect(table.name == "recipeIngredients") + #expect(table.dependsOn == ["recipes"]) // a record type, not a string that can rot + #expect(table.jsonColumns == ["amount"]) + #expect(table.scopeColumn == "userId") +} + +@Test func typedAndStringManifestsProduceTheSameTable() { + let typed = SyncTable.table( + Recipe.self, + primaryKey: Recipe.Columns.id, + dependsOn: [RecipeIngredient.self], + jsonColumns: [Recipe.Columns.nutrition], + scopeColumn: Recipe.Columns.userId + ) + let strings = SyncTable( + name: "recipes", + primaryKey: "id", + dependsOn: ["recipeIngredients"], + jsonColumns: ["nutrition"], + scopeColumn: "userId" + ) + + #expect(typed == strings) // one manifest type, two ways to spell it +} + +@Test func aTypedManifestDrivesTheEngine() async throws { + // The point of the equivalence: everything downstream — validation, trigger install, the drain — + // is the same code path, so a typed manifest is a spelling choice rather than a mode. + let db = try recipesDB() + let remote = InMemorySyncRemote() + let engine = try SyncEngine.forTesting( + db: db, + remote: remote, + tables: [SyncTable.table(Recipe.self, jsonColumns: [Recipe.Columns.nutrition])] + ) + + try await write(db, "INSERT INTO recipes (id, title) VALUES ('r1', 'Soup')") + try await engine.drainOutbox() + + #expect(await remote.upsertCalls.map(\.table) == ["recipes"]) + #expect(try await db.read { try Int.fetchOne($0, sql: "SELECT COUNT(*) FROM _sync_outbox") } == 0) +} + +@Test func aTypedManifestIsStillCheckedAgainstTheSchema() throws { + // Compile-time checking is not a substitute for the launch check: the record's table name is + // checked against the *record*, and the record can be ahead of the migration that creates it. + let db = try recipesDB() + + #expect(throws: SyncError.self) { + _ = try SyncEngine.forTesting(db: db, remote: InMemorySyncRemote(), tables: [.table(Ghost.self)]) + } +} diff --git a/docs/SYNC-CONTRACT.md b/docs/SYNC-CONTRACT.md index 51f5ff5..954a218 100644 --- a/docs/SYNC-CONTRACT.md +++ b/docs/SYNC-CONTRACT.md @@ -74,6 +74,14 @@ newer `now()` and wins; a plain PostgREST upsert is sufficient. - **PostgREST upsert** with `Prefer: return=representation` for `.upsert`; soft-delete for `.delete`. Both are **idempotent by primary key**, so retries are safe; back off exponentially **per entry** (`last_attempt_at` gates the window) and count `attempts`. +- **Uploads are batched, failures are not (#54).** Consecutive entries sharing a table and an op go + out as **one** request — an array body for the upsert, `in.(…)` for the tombstone update — capped + at 256 rows, so a backlog costs ⌈N/256⌉ round trips per run rather than N. Entries are never + reordered to make a bigger batch, so the FK order below survives. A rejected batch identifies the + request, not the row, so the chunk is **re-run one entry at a time** and only the entries that + genuinely fail are classified, retried and parked: batch speed on the healthy path, per-entry + diagnostics on the unhealthy one. The failed batch itself charges no attempts — otherwise one + poison row would spend the retry budget of every healthy row beside it. - **The full representation is written back locally (APPS-506).** On a successful upsert the server row — column defaults, trigger-normalized fields, recomputed `serverOwnedColumns`, and (for a `conflictColumns` upsert) the **merged** row re-keyed to the client's pk (APPS-478) — is applied to diff --git a/wiki/API-Reference.md b/wiki/API-Reference.md index de8c54f..c934360 100644 --- a/wiki/API-Reference.md +++ b/wiki/API-Reference.md @@ -1,233 +1,13 @@ # API Reference -All types are in the `HappySync` module. `SyncEngine` is an `actor`; its async methods are `await`ed -and safe to call from any task. +**The API reference is now generated from the source**, so a signature here can no longer drift from +the one you actually call: -## SyncEngine +### → [HappySync documentation](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync) -```swift -public init( - db: any DatabaseWriter, // GRDB writer; local source of truth - supabase: SupabaseClient, - tables: [SyncTable], // the manifest, declared once - auth: @escaping @Sendable () async -> String, // fresh access token per call - scope: @escaping @Sendable () async -> String? = { nil }, // partition value for scopeColumn tables - maxOfflineGap: TimeInterval = 30 * 24 * 3600 // stale-cursor full-resync threshold -) throws -``` +Every public type carries its documentation in the source it documents. Start with +[`SyncEngine`](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/syncengine) and [`SyncTable`](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/synctable). -| Method | Behavior | -|---|---| -| `func start()` | Begins background sync. Idempotent. Immediate convergence sync, then drives from the Realtime doorbell, a periodic poll, and retry backoff through one serialized runner. | -| `func stop() async` | **Async.** Awaits the in-flight pass, quiesces (no more DB writes / network), unsubscribes Realtime. Settles `status` to `.idle` but leaves subscriptions open. `start()` re-subscribes cleanly. Call **before** wiping/replacing the DB. | -| `func enqueue(_ op: SyncOp, table: String, row: some Encodable & Sendable) throws` | **Deprecated.** Performs the write; the table's capture trigger queues it. Write GRDB directly instead. Throws `SyncError`. | -| `func syncNow()` | Fire-and-forget nudge (foreground, pull-to-refresh). Not needed after a write — the engine wakes on queued writes itself. | -| `func pullNow() async throws` | Cursor pull now — the app-driven nudge (foreground return, pull-to-refresh). Returns nothing. | -| `var status: AsyncStream` | `nonisolated`. Live status for the UI. Each access is an independent stream replaying the latest snapshot; it survives `stop()`/`start()` and only ends when the engine is deallocated. | -| `func deadLetters() async throws -> [DeadLetter]` | Inspect parked entries. | -| `func retryDeadLetters(_ seqs: [Int64]? = nil) async throws` | Re-queue parked writes (specific `seq`s, or all). | -| `func discardDeadLetters(_ seqs: [Int64]? = nil) async throws` | Drop parked writes and re-pull affected rows so local converges to the server. | +--- -## SyncTable - -```swift -public init( - name: String, // identical local + remote table name - primaryKey: String = "id", - cursorColumn: String = "updatedAt", // change-time column the cursor orders/filters/advances by - // — server-stamped: never included in an upload payload - dependsOn: [String]? = nil, // nil = derive from the schema's FKs; drives sync + delete ordering - jsonColumns: [String] = [], // JSON text locally ↔ json/jsonb remote - serverOwnedColumns: [String] = [], // *extra* RPC-managed columns; stripped too, applied on download - scopeColumn: String? = nil, // partition column when RLS is broader than the partition - conflictColumns: [String] = [], // secondary-unique upsert target; LEAF tables only - serverColumns: [String] = [] // optional allow-list to survive a dropped/renamed column -) -``` - -| Field | Notes | -|---|---| -| `dependsOn` | FK dependencies. Parents upsert/download before children; children tombstone before parents. Leave it `nil` (the default) and the engine reads `PRAGMA foreign_key_list` at init and uses the table's real FK parents. Declare it only to **override** that for a *logical* parent with no FK constraint — an explicit list replaces the derived one rather than adding to it, and every name in it must be a declared `SyncTable`. A self-reference (a tree table) is dropped from either form. | -| `scopeColumn` | Requires the engine `scope:` closure. While no partition value is available (cold launch, signed out), the table's pull is skipped and the stale-resync check stays armed — never wipes an un-pulled table. | -| `conflictColumns` | Merges a fresh-pk insert onto an existing server row by a secondary `UNIQUE` (avoids a permanent 409). The merge **re-keys the server row to the client's pk** → **leaf tables only**, or you orphan children. | -| `serverColumns` | Opt-in backstop; a stale list silently stops uploading a real column. Prefer the client-first removal rule. Downloads need no equivalent (they intersect against the local schema). | -| `serverOwnedColumns` | Stripped from every upsert so a stale client value never clobbers the authoritative one; still arrive on download. Declare only the *extra* ones — the `cursorColumn` is always stripped. | -| `cursorColumn` | Server-stamped by contract (§1/§4), so the engine **never uploads it** — nothing to declare, and no way to opt out. Defence in depth for the one server-side requirement the engine can't verify: on a table whose trigger is missing, a client-sent value would silently make that device's clock the LWW authority. `deletedAt` is *not* stripped — an upsert carrying `deletedAt = null` is how a re-created row un-tombstones. | - -## SyncStatus - -```swift -public struct SyncStatus { - public enum Phase { - case idle // settled and clean — genuinely healthy - case syncing - case degraded // settled, but writes are still failing or parked - case failed(SyncFailure) - } - public var phase: Phase - public var lastSyncedAt: Date? // last successful pull/push, or nil - public var failedUploads: Int // entries retrying with backoff (attempts > 0, not parked) - public var deadLetters: Int // entries parked after a permanent 4xx or exhausted retries - public var isHealthy: Bool // phase == .idle && both counts zero -} -``` - -**Ask `status.isHealthy`.** `.idle` on a status the engine broadcast already implies it — the engine -derives `.idle` vs `.degraded` from the counts — but `isHealthy` is the one read that stays correct -on a `SyncStatus` you construct yourself, and it's the one that documents the intent. - -## SyncFailure - -The classified cause carried by `Phase.failed` and `DeadLetter.failure` — so an app can branch on -*why* instead of substring-matching PostgREST prose. - -```swift -public enum SyncFailure: Error, Sendable, Equatable { - case network // URLError, dropped connection, Postgres 08xxx - case authExpired // 401/403, PGRST301/302 — prompt re-auth - case permissionDenied // 42501 — RLS rejected the write - case constraintViolation(code: String) // 23xxx — unique, FK, not-null, check - case schemaMismatch(column: String?) // 42703 / PGRST204 — prompt an app update - case server(status: Int) // the server answered, including 5xx - case other(String) // unrecognised — original text preserved -} -``` - -The kind says *what to show*, not whether to retry: `failedUploads` vs `deadLetters` is what tells -you a write is still being retried or has been parked. - -## DeadLetter - -```swift -public struct DeadLetter { - public let seq: Int64 // pass to retryDeadLetters / discardDeadLetters - public let table: String - public let pk: String - public let op: SyncOp // .upsert | .delete - public let attempts: Int // upload attempts charged before parking - public let failure: SyncFailure // classified cause — branch on it to offer the right repair - public let lastError: String?// the raw breadcrumb behind `failure`, for logs and bug reports - public let queuedAt: Date -} -``` - -## SyncOp / SyncError - -```swift -public enum SyncOp: String, Codable { case upsert; case delete } - -public enum SyncError: Error, CustomStringConvertible { - case unknownTable(String) // deprecated enqueue on an undeclared table - case missingPrimaryKey(table: String, column: String) - case encoding(String) - case invalidManifest(table: String, reason: ManifestProblem) // the manifest doesn't match the schema (init) - case recursiveTriggersUnavailable // PRAGMA recursive_triggers could not be enabled -} - -public enum ManifestProblem: Equatable, CustomStringConvertible { - case duplicateDeclaration // the same table declared twice - case noSuchTable(didYouMean: String?) // no local table of that name - case noSuchColumn(field: String, column: String, didYouMean: String?) - case unknownDependency(String) // dependsOn names an undeclared SyncTable - case conflictColumnsOnParentTable(referencedBy: String) - case serverColumnsOmitPrimaryKey(String) -} -``` - -### Manifest validation - -`SyncEngine.init` checks every declared `SyncTable` against the local schema before it installs a -single trigger, and throws `SyncError.invalidManifest` naming the table, the field and the value. -Each of these used to fail silently and late — the wrong partition downloads, a column corrupts in -both directions, an FK-order violation surfaces days later as a dead letter on another device. - -- `name` must be a local table; `primaryKey`, `cursorColumn`, `scopeColumn`, `jsonColumns`, - `conflictColumns` and `serverOwnedColumns` must each name a column on it, **spelled exactly** as - the schema spells it. Case counts even though SQLite would resolve either spelling: these names go - to PostgREST verbatim, and a camelCase Postgres identifier is quoted and matched - case-sensitively — so `scopeColumn: "userID"` against a `userId` column works locally and 400s (or - silently filters on nothing) on the wire. A case-only mismatch is reported with a "did you mean". -- `serverColumns` is **not** checked against the local schema — it describes the *server's* schema, - which is allowed to differ. It may not omit the primary key, since the payload is intersected - against it. -- `conflictColumns` is rejected on a table anything else foreign-keys, enforcing the leaf-only rule - that was previously documentation alone. The scan covers unsynced children too. -- `dependsOn` entries must name declared tables. -- The same table may not be declared twice. - -`SyncError` and `ManifestProblem` both print a full sentence, so the message stands on its own in a -crash log. - -## Writing - -There is no write API. Each declared table gets `AFTER INSERT` / `AFTER UPDATE` / `AFTER DELETE` -triggers that append to `_sync_outbox` inside your own transaction, so any GRDB write syncs: - -```swift -try await db.write { try recipe.save($0) } -try await db.write { try $0.execute(sql: "UPDATE recipes SET title = ? WHERE id = ?", arguments: [t, id]) } -try await db.write { try $0.execute(sql: "DELETE FROM recipes WHERE id = ?", arguments: [id]) } -``` - -Declare `ON DELETE CASCADE` on child foreign keys to have a parent delete tombstone its children. -The engine enables `PRAGMA recursive_triggers` on the writer connection at init. - -## Testing seams - -The network seams are public, so a consumer can drive sync with fakes instead of a live Supabase -project. Full guide: **[[Testing]]**. - -```swift -// The engine, with the network injected. Otherwise identical to the production one. -static func SyncEngine.forTesting( - db: any DatabaseWriter, remote: any SyncRemote, tables: [SyncTable], - pageSize: Int = 500, doorbell: any SyncDoorbell = SilentDoorbell(), - pollInterval: TimeInterval = 30, debounceInterval: TimeInterval = 0.3, - scope: @Sendable () async -> String? = { nil }, - deadLetterAfter: Int = 8, maxOfflineGap: TimeInterval = 30 * 24 * 3600 -) throws -> SyncEngine - -protocol SyncRemote: Sendable { // the upload/download surface - func upsert(table: String, row: [String: AnyJSON], onConflict: String?) async throws -> [String: AnyJSON] - func delete(table: String, primaryKey: String, pk: String) async throws - func fetch(table: String, cursorColumn: String, since: SyncCursor?, primaryKey: String, - scope: ScopeFilter?, limit: Int) async throws -> [[String: AnyJSON]] -} -protocol SyncDoorbell: Sendable { // the Realtime wake signal - func ring(scope: String?) -> AsyncStream -} -struct SilentDoorbell: SyncDoorbell // never rings; the default -struct SyncCursor { var updatedAt: String; var id: String } // per-table download position -struct ScopeFilter { let column: String; let value: String } // resolved partition filter - -protocol ClassifiedSyncError: Error { var isPermanent: Bool { get } } // retryable, or park it? -struct RemoteFailure: ClassifiedSyncError { - init(isPermanent: Bool, underlying: Error) - init(classifying underlying: Error) // production classification of a real transport error -} -``` - -An error that conforms to neither is treated as **transient** — safer to retry than to silently drop -a user's write. - -### HappySyncTestSupport - -A separate product; add it to your test target only. - -```swift -actor InMemorySyncRemote: SyncRemote // seedable server, recorded calls, injectable failures -final class ManualDoorbell: SyncDoorbell // rings on fire(); records every re-subscription -actor AsyncSignal // one-shot rendezvous: fire() / wait() / isFired -func eventually(timeout: Duration = .seconds(5), _ condition: () async -> Bool) async -> Bool -enum SimulatedFailure { case transient, permanent, error(any Error) } -``` - -## SyncSchema (migrator sharing) - -```swift -static func migrator() -> DatabaseMigrator // standalone; used when the engine owns the DB -static func register(into migrator: inout DatabaseMigrator) // prefer this when the app owns the migrator -// The capture triggers are NOT migrations — the engine installs them idempotently at init. -``` - -Registers `happysync_v1..v3` — the `_sync_outbox`, `_sync_state`, and `_sync_meta` internal tables. +- [[Home]] — this wiki's index diff --git a/wiki/Architecture.md b/wiki/Architecture.md index 20d2709..fd3918a 100644 --- a/wiki/Architecture.md +++ b/wiki/Architecture.md @@ -1,119 +1,14 @@ # Architecture -HappySync is deliberately small. Its correctness rests on one enabling constraint and a handful of -mechanisms that compose cleanly. +**This page has moved into the API documentation**, where it sits alongside the symbols it talks +about — and where there is one copy to maintain instead of two that drift apart: -## The enabling constraint +### → [Architecture](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/architecture) -Personal data is **single-user, multi-device — not collaborative**. One user edits their own rows, -occasionally from two devices. Because two devices rarely edit the *same* row at the *same* instant, -**last-write-wins (LWW) by server timestamp** is correct. No CRDTs, no operational transforms, no -conflict-resolution RPC. Revisit only if the data ever becomes collaborative. +The source is [`Sources/HappySync/Documentation.docc/Architecture.md`](https://github.com/happyface-studio/HappySync/blob/main/Sources/HappySync/Documentation.docc/Architecture.md); the page above is +what Swift Package Index builds from it. Everything that used to be here is there, unabridged. -## The pieces +--- -``` - write read - │ ▲ - ▼ │ - ┌──────────────┐ ValueObservation │ - │ GRDB SQLite │───────────────────► UI - │ (local truth)│ - └──────┬───────┘ - │ same transaction - ▼ - ┌──────────────┐ drain (upsert) ┌───────────────┐ - │ _sync_outbox│───────────────────► │ Supabase │ - └──────────────┘ │ Postgres │ - ▲ cursor pull (LWW) │ (server truth│ - └───────────────────────────── │ for writes) │ - └──────┬────────┘ - Realtime doorbell │ (debounced pullNow) - ◄───────┘ -``` - -### Upload — outbox drain - -A write appends to `_sync_outbox` **in the same transaction** as the domain write, then returns -optimistically. A background drain processes entries in `seq` order: - -- **PostgREST upsert** with `Prefer: return=representation` for `.upsert`; soft-delete for - `.delete`. Both are **idempotent by primary key**, so retries are safe. -- **The cursor column never leaves the device.** It's server-stamped by contract, so the payload - drops it (along with any declared `serverOwnedColumns`) with nothing declared — a table whose - `updatedAt` trigger is missing can't quietly promote a device's clock to the LWW authority. - `deletedAt` still ships, as `null`, which is how a re-created row un-tombstones. -- **The full server representation is written back locally** on success — column defaults, - trigger-normalized fields, recomputed server-owned columns, and (for a `conflictColumns` upsert) - the merged row re-keyed to the client's pk. Otherwise the writing device would be the one device - that never sees the server's version of its own write (its local `updatedAt` would equal the - server's, so LWW would skip the row forever). The write-back is skipped for a row that gained a - newer outbox entry mid-flight — the pending edit wins. -- **FK ordering:** upsert parents before children; tombstone children before parents. -- **Per-entry backoff** with jitter; failures are classified permanent vs transient (see - [[Operations and Troubleshooting]]). - -### Download — cursor pull with LWW - -Per table: `SELECT * WHERE updatedAt > :cursor [AND scopeColumn = :partition] ORDER BY (updatedAt, -id)`, RLS-scoped. - -- **Tuple cursor `(updatedAt, id)`** — not a bare timestamp — so rows sharing a millisecond at a - page boundary aren't dropped. -- **LWW apply:** apply a remote row only if `remote.updatedAt > local.updatedAt` **and** the local - row is not dirty (a pending local edit is never clobbered). The compare is at **microsecond** - precision — Postgres `now()` is µs-precise, so two writes inside the same millisecond don't tie. -- **Tombstones** (`deletedAt` set) arrive through the same pull and delete locally. -- **Schema-drift tolerance:** each wire row is intersected against the local table's columns, so a - server that migrated ahead doesn't brick an old client's pulls. - -### Realtime is a doorbell - -A Realtime change event triggers a **debounced `pullNow()`** — it never applies payloads directly. -All correctness lives in the idempotent cursor pull, so sync converges even if Realtime drops; the -doorbell just makes it feel instant. Foreground and periodic pulls converge on their own. - -### Writes and capture - -There is no write API to remember. Each declared table carries `AFTER INSERT` / `AFTER UPDATE` / -`AFTER DELETE` triggers that append to `_sync_outbox`, so any GRDB write — `save`, a partial update, -raw SQL, a whole multi-table transaction — is queued **in the caller's transaction**. The unit of -atomicity is the user's intent, not one row: "create a recipe + 12 ingredients" commits the rows and -their queued uploads together or not at all. - -Each trigger's `WHEN` clause reads a one-row `_sync_control.applying` flag that the engine raises -while it writes *server* state locally (a pulled page, a tombstone, an upload's write-back, a resync -reconcile, a discarded dead letter). Without it every downloaded row would queue itself straight back -for upload and the sync would never settle. - -The triggers come from the `tables` manifest, which is source code rather than a numbered migration, -so the engine (re)installs them idempotently at init and drops the ones for tables you've stopped -syncing. A declared table that doesn't exist locally throws at init — the manifest and the schema are -not allowed to drift silently. - -A GRDB `TransactionObserver` watches for **inserts into the outbox** and wakes the runner, so a plain -write uploads as promptly as an explicit call used to. Drain bookkeeping (updates and deletes on the -outbox) is deliberately not observed, so a drain can't feed itself another pass. - -### Deletes and cascades - -A plain `DELETE` removes the row locally and queues a tombstone. For a parent, the schema's -`ON DELETE CASCADE` does the graph walk and each removed child's own delete trigger queues its -tombstone — mirroring the server's child-tombstone trigger. Result: the local and server deleted sets -stay symmetric, with no orphan window and no round-trip. A table that only `dependsOn` a parent -without an FK action is not cascaded; `dependsOn` orders uploads, it doesn't delete rows. - -## What HappySync owns (and doesn't) - -**Owns:** the write-capture triggers on your declared tables, the outbox drain, cursor pull, -tombstones, FK ordering, the Realtime doorbell, status, and retry/backoff. - -**Does not own:** your reads (observe GRDB directly) or your schema (you declare tables; the server -holds the authoritative schema). - -## The contract - -Every mechanism here has a precise, language-neutral spec — server conventions, wire semantics, -field mapping, schema-evolution rules — in -[`docs/SYNC-CONTRACT.md`](https://github.com/happyface-studio/HappySync/blob/main/docs/SYNC-CONTRACT.md). -A future non-Swift client would implement that contract, not reuse this engine's code. +- [Documentation home](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync) — the landing page, and the full symbol reference +- [[Home]] — this wiki's index diff --git a/wiki/Getting-Started.md b/wiki/Getting-Started.md index eb931f3..1d8fc3f 100644 --- a/wiki/Getting-Started.md +++ b/wiki/Getting-Started.md @@ -1,187 +1,14 @@ # Getting Started -This walks you from zero to a working sync loop. Before your first sync completes, you also need the -server schema in **[[Server Setup]]** — the engine cannot correct a server that lacks the -`updatedAt` trigger. +**This page has moved into the API documentation**, where it sits alongside the symbols it talks +about — and where there is one copy to maintain instead of two that drift apart: -## 1. Add the package +### → [Getting Started](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/gettingstarted) -In `Package.swift`: +The source is [`Sources/HappySync/Documentation.docc/GettingStarted.md`](https://github.com/happyface-studio/HappySync/blob/main/Sources/HappySync/Documentation.docc/GettingStarted.md); the page above is +what Swift Package Index builds from it. Everything that used to be here is there, unabridged. -```swift -dependencies: [ - .package(url: "https://github.com/happyface-studio/HappySync.git", from: "0.4.0"), -], -targets: [ - .target(name: "MyApp", dependencies: ["HappySync"]), -] -``` +--- -Or in Xcode: **File → Add Package Dependencies →** `https://github.com/happyface-studio/HappySync`. - -## 2. Declare your tables - -Each synced table is a `SyncTable`, declared once, **in any order** — the engine sorts the manifest -by FK dependency itself. Only `name` is required. - -Every declared table must **already exist** in your local schema when the engine is constructed, so -run your migrations first. `SyncEngine.init` validates the whole manifest against that schema and -throws `SyncError.invalidManifest(table:reason:)` — naming the table, the field and the value — if -anything disagrees: a table that isn't there, or a `cursorColumn` / `scopeColumn` / `jsonColumns` / -`conflictColumns` / `serverOwnedColumns` entry naming a column the table doesn't have. Each of those -used to fail silently and late, so it's worth reading the message rather than working around it. - -You don't declare `dependsOn`: the engine reads your foreign keys at init and derives it. Pass one -explicitly only for a *logical* parent your schema has no FK for (see -[Cascade](#delete-with-cascade)). - -Declare `ON DELETE CASCADE` on child foreign keys if you want a parent delete to remove (and -tombstone) its children — see [Delete](#delete-with-cascade) below. - -```swift -import HappySync - -let tables = [ - SyncTable(name: "recipes", jsonColumns: ["nutrition"]), - SyncTable(name: "recipeIngredients"), // dependsOn derived from the FK - SyncTable(name: "recipeSteps", jsonColumns: ["temperature"]), -] -``` - -See **[[API Reference]]** for every `SyncTable` field (`scopeColumn`, `conflictColumns`, -`serverOwnedColumns`, `serverColumns`, `cursorColumn`). - -## 3. Create and start the engine - -```swift -let engine = try SyncEngine( - db: databaseQueue, // any GRDB DatabaseWriter — your local source of truth - supabase: client, // your SupabaseClient - tables: tables, - auth: { await session.accessToken } // return a FRESH access token each call -) - -await engine.start() -``` - -`start()` is idempotent. It runs an immediate convergence sync, then keeps syncing from three -triggers — the Realtime doorbell (debounced), a periodic poll (converges even if Realtime drops), -and retry backoff — all funnelled through one serialized runner. - -### Sharing a migrator - -The engine's internal tables (`_sync_outbox`, `_sync_state`, `_sync_meta`, `_sync_control`) migrate -at init. If your app owns the GRDB `DatabaseMigrator`, register HappySync's migrations into it so -both share one `grdb_migrations` table: - -```swift -var migrator = DatabaseMigrator() -SyncSchema.register(into: &migrator) // happysync_v1..v4 -// … your app migrations … -try migrator.migrate(databaseQueue) -``` - -The **write-capture triggers** are not part of that migrator. They're derived from your `tables` -manifest — source code, which ships without a schema version bump — so the engine (re)installs them -idempotently at every init and drops the ones for tables you've stopped syncing. - -## 4. Write - -Write GRDB **normally**. Each declared table carries `AFTER INSERT` / `UPDATE` / `DELETE` triggers -that append to the outbox in your transaction, so there is no second API to call and nothing to -forget. The engine notices the queued write and uploads promptly, so there's **no `syncNow()` needed -after a write** — a burst debounces into one drain pass. - -```swift -try await db.write { try recipe.save($0) } // PersistableRecord - -try await db.write { db in // partial / expression updates - try db.execute(sql: "UPDATE recipes SET title = ? WHERE id = ?", arguments: [title, id]) - try db.execute(sql: "UPDATE recipes SET cookCount = cookCount + 1 WHERE id = ?", arguments: [id]) -} - -try await db.write { db in // one transaction = one user intent - try recipe.save(db) - for ingredient in ingredients { try ingredient.save(db) } -} -``` - -That last one matters: the rows and their queued uploads commit together, or neither does. A crash -part-way through leaves no half-recipe *and* no half-batch in the outbox. - -> `engine.enqueue(op:table:row:)` still works but is **deprecated**. It now just performs the write -> and lets the trigger queue it. Replace `enqueue(.upsert, table: t, row: r)` with -> `try await db.write { try r.save($0) }`, and `enqueue(.delete, …)` with a plain `DELETE`. - -### Delete (with cascade) - -A plain `DELETE` removes the row locally and queues a tombstone. For a **parent** row, declare -`ON DELETE CASCADE` on the child foreign keys — SQLite then walks the graph and each removed child's -own trigger queues its tombstone, so you never delete children by hand and the UI never shows -orphans. - -```swift -// In your migration: -try db.create(table: "recipeIngredients") { t in - t.column("id", .text).primaryKey() - t.column("recipeId", .text).references("recipes", onDelete: .cascade) - // … -} - -// Then: -try await db.write { try $0.execute(sql: "DELETE FROM recipes WHERE id = ?", arguments: [recipeID]) } -``` - -A table that only `dependsOn` a parent without an FK action is **not** cascaded — `dependsOn` orders -uploads, it doesn't delete rows. Its orphans reconcile on the next pull via the server tombstone. - -## 5. Pull - -Downloads happen automatically (doorbell + periodic). Call `pullNow()` for app-driven nudges — -returning to the foreground, pull-to-refresh: - -```swift -try await engine.pullNow() -``` - -## 6. Show sync status - -Ask `isHealthy` — never `phase == .idle` alone. A settled pass with writes still failing or parked -is `.degraded`, so switching on the phase forces you to confront it: - -```swift -for await status in engine.status { - guard !status.isHealthy else { return showSynced() } - switch status.phase { - case .idle, .syncing: break // .idle here means healthy; .syncing is in flight - case .degraded: showPending(status.failedUploads + status.deadLetters) - case .failed(.authExpired): promptReauth() // classified — no string matching - case .failed(.network): break // recovers on its own; show nothing - case .failed(let failure): showBanner(failure) - } -} -``` - -See **[[Operations and Troubleshooting]]** for what `failedUploads` / `deadLetters` mean and how to -repair parked writes. - -## 7. Tear down on sign-out (order matters) - -`stop()` is **async** and awaits the in-flight pass. Always `await engine.stop()` **before** wiping -or replacing the database on sign-out / account switch — otherwise an in-flight pass could write to -the store you're about to delete, or upload the old user's rows with the new user's token. - -```swift -await engine.stop() // engine is quiesced here -try await wipeLocalDatabase() -``` - -Status subscriptions survive teardown: `stop()` broadcasts a settled `.idle` status but doesn't end -the streams, so a `for await status in engine.status` loop keeps delivering after the next `start()` -— no re-subscribing needed. - -## Next - -- **[[Server Setup]]** — required before the first successful sync. -- **[[Operations and Troubleshooting]]** — health, dead letters, schema evolution. -- **[[Claude Skill]]** — let your AI assistant wire this correctly for you. +- [Documentation home](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync) — the landing page, and the full symbol reference +- [[Home]] — this wiki's index diff --git a/wiki/Home.md b/wiki/Home.md index bc55f99..3c019a9 100644 --- a/wiki/Home.md +++ b/wiki/Home.md @@ -23,16 +23,22 @@ status, and retry/backoff. It does **not** own reads or schema. ## Start here +The developer documentation is **[built from the source](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync)** — one copy, next to the symbols it +describes. These wiki pages point at it: + | Page | What's on it | |---|---| -| **[[Getting Started]]** | Add the package, wire a `SyncEngine`, first write and pull. | -| **[[Server Setup]]** | The required Supabase schema: `updatedAt` trigger, `deletedAt` tombstones, RLS, Realtime. | -| **[[API Reference]]** | Every public type and method with signatures and field semantics. | -| **[[Operations and Troubleshooting]]** | Status UI, dead-letter repair, teardown, schema evolution, a troubleshooting table. | -| **[[Testing]]** | Test your sync integration offline with the shipped fakes — no Supabase project. | -| **[[Architecture]]** | How upload, download, LWW, the doorbell, and cascades fit together. | +| **[Getting Started](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/gettingstarted)** | Add the package, wire a `SyncEngine`, first write and pull. | +| **[Server Setup](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/serversetup)** | The required Supabase schema: `updatedAt` trigger, `deletedAt` tombstones, RLS, Realtime. | +| **[API Reference](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync)** | Every public type and method, generated from the source. | +| **[Operations and Troubleshooting](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/operations)** | Status UI, dead-letter repair, teardown, schema evolution, a troubleshooting table. | +| **[Testing](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/testingyourintegration)** | Test your sync integration offline with the shipped fakes — no Supabase project. | +| **[Architecture](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/architecture)** | How upload, download, LWW, the doorbell, and cascades fit together. | | **[[Claude Skill]]** | Install the HappySync Claude skill so your AI assistant integrates it correctly. | +Want to *see* it work first? [`Examples/`](https://github.com/happyface-studio/HappySync/tree/main/Examples) +is a two-table SwiftUI app wired to the in-memory fake — no Supabase project needed. + ## Requirements - Swift 6, iOS 16+ / macOS 13+ diff --git a/wiki/Operations-and-Troubleshooting.md b/wiki/Operations-and-Troubleshooting.md index c57f5db..e7f3cb2 100644 --- a/wiki/Operations-and-Troubleshooting.md +++ b/wiki/Operations-and-Troubleshooting.md @@ -1,139 +1,14 @@ # Operations and Troubleshooting -## Sync-status UI +**This page has moved into the API documentation**, where it sits alongside the symbols it talks +about — and where there is one copy to maintain instead of two that drift apart: -Consume `engine.status` and ask `isHealthy`. `.idle` means settled **and** clean; a pass that -completed while writes are still failing or parked settles as `.degraded`. +### → [Operations and Troubleshooting](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/operations) -```swift -for await status in engine.status { - switch status.phase { - case .idle: showSynced() // settled and clean - case .syncing: showSpinner() - case .degraded: showPending(status.failedUploads + status.deadLetters) - case .failed(let failure): showBanner(failure) // classified — see below - } -} -``` +The source is [`Sources/HappySync/Documentation.docc/Operations.md`](https://github.com/happyface-studio/HappySync/blob/main/Sources/HappySync/Documentation.docc/Operations.md); the page above is +what Swift Package Index builds from it. Everything that used to be here is there, unabridged. -- `failedUploads > 0` → some local changes haven't reached the server yet (degraded, still - retrying with backoff). -- `deadLetters > 0` → some writes parked and need inspection/repair. +--- -## Failure model - -- **Permanent** (constraint `23xxx`, RLS `42501`, undefined-column `42703`, other 4xx) → - dead-lettered immediately. -- **Transient** (network, 5xx, 408/429, transient Postgres `40001`/`40P01`/`53xxx`/`08xxx`, unknown - codes) → retried with per-entry exponential backoff (jittered) until a cap, then dead-lettered. -- **Auth-shaped** (401/403, `PGRST301`/`PGRST302`) → transient **and exempt from the retry budget**, - so an expired-token stretch never dead-letters the whole outbox; it recovers when the token - refreshes out-of-band. - -A dead-lettered entry stops retrying **and** stops counting as a dirty row, so it never permanently -blocks downloads for its key. - -### What the app sees - -That classification is surfaced as `SyncFailure`, on `Phase.failed` and on `DeadLetter.failure`. -It answers *what to show the user* — the retry decision is already made by then, and is visible as -`failedUploads` (still retrying) vs `deadLetters` (parked): - -| `SyncFailure` | Raised by | What the app should do | -|---|---|---| -| `.network` | `URLError`, Postgres `08xxx` | Nothing — it recovers | -| `.authExpired` | 401/403, `PGRST301`/`PGRST302` | Nothing at first; prompt re-auth if it persists | -| `.permissionDenied` | `42501` | Surface it — a policy bug or a real permissions problem | -| `.constraintViolation(code:)` | `23xxx` | Inspect the row; the write will never land as-is | -| `.schemaMismatch(column:)` | `42703`, `PGRST204` | Prompt an app update (or restore the server column) | -| `.server(status:)` | any other HTTP status, incl. 5xx | Nothing — retried; surface if it persists | -| `.other(String)` | anything unrecognised, transient Postgres states | Log the text | - -`.network` means the request got no answer at all; a server that answered with a 5xx is -`.server(status:)`, so the status code isn't thrown away. - -The kind and the retry decision agree everywhere except `PGRST204`: it classifies as -`.schemaMismatch` but is **retried** rather than parked immediately, because PostgREST also raises it -from a stale schema cache, which clears on its own. - -Every case renders via `CustomStringConvertible`, so `"\(failure)"` is safe to put in a banner. The -classified cases carry no server prose of their own — the full text lives in the engine's log, and on -`DeadLetter.lastError` for a parked write. - -## Repairing dead letters - -```swift -let parked = try await engine.deadLetters() -for letter in parked { - switch letter.failure { - case .permissionDenied: reportPolicyBug(letter) // fix RLS, then retryDeadLetters() - case .schemaMismatch(let column): promptAppUpdate(column) - case .constraintViolation(let code): inspect(letter, code) - default: log("\(letter.op) \(letter.table)/\(letter.pk): \(letter.lastError ?? "unknown")") - } -} - -// After fixing the cause (RLS/policy change, schema migration, app update): -try await engine.retryDeadLetters() // all, or pass [seq] for specific entries - -// Or abandon the local write and accept the server's version — discard drops the entries and -// re-pulls the affected rows so local converges to the server (including changes missed while parked): -try await engine.discardDeadLetters([badSeq]) // specific seqs, or omit for all -``` - -Both mutations refresh the status stream immediately, so `deadLetters` drops as soon as they return. - -## Teardown / account switch (order matters) - -`stop()` is async and awaits the in-flight pass. A consumer that wipes or replaces the database on -sign-out / account switch **must `await engine.stop()` before touching the DB file** — otherwise an -in-flight pass could write to the store you're about to delete, or (mid-account-switch) upload the -old user's rows with the new user's token. - -```swift -await engine.stop() // quiesced here -try await wipeLocalDatabase() -``` - -`stop()` settles the status stream (`.idle`, keeping the outbox counts) but leaves subscriptions -open, so the sync-status UI resumes updating on the next `start()` without being re-subscribed. - -## Stale-cursor full resync - -A device offline past `maxOfflineGap` (default 30 days) can't trust its cursors — a tombstone it -never saw may already be purged. Once `now − lastSyncedAt > maxOfflineGap` the engine clears -cursors, re-pulls, and reconciles purged deletes instead of resurrecting them. Keep `maxOfflineGap` -**≤ the server's tombstone-purge retention** (see [[Server Setup]]). For a `scopeColumn` table the -resync is deferred until a partition value resolves, so it never wipes an un-pulled table. - -## Schema evolution (staggered clients) - -Shipped clients and the server are **not lockstep** — App Store review + staggered updates mean old -builds run against a newer server for weeks. The engine tolerates drift both ways, but follow these -rules so any single migration is safe for every client in the field: - -- **Additive-only.** Never drop-and-recreate or repurpose a column in place. -- **Never rename in place.** Add the new name, dual-write server-side, wait for adoption, then - remove the old name by the removal rule. -- **Adding a column — server-first.** Deploy it (nullable/default) before any client reads/writes - it. Old clients drop the unknown column on download and don't send it on upload. -- **Removing a column — client-first.** Drop it server-side **only after no shipped client still - writes it**: (1) ship a build that stops sending it — optionally declaring `serverColumns` so the - payload is intersected immediately; (2) wait out the update-lag window; (3) then drop it. Skipping - step 2 dead-letters every write from clients still sending the column. - -## Troubleshooting - -| Symptom | Likely cause | Fix | -|---|---|---| -| Writes never reach the server; `deadLetters` climbing | RLS rejects (`42501`) or a constraint violation | `deadLetters()` → switch on `failure` (`lastError` for the raw text); fix the policy/data, then `retryDeadLetters()`. | -| A device never sees the server's version of its **own** write | Missing `updatedAt` trigger (the engine already strips the client's `updatedAt` from every upload, so a stale client value can't be the cause) | Add the server trigger. | -| Rows on an insert-only table never download to other devices | The table's `cursorColumn` has no server default — the engine never uploads it, so it lands null | `default now()` (or a trigger) on that column server-side. | -| Whole public catalog downloads to every device | RLS broader than the partition, no `scopeColumn` | Declare `scopeColumn` + supply the engine `scope:` closure. | -| A fresh-pk insert 409s forever | Row already exists under a secondary `UNIQUE` | Declare `conflictColumns` (leaf tables only). | -| Every write on a table dead-letters after a server migration | Dropped/renamed server column (`PGRST204`) | Restore the column until clients stop sending it; or declare `serverColumns`. | -| Deleted parent leaves orphaned children in the UI | Child FK not declared `ON DELETE CASCADE` locally, or missing server child-tombstone trigger | Declare `ON DELETE CASCADE` on the child FKs (`dependsOn` alone deletes nothing) and add the server child-tombstone trigger. | -| `SyncEngine.init` throws `invalidManifest` | A declared `SyncTable` disagrees with the local schema — a table or column that isn't there, a duplicate declaration, `dependsOn` naming an undeclared table, `conflictColumns` on a table something foreign-keys, or `serverColumns` without the primary key | Read the message: it names the table, the field and the value. Run your app's migrations **before** constructing the engine, then fix the manifest. It throws rather than silently syncing the wrong thing — every one of these used to be invisible until the damage showed up on another device. | -| Some writes sync, one code path never does | A write that bypasses the declared tables — writing a table not in the manifest, or a `DatabasePool` reader-side hack | Add the table to the manifest. Writes to declared tables are captured by triggers, so there is no way to "forget" to enqueue one. | -| Offline-long device resurrects deleted rows | `maxOfflineGap` > server tombstone retention | Set `maxOfflineGap` ≤ retention. | -| Idle status shows healthy but changes aren't syncing | Reading `phase == .idle` as health | Ask `status.isHealthy`. The engine now settles such a pass as `.degraded`, so a `switch` over the phase surfaces it. | +- [Documentation home](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync) — the landing page, and the full symbol reference +- [[Home]] — this wiki's index diff --git a/wiki/README.md b/wiki/README.md index 12a003a..04e5eaa 100644 --- a/wiki/README.md +++ b/wiki/README.md @@ -7,17 +7,23 @@ GitHub resolves `Getting Started` → `Getting-Started.md`, so keep filenames hy ## Pages -| File | Wiki page | -|---|---| -| `Home.md` | Landing page | -| `Getting-Started.md` | Getting Started | -| `Server-Setup.md` | Server Setup | -| `API-Reference.md` | API Reference | -| `Operations-and-Troubleshooting.md` | Operations and Troubleshooting | -| `Testing.md` | Testing | -| `Architecture.md` | Architecture | -| `Claude-Skill.md` | Claude Skill | -| `_Sidebar.md` | Sidebar nav (all pages) | +**The developer documentation lives in DocC**, not here — `Sources/HappySync/Documentation.docc/`, +published by [Swift Package Index](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync) +and readable in Xcode. It sits next to the symbols it documents, which is the only arrangement that +keeps the two from drifting (issue #57). The pages below that used to hold that content are now +one-paragraph pointers to it; edit the DocC article instead. + +| File | Wiki page | | +|---|---|---| +| `Home.md` | Landing page | index | +| `Claude-Skill.md` | Claude Skill | the only full article left here | +| `_Sidebar.md` | Sidebar nav | index | +| `Getting-Started.md` | Getting Started | → DocC | +| `Server-Setup.md` | Server Setup | → DocC | +| `API-Reference.md` | API Reference | → DocC | +| `Operations-and-Troubleshooting.md` | Operations and Troubleshooting | → DocC | +| `Testing.md` | Testing | → DocC | +| `Architecture.md` | Architecture | → DocC | ## Publish diff --git a/wiki/Server-Setup.md b/wiki/Server-Setup.md index 86cdb9e..ce1c313 100644 --- a/wiki/Server-Setup.md +++ b/wiki/Server-Setup.md @@ -1,135 +1,14 @@ # Server Setup -Every synced table needs four things on the Supabase side. These are **non-negotiable** — LWW -compares a *server* clock, and deletes, scoping, and live updates all depend on this contract. The -engine will not paper over a server that violates it. +**This page has moved into the API documentation**, where it sits alongside the symbols it talks +about — and where there is one copy to maintain instead of two that drift apart: -Run the SQL below per synced table (swap `recipes` for your table). +### → [Server Setup](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/serversetup) -## 1. `updatedAt` — server-stamped change time +The source is [`Sources/HappySync/Documentation.docc/ServerSetup.md`](https://github.com/happyface-studio/HappySync/blob/main/Sources/HappySync/Documentation.docc/ServerSetup.md); the page above is +what Swift Package Index builds from it. Everything that used to be here is there, unabridged. -LWW compares `updatedAt`, and a *server* clock is what stops two devices' clock skew from silently -losing writes. **Never trust a client-sent `updatedAt`.** The engine holds up its half unasked: a -table's `cursorColumn` is stripped from every upload payload, so no HappySync client ever sends one -— you don't declare anything for this. The trigger below is still required (it's what advances the -column on update, and what the cursor pull depends on); the stripping just means a table that lost -its trigger can't quietly promote a device's clock to the ordering authority. +--- -```sql -alter table public.recipes - add column if not exists "updatedAt" timestamptz not null default now(); - -create or replace function public.stamp_updated_at() -returns trigger language plpgsql as $$ -begin - new."updatedAt" := now(); -- server clock, always; ignore any client value - return new; -end $$; - -drop trigger if exists stamp_updated_at on public.recipes; -create trigger stamp_updated_at - before insert or update on public.recipes - for each row execute function public.stamp_updated_at(); -``` - -> An **insert-only / immutable** table needs no update trigger. Give it a monotonic stamp column -> (e.g. `translatedAt`) and set that table's `cursorColumn` in its `SyncTable` descriptor. Because -> the engine never uploads the cursor column, that column must be `default now()` server-side — -> otherwise inserts land with a null stamp the cursor can't order. - -## 2. `deletedAt` — soft-delete tombstones - -Deletes are **soft**: set `deletedAt` instead of removing the row, so deletions propagate on the -next cursor pull. Deleting a parent must **tombstone its children too** — use a trigger, *not* -`ON DELETE CASCADE` (a hard cascade leaves no tombstone for other devices to pull). - -```sql -alter table public.recipes - add column if not exists "deletedAt" timestamptz; - -create or replace function public.tombstone_recipe_children() -returns trigger language plpgsql as $$ -begin - if new."deletedAt" is not null and old."deletedAt" is null then - update public."recipeIngredients" - set "deletedAt" = new."deletedAt" - where "recipeId" = new.id and "deletedAt" is null; - -- repeat per child table - end if; - return new; -end $$; - -drop trigger if exists tombstone_recipe_children on public.recipes; -create trigger tombstone_recipe_children - after update on public.recipes - for each row execute function public.tombstone_recipe_children(); -``` - -Locally, declare the child foreign keys `ON DELETE CASCADE` in your **SQLite** schema: deleting a -parent then removes its children in the same transaction and each child's capture trigger queues its -tombstone, keeping the local and server deleted sets symmetric with no orphan window. The server -trigger above is what makes *other* devices converge. - -### Tombstone purge and the `maxOfflineGap` invariant - -Purge old tombstones on a schedule, but respect this **invariant**: the purge retention must exceed -the engine's `maxOfflineGap` (default 30 days). A device offline longer than the retention would -never see a purged tombstone and would resurrect the row. Set `maxOfflineGap` **≤ your server's -retention**. - -```sql --- e.g. a pg_cron job; retention (90d) > engine maxOfflineGap (30d default) -delete from public.recipes where "deletedAt" < now() - interval '90 days'; -``` - -## 3. RLS scoped to `auth.uid()` - -RLS is the **security boundary**. Enable it and add policies keyed on `auth.uid()`. - -```sql -alter table public.recipes enable row level security; - -create policy "own rows: select" on public.recipes - for select using ("userId" = auth.uid()); -create policy "own rows: write" on public.recipes - for all using ("userId" = auth.uid()) with check ("userId" = auth.uid()); -``` - -**RLS is not necessarily the sync partition.** If a policy is deliberately broader than what a -device should download — e.g. `isPublic = true OR userId = auth.uid()` — an unfiltered pull drags -the whole public catalog to every device. Declare a `scopeColumn` (e.g. `userId`) on the `SyncTable` -and supply the engine's `scope:` closure; the engine filters the pull **and** the doorbell to -`scopeColumn = `. Omit `scopeColumn` when RLS already equals the partition. - -## 4. Realtime publication (the doorbell) - -```sql -alter publication supabase_realtime add table public.recipes; -``` - -Realtime is a **doorbell only** — an event triggers a debounced `pullNow()`; payloads are never -applied directly. Sync converges without it; Realtime just makes it feel instant. - -## Field-mapping conventions - -| Concern | Convention | -|---|---| -| Column names | **camelCase, identical** in SQLite and Postgres — no snake_case mapping. | -| Dates | ISO-8601 with fractional seconds; the LWW gate compares instants at **microsecond** precision. | -| UUID | text locally ↔ `uuid` in Postgres. | -| Bool | integer `0/1` locally ↔ `boolean`. | -| Enum | `rawValue` string. | -| JSON columns | JSON **text** locally ↔ `json`/`jsonb`; declare in `jsonColumns`. | -| Server-owned | RPC-managed columns; declare in `serverOwnedColumns`, never in an upsert. The `cursorColumn` is server-owned too and the engine strips it with nothing declared. | - -## Checklist - -- [ ] `updatedAt` + `BEFORE INSERT/UPDATE` trigger on every table (`default now()` on an - insert-only table's stamp column — the client never sends it). -- [ ] `deletedAt` on every table + a child-tombstone trigger on every parent. -- [ ] Tombstone purge scheduled, retention **>** engine `maxOfflineGap`. -- [ ] RLS enabled + `auth.uid()` policies on every table. -- [ ] `scopeColumn` declared for any table whose RLS is broader than the partition. -- [ ] Every synced table in the `supabase_realtime` publication. - -See **[[Operations and Troubleshooting]] → Schema evolution** before you change a live schema. +- [Documentation home](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync) — the landing page, and the full symbol reference +- [[Home]] — this wiki's index diff --git a/wiki/Testing.md b/wiki/Testing.md index 28617c7..b2f743f 100644 --- a/wiki/Testing.md +++ b/wiki/Testing.md @@ -1,220 +1,14 @@ # Testing -Sync is the part of an app where bugs are silent and data-losing, and it is usually the part with no -tests — because testing it seems to require a live Supabase project. +**This page has moved into the API documentation**, where it sits alongside the symbols it talks +about — and where there is one copy to maintain instead of two that drift apart: -It doesn't. `SyncRemote` and `SyncDoorbell` are the seams the engine's own 2,700 lines of tests run -on, and both are public. The `HappySyncTestSupport` product ships the fakes, so a test can answer -questions like *does deleting a recipe queue tombstones for all its children?* in milliseconds, with -no network. +### → [Testing](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync/testingyourintegration) -## Add the test-support product +The source is [`Sources/HappySync/Documentation.docc/TestingYourIntegration.md`](https://github.com/happyface-studio/HappySync/blob/main/Sources/HappySync/Documentation.docc/TestingYourIntegration.md); the page above is +what Swift Package Index builds from it. Everything that used to be here is there, unabridged. -Add it to your **test** target only — it contains no production code: +--- -```swift -.testTarget( - name: "MyAppTests", - dependencies: [ - "MyApp", - .product(name: "HappySync", package: "HappySync"), - .product(name: "HappySyncTestSupport", package: "HappySync"), - ] -), -``` - -## Build an engine with no Supabase - -`SyncEngine.forTesting(db:remote:tables:…)` is the production engine with the network seams -injected. Everything else — drain, pull, retry classification, dead-lettering, cascade ordering — is -the code that ships, so what a test proves here is what the app does. - -```swift -import HappySync -import HappySyncTestSupport - -let db = try DatabaseQueue() -try MyApp.migrator.migrate(db) // your migrations, as at launch - -let remote = InMemorySyncRemote() -let engine = try SyncEngine.forTesting( - db: db, - remote: remote, - tables: MyApp.syncTables, // the real manifest — a typo in it fails here, too - pollInterval: 999, // only the passes the test triggers - debounceInterval: 0.02 -) -``` - -The parameters worth knowing: `pageSize` (shrink to `1` to force pagination), `pollInterval` (raise -to silence the background poll), `debounceInterval` (shrink so a doorbell ring converges promptly), -`deadLetterAfter` (lower to reach the parked state without driving eight retries), and `scope` (the -partition value, to test a user switch). - -## `InMemorySyncRemote` - -A server that lives in the test process. It records every upload, serves downloads from a seedable -per-table row store honouring the `(cursorColumn, primaryKey)` tuple cursor and the page limit, and -can be told to fail. - -```swift -// What the server holds — rows are [String: AnyJSON], exactly what PostgREST returns. -let remote = InMemorySyncRemote(dataset: [ - "recipes": [["id": "r1", "title": "Soup", "updatedAt": "2026-06-30T12:00:00.000Z"]], -]) - -// What the client sent. -await remote.upsertCalls // [(table, row, onConflict)] in order -await remote.deleteCalls // [(table, pk)] — the tombstones -await remote.fetchCalls // download pages requested; the usual "a pass ran" signal -await remote.lastScope // the partition filter the engine resolved - -// Publish a change "another device" made, between two pulls. -await remote.seed("recipes", rows: [updatedRow]) -``` - -### Failure injection - -The one knob that matters is whether the drain **retries** the write or **parks** it: - -```swift -InMemorySyncRemote(failUpserts: 1, upsertFailure: .transient) // retried, then succeeds -InMemorySyncRemote(failUpserts: 1, upsertFailure: .permanent) // dead-lettered on the first attempt -InMemorySyncRemote(failUpserts: 1, upsertFailure: .error(HTTPError(…))) // real error, real classification -``` - -`.error` wraps a genuine transport error (`HTTPError`, `PostgrestError`, `URLError`) with the -classification the production remote applies — so *"an RLS reject parks, an expired token doesn't"* -is asserted against the shipped table rather than a hand-picked answer. `failFetches` / -`fetchFailure` do the same for downloads, which drives a whole pass to `.failed(SyncFailure)`. - -Use `99` for "always", and `failNextUpserts(_:with:)` / `failNextFetches(_:with:)` to change the -injection mid-test — e.g. fail, assert parked, then let the retry succeed. - -### Simulating server-side computation - -`representation` maps an uploaded row to the row the server returns. The drain writes the whole -representation back locally, so this is how you test server-normalized values, defaulted columns, or -a `conflictColumns` merge: - -```swift -let remote = InMemorySyncRemote(representation: { row in - var server = row - server["cookedCount"] = .integer(7) // the merged server total, not what this client sent - server["updatedAt"] = .string("2026-06-30T12:00:00.000Z") - return server -}) -``` - -### Holding a call in flight - -`onUpsert` / `onFetch` run at the start of each call with its 1-based number. Because the remote -suspends there, this is how a test races a local write against an in-flight upload, or stops the -engine mid-pull: - -```swift -let started = AsyncSignal(), gate = AsyncSignal() -await remote.onUpsert { _ in await started.fire(); await gate.wait() } - -let pass = Task { await engine.syncNow() } -await started.wait() // the upload is blocked in the remote -try await db.write { … } // a competing local edit lands behind it -await gate.fire() // let the upload finish -``` - -## `ManualDoorbell` - -The Realtime channel, made deterministic — it rings only when you say so. - -```swift -let doorbell = ManualDoorbell() -let engine = try SyncEngine.forTesting(db: db, remote: remote, tables: tables, doorbell: doorbell) -await engine.start() - -doorbell.fire() // a remote change lands -#expect(await eventually { await remote.fetchCalls >= 2 }) - -doorbell.ringScopes // [nil, "u1"] proves a sign-in re-scoped the subscription -doorbell.liveSubscriptions // settles to 1 — the old subscription was torn down -doorbell.fire(subscription: 0) // ring a torn-down one; it must not poke the engine -``` - -`SilentDoorbell` (the default) never rings, so convergence comes only from the periodic poll and the -passes you trigger. - -## Async helpers - -`eventually` polls until a condition holds or the timeout elapses. Reach for it any time the engine -converges in the background — a fixed sleep plus one assertion is the alternative, and it flakes -under a loaded CI runner: - -```swift -#expect(await eventually { await remote.fetchCalls >= 1 }) -``` - -For a *negative* assertion ("no further pull happened"), poll until the count stops moving, then -assert it doesn't move again. - -`AsyncSignal` is a one-shot rendezvous — `fire()` releases every `wait()`, now and later, so a wait -can't miss a signal that arrived first. - -## Three tests worth having - -These are the ones that catch real bugs, and none of them need a network. - -**A cascade delete queues every child's tombstone.** The app deletes one row; SQLite cascades and -each child's capture trigger queues its own tombstone. If a child table is missing from your -manifest, this test fails and the field doesn't. - -```swift -try await db.write { try $0.execute(sql: "DELETE FROM recipes WHERE id = 'r1'") } -#expect(await eventually { await remote.deleteCalls.count == 3 }) -``` - -**The status UI shows degraded when a write parks.** `.degraded` is the case everyone gets wrong: the -pass completed, so nothing *looks* broken, but the user's write never reached the server. Assert on -`isHealthy`, never on the phase alone. - -```swift -let remote = InMemorySyncRemote(failUpserts: 99, upsertFailure: .permanent) -… -#expect(await eventually { await latestStatus(engine)?.phase == .degraded }) -#expect(try await engine.deadLetters().count == 1) -``` - -**Sign-out quiesces the engine before the wipe.** After `await engine.stop()` returns, nothing may -still be in flight — a late pull writes into the store you're about to delete, and mid-account-switch -uploads the previous user's rows under the new user's token. - -```swift -await engine.stop() -let atStop = await remote.fetchCalls -doorbell.fire() // a Realtime event racing the sign-out -try await Task.sleep(for: .milliseconds(200)) -#expect(await remote.fetchCalls == atStop) // nothing restarted -``` - -A worked version of all three lives in -[`Tests/HappySyncTests/PublicSeamsTests.swift`](https://github.com/happyface-studio/HappySync/blob/main/Tests/HappySyncTests/PublicSeamsTests.swift), -which imports HappySync the way your app does — no `@testable` — so it stops compiling if these -seams ever regress. - -## Writing your own remote - -`SyncRemote` and `SyncDoorbell` are protocols, so a bespoke double is three methods. Signal whether a -failure is retryable by conforming your error to `ClassifiedSyncError`, or wrap a real transport -error with `RemoteFailure(classifying:)` to get the production classification: - -```swift -struct RejectingRemote: SyncRemote { - func upsert(table: String, row: [String: AnyJSON], onConflict: String?) async throws -> [String: AnyJSON] { - throw RemoteFailure(classifying: PostgrestError(code: "42501", message: "rls")) - } - func delete(table: String, primaryKey: String, pk: String) async throws {} - func fetch(table: String, cursorColumn: String, since cursor: SyncCursor?, primaryKey: String, - scope: ScopeFilter?, limit: Int) async throws -> [[String: AnyJSON]] { [] } -} -``` - -An error the engine can't classify is treated as **transient** — safer to retry than to silently drop -a user's write. +- [Documentation home](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync) — the landing page, and the full symbol reference +- [[Home]] — this wiki's index diff --git a/wiki/_Sidebar.md b/wiki/_Sidebar.md index aa29143..eb59999 100644 --- a/wiki/_Sidebar.md +++ b/wiki/_Sidebar.md @@ -1,16 +1,12 @@ ### HappySync - [[Home]] -- [[Getting Started]] -- [[Server Setup]] -- [[API Reference]] -- [[Operations and Troubleshooting]] -- [[Testing]] -- [[Architecture]] +- [Documentation](https://swiftpackageindex.com/happyface-studio/HappySync/documentation/happysync) — Getting Started, Server Setup, API, Operations, Testing, Architecture - [[Claude Skill]] --- +- [Example app](https://github.com/happyface-studio/HappySync/tree/main/Examples) - [Releases](https://github.com/happyface-studio/HappySync/releases) - [Sync contract](https://github.com/happyface-studio/HappySync/blob/main/docs/SYNC-CONTRACT.md) - [Source](https://github.com/happyface-studio/HappySync)