Repository navigation
feat(migrations): complete the reusable database upgrade framework - #2168
Zxf-xufeng wants to merge 118 commits into
Conversation
Implement issue01 of the Source evidence declarations specification. Bind trusted native and remote declarations to exact Source material and retain per-revision Atomic Memory snapshots in the existing transaction. Preserve neutral fallback, exact identities, scope authorization and public retrieval contracts. Add versioned p0004 migrations and SQLite persistence regressions. Refs #1975. Full suite: 5255 passed, 431 skipped, 7 failed; introduced observability adapter failures fixed and its file passes 18 tests. Topic Memory vector failure also reproduces at the unchanged baseline.
Cover persisted declarations, scope isolation, historical neutrality, synthetic model trust claims, and unchanged recall consumers on SQLite and OceanBase.
Preserve the published p0004 artifact handle revision and introduce evidence as p0005. Retain both schema intents, recompute exact SQLite fingerprints, and update MySQL migration head expectations.
Retain native Source identity independently of adapter aliases and mutable description defaults, preserve legacy read/replay behavior, and keep serialized business fields intact. Capture evidence at the shared Atomic Memory body-commit boundary, including restoration revisions, without retroactively attesting state-only writes. Refs #1975.
Check native serialization compatibility at acceptance without blocking valid captures or Memory writes. Keep previously accepted incompatible declarations neutral while preserving exact material and digest checks. Reject legacy identities that cannot be decoded exactly instead of supplying a default Definition version.
Preserve Source evidence verification and upstream revision metadata initialization in Atomic Memory projection rebuild.
Reject native serialized material that omits an identity field before journal allocation or persistence, rather than accepting Sources that later fail evidence capture. Preserve strict retained identity checks.
Validate serialized identity values as well as key presence before persistence, matching historical evidence validation. Cover round-trippable codecs for all three identity fields without weakening retained-material checks.
Preserve exact evidence inspection alongside upstream bounded list queries and reuse its authorized read snapshot lifecycle.
Canonicalize worker envelope keys and their manifest without changing native persisted schemas. Read retained canonical identities before consulting current adapter aliases, preserving neutral historical evidence.
Only trust retained canonical identity keys when the registered model does not claim those spellings for business fields. Keep headerless business-aliased material on the declared decoder, and format the regression file with the pinned hook version.
Resolve referenced root schemas before canonicalizing transport identity keys while preserving nested serialization aliases. Cover manifest registration, observation submission, and SQLite readback.
Preserve Source evidence persistence alongside upstream Atomic Memory startup vector validation, including both contracts in the observability test index.
Keep remote manifests and payloads on the canonical field-name contract so existing Definition fingerprints remain stable. Preserve native historical identity and retained alias spellings across upgrades, replay, and rebuild.
Generate remote transport schemas with the same alias policy as their payloads so existing serialize_by_alias Definitions keep their fingerprint and observations after upgrade. Recover explicit canonical historical versions beyond version 1 instead of handing accepted material to a newer aliased Definition.
Zxf-xufeng
changed the title
Allow complete Server setup within the aggregate long-query CI step. Eight healthy isolated cases can exceed the former ten-minute limit while each remains inside the existing three-minute process deadline. Constraint: Keep all assertions, the 180-second case limit and the 100-minute job limit. Scope-risk: CI aggregate execution budget only; runtime and migration resources are unchanged. Tested: Eight real OceanBase cases passed without skips in 665.9 seconds; YAML structure and git diff --check pass; Ty and other validators pass in the existing environment, while the hook wrapper reports JSON normalizations without a tracked JSON diff.
hireview001
left a comment
hireview001
left a comment
There was a problem hiding this comment.
- Resolve merge conflicts with the target branch.
|
Resolved the conflicts with master at 1da9974 and pushed 8f10c560. Both inline implementation findings have separate replies. The merge also requires registering storage already introduced on master. A new s1402 revision adds portable restore receipts and the four native DATETIME(6) columns while preserving the earlier revision resources/checksums. Ordinary managed startup remains read-only and rejects incomplete restores; the existing explicit archive-recovery entry can finish projections after layout and configuration checks. Validation: make check passed; the persistence/managed-maintenance/upgrade-policy selection passed 709 tests (102 skipped, including unavailable native backend cases); CLI/startup selection passed 47; final initialization/policy regressions passed 68; Compose launcher regressions passed 7. SQLite archive restoration was also exercised with and without vectors. These results overlap and are not a combined test count. The earlier native release-gate and long-query acceptance findings remain pending the CI results for this new commit. Local modeled tests do not establish seekDB or OceanBase release acceptance. GitHub now reports MERGEABLE. The new Upgrade release gate is queued. |
hidb4ai
left a comment
hidb4ai
left a comment
There was a problem hiding this comment.
Reviewed 9b457be2d21f. Requesting changes for the embedding-profile error classification described inline.
The upgrade release gate also remains incomplete: the seekDB no-vector, Code, and vector+Code profiles stop at native CREATE TABLE pc_schema_revision with -6210. This does not establish a migration-code root cause. The OceanBase Recall job fails with 4184: Server out of disk space, an environment-capacity failure. The previously reported OceanBase Atomic Memory/long-query check now passes.
|
@Zxf-xufeng Please prioritize the remaining 1.4.1 blockers in this PR: resolve the master merge conflict, fix the embedding-profile error classification, and complete installed-release acceptance for all twelve backend/configuration profiles. The latest retained evidence still has no passing seekDB profile. If the native |
hidb4ai
left a comment
hidb4ai
left a comment
There was a problem hiding this comment.
Reviewed 9b457be2d21f with independent installed-package tests. Changes are still required for the existing P2: changing only the embedding profile ID after a successful seekDB upgrade reports checksum_conflict instead of embedding_profile_mismatch. Startup still rejects the mismatch without changing rows.
I installed the actual PyPI 1.3.0 and 1.4.0 wheels separately, created populated databases through their HTTP Servers, and migrated with the wheel built from this head (1.4.1.dev246+g9b457be2d). These runs used real SQLite, embedded seekDB (pylibseekdb==1.4.0.post1), OceanBase CE 4.3.5.4, and real generation/embedding providers. Configuration: Code disabled, 1024-dimensional unit-normalized L2 vectors.
| Source → this PR | SQLite | seekDB | OceanBase |
|---|---|---|---|
| 1.3.0 | Passed | Migration and business checks passed; P2 reproduced | Passed |
| 1.4.0 | Blocked: unknown_baseline |
Blocked: unknown_baseline |
Blocked: unknown_baseline |
The 1.3.0 runs verified historical/new reads and writes, references, Candidate history and approval retries/CAS, access bindings, cursors, FTS/vector retrieval, restart, unchanged historical embedding hashes, repeat apply, real background Memory extraction with Source citations and Worker usage reporting, and fresh/upgraded schema equivalence.
SQLite automatic-backup restoration passed. Actual SIGKILL after s1400 committed recovered to s1402, preserving the original backup and maintenance window, with inference endpoints unavailable. OceanBase fresh initialization exceeded the initial test timeout; after confirming no active SQL or pending DDL for that dedicated database, explicit acknowledgment of the original run allowed recovery and fresh-storage verification.
Support boundary: 1.4.0 databases remained unchanged and readable by 1.4.0 after rejection. A separate SQLite case without vectors also returned unknown_baseline. The current policy selects only 1.3.0; if 1.4.0 users are in release scope, register their actual baseline/conversion and add installed-release acceptance.
These small fixtures do not cover Code-enabled configurations, production-scale performance, remote backup restoration, or observer crashes. Local success does not replace the required twelve-profile upgrade gate, which remains failed. The OceanBase Recall rerun now passes. The PR still conflicts with master in four Desktop files; resolve those conflicts and rerun acceptance against the resulting head.
|
Follow-up to the latest review and the release-blocker request:
|
hidb4ai
left a comment
hidb4ai
left a comment
There was a problem hiding this comment.
Design note (non-blocking): simplify schema registration.
I recommend keeping the existing Alembic dependency and using frozen initial baselines, immutable incremental revisions, and CI-generated validation artifacts. The current s1400 registration pairs 41 SQL statements with 41 structure transitions; pc_candidate_heads alone has 11 intermediate definitions. There is room to reduce duplication while preserving recovery checks.
- Freeze each supported historical baseline once. Share identical definitions where practical, while retaining backend-specific differences and capture provenance.
- Maintain incremental Alembic revisions, domain data transformations, and explicit recovery preconditions/postconditions. Historical migrations must remain independent of current application metadata.
- Replay the migration history on real SQLite, seekDB, and OceanBase in CI to generate the expected structures, digests, and necessary DDL recovery checkpoints. Package these generated artifacts with the wheel so developers do not have to maintain a separate full-table definition for each SQL operation.
Keep target locks, backups, resource-integrity checks, domain validators, and the original maintenance evidence. Recovery must verify actual database state; a matching checksum or a completed-step log is insufficient.
Start with an isolated prototype of the s1400 Candidate conversion and require equivalent installed-release, interruption/recovery, repeated-apply, and historical-data acceptance before expanding it. Any production refactor must preserve compatibility with already-applied revision checksums and existing recovery evidence.
|
Implemented the bounded prototype requested in the schema-registration design review, with the final changes in
The suite covers a complete run and interruption after copy/rename, distinguishing SQLite transaction rollback from committed native DDL. It checks preserved Candidate history, the original maintenance evidence, recovery, and a no-change repeated apply. The native repeated-apply check now uses a fresh runner after the recovery acknowledgment has been consumed; production rejection of stale acknowledgments remains enforced. The generator and its evidence are outside the production bundle. Existing historical resource bytes, revision checksums, locks, backup policies, actual-state validation and recovery evidence remain unchanged. This implements the first-stage replay/equivalence prototype; it does not replace all production registrations or claim a new generated registration format is ready for rollout. Generated structures still require review and data-correctness acceptance before being used for new migrations. Validation: the three-backend Candidate replay CI at |
|
Current status for the remaining release blockers and installed-package review, at
The schema-generation suggestion is implemented as a bounded Candidate replay prototype, with its scope and evidence recorded in the original design reply. It does not replace production registrations or relax locks, backups, actual-state checks or recovery evidence. Other released versions still require explicit source registration and installed-release validation; successful 1.3 and 1.4 paths do not prove arbitrary future sources. |
hidb4ai
left a comment
hidb4ai
left a comment
There was a problem hiding this comment.
Reviewed 3a3569bd16b3. Requesting changes for the P1 described inline, reproduced locally on real seekDB and OceanBase.
I upgraded populated databases created by the actual PyPI 1.3.0 and 1.4.0 wheels using the installed target wheel 1.4.1.dev290+g3a3569bd1. Configuration: Code disabled, real 1024-dimensional unit-normalized L2 embeddings and real generation/embedding providers.
| Source | SQLite | Embedded seekDB | OceanBase CE 4.3.5.4 |
|---|---|---|---|
| 1.3.0 | Passed | Apply timed out after 900s | Passed |
| 1.4.0 | Passed | Apply timed out after 900s | Passed |
The passing paths verified preserved history/references/permissions/cursors, Candidate CAS and retries, unchanged historical vectors, repeat apply, restart/new writes, real background Memory processing with Source citations, and fresh/upgraded schema equivalence. Both SQLite sources also passed automatic-backup restoration and recovery after an actual SIGKILL following committed s1400, retaining the original backup and maintenance window with inference endpoints unavailable.
Both seekDB timeouts retained active baseline-adoption journals. Subsequent public plans returned recovery_required without changing that evidence; successful recovery apply and post-upgrade operation remain unproven.
The current installed-release gate passed 18/24 cases. All six failures are seekDB cases whose native logs record CREATE TABLE pc_schema_revision returning -6210. This is separate from the reproducible startup/publication locking defect below; the local timeouts alone do not establish an upstream root cause. Keep the complete acceptance gate enforced.
| from powercontext.builtin.runtime.atomic_memory_rebuild import verify_atomic_memory_current | ||
|
|
||
| atomic_index = SQLiteAtomicMemoryIndex() if connection.dialect.name == "sqlite" else OceanBaseAtomicMemoryIndex() | ||
| report = finish(verify_atomic_memory_current(pinned(connection), atomic_index)) |
There was a problem hiding this comment.
[P1] Keep startup validation on non-locking snapshot reads
verify_atomic_memory_current() calls _record(), which invokes AtomicMemoryStateRepository.require_summary() and issues SELECT ... FOR UPDATE on pc_artifact_heads. These exclusive locks remain until the full readiness transaction exits; START TRANSACTION WITH CONSISTENT SNAPSHOT neither removes them nor makes the locking reads snapshot-consistent. I reproduced this with the installed target on both native backends: on OceanBase, a normal public Memory PUT returned 200, but the same operation interleaved with a second Server startup waited 10.27s and returned 500/1205; the new startup then raised domain_validation_failed. The real seekDB concurrent-startup test fails the same way. Readiness can therefore block normal publication for the duration of its scan and reject startup on healthy storage. Use non-locking snapshot reads throughout verification while preserving the locking semantics of mutation/rebuild operations. The vector validator calls the same verifier and needs the same correction.
There was a problem hiding this comment.
Fixed in 65ab7ef.
The shared verifier now reads Atomic Memory state and content without the locking require_summary() recheck. Both vector and non-vector validators use this path for Server, SDK and background admission. All three repair reads explicitly retain locking, including the existing state comparison, head locks and write-side CAS checks. Frozen migration resources and historical checksums are unchanged.
The existing concurrent public-update/startup regression now also checks generated MySQL SQL for locking clauses, so SQLite runs detect this regression despite SQLite omitting those clauses. It failed before the correction and passes afterward; projection repair and damaged-authority regressions also pass locally.
Native confirmation is still pending on this exact commit: Main and E2E. The separate seekDB CREATE TABLE pc_schema_revision / -6210 failures remain outside this correction, and the complete installed-release gate remains enforced. I am leaving this thread open pending native results and re-review.
Which issue or RFC does this PR close?
Refs #2154 and implements the reusable database migration framework described in RFC #1771.
Includes the installed-release acceptance and publishing gate from #2181 for #2161. Release support remains conditional on the complete gate passing for the exact target artifact.
Rationale for this change
A partial schema upgrade does not establish that a complete Server can use an existing database. The framework recognizes registered historical storage, applies ordered schema and data transformations, preserves business history, and validates the resulting database before APIs or workers start.
The declared target release is PowerContext 1.4.1. Actual PowerContext 1.3.0 and 1.4.0 have separate registered source paths. Application versions and database revision identifiers are distinct: the released 1.4.0 database is
r1400, not the framework'ss1400checkpoint. Other releases require reviewed source registration and acceptance; upgrading from the oldest source does not certify every intermediate release.What changes are included in this PR?
s1300 → s1400 → s1401 → s1402; actual 1.4.0 followsr1400 → s1400 → s1401 → s1402. The 1.4 bridge packages real three-backend captures and provenance, adds missing Topic budget/block storage, preserves current Candidate/Artifact/task data and fences old leases. Empty retained tables align the final schema without inventing 1.3 Candidate history. Original 1.3 resources and checksums remain unchanged.v1_3_0_000andv1_4_0_000; the shared target steps arev1_4_1_001,v1_4_1_002andv1_4_1_003. Public command output includes the software version, sequence and original revision ID. Existing IDs, frozen checksums, plan identities, version tables and recovery evidence remain unchanged. Future Server revisions use the release-based name directly and reference the actual predecessor ID. Release metadata and installation references are synchronized to 1.4.1; no tag or package is published by this change.atomic-memory-migraterecognizes Server ownership and delegates source admission to the shared framework, including 1.4.0. Historical Server storage must finish the complete upgrade before Processing or projection maintenance. Unknown or misconfigured Server databases cannot fall back to legacy writes. Standalone unmanaged SDK maintenance remains available.s1401registers the offline Memory archive;s1402registers portable restore receipts and native microsecond timestamp precision. These adapt existing capabilities rather than adding unrelated business features. Pending archive restores block ordinary startup and require explicit recovery.s1400Candidate replay observes real database inventories, generates structures/deltas/digests with provenance and compares them with registered checkpoints. It covers cross-table foreign keys, SQLite rollback, native committed-DDL recovery, preserved history and repeat apply. Generated evidence remains outside the production bundle; historical registrations are unchanged. Production registration-format replacement remains a separate stage.Are there any user-facing changes?
Use
powercontext server db-migrate status,plan,applyandverify; the default bundle isserver. Only complete validation reportsstate=ready,readiness_scope=serverandserver_ready=true.Migration names use
v<major>_<minor>_<patch>_<sequence>, with000for a registered historical baseline. Both registered sources reachv1_4_1_003; CI upgrade acceptance selects only 1.3.0.status/planreturnsource_migration,target_migrationandpending_migrations;apply/verifyreturnmigration. The original machine fields retains1300,r1400ands1400–s1402for compatibility. The new names are presentation aliases for these existing IDs, not replacement values to stamp into version tables. A release without storage changes can reuse a database revision; names do not establish upgrade acceptance.Upgrades require stopped writers, a reviewed plan and an explicit backup policy. Drain queued/running 1.3.0 Dream tasks because their old prompt contract cannot be executed by the target. Current-format 1.4.0 tasks and Processing counters are preserved. OceanBase requires a fixed coordination host and persistent evidence directory; interrupted remote runs require acknowledgment after confirming previous execution has stopped. Manual backup confirmation records the operator's declaration, not independently verified restoration.
Registered configurations cover SQLite, embedded seekDB and OceanBase with Code and unit-normalized L2 vectors enabled/disabled. Historical profile ID, model, dimension, distance and normalization must match. Migration does not call an embedding provider or implicitly change vector/embedded Code configuration. Processing and Atomic projection partial maintenance remain restricted to their registered no-vector configurations.
The support policy remains
upgrade-requireduntil the exact target artifact passes all twelve 1.3.0 combinations. Source registration, schema capture and local behavior tests do not constitute release acceptance.How was this change tested?
Current source:
dd62723dceabae8b41e293531f0080a819f87c68, based on master31ecdd61.make checkpassed, including upgrade policy, formatting, lint, type checks and 50 integration checks.Current-source CI has been triggered: Main, E2E harness. Results are pending. Current-artifact SQLite, embedded seekDB and OceanBase acceptance for 1.3.0 is not yet established. 1.4.0 is outside this CI upgrade matrix.
Available native evidence is tied to source
3a3569bd:db-migrate apply, with nativeCREATE TABLE pc_schema_revisionreturning-6210. This remains a separate unresolved acceptance failure.1205. The readiness validator acquired a head-row lock throughrequire_summary; the interleaved HTTP update then waited for that row while readiness waited for the update. The current correction removes that locking read from verification and preserves it in repair. Native reruns must verify the correction.preparetimeout. No migration failure was present in that stack; it is not covered by the readiness correction.The support policy remains
upgrade-required. No failing, skipped or missing acceptance case is waived. SQLite automatic-backup restoration has installed-release coverage; seekDB/OceanBase automatic-backup restoration and production-scale performance require separate evidence.AI usage statement
OpenAI Codex assisted with implementation, regression tests, documentation, review and CI diagnosis. Observed results and remaining acceptance boundaries are reported separately from generated code.