Skip to content

feat: add portable archive export and restore - #1561

Open
larry-zy wants to merge 45 commits into
oceanbase:masterfrom
larry-zy:feat/1421-portable-export-backup-restore
Open

larry-zy wants to merge 45 commits into
oceanbase:masterfrom
larry-zy:feat/1421-portable-export-backup-restore

Conversation

@larry-zy

@larry-zy larry-zy commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Closes #1421

Rationale for this change

PowerContext needs a backend-neutral logical archive for disaster recovery, offline transfer, customer-controlled backup, and SQLite/SeekDB/OceanBase migration without losing immutable history, lineage, review decisions, or Handoff evidence. Database-native backup remains a separate operational recovery mechanism.

What changes are included in this PR?

  • Adds a versioned .pcb contract with deterministic manifests and ZIP output, producer/schema versions, per-record SHA-256 digests, an aggregate checksum, and optional compression.
  • Streams scope-selective export, inspect, target-aware dry-run validation, and transactional/idempotent restore with content-free progress.
  • Preserves Scope relationships, Sources and journal positions, remote Source Definition manifests, immutable Artifact Revisions including timestamps/actors/rollback metadata, lineage, cross-Scope publication provenance, Memory versions/heads, Candidate decisions, Work Sources, Handoffs, receipts, and frozen recurrence evidence.
  • Requires export authorization before database enumeration. Excludes credentials, provider secrets, access bindings, audit/usage/evaluation data, host-local state, cursors, and derived search projections.
  • Validates Scope external references through their domain model, Sources through the configured target decoder and transported schemas, and tag display labels through the normal tag rules. A tag's normalized key and key hash must agree with its label; malformed checksum-valid input is rejected before domain writes. Unicode display labels are preserved rather than silently normalized.
  • Enforces identity/storage bounds and dependency/uniqueness constraints, detects immutable conflicts before writes, and preserves rollback-safe retries with durable readiness receipts. Incomplete projection recovery blocks normal Runtime/worker startup until an explicit restore completes it.
  • Rebuilds Atomic Memory from authoritative heads. Explicitly rebuilds Experience/Skill search projections, including SQLite FTS, instead of relying on version-gated initialization; restore cannot report ready while supported restored heads lack searchable text.
  • Retries only MySQL/OceanBase error 1205 during vector-index creation, with bounded backoff and a fresh existence check on each attempt. Other errors and cancellation remain visible; the caller's transaction is not committed or rolled back by the retry handler.
  • Recovers canonical default-Scope identity without bootstrapping a competing Scope or starting background Artifact processing. Preserves microsecond MySQL revision timestamps and unknown legacy metadata as NULL; read-only validation reports incompatible schemas without migrating them.
  • Prevents export from replacing live SQLite databases/journals, including case aliases, and rejects malformed or unsupported ZIP input with the documented invalid-bundle response.
  • Adds operator documentation for format compatibility, identity/conflict policy, recovery, exclusions, and distinct SQLite/OceanBase native backup procedures.

Are there any user-facing changes?

Adds powercontext archive export, powercontext archive inspect, and powercontext archive restore. Restore writes require --yes; --dry-run performs compatibility/conflict checks without domain writes. This is an offline operator boundary, not a new remote HTTP/MCP endpoint. Writers must be stopped for restore; export takes a consistent database-transaction snapshot.

Introduces portable format version 1 and pc_portable_restore_receipts. Versioned record schemas reject incomplete authoritative state rather than silently defaulting it; revision metadata and candidate/head state use their declared schema versions. Normal target initialization owns schema upgrades.

The CLI extra includes the Builtin runtime needed for offline archive operations; it is not a remote-only installation. CLI-only installations do not require FastAPI or the Server role. The isolated-role smoke test checks this advertised dependency and still rejects unavailable Server/service commands.

How was this change tested?

  • UV_NO_SYNC=true make check contract-test: lock consistency, pre-commit hooks, Ruff, type checks, integration manifests, and API/JS/Desktop contracts passed on the final source tree.
  • python -m pytest -n 4 --doctest-modules -q with Python 3.14, isolated HOME, Node 24, and case-sensitive temporary storage: 6431 passed, 585 skipped. Skipped external-host/service tests do not establish acceptance for those integrations.
  • The new tag regression was reproduced before the fix: eight checksum-valid label/key mutations were incorrectly accepted. Inspection, validation, and restore now reject them without changing existing Scopes; valid archives retain exact tag readback and filtered search after restart/idempotent retry. Python 3.11, 3.12, and 3.13 each passed all ten selected tag/family/restart cases on the final source tree.
  • The Experience/Skill search regression was reproduced on SQLite before the fix. Both searches now work before export, immediately after restore, and after retry/restart on SQLite, native SeekDB, and actual OceanBase. The final full suite includes all 27 native SeekDB archive scenarios.
  • Actual OceanBase CE 4.3.5.6: 27 archive scenarios and 10 authorization/read-snapshot cases passed on the first merged source tree. After the source-migration merge, all 21 unique cases from the originally failing acceptance cohort passed, including the two opt-in migration cases; the Experience/Skill package scenario also passed. These live runs precede the final atomic-ownership merge and tag-only fix; the final tree has separate local regression coverage and requires its own remote CI result.
  • Simulated adapter cases verify transient/persistent 1205, non-retryable 1142/1213, and cancellation handling. The original GitHub failure establishes that the real server emitted 1205 at vector-index creation; these mocks do not claim a forced live-server lock-timeout recovery.
  • Clean Python 3.11 CLI-only wheel installation outside the checkout: isolated-role smoke plus archive export, inspect, dry-run, restore, idempotent retry, Unicode tag readback, and filtered search passed without FastAPI or pytest.
  • Website lint/build verified 956 public pages and their internal links before the final upstream RFC text merge; no website implementation changed in the tag fix.
  • Windows portability cases are included in windows-unit-portability; they are not executed locally. Remote checks and reviewer approval remain separate merge gates.

Limitations

Target compatibility checks remain per record. Experience/Skill projection rebuilding traverses the target's supported heads, not just restored Scopes. These are offline maintenance costs; no representative large-archive throughput or bounded-memory-at-scale claim is made. Batching is a non-blocking optimization opportunity that requires backend measurements.

AI usage statement

AI assistance was used for implementation, tests, documentation, and review. The changes were reviewed and validated locally.

…r and Client

The ready-to-run Server and the primary Python Client did not enforce the
network safety policy already applied by the Agent integrations, allowing two
unsafe transports:

- ServerSettings accepted a non-loopback bind (e.g. 0.0.0.0) while bearer
  authentication was disabled, silently exposing an unauthenticated Server.
- ClientSettings / PowerContextClient accepted a non-loopback http:// URL and
  would send a bearer token over plaintext.

Introduce a single shared transport-policy module (powercontext.transport)
and consume it everywhere:

- ClientSettings rejects non-loopback plaintext http:// Server URLs.
- PowerContextClient refuses to send a bearer token over an unencrypted
  non-loopback connection.
- ServerSettings rejects an unauthenticated non-loopback bind unless the
  operator opts in via POWERCONTEXT_SERVER_ALLOW_UNAUTHENTICATED_NON_LOOPBACK
  (for TLS-terminated / controlled-network deployments).
- The `server run` CLI applies the same guard, since model_copy bypasses the
  settings validator on --host overrides.

Add a cross-surface contract test suite and update the EN/ZH configuration
reference.

Closes oceanbase#1319
# Conflicts:
#	docs/en/docs/reference/configuration.md
#	docs/zh/docs/reference/configuration.md
#	src/powercontext/server/settings.py
Load the Codex plugin's settings module by path and assert its private
loopback host set and _http_base_url behavior match the shared transport
contract, so the isolated plugin copy cannot silently drift. Tighten the
configuration docs to note the plaintext-URL rule applies to configured
Server URLs.
The loopback check only matched the three named hosts 127.0.0.1, ::1 and
localhost, so plaintext HTTP to any other 127.0.0.0/8 address (e.g.
127.0.0.2) was wrongly rejected as non-loopback. Detect loopback via
ipaddress.ip_address(...).is_loopback, which covers the whole IPv4
loopback block, and route the Client CLI through the shared helper.

Both agent plugins ship isolated and cannot import powercontext, so each
vendors its own copy of the check; parametrised drift-guard tests pin both
copies to the shared contract. Surface ServerSettings validation errors as
a typer.BadParameter instead of a raw traceback.
…ports

The bearer-token-over-plaintext guard is only meaningful for the transport
the client opens itself, where base_url's scheme reflects what crosses the
wire. A caller-supplied http_client owns its own transport (ASGI in-process,
a Unix socket, a TLS-terminating proxy -- all carrying an http:// label), so
the scheme is no longer a reliable signal and enforcing it only produced
false positives, breaking the authenticated in-memory e2e flow. Enforce the
guard only when the client creates the transport.
The Docker image binds POWERCONTEXT_SERVER_HTTP_HOST=0.0.0.0 so the port is
reachable from the harness container, but the unauthenticated-non-loopback
guard now refuses that bind and the Server fails to start. The Compose
network is isolated, so opt in via
POWERCONTEXT_SERVER_ALLOW_UNAUTHENTICATED_NON_LOOPBACK to restore startup.
 review

- client: gate the plaintext-token guard on an explicit trust_transport_security
  opt-in; supplying an http_client is no longer treated as evidence the transport
  is safe, so a bearer token is still refused over unencrypted non-loopback HTTP.
- docker: declare POWERCONTEXT_SERVER_ALLOW_UNAUTHENTICATED_NON_LOOPBACK in the
  image itself so the documented `docker run` starts out of the box, document the
  network-exposure trade-off, and drop the masking env from the e2e compose so it
  exercises the real image contract; add a Dockerfile smoke test.
- server cli: merge --host/--port before validation so a safe override can repair
  an unsafe environment bind, and translate the actionable settings failures
  (unauthenticated non-loopback bind, missing bearer token) into friendly CLI
  errors recognised by exception identity rather than raw validation text.
- cli: unpack the ServerSettings kwargs so the partial ``http`` mapping is not
  type-checked against the full HttpConfig annotation (ty invalid-argument-type).
- tests: pin a wide COLUMNS in the friendly-error tests so the rich panel does
  not hard-wrap the asserted --host / env-var tokens on a narrow CI terminal.
…ntract

Pi's isLoopback only accepted localhost, 127.0.0.1 and [::1], so its production
resolveConfig rejected http://127.0.0.2:8000 as "must use HTTPS outside loopback"
while the shared policy and the Python plugins already trust the whole 127.0.0.0/8
block. Rewrite it to mirror is_loopback_host (strip brackets, lowercase, accept
127.0.0.0/8, localhost and ::1).

Add a Pi Vitest drift guard and a JSON fixture of loopback/non-loopback host
vectors that both the Python drift guard and the Pi suite consume, so a future
divergence from the shared contract fails in at least one language.
…error assertions

pretty-format-json rewrites the fixture with one array element per line; apply that
formatting so the quality hook passes in CI.

The friendly-error CLI tests asserted tokens (--host, the opt-in env var) against
typer's rich error panel. Under GitHub Actions typer forces force_terminal=True and
the runner's TERM=dumb pins rich to 80 columns while ignoring COLUMNS, so the panel
hyphen-breaks --host and force-splits the long env var. Replace the COLUMNS override
with a fixture that neutralises the forced terminal and sets an explicit wide width,
making the assertions independent of the terminal environment.
…hes for the transport

The loopback guard only tripped when a bearer token was present, so an
unauthenticated client would still ship Memory content -- carried in the request
body -- over unencrypted non-loopback HTTP. Broaden the guard to reject *any*
plaintext non-loopback request, and let a caller that knows its transport is
secure opt in explicitly with `trust_transport_security=True`; supplying an
`http_client` is not itself evidence of safety.

Thread that opt-in through the LangGraph adapter's `shared_http_client` /
`open_client`: a plain pooling client stays untrusted, while in-process ASGI
harnesses declare trust. Update the in-process test harnesses accordingly.

Also stop the remote-access guides from documenting a bare
`server run --host 0.0.0.0`, which the bind policy now rejects; show the
authenticated bind and the explicit unauthenticated opt-in instead, and add a
command-level contract test so a copy-pasteable command that fails to start
fails in CI rather than only for a reader.
Resolve tests/e2e/test_runtime_server.py: upstream dropped the explicit
handoff-report project/workstream setup in the delegation-loop e2e; keep that
refactor and re-apply this branch's `trust_transport_security=True` on the
in-process ASGI client.
The hardened client guard now refuses plaintext non-loopback URLs
unconditionally, which broke the e2e harness where agent containers
reach the server over the private Compose bridge via host-gateway.

Add an explicit POWERCONTEXT_BUB_TRUST_TRANSPORT_SECURITY opt-in
(default false) that routes through the client's endorsed bypass: a
caller-supplied httpx transport plus trust_transport_security=True.
Enable it only in the e2e Compose harness, mirror the server-side
ALLOW_UNAUTHENTICATED_NON_LOOPBACK opt-in shape, document the switch,
and cover both the refusal default and the trusted construction.
…t-policy

# Conflicts:
#	e2e/bub/uv.lock
#	integrations/bub/pyproject.toml
#	integrations/bub/src/powercontext_bub/plugin.py
…t policy

The LangChain middleware and pydantic-ai tests merged from master were
written before the client started refusing all plaintext non-loopback
URLs, so their in-process ASGI transports now hit the guard.

Port the langgraph-style trust plumbing to the LangChain shared HTTP
client and let the affected tests vouch for their ASGI transports with
trust_transport_security=True.
The plugin hooks honoured POWERCONTEXT_BUB_TRUST_TRANSPORT_SECURITY, but the
powercontext.search/remember/context tools still constructed bare clients, so
the acceptance harness failed on the plaintext host-gateway hop. Hoist the
vouched construction into a module-level open_client helper, carry the flag in
the per-turn tool settings, and cover both tool paths in the trust tests.
…k requests

The reference still described the pre-hardening guard, which only refused to
attach a bearer token. Align both language variants with the shipped behaviour:
every request over unencrypted non-loopback HTTP is refused, and callers whose
http:// base URL merely labels a secure transport must supply their own
http_client with an explicit trust_transport_security=True.
@larry-zy
larry-zy force-pushed the feat/1421-portable-export-backup-restore branch from ea1688e to fc1521a Compare September 11, 2026 13:26
@larry-zy
larry-zy force-pushed the feat/1421-portable-export-backup-restore branch from fc1521a to 3c0e6fb Compare September 11, 2026 19:27
@larry-zy
larry-zy marked this pull request as ready for review September 11, 2026 19:28
@larry-zy larry-zy changed the title feat: add local portable archive CLI feat: add portable archive export and restore Sep 11, 2026
@AlexStocks AlexStocks removed the 1.3.0 label Oct 11, 2026
Comment thread src/powercontext/builtin/portability/bundle.py Outdated

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two correctness issues remain: the merge relationship loss discussed in the existing P1 thread, and the Candidate proposal validation gap below. Both were reproduced on SQLite at this commit.

Comment thread src/powercontext/builtin/portability/bundle.py
Comment thread src/powercontext/builtin/portability/bundle.py Outdated
Comment thread src/powercontext/builtin/portability/bundle.py
Comment thread src/powercontext/builtin/portability/bundle.py
Comment thread src/powercontext/cli/archive.py Outdated
@hidb4ai

hidb4ai commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

please update

@hidb4ai

hidb4ai commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

please update & resolve conflicts

@PsiACE PsiACE left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design note (non-blocking): I understand the goal of preserving history across backends, but I don’t see why backups and database migrations require a private export/import protocol. Please identify the concrete gaps in existing backup, dump/load, and migration tools before introducing a separate mechanism.

Comment thread src/powercontext/builtin/portability/bundle.py
Comment thread src/powercontext/builtin/portability/bundle.py
Comment thread src/powercontext/cli/archive.py Outdated
Comment thread src/powercontext/builtin/runtime/relational.py

@AlexStocks AlexStocks left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

审查收尾对账 — head 776a8fdbd08c5dd5387ad1fb1184b1f54cb6fe68

审查在隔离镜像中进行(未 checkout 开发目录),快照固定为 head 776a8fdb、base 40bd8481(= master merge-base),并发数与 API 一致:30 文件、+5804/−50、38 commits。git archive 侧与 API 侧逐文件对账一致(only-in-api 与 only-in-diff 均为空)。

既有讨论复验(全部对当前 head 行为复验,均已由作者标记 resolved)

线程 结论 行为复验方式
4256690235 / 4256841927(merge 状态与 merged_into_id) 已修复 test_archive_preserves_merge_state_and_undo_after_restart[sqlite]、test_archive_detects_conflicting_merge_topology[merged_into_id|is_merge_input]、test_archive_rejects_invalid_merge_records_before_writing[missing_result|old_head_schema|old_lineage_schema|invalid_marker] 通过
4278062386(同时保留 is_merge_input) 已修复 同上(lineage 侧 schema_version=2 + is_merge_input 的布尔校验经 invalid_marker 用例覆盖)
4256842752 / 4278062403(candidate proposal 解码) 已修复 test_archive_rejects_invalid_candidate_proposals_before_writing[empty|invalid_envelope|invalid_audit]×3 family、test_archive_rejects_invalid_candidate_envelopes_before_writing[5 种损坏]、test_archive_restores_candidates_through_supported_read_paths[sqlite×audited×3 family] 通过
4278062393(迁移后保留的 legacy entry 行) 已修复 test_archive_exports_migrated_scopes_without_deleting_offline_history 通过:断言 pc_memory_entry_versions 与 archive 表的保留行前后不变、bundle 中不含 memory_entry_version、restore 为 ready、且往返 total_digest 相等
4278062408(SQLite file: URI 存在性判定) 部分修复 test_archive_cli_dry_run_checks_existing_sqlite_uri_target[relative] 通过;[absolute] 与 [escaped] 在 Windows 上仍失败,已在 4293611535 补充残留范围

本次新增发现

  • 4293611535(P2,cli/archive.py):Windows 下绝对 file: URI(file:///C:/…,即 Path.as_uri() 产出形式)仍解析为 /C:/...,dry-run 退回内存库并返回假绿。与 4278062408 同根因,属未覆盖的输入形式。
  • 4293611962(P2,runtime/relational.py):中断的 restore 会让 receipt 停在 authoritative_restored,而 Atomic Memory 投影没有启动期完整性校验;作用域只含 Atomic Memory 产物时正常启动与服务入口都会放行,搜索静默缺失已恢复记忆。文档中「正常启动拒绝不完整投影」的承诺对该作用域不成立。

平台与覆盖说明(不计入 PR 结论)

  • 本机 Windows 上 tests/e2e/test_portable_archive.py 为 64 passed / 47 skipped / 1 failed。唯一失败 test_archive_restores_package_files_and_topic_search_after_restart[sqlite] 是测试夹具的平台产物:夹具用 Path.write_text("…\n") 落盘,Windows 文本模式把 \n 翻译为 \r\n,断言比对 b"…\n" 因而失配,与产品代码无关。
  • 47 个 skip 全部来自 target_config 的 seekdb(缺 pylibseekdb)与 server(未设 POWERCONTEXT_TEST_OCEANBASE_URL)参数化,因此 OceanBase / SeekDB 路径本次未在本地执行,也未对其作任何等价推断。
  • tests/builtin/portability/test_bundle.py 与 tests/test_archive_cli.py 为 27 passed / 2 failed(失败即上面 4293611535 的两例)。
  • 单元测试层没有任何用例注入 candidate_decoder / artifact_content_decoder(tests/ 内检索为空),新增的两个解码器校验路径仅由 e2e 覆盖。
  • master.yml 的 windows-unit-portability 只执行 make opendal-test,不运行 Python 测试套件,故上述 Windows 差异在 CI 中不可见。

Merge 结论

state=open、merged=false、draft=false、mergeable=true、mergeable_state=blocked、review_decision=null。head 上 37 个 job 的最新一次运行全部为 36 success + 1 skipped、0 非成功,且本 ruleset 不含 required_status_checks,故 blocked 来自缺少 1 个 approving review。在 4293611535 与 4293611962 两条 P2 处理前不建议合并;两者都不涉及权威数据损坏,处理方式可以只是收窄文档承诺或补一处就绪判定。

Comment thread src/powercontext/cli/archive.py
Comment thread src/powercontext/builtin/portability/bundle.py
Comment thread src/powercontext/builtin/portability/bundle.py Outdated
@hidb4ai hidb4ai added 1.5.0 v1.5.0 release tracking and removed 1.4.0 v1.4.0 release tracking labels Oct 19, 2026
@AlexStocks AlexStocks added the 1.4.1 v1.4.1 release tracking label Oct 20, 2026

@knqiufan knqiufan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One reproduced restore-validation defect remains: a checksum-valid bundle can be committed as ready while leaving its Scope unreadable through the supported API. Details are attached inline.

Comment thread src/powercontext/builtin/portability/bundle.py

Copy link
Copy Markdown
Collaborator

Design note (non-blocking): batch target compatibility checks for large archives

In _target_compatibility, lines 773–774, each record whose target table exists triggers a separate SELECT within the validation transaction. Thus N such records require N compatibility queries, and restore() repeats this validation after an operator's separate dry-run. The format permits up to 2,000,000 records, so database round trips may dominate large OceanBase/SeekDB restores.

Consider benchmarking representative archive sizes on the actual backends, then batching compatibility lookups by record type while preserving checks of both primary and alternate unique identities. _write_batch provides an existing batching example, though compatibility checks also need the alternate uniqueness constraints.

Open Code Review raised this as a medium-severity performance concern. I am treating it as non-blocking: the query pattern is confirmed from the code, but I have not measured real OceanBase/SeekDB latency or demonstrated a violated performance budget. This is separate from the reproduced P2 Scope readback defect.

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A reproduced projection-recovery defect remains: restore reports readiness while restored Experience and Skill artifacts are absent from search. Details inline.

Comment thread src/powercontext/builtin/runtime/relational.py Outdated
Comment thread src/powercontext/builtin/portability/bundle.py

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

1.4.1 v1.4.1 release tracking 1.5.0 v1.5.0 release tracking

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants