Skip to content

feat(sources): add opt-in image evidence and Codex capture - #1956

Open
knqiufan wants to merge 14 commits into
oceanbase:masterfrom
knqiufan:codex/experiment-image-source
Open

knqiufan wants to merge 14 commits into
oceanbase:masterfrom
knqiufan:codex/experiment-image-source

Conversation

@knqiufan

@knqiufan knqiufan commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Which issue or RFC does this PR close?

Refs #1932 — image evidence foundation and initial Codex integration. Managed-local model provisioning is a separate follow-up; this PR does not close the entire proposal.

Based on master 2d11abd6ab24592427cc4a42132101ef77884245. Image evidence remains experimental and disabled by default.

Rationale for this change

Screenshots and diagrams contain useful project evidence that text capture misses. Preserve the original image, save a validated interpretation as immutable Source evidence, and let extraction and recall use the derived text with a link to the exact original material.

What changes are included in this PR?

  • Durable PNG/JPEG assets and original/analysis Source definitions; recoverable vision jobs, bounded retry/reparse, scoped readback, idempotency, and storage/queue budgets. Reads do not call the model again.
  • Default-off wizard/.env configuration, with explicit reuse of a multimodal generation connection or a dedicated vision endpoint. Automatic model downloads and managed-local inference are not included.
  • Codex Desktop/CLI attachment adapters sharing a durable outbox, persistent project consent, incremental collection, managed client lifecycle, and standard setup codex / doctor codex integration.
  • Seven experimental OpenAPI operations, generated Python/JS/Desktop bindings, SDK/CLI operations, Dashboard image lists/thumbnails/details, and exact Artifact evidence links.
  • Atomic Memory Source-only extraction, evidence-child and revision provenance, and bounded structured-generation repair for missing reconciliation content.
  • Collector recovery preserves unread cursors when staging capacity fills. Delivery filters disabled workspaces before pagination and rechecks consent before upload; re-enabling preserves staged bytes without importing messages from the disabled period.
  • Integration with current startup context/receipts, Working Note eligibility, Atomic Artifact search, Dream/Candidate contracts, and Dashboard review/history/Scope reads. Python/JS/Desktop bindings and the DSH bundle are generated from the combined contract; the current master ZCode Skill pin is retained.
  • Runtime composition preserves atomic local ownership writes and the Handoff committer alongside the optional image service. Generated bindings use the combined OpenAPI contract and the current Desktop generator.
  • Codex image consent and diagnostics use the current unified setup / doctor installer contracts, including explicit --capture-images / --no-capture-images options. Background infrastructure apps retain their restricted route surface; image errors use the shared HTTP error boundary.
  • Windows startup receipts close and fsync the temporary file before atomic replacement in both Codex and Claude Code. The prior open handle caused WinError 32; the defect was reproduced from unmodified master. Locking and receipt identity remain unchanged.
  • Bilingual user documentation, synthetic fixtures, recovery tests, and an implementation/acceptance guide.

Are there any user-facing changes?

Images remain disabled by default. After optional configuration and normal host setup, users start the existing Server; a separate image env file or manually launched collector is unnecessary. Plugin activation and Hook trust remain host-required first-use steps.

Image receipt, interpretation, Memory extraction, and fresh-session recall are separate outcomes. Existing Memory governance is retained; no image-specific confirmation UI is introduced. The initial backend is file-backed SQLite with static PNG/JPEG. Older experimental databases require an explicit upgrade and paired DB/assets backup. Exact historical evidence must remain readable. Other storage backends, image embeddings, and joint multi-image reasoning are outside this PR.

How was this change tested?

Validation on Windows / Python 3.14, against master 2d11abd6:

  • Access-control regressions, image evidence/processing/recovery/product/Dashboard, Topic Memory processing, and API/JS/Desktop contract collection: 237 passed with two pytest workers.
  • Atomic Memory flush concurrency, scheduled Scope selection, Topic Memory recovery, Atomic Memory reconciliation, CLI setup, and image diagnostics: 156 passed in a separate serial collection.
  • pnpm --dir desktop lint, test, and build: passed, including 32 UI tests. The current Desktop OpenAPI generator ran with Java 21; generated Rust models match the base branch.
  • DSH official build: passed, with no drift in the committed bundle.
  • Python/JS API generation checks, Desktop contract generation, integration capability documentation and packaged guidance checks, root/harness lock checks, and git diff --check: passed.
  • Both lockfiles were generated with the project's uv 0.13.4 to include the inherited httpx dependency for the images extra; no additional package version upgrades.
  • All non-type pre-commit hooks and uv run --no-sync ty check --python-platform linux: passed. This is a Linux-targeted static check on Windows, not a claim that native Windows whole-tree type checking passes.

GitHub CI runs independently for the pushed head. No workflow or acceptance assertion is disabled.

Prior manual evidence includes real Codex CLI execution with controlled providers, an actual Desktop file-picker upload interpreted by MiniCPM-V 35B, and an Atomic Memory migration preserving existing Source identities and image assets. These were not rerun as part of the rebase.

Remaining acceptance and review limitations: real Atomic Memory background publication and fresh Desktop recall; paired DB/assets restore; drag/paste and remaining lifecycle variants; native macOS/Linux collector lifecycle; and complete external Open Code Review. These remain documented follow-up validation and are not claimed complete.

Details: acceptance status, implementation and validation log, desktop testing, contracts.

AI usage statement

OpenAI Codex assisted with design, implementation, rebase/conflict resolution, CI repairs, documentation, and testing. Open Code Review was used for a partial external-model review and delegate file/rule selection. The author is responsible for the implementation and will complete the documented acceptance and review before requesting merge.

@knqiufan
knqiufan force-pushed the codex/experiment-image-source branch from df4c766 to 078cf6f Compare October 13, 2026 01:23

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two reproducible collector recovery issues remain: temporary outbox capacity pressure permanently stops reading the affected turn, and pending attachments from a disabled workspace can block delivery for other enabled workspaces. Details are inline.

outbox.finish(row["id"], state="done", error="image_consent_changed")
except (ValueError, TypeError, KeyError, OSError, AttributeError, RecursionError) as error:
code = str(error) if str(error).startswith("image_") and len(str(error)) < 80 else "image_read_unavailable"
outbox.finish(row["id"], state="error", error=code)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep temporary outbox capacity pressure recoverable

When the shared 128 MiB staging budget fills, Outbox.advance() raises image_outbox_full, and this handler permanently marks the turn's reader as error. Outbox.jobs() subsequently selects only watching jobs, while reset_delivery() resets attachments but never restores the reader. Thus an offline backlog can cause all remaining images in that turn to be omitted even after uploads free space or the user explicitly retries. I reproduced this with a scaled capacity limit: the first attachment was staged and then uploaded, but the second remained unread after capacity was freed and delivery reset. Keep the saved cursor recoverable for capacity pressure so reading can resume without changing its consent boundary.

for row in outbox.pending():
try:
if not _permitted(json.loads(row["job"]), settings):
continue

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Allow enabled workspaces to progress past disabled deliveries

Outbox.pending() selects the oldest eight due attachments globally before this consent check. If workspace A accumulates eight pending attachments while offline and capture is then disabled for A, every pass selects and skips those same rows; they remain pending and due forever. Images from an enabled workspace B are consequently never selected. I reproduced this with eight staged attachments for A and one for B: after disabling only A, three delivery passes made zero uploads and B remained pending. Preserve A's consent and staged data, but ensure ineligible rows cannot monopolize the delivery page, for example by suspending them or advancing through a bounded eligible selection.

@knqiufan
knqiufan force-pushed the codex/experiment-image-source branch from 9090294 to f71748a Compare October 13, 2026 03:31
@hidb4ai

hidb4ai commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Before moving this capability toward merge, I would like us to review the design through an RFC:

  1. RFC first. This introduces image ingestion, durable assets, background vision processing, host capture, and new public operations. Please prepare an RFC and have the design reviewed before proceeding with implementation approval. It should describe the user workflow, architecture boundaries, API and storage changes, and acceptance criteria.

  2. Evaluate extending the existing Artifact framework. Could the user-facing image capability be implemented by extending existing Artifact families and their evidence/provenance flows? Please explain which responsibilities fit the existing Source/Artifact contracts and which require a separate image service. We should reuse existing versioning, authorization, retrieval, and evidence inspection where their contracts fit.

  3. Be conservative about new database tables. The project already has many tables, and adding seven more deserves careful justification. Please evaluate reuse of existing storage and consolidation of the proposed tables first. For each table that remains necessary, explain its distinct responsibility and why existing structures cannot meet the requirement. Reuse should preserve idempotency, recovery, historical evidence, and concurrency guarantees; it should not simply mix unrelated business state into an existing table.

The feature has a useful direction, but I would prefer to settle these architectural choices in the RFC before merging it.

knqiufan commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator Author

Thank you for the architectural feedback. I agree that this capability should go through RFC review first. I’ve opened RFC #1972, linked to #1932, for design review.

It compares the Source/Artifact boundaries, evaluates existing binary storage and shared processing contracts, and gives a table-by-table analysis with a proposed three-table design. The proposal keeps idempotency, recovery, historical evidence and concurrency guarantees explicit.

I’ll keep #1956 as a draft prototype and wait for the RFC to be reviewed and accepted before proceeding with further implementation changes. After that, I’ll align the implementation with the agreed design and split it into focused PRs where appropriate.

@hidb4ai

hidb4ai commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

please update & resolve conflicts

@hidb4ai hidb4ai removed the 1.4.0 v1.4.0 release tracking label Oct 16, 2026
@AlexStocks AlexStocks added the 1.4.1 v1.4.1 release tracking label Oct 17, 2026
@knqiufan
knqiufan force-pushed the codex/experiment-image-source branch 2 times, most recently from eca666c to 072bcfc Compare October 21, 2026 01:49
@knqiufan
knqiufan marked this pull request as ready for review October 21, 2026 01:49
@knqiufan
knqiufan force-pushed the codex/experiment-image-source branch from 072bcfc to 3fde6d5 Compare October 21, 2026 15:00
ExperimentalImageTaskPage,
ExperimentalListImageTasksRequest,
)
from powercontext.paths import powercontext_data_dir

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] powercontext[builtin] 安装后无法导入内置运行时

builtin/runtime/application.py:151 在模块层导入 powercontext.builtin.images.service,该文件第 64 行导入 powercontext.paths,而 src/powercontext/paths.py 在模块层执行 from platformdirs import user_data_path。pyproject.toml 的 builtin extra 没有声明 platformdirs(只有 server 与 cli 声明),images extra 又只组合 powercontext[builtin] + pillow。

后果:pip install "powercontext[builtin]" 之后任何 import powercontext.builtin.runtime 都抛 ModuleNotFoundError: No module named 'platformdirs',powercontext[images] 同样不可用。

证据:

  • CI 同签名失败:isolated-installations (3.11, builtin) job 119297345299 与 isolated-installations (3.14, builtin) job 119297345844,栈为 ci_role_smoke.py:31 → builtin/runtime/__init__.py → builtin/runtime/application.py:151 → builtin/images/service.py:64 → paths.py:22 → ModuleNotFoundError: platformdirs。
  • 同一 Main 工作流在 base 2d11abd6 上 success(run 39602208746),即该 job 在 base 是绿的。
  • 静态核对模块级导入闭包:base 2d11abd6 的 powercontext.builtin.runtime 闭包不含 powercontext.paths,本 head 含(唯一新增路径就是 builtin/images/service.py)。所以缺失依赖是本 PR 引入,不是既有形态。

修复方向(二选一):

  1. 在 builtin extra 增加 platformdirs>=4,<5(与 server/cli 一致);
  2. 或者让 images 不依赖 paths:把数据目录通过 ImageConfig/BuiltinConfig 注入,或在 service.py 内延迟导入。

无论选哪条,请保留 isolated-installations (x, builtin) 的覆盖——它正是这次唯一抓到该问题的门禁;tests (x, unit) 与 quality 都因为工作区已装 server 依赖而看不到它。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

1.4.1 v1.4.1 release tracking

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants