Skip to content

fix(manual-docs): advance audit cursor on uncertain results - #33

Merged
Bosheng0422 merged 8 commits into
TencentCloud:developfrom
TrueFurina:fix/manual-docs-uncertain-cursor-advance
Nov 4, 2026
Merged

Bosheng0422 merged 8 commits into
TencentCloud:developfrom
TrueFurina:fix/manual-docs-uncertain-cursor-advance

Conversation

@TrueFurina

@TrueFurina TrueFurina commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Rebased onto develop and narrowed to the delta that develop does not already have.

develop already refuses to report complete on a pass that performs an uncertain comparison. What it still loses is the state across passes: the cursor advanced, the next pass has nothing left to compare, matches is empty, and the audit reports complete for an atom nobody ever verified.

Change

  • Keep unresolved atoms recorded against their assertion hash, and carry that record over while the atom is unchanged.
  • Report uncertain: <count> in the audit result; a pass with nothing left to compare reports uncertain, not complete.
  • Apply the same rule in manual_document_review_status, so a restart cannot resurrect complete.
  • A new document revision re-compares the atoms and can clear the state back to complete.
  • No document-revision short-circuit (fix(manual-docs): 文档审计不再原地打转(uncertain 推进游标 + 同 revision 只审一次) #28's other half) is taken: an unchanged document with a new contradictory atom is still audited.

History
The branch was based on main; develop has since rewritten this function (defer/retry of failed batches, max_atoms). This commit merges develop in and takes develop's implementation, so the diff is only the delta above. No force-push.

Verification (this machine, this branch)

  • pytest tests/test_manual_documents.py -q → 31 passed (2 new tests).
  • Full suite: 27 failed / 1343 passed / 161 skipped. The same 27 tests fail identically on unpatched develop@8d0c906 (27 failed / 1341 passed), so the failures are pre-existing in this Windows environment and not caused by the patch.
  • ruff check src tests clean; ruff format --check src tests clean.
  • Repro (single atom, model answers confidence="uncertain"): unpatched → first pass uncertain / checked=1 / remaining=0, second pass complete; patched → both passes uncertain, model not called again. The two new tests fail on unpatched develop.

Not verified

  • PostgreSQL 16 parity: TEST_POSTGRES_DSN is unset here, so the whole PostgreSQL half of test_backend_parity.py is skipped (161 skips). The assertions are backend-agnostic (Memory API + meta JSON), but I have not run them against PostgreSQL.
  • make all type check.

…ared

The previous version recorded the hash before the match check, so three
different cases all counted as audited: (1) uncertain — intended, (2) no
match at all because the LLM failed or the batch broke — this would mark
facts as audited after a single transient failure and persist it in
`checked`, (3) stale/deprecated atoms that still need review. Split them:
advance only when a comparison actually happened.
@Bosheng0422

Copy link
Copy Markdown
Collaborator

本轮统一评审:手动文档审计的 uncertain 游标推进以本 PR 为主推进,#28 的有效状态保护和测试应向这里收敛。

选择此方案是因为范围较小:只推进已经比较、事实未变化的 atom,不引入按文档 revision 跳过后续新增事实审计的短路。不过当前 head 543bf4f 仍需修改,暂不进入合并。

请完成以下验收:

  1. 将目标分支从 main 调整为 develop。
  2. 保留 uncertain 的真实状态。本轮 SQLite / PostgreSQL 16 均复现:仅 1 个 atom、模型返回 confidence="uncertain" 时,checked=1、remaining=0,但 status 被报告为 complete。游标推进不等于事实已验证。
  3. 吸收 fix(manual-docs): 文档审计不再原地打转(uncertain 推进游标 + 同 revision 只审一次) #28 中 uncertain 状态保护及其测试,注明来源;不要带入仅根据文档 revision 缓存整次审计的逻辑。
  4. 补充行为回归:uncertain 推进游标但状态仍 uncertain;未比较/模型失败不推进;文档不变但新增矛盾 atom 时仍会审计;SQLite 和 PostgreSQL 同组断言均需通过。

本轮新增事实审计对照在当前 develop 上 2 passed,在 #28 上两后端均失败;本 PR 的 uncertain 状态断言也在两后端失败。完成修订后再跑完整门禁。

@TrueFurina
TrueFurina changed the base branch from main to develop November 2, 2026 08:33
@TrueFurina

TrueFurina commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor Author

本轮只完成验收第 1 项,其余未做,先如实同步状态,避免误以为已在推进。

已改:目标分支 main → develop(PR 元数据改动,非代码提交;head 仍 543bf4f)。
改完立即复核:mergeable=false / mergeable_state=dirty,diff 范围不变(1 文件 src/octop_memory/operations/retention.py,+10/−7)。
也就是说切到 develop 后与目标分支产生冲突,需要先 rebase 到当前 develop 才能谈后续。此前 base=main 时的 mergeable=true 是相对 main 算的,对 develop 没有意义。

没做:

未验证:本轮零测试执行,SQLite 与 PostgreSQL 16 都没跑;没有 rebase,也没有 push 任何 commit。
下一步顺序是 rebase → 补第 2/3 项代码 → 写第 4 项双后端断言,再跑完整门禁。

@Bosheng0422 Bosheng0422 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复核结论:uncertain 已比较项推进游标的方向成立,继续保留本 PR;已看到目标分支改为 develop,当前代码仍需修改。

[P2] checked 的推进不能代表结论已确认。仅一个 Atom、模型 confidence="uncertain" 时,两后端均返回 checked=1、remaining=0、status="complete"。原因是把 uncertain 写入 checked 后,末尾仅在 remaining 非零时设置 uncertain 状态,导致“审核完成”假绿。

锁定依赖下原有测试结果 230 passed / 1 failed,失败就是 tests/test_manual_documents.py::test_unverified_classification_never_quarantines[uncertain];新增 SQLite / PostgreSQL 同组状态回归也均失败。

请分别保留比较进度与 uncertain 真实状态,后续空批次/维护 tick 也不能把未决项改报 complete;模型失败/未比较的项不能推进。吸收已关闭 #28 中有价值的状态保护及测试并注明来源,不带入仅按文档 revision 跳过新增 Atom 审核的短路。补文档不变但新增矛盾 Atom 仍受审计的回归,解决 develop 冲突,再跑双后端及完整门禁。

A comparison that returns confidence="uncertain" still advances the audit
cursor -- that part is right, otherwise the same atoms are re-sent to the
model on every pass. What was missing is that the state itself has to
survive: the next pass has nothing left to compare, so it reported
"complete" for atoms that were never verified.

Record unresolved atoms against their assertion hash, report them as
`uncertain` in the audit result, and keep the persisted status from
flipping back to "complete" while any of them is still current. A new
document revision re-compares the atoms and can clear the state.

Verified on SQLite (tests/test_manual_documents.py, 27 passed); the
PostgreSQL half of the parity suite is skipped locally (no dedicated
server configured).
@TrueFurina

Copy link
Copy Markdown
Contributor Author

按上一轮的验收项做完了第 1–4 项,代码已落到分支:7d9bfc0aa19c(4 commits,3 文件,+94/−2),base 已是 develop,当前 mergeable=true。

改了什么

  1. 目标分支 main → develop(上一轮已完成),本轮把 develop 合并进分支并采用 develop 的实现解决冲突 —— 没有 rebase、没有 force-push。原因是 develop 已经把这段函数重写过(失败批次 defer/重试、max_atoms),单纯 rebase 会丢掉这些改动。
  2. 保留 uncertain 的真实状态:develop 只在“本轮确实做了 uncertain 比对”时改判 uncertain,下一个 pass 没有待比对项、matches 为空,于是又回到 complete。现在把未验证的 atom 连同断言哈希一起记入 uncertain 字段,无待比对项的 pass 仍报 uncertain,manual_document_review_status 也按同一规则读,重启不会复活 complete。
  3. 吸收了 fix(manual-docs): 文档审计不再原地打转(uncertain 推进游标 + 同 revision 只审一次) #28 中 uncertain 状态保护对应的测试意图(单 atom + 全 uncertain 的用例),并在 CHANGELOG 注明;没有搬 fix(manual-docs): 文档审计不再原地打转(uncertain 推进游标 + 同 revision 只审一次) #28 的“同 revision 跳过审计”短路——文档不变但新增矛盾 atom 仍会重审,这一点保留 develop 现有行为。
  4. 新增 2 个回归断言:uncertain 推进游标但状态仍 uncertain(含持久化读回);新文档 revision 重新比对后可恢复 complete。

验证了什么(本机,本分支)

  • pytest tests/test_manual_documents.py -q → 31 passed。
  • 全量:27 failed / 1343 passed / 161 skipped;未打补丁的 develop@8d0c906 是 27 failed / 1341 passed,失败集合逐条相同,即这 27 个是这台 Windows 上的既有失败,不是本次引入。
  • ruff check 与 ruff format --check 均 clean。
  • 复现(1 atom、模型返回 uncertain):打补丁前第二个 pass 报 complete,打补丁后两次都是 uncertain 且模型没有被再次调用;两个新测试在未打补丁的 develop 上失败(KeyError/断言),不是空转。

没做 / 没验证

  • PostgreSQL 16:本机没有 TEST_POSTGRES_DSN,test_backend_parity.py 的 PG 部分整体 skip(161 项),所以“双后端同组断言”目前只有 SQLite 这一半有实测。
  • make all 的 typecheck 未跑。

如果验收方向是“uncertain 只影响本轮报告、不跨轮持久化”,那第 2 项的做法需要改成另一种口径,请指出来,我按那个口径调整。

@Bosheng0422 Bosheng0422 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

新版 7d9bfc0 已复核:跨轮持久化 uncertain 的方向符合前次验收,原先“下一轮误报 complete”的用例在 SQLite / PostgreSQL 16 均已通过;这个进展确认有效,不再按旧提交的问题评价新版。

仍有一个 [P2] 需要修复:已停用的 Atom 会永久留下 uncertain 状态。

两后端相同复现:

  1. 只存一个 Atom,模型返回 uncertain,首次审核得到 checked=1 / remaining=0 / uncertain=1。
  2. 用户停用该 Atom,确认 memory.list_atoms() 已为空,文档内容保持不变。
  3. 再次 reconcile,并读取 manual_document_review_status。

实际两处仍返回 status="uncertain"、uncertain=1、remaining=0、scanned=0;活跃事实已经为零,却必须改文档 revision 才能清掉这个状态。新增两后端回归均失败;原跨轮 uncertain 两项通过(2 passed / 2 failed)。

原因:unresolved 的保留仅对照历史 checked[atom_id],没有对照当前活跃 atoms / 当前 assertion hash;状态读取函数也只比较两个历史字典。请让两条路径都过滤已停用、已删除或断言已变的记录,并按当前有效未决集合计算状态,不能让历史条目永久锁住审核。补对应双后端回归,保留原先模型失败/未比较不推进、文档不变但新增事实仍审计的保护;再补 strict mypy 与完整门禁。继续保留本 PR,不迁回 #28 的 revision 短路。

A comparison that returns confidence="uncertain" is recorded per atom id so that
a later pass -- which has nothing left to compare -- cannot report "complete" for
a fact nobody verified. The stored ids were only checked against the historical
`checked` map, so an atom the user disabled (or deleted, or reworded) stayed in
the set forever: every later pass reported status="uncertain" with zero live
facts pending, and only editing the documents could clear it.

Both paths now intersect the stored set with the current live atoms and their
assertion hashes -- the reconcile write path and manual_document_review_status.
`remaining` is computed over live atoms as well, and a persisted "uncertain"
downgrades to partial/complete once nothing unresolved is left. Model failures
and unverified comparisons still hold the audit, and a document edit that leaves
the facts untouched still leaves them audited.

Verified locally: tests/test_manual_documents.py 32 passed / 1 skipped (SQLite;
the PostgreSQL half skips without TEST_POSTGRES_DSN, so it is unverified here).
Full suite and strict mypy keep the same failure sets as the unpatched tree.
@TrueFurina

Copy link
Copy Markdown
Contributor Author

已按上一条评审的 [P2] 改好并推到分支:58f6bb4273ad(1 commit,3 文件 +166/−6,head 已换;当前 mergeable=true、mergeable_state=blocked,blocked 来自上一条 CHANGES_REQUESTED)。

改了什么

  1. 未决集合改按当前活跃 atom 计算。写路径 reconcile_manual_documents 里,unresolved 现在要求同时满足:本轮没拿到结论(atom_id not in matches)、checked 里的哈希与该条记录一致、且该 atom 仍在当前 hashes 中、断言未变。已停用/已删除的 atom 不进 hashes,断言已改的 digest 不同,两类都会被剔除。
  2. 读路径 manual_document_review_status 同样改为先用活跃 atom 建 live 映射:remaining 只统计活跃 atom;unverified 要求 live[atom_id] == digest 且历史上确实比较过。未决集合清空而持久化状态还是 uncertain 时,按「是否仍有未审项」降为 complete / partial —— 不再需要改文档 revision 才能清状态。
  3. 新增双后端回归 test_a_disabled_atom_cannot_pin_the_uncertain_state(both_backends 夹具 sqlite + postgres,无 TEST_POSTGRES_DSN 时跳过):单 atom → 首次对账 uncertain=1 → 停用该 atom、list_atoms() 为空 → 再次对账与读状态都必须 uncertain=0 且 complete,文档 revision 全程不变。
  4. 保留原有保护不动:模型失败/未比较不推进结论、无待比对项时不回到 complete、文档不变但新增事实仍受审计(原有 31 个用例未改动,全部通过)。CHANGELOG 补一条对应说明。

没做什么

  • 没有迁回 #28 的「按文档 revision 跳过新增 Atom 审核」短路,也没有复活其中的状态保护。
  • 没有改 schema、没有新增依赖、没有改默认 max_atoms / 批大小。
  • 没有动 PR base(仍 develop),没有 rebase 之外的合并动作:当前分支相对 develop 仍 mergeable=true,无需再解冲突。

没验证什么

  • PostgreSQL 一半未验证:本机没有可用 PG 实例,both_backends 的 postgres 参数与仓库其余 PG 用例一样被 skip(本机 tests/test_manual_documents.py 32 passed / 1 skipped)。我没有在真实 PG 16 上跑这个新用例,因此不能声称双后端通过。
  • 完整 make all 未跑:本机缺可选依赖(fastapi / click / psycopg),全量 pytest 有 63 项失败或收集错误,打补丁前后的失败集合逐条相同(diff 为空),只能说明未新增回归,不等于门禁通过。
  • strict mypy 在本机跑出 355 条错误,与未打补丁的树逐条相同、retention.py 0 条;同样受缺可选依赖影响,不能等同于仓库 CI 的干净结果。
  • ruff format --check 与 ruff check 对两个改动文件通过。

@Bosheng0422
Bosheng0422 self-requested a review November 3, 2026 03:38
@Bosheng0422 Bosheng0422 self-assigned this Nov 3, 2026
@Bosheng0422
Bosheng0422 merged commit 4398189 into TencentCloud:develop Nov 4, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants