Repository navigation
fix(manual-docs): advance audit cursor on uncertain results - #33
Bosheng0422 merged 8 commits into
Conversation
…ared The previous version recorded the hash before the match check, so three different cases all counted as audited: (1) uncertain — intended, (2) no match at all because the LLM failed or the batch broke — this would mark facts as audited after a single transient failure and persist it in `checked`, (3) stale/deprecated atoms that still need review. Split them: advance only when a comparison actually happened.
|
本轮统一评审:手动文档审计的 uncertain 游标推进以本 PR 为主推进,#28 的有效状态保护和测试应向这里收敛。 选择此方案是因为范围较小:只推进已经比较、事实未变化的 atom,不引入按文档 revision 跳过后续新增事实审计的短路。不过当前 head 543bf4f 仍需修改,暂不进入合并。 请完成以下验收:
本轮新增事实审计对照在当前 develop 上 2 passed,在 #28 上两后端均失败;本 PR 的 uncertain 状态断言也在两后端失败。完成修订后再跑完整门禁。 |
|
本轮只完成验收第 1 项,其余未做,先如实同步状态,避免误以为已在推进。 已改:目标分支 main → develop(PR 元数据改动,非代码提交;head 仍 543bf4f)。 没做:
未验证:本轮零测试执行,SQLite 与 PostgreSQL 16 都没跑;没有 rebase,也没有 push 任何 commit。 |
Bosheng0422
left a comment
There was a problem hiding this comment.
复核结论:uncertain 已比较项推进游标的方向成立,继续保留本 PR;已看到目标分支改为 develop,当前代码仍需修改。
[P2] checked 的推进不能代表结论已确认。仅一个 Atom、模型 confidence="uncertain" 时,两后端均返回 checked=1、remaining=0、status="complete"。原因是把 uncertain 写入 checked 后,末尾仅在 remaining 非零时设置 uncertain 状态,导致“审核完成”假绿。
锁定依赖下原有测试结果 230 passed / 1 failed,失败就是 tests/test_manual_documents.py::test_unverified_classification_never_quarantines[uncertain];新增 SQLite / PostgreSQL 同组状态回归也均失败。
请分别保留比较进度与 uncertain 真实状态,后续空批次/维护 tick 也不能把未决项改报 complete;模型失败/未比较的项不能推进。吸收已关闭 #28 中有价值的状态保护及测试并注明来源,不带入仅按文档 revision 跳过新增 Atom 审核的短路。补文档不变但新增矛盾 Atom 仍受审计的回归,解决 develop 冲突,再跑双后端及完整门禁。
A comparison that returns confidence="uncertain" still advances the audit cursor -- that part is right, otherwise the same atoms are re-sent to the model on every pass. What was missing is that the state itself has to survive: the next pass has nothing left to compare, so it reported "complete" for atoms that were never verified. Record unresolved atoms against their assertion hash, report them as `uncertain` in the audit result, and keep the persisted status from flipping back to "complete" while any of them is still current. A new document revision re-compares the atoms and can clear the state. Verified on SQLite (tests/test_manual_documents.py, 27 passed); the PostgreSQL half of the parity suite is skipped locally (no dedicated server configured).
|
按上一轮的验收项做完了第 1–4 项,代码已落到分支: 改了什么
验证了什么(本机,本分支)
没做 / 没验证
如果验收方向是“uncertain 只影响本轮报告、不跨轮持久化”,那第 2 项的做法需要改成另一种口径,请指出来,我按那个口径调整。 |
Bosheng0422
left a comment
There was a problem hiding this comment.
新版 7d9bfc0 已复核:跨轮持久化 uncertain 的方向符合前次验收,原先“下一轮误报 complete”的用例在 SQLite / PostgreSQL 16 均已通过;这个进展确认有效,不再按旧提交的问题评价新版。
仍有一个 [P2] 需要修复:已停用的 Atom 会永久留下 uncertain 状态。
两后端相同复现:
- 只存一个 Atom,模型返回 uncertain,首次审核得到 checked=1 / remaining=0 / uncertain=1。
- 用户停用该 Atom,确认 memory.list_atoms() 已为空,文档内容保持不变。
- 再次 reconcile,并读取 manual_document_review_status。
实际两处仍返回 status="uncertain"、uncertain=1、remaining=0、scanned=0;活跃事实已经为零,却必须改文档 revision 才能清掉这个状态。新增两后端回归均失败;原跨轮 uncertain 两项通过(2 passed / 2 failed)。
原因:unresolved 的保留仅对照历史 checked[atom_id],没有对照当前活跃 atoms / 当前 assertion hash;状态读取函数也只比较两个历史字典。请让两条路径都过滤已停用、已删除或断言已变的记录,并按当前有效未决集合计算状态,不能让历史条目永久锁住审核。补对应双后端回归,保留原先模型失败/未比较不推进、文档不变但新增事实仍审计的保护;再补 strict mypy 与完整门禁。继续保留本 PR,不迁回 #28 的 revision 短路。
A comparison that returns confidence="uncertain" is recorded per atom id so that a later pass -- which has nothing left to compare -- cannot report "complete" for a fact nobody verified. The stored ids were only checked against the historical `checked` map, so an atom the user disabled (or deleted, or reworded) stayed in the set forever: every later pass reported status="uncertain" with zero live facts pending, and only editing the documents could clear it. Both paths now intersect the stored set with the current live atoms and their assertion hashes -- the reconcile write path and manual_document_review_status. `remaining` is computed over live atoms as well, and a persisted "uncertain" downgrades to partial/complete once nothing unresolved is left. Model failures and unverified comparisons still hold the audit, and a document edit that leaves the facts untouched still leaves them audited. Verified locally: tests/test_manual_documents.py 32 passed / 1 skipped (SQLite; the PostgreSQL half skips without TEST_POSTGRES_DSN, so it is unverified here). Full suite and strict mypy keep the same failure sets as the unpatched tree.
|
已按上一条评审的 [P2] 改好并推到分支: 改了什么
没做什么
没验证什么
|
Rebased onto
developand narrowed to the delta thatdevelopdoes not already have.developalready refuses to reportcompleteon a pass that performs an uncertain comparison. What it still loses is the state across passes: the cursor advanced, the next pass has nothing left to compare,matchesis empty, and the audit reportscompletefor an atom nobody ever verified.Change
uncertain: <count>in the audit result; a pass with nothing left to compare reportsuncertain, notcomplete.manual_document_review_status, so a restart cannot resurrectcomplete.complete.History
The branch was based on
main;develophas since rewritten this function (defer/retry of failed batches,max_atoms). This commit mergesdevelopin and takesdevelop's implementation, so the diff is only the delta above. No force-push.Verification (this machine, this branch)
pytest tests/test_manual_documents.py -q→ 31 passed (2 new tests).develop@8d0c906(27 failed / 1341 passed), so the failures are pre-existing in this Windows environment and not caused by the patch.ruff check src testsclean;ruff format --check src testsclean.confidence="uncertain"): unpatched → first passuncertain / checked=1 / remaining=0, second passcomplete; patched → both passesuncertain, model not called again. The two new tests fail on unpatcheddevelop.Not verified
TEST_POSTGRES_DSNis unset here, so the whole PostgreSQL half oftest_backend_parity.pyis skipped (161 skips). The assertions are backend-agnostic (MemoryAPI + meta JSON), but I have not run them against PostgreSQL.make alltype check.