[Bug] Content between embedder limit and chat_window_max_tokens is silently stored without a vector | 内容落在嵌入上限与切分阈值之间时静默丢失向量 #2461

Activity

1. Memtensor-AI

   added

   ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务Dispatched to AI coding agent | 已派发给 AI 编码任务

   area:coreMOS 编排层 / 框架底座 / 跨模块问题MOS 编排层 / 框架底座 / 跨模块问题

   status:in-progressSomeone or AI is working on it | 人工或 AI 正在处理Someone or AI is working on it | 人工或 AI 正在处理

   types:bugSomething isn't working | 功能异常Something isn't working | 功能异常

   on Oct 8, 2026
2. Memtensor-AI

   assigned

   syzsunshine219

   and

   WeiminLee

   on Oct 8, 2026
3. Memtensor-AI

   added

   ai:codingAI agent is coding | AI 正在编码AI agent is coding | AI 正在编码

   on Oct 8, 2026
4. Memtensor-AI

   self-assigned this

   on Oct 8, 2026
5. Memtensor-AI commented on Oct 8, 2026

   

   Memtensor-AI

   on Oct 8, 2026

   Collaborator

   More actions

   🤖 AutoDev has picked up this issue and started working on it.

   Task ID: 00c0f8f3534623df
   Working branch: bugfix/autodev-2461-20261008063231554
   Target branch: latest dev* branch
   Workflow: opsp (analysis → coding → testing → PR)

   I will post the PR link here once done. If I need more information, I will ask in the comments.
6. KongFUN2018 commented on Oct 8, 2026

   

   KongFUN2018

   on Oct 8, 2026

   Author

   More actions

   补充：我们已在本地 v2.0.33 上实现了这个修复，并做了判别性验证，实现细节供参考（如需完整 patch 可提供）。

   修复方案（两层，缺一不可）

   第 1 层——装配时把分段阈值压到嵌入上限之下（src/memos/api/config.py，两处 mem_reader 装配的 embedder 之后）：

   "chat_window_max_tokens": int(
       os.getenv("MEM_READER_CHAT_WINDOW_MAX_TOKENS", "480")
   ),

   默认 480：本地 embedder 单条上限 512，留约 6% 余量；env 可调。

   第 2 层——chunker 输出后加硬切兜底（multi_modal_struct.py 的 _split_large_memory_item，chunks = self.chunker.chunk(item_text) 之后、chunk_total = len(chunks) 之前）：

   budget = max(128, int(max_tokens * 0.85))
   normalized = []
   for chunk in chunks:
       chunk_text = chunk.text if hasattr(chunk, "text") else chunk
       if chunk_text and self._count_tokens(chunk_text) > budget:
           normalized.extend(self._hard_split_text(chunk_text, budget))
       else:
           normalized.append(chunk)
   chunks = normalized

   _hard_split_text 实现（二分字符窗 + 标点优先硬切）

   def _hard_split_text(self, text: str, budget_tokens: int) -> list[str]:
       """Split text by binary-searched character windows so every piece counts
       <= budget_tokens under self._count_tokens (cut preferring punctuation)."""
       pieces = []
       start, n = 0, len(text)
       while start < n:
           lo, hi, best = start + 1, n, start + 1
           while lo <= hi:
               mid = (lo + hi) // 2
               if self._count_tokens(text[start:mid]) <= budget_tokens:
                   best = mid
                   lo = mid + 1
               else:
                   hi = mid - 1
           end = best
           window_start = end - max(1, (end - start) // 5)
           for i in range(end - 1, window_start - 1, -1):
               if text[i] in "。！？；!?;\n":
                   end = i + 1
                   break
           piece = text[start:end].strip()
           if piece:
               pieces.append(piece)
           start = end
       return pieces

   为什么两层都要：只调阈值——分句器失明时（不识别中文标点、整段穿过）仍可能产出超限单块；只做硬切——(嵌入上限, chat_window_max_tokens] 区间根本不触发 _split_large_memory_item，硬切没有机会执行。

   判别性验证（v2.0.33 实测）

   注意：直接写长文本测试不能判别修复是否生效——LLM 抽取会把长内容压缩成短记忆，走不到切块路径（修复前后都会"通过"）。需要 stub 掉 chunker 构造"分句器整段穿过"的场景：

   版本	1207 字（1035 token）输入结果
   上游 a7367d0	1 块 / 1035 token → 超嵌入上限，静默无向量
   修复后	3 块 / 最大 401 token → 全部 vector_sync=success、可搜索

   另：本机曾因此累计 22 条记忆无向量（事后对照元数据逐条重灌恢复），问题真实且全程无任何错误上报。
7. Memtensor-AI

   added a commit that references this issue on Oct 8, 2026

   fix: split memory items that exceed the embedder per-item limit

   ...

   c3572b0
8. Memtensor-AI commented on Oct 8, 2026

   

   Memtensor-AI

   on Oct 8, 2026

   Collaborator

   More actions

   ✅ AutoDev task 00c0f8f3534623df completed.

   Summary: Fixed the silent no-vector write for memory items whose token count falls in (embedder per-item limit, chat_window_max_tokens] (issue #2461). Such an item was previously neither split (the splitter only fired above chat_window_max_tokens, hard-coded to 1024 and never passed by either api/config.py mem_reader assembly site) nor embeddable (it exceeded the embedder's limit). /product/add still returned 200, but the item was persisted with metadata.embedding=None and vector_sync != "success", and search_by_embedding filters on vector_sync == "success" — so the memory was permanently invisible to semantic search. A secondary cause: _split_large_memory_item trusted chunker output and never re-checked the token budget per chunk, and the sentence chunker (chonkie) does not split CJK punctuation (it can also raise on an incompatible installed version), in which case the except branch returned the over-budget item unchanged.

   The fix adds an explicit, env-configurable chat_window_max_tokens field on BaseMemReaderConfig (MEM_READER_CHAT_WINDOW_MAX_TOKENS, default 1024 so existing deployments are unchanged), wired into all three mem_reader assembly sites via APIConfig.get_chat_window_max_tokens() with fault-tolerant parsing. The effective split budget is now min(chat_window_max_tokens, embedder per-item limit). A new _chunk_within_budget helper re-checks every chunker-produced chunk and hard-splits any that still exceeds the budget — preferring a punctuation boundary (CJK and Latin terminators) and falling back to a binary-search character window — and a chunker that raises or yields nothing now hard-splits instead of passing the over-budget item through. _concat_multi_modal_memories uses the effective budget for both the split trigger and the window accumulator, plus a guard that emits over-budget items directly rather than aggregating them into a larger window. Finally, the per-item embedding retry (and the base reader's _make_memory_item path) truncates to budget before calling the embedder, so one over-limit item can no longer leave a whole batch without vectors.

   Verification: a new 18-case regression suite (tests/mem_reader/test_embed_budget_split.py) passes, covering budget derivation, punctuation-free CJK splitting within budget, boundary preference, empty/failed chunker fallback, contiguous re-indexing, the 700-token in-window case, and the mixed-batch embedding fallback. Affected suites (mem_reader, configs, api, memories, mem_cube) report 341 passed with 4 pre-existing failures confirmed on the untouched baseline (markitdown extra not installed; unrelated kv-cache API). ruff check and ruff format both clean. An end-to-end reproduction using the real SentenceChunker and tiktoken counter with a 512-token embedder limit turns the issue's 780-token Chinese paragraph into chunks of 512 + 268 tokens, all embedded, while the same script's pre-fix branch reproduces the reported embedded=[False].

   Out of scope: backfilling vectors for memories already stored without one.

   Base branch: main
   Branch: bugfix/autodev-2461-20261008063231554
   Commit: c3572b0
   PR: #2462
   Assigned to: @syzsunshine219
   Reviewers: @WeiminLee
9. Memtensor-AI

   added

   ai:testingAI agent is running tests | AI 正在运行测试AI agent is running tests | AI 正在运行测试

   and removed

   ai:codingAI agent is coding | AI 正在编码AI agent is coding | AI 正在编码

   on Oct 8, 2026
10. Memtensor-AI

    added a commit that references this issue on Oct 8, 2026

    fix: keep split budget decisions on the safe token counter

    ...

    243ef50