Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 65 additions & 5 deletions apps/memos-local-plugin/core/llm/prompts/l2-induction.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,22 @@ import type { PromptDef } from "./index.js";
* Boundary contract (see `docs/GRANULARITY-AND-MEMORY-LAYERS.md` §6):
* an L2 policy is **procedural** ("how to do it") — it MUST contain an
* action template. Anything declarative ("the environment looks like X")
* belongs to the L3 world model, not here. The system prompt explicitly
* rejects environment-fact drift to keep the two layers semantically
* orthogonal. Bumping the version to v2 captures that change.
* belongs to the L3 world model, not here.
*
* v3 (issue #2318): adds a second boundary against **conversational acts**
* (asking, confirming, notifying, reporting status) leaking into the
* `action` field. Such policies passed schema validation but crystallised
* into dead skills — always retrieved, never callable. The revised prompt
* (a) enumerates the dialogue verbs to reject and (b) tells the model to
* abstain when the cluster's only shared behaviour is dialogue.
*
* v2 history: added the L3 world-model drift guard.
*/
export const L2_INDUCTION_PROMPT: PromptDef = {
id: "l2.induction",
version: 2,
version: 3,
description:
"Distill an L2 policy (procedural sub-task strategy) from a cluster of similar L1 traces, with explicit boundaries against L3 world-model drift.",
"Distill an L2 policy (procedural sub-task strategy) from a cluster of similar L1 traces, with explicit boundaries against L3 world-model drift and against conversational-act 'actions'.",
system: `You induce reusable **procedural policies** from agent experience.

A policy is a "how-to": "when you see condition X in the agent's state,
Expand Down Expand Up @@ -83,6 +90,59 @@ libs by default":
"Alpine container images ship only the pure-Python tier of the
Python dependency stack."

──────────────────── Boundaries — conversational acts are NOT policies ────────────────────

A **conversational act** is an I/O behaviour aimed at the user (or
another agent) — not a procedure applied to the environment. When a
trace cluster's dominant shared behaviour is dialogue, do NOT wrap that
dialogue as an ACTION. Such policies pass schema validation but produce
"dead skills" — the crystalliser promotes them, retrieval surfaces
them, and no agent can meaningfully invoke them because there is
nothing to invoke.

Dialogue verbs / patterns to REJECT as ACTION templates (non-exhaustive):
- **asking the user** to clarify, choose, or provide input
("ask the user to confirm the skill name before viewing it")
- **requesting confirmation** before proceeding
("confirm with the user before deleting the file")
- **notifying / informing** the user about state
("notify the user that the build finished")
- **reporting status** back to the user
("report the current pipeline status to the user")
- **explaining / clarifying** to the user
("explain to the user why the request failed")

Contrast:

Wrong (dialogue as action — dead skill):
trigger: "user hasn't specified which skill they want to view"
action: "ask the user to confirm the skill name before viewing it"

Right (dialogue folded into trigger; real procedure as action):
trigger: "user's view-skill request omits an unambiguous skill id
AND at least two skills fuzzy-match the phrase they used"
action: "1. look up all skills whose name/tags fuzzy-match the
user's phrase; 2. if exactly one hits, resolve to that
skill's id; 3. if multiple hit, present the top 3 with
ids and let the user pick by id in the next turn"

The distinction: the "wrong" version's action IS the dialogue; the
"right" version's action is a resolution procedure the agent can carry
out, and asking the user is at most a fallback branch, not the main
step. If your only shared behaviour across traces is "the agent talks
to the user", you have NO policy to induce — see the abstain rule
below.

Abstain when the cluster IS dialogue-only: if every trace in the input
reduces to the same dialogue act with no follow-up procedure the agent
executes on the environment, **do not emit a policy**. Instead return:

{ "abstain": true, "reason": "cluster's shared behaviour is a
conversational act, not a reusable procedure — no policy to induce" }

An abstained induction is a successful outcome, not a failure — it
prevents dead skills from entering the pool.

──────────────────── Output ─────────────────────

Return JSON:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ function buildFullChainLlm(): LlmClient {
}),

// L2 induction — distills a policy from ≥2 similar traces.
"l2.l2.induction.v2": (input: unknown) => {
"l2.l2.induction.v3": (input: unknown) => {
const text = lastUserMessage(input);
const isPython = /python|pip|\.py\b/i.test(text);
return {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
* - `capture.alpha.reflection.score.v1`— α scoring
* - `capture.summarize` — trace-level summaries
* - `reward.reward.r_human.v3` — R_human axis scoring
* - `l2.l2.induction.v2` — L2 policy induction
* - `l2.l2.induction.v3` — L2 policy induction
* - `l3.abstraction.v2` — L3 world-model abstraction
* - `skill.crystallize` — skill draft
*
Expand Down Expand Up @@ -234,7 +234,7 @@ function buildLlm(): LlmClient {
reason: "concrete root-cause reflection",
}),

"l2.l2.induction.v2": (input: unknown) => {
"l2.l2.induction.v3": (input: unknown) => {
const evidence = (input as { evidenceTraces?: Array<{ id: string }> })
?.evidenceTraces ?? [];
return {
Expand Down
29 changes: 29 additions & 0 deletions apps/memos-local-plugin/tests/unit/llm/prompts.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,35 @@ describe("llm/prompts", () => {
expect(detectDominantLanguage(["GRPO / TRL / reward_fn.py"])).toBe("en");
});

it("l2 induction prompt guards against conversational-act actions (issue #2318)", () => {
// Version must bump when we tighten guidance so downstream
// `promptId@version` records attribute the new behaviour correctly.
expect(L2_INDUCTION_PROMPT.version).toBe(3);

const sys = L2_INDUCTION_PROMPT.system;

// The boundary block must name conversational acts as a rejected
// ACTION shape (symmetric to the existing L3 world-model drift guard).
expect(sys).toMatch(/conversational act/i);

// Enumerate the dialogue verbs we want the model to recognise as
// dead-skill precursors — must appear in the prompt so the LLM has a
// concrete list to check itself against.
expect(sys).toMatch(/\bask(ing)?\b.*\bthe user\b|\bask the user\b/i);
expect(sys).toMatch(/\bconfirm(ation)?\b/i);
expect(sys).toMatch(/\bnotify(ing)?\b|\bnotification\b/i);
expect(sys).toMatch(/\breport(ing)? status\b|\bstatus report\b/i);

// When the trace cluster's ONLY shared behaviour is dialogue, the
// prompt must offer the model an "abstain" escape hatch instead of
// forcing it to fabricate a policy.
expect(sys).toMatch(/abstain|no policy|do not emit|omit the policy/i);

// The existing L3-drift guard must still be present — this change is
// additive, not a rewrite.
expect(sys).toMatch(/L3|world model|world-model/i);
});

it("retrieval filter prompt asks for ranked output without selected-field leftovers", () => {
expect(RETRIEVAL_FILTER_PROMPT.system).toContain('"ranked"');
expect(RETRIEVAL_FILTER_PROMPT.system).not.toContain('"selected"');
Expand Down
6 changes: 3 additions & 3 deletions apps/memos-local-plugin/tests/unit/memory/l2/induce.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ describe("memory/l2/induce", () => {
it("returns {ok:true, draft} and fills support_trace_ids when the LLM omits them", async () => {
const llm = fakeLlm({
completeJson: {
"l2.l2.induction.v2": {
"l2.l2.induction.v3": {
title: "install system libs first",
trigger: "pip install fails in container with missing system library",
procedure: "1. detect missing lib 2. apk/apt-get install 3. retry pip",
Expand Down Expand Up @@ -75,7 +75,7 @@ describe("memory/l2/induce", () => {
it("cleans unsafe markup from LLM-derived policy fields", async () => {
const llm = fakeLlm({
completeJson: {
"l2.l2.induction.v2": {
"l2.l2.induction.v3": {
title: "<img src=x onerror=alert(1)> install system libs",
trigger: "<script>alert(1)</script>pip fails [bad](javascript:alert(1))",
procedure: "Use [safe](https://example.com), ignore [bad](javascript:alert(1))",
Expand Down Expand Up @@ -149,7 +149,7 @@ describe("memory/l2/induce", () => {
it("reason=llm_failed when the LLM draft is malformed (missing title)", async () => {
const llm = fakeLlm({
completeJson: {
"l2.l2.induction.v2": { trigger: "no title", procedure: "..." },
"l2.l2.induction.v3": { trigger: "no title", procedure: "..." },
},
});
const res = await induceDraft(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ describe("memory/l2/integration", () => {

const llm = fakeLlm({
completeJson: {
"l2.l2.induction.v2": {
"l2.l2.induction.v3": {
title: "install missing system libs in container",
trigger: "pip install fails in container with MODULE_NOT_FOUND due to missing system lib",
procedure: "1. detect lib 2. use distro pkg manager 3. retry pip",
Expand Down Expand Up @@ -367,7 +367,7 @@ describe("memory/l2/integration", () => {
repos: handle.repos,
llm: fakeLlm({
completeJson: {
"l2.l2.induction.v2": {
"l2.l2.induction.v3": {
title: " 闲聊场景下避免使用emoji ",
trigger: "用户发起非任务性闲聊,且未明确要求使用emoji",
procedure: "以简洁、自然的文字回应,但不添加任何emoji或表情符号",
Expand Down Expand Up @@ -457,7 +457,7 @@ describe("memory/l2/integration", () => {
repos: handle.repos,
llm: fakeLlm({
completeJson: {
"l2.l2.induction.v2": {
"l2.l2.induction.v3": {
title: "结构化呈现外部数据查询结果",
trigger: "agent通过工具调用获取到外部数据源的原始响应(搜索结果、API返回、数据库查询等),需要向用户呈现信息",
procedure: "1) 从原始数据中提取关键信息要素;2) 按逻辑分类组织信息(时间、地点、数值、状态等);3) 使用结构化格式呈现(分类标题、列表、表格等);4) 可选:基于数据添加简短的实用性总结或建议",
Expand Down Expand Up @@ -548,7 +548,7 @@ describe("memory/l2/integration", () => {
repos: handle.repos,
llm: fakeLlm({
completeJson: {
"l2.l2.induction.v2": {
"l2.l2.induction.v3": {
...shared,
boundary: "仅适用于数据库事务和写入操作,不适用于天气API或公开搜索结果",
rationale: "边界不同,不能复用天气呈现经验",
Expand Down Expand Up @@ -630,7 +630,7 @@ describe("memory/l2/integration", () => {
repos: handle.repos,
llm: fakeLlm({
completeJson: {
"l2.l2.induction.v2": {
"l2.l2.induction.v3": {
title: "工具全部失效时坦诚说明并回退到已有知识",
trigger: "连续尝试多个同类工具(如多个搜索引擎、多个新闻源)后全部因反爬、封禁或网络错误失效,且用户问题需要实时信息",
procedure: "1) 明确告知用户所有尝试过的工具都已失效(列出具体工具名);2) 基于训练数据中的已有知识提供最接近的答案;3) 明确标注该信息的时间戳或来源时间,并提醒用户可能存在时效性差异;4) 建议用户通过其他渠道(如直接搜索、官方网站)验证",
Expand Down Expand Up @@ -663,7 +663,7 @@ describe("memory/l2/integration", () => {

const llm = fakeLlm({
completeJson: {
"l2.l2.induction.v2": {
"l2.l2.induction.v3": {
title: "t",
trigger: "tr",
procedure: "pr",
Expand Down
Loading