install deletes foreign tables placed between the managed MCP markers in ~/.codex/config.toml (0.10.4 → 0.11.0 upgrade)

Version

Installed by install.sh: codebase-memory-mcp 0.11.0 (upgrading from 0.10.4)
Codex CLI 0.154.0 / Codex Desktop present on the same machine

Platform

macOS 15 (Darwin 24.6.0), Intel (darwin-amd64)

Install channel

GitHub release archive via install.sh (bash ~/.local/bin/install.sh, installs to ~/.local/bin)

Binary variant

standard

What happened, and what did you expect?

codebase-memory-mcp install (run by install.sh during the 0.10.4 → 0.11.0 upgrade) rewrote the managed # >>> codebase-memory-mcp MCP >>> … # <<< codebase-memory-mcp MCP <<< region of ~/.codex/config.toml and deleted every table that other software had placed between those two markers: [mcp_servers.node_repl] (+ its .env table), [desktop], and all four [marketplaces.*] tables. ~/.codex/config.toml went from 5,841 bytes to 4,371 bytes. Codex Desktop lost its marketplaces, desktop settings and its bundled node_repl MCP server. Nothing was printed for Codex; the install summary reported mcp: ~/.codex/config.toml as if it had succeeded.

How the foreign tables got inside the managed region: the 0.10.4 install had appended its block at the end of the file, so the closing marker was the last line. Codex Desktop later appended its own tables, and the trailing comment line stayed last (apparently its TOML editor treats a trailing comment as attached to the document end — I did not verify that in Codex itself), which left the closing marker below Codex's tables. From that point on, the region between the markers contained one cbm-owned table and four foreign ones.

Expected: the installer should never remove a table it did not write. cbm_toml_upsert_managed_block should fail closed (as toml_find_markers already does for orphan markers, #1558) when the span between its markers contains any table header other than [mcp_servers.codebase-memory-mcp] — or, better, replace only the owned table and keep the rest of the span verbatim. The user-added key inside the owned table (startup_timeout_sec = 90, needed because the MCP server takes >10 s to start on a 1.5 GB index) was also dropped; preserving unknown keys, or at least printing a note, would be kinder.

Reproduction

The real-world trigger was simply bash ~/.local/bin/install.sh on a machine whose ~/.codex/config.toml had foreign tables between the cbm markers (state shown in the diff below).

Sandboxed steps that should reproduce it without touching a real Codex configuration (HOME, CODEX_HOME, TMPDIR and the cbm cache/runtime dirs all point at a temp directory). Caveat: I could not execute this on the affected machine — with a live cbm daemon running, install stops with activation could not reserve exclusive access; no activation was committed, because the version-cohort locks live in /private/tmp/cbm-daemon-<uid>/ and are account-wide rather than under CBM_CACHE_DIR/CBM_RUNTIME_DIR. It should run on a machine (or CI job) with no daemon. The evidence for the bug is the real diff and the code path, both below.

sandbox="$(mktemp -d)"
mkdir -p "$sandbox/.codex" "$sandbox/bin" "$sandbox/cache" "$sandbox/runtime" "$sandbox/tmp"

cat > "$sandbox/.codex/config.toml" <<'EOF'
model = "gpt-5"

[mcp_servers.codegraph]
command = "codegraph"
args = ["serve", "--mcp"]
# >>> codebase-memory-mcp MCP >>>
[mcp_servers.codebase-memory-mcp]
command = "/tmp/old/codebase-memory-mcp"
args = []
startup_timeout_sec = 90

[mcp_servers.node_repl]
args = []
command = "/Applications/ChatGPT.app/Contents/Resources/cua_node/bin/node_repl"
startup_timeout_sec = 120

[desktop]
followUpQueueMode = "steer"

[marketplaces.openai-codex]
source_type = "git"
source = "https://github.com/openai/codex-plugin-cc.git"
# <<< codebase-memory-mcp MCP <<<
EOF
cp "$sandbox/.codex/config.toml" "$sandbox/before.toml"

HOME="$sandbox" CODEX_HOME="$sandbox/.codex" XDG_CONFIG_HOME="$sandbox/.config" TMPDIR="$sandbox/tmp" \
CBM_CACHE_DIR="$sandbox/cache" CBM_RUNTIME_DIR="$sandbox/runtime" \
  codebase-memory-mcp install -y --dir="$sandbox/bin" --clients=codex

diff "$sandbox/before.toml" "$sandbox/.codex/config.toml"

Expected result (matches what happened for real): [mcp_servers.node_repl], [desktop] and [marketplaces.openai-codex] are gone; only the regenerated [mcp_servers.codebase-memory-mcp] table remains between the markers.

Logs

Diff of the real file, before → after the upgrade (values in node_repl.env elided, $HOME written as ~). Lines 133–178 were replaced by the single env_vars line:

133,178c139
< startup_timeout_sec = 90
<
< [mcp_servers.node_repl]
< args = []
< command = "/Applications/ChatGPT.app/Contents/Resources/cua_node/bin/node_repl"
< startup_timeout_sec = 120
<
< [mcp_servers.node_repl.env]
< NODE_REPL_NATIVE_PIPE_CONNECT_TIMEOUT_MS = "…"
< … (12 more env keys)
<
< [desktop]
< followUpQueueMode = "steer"
< external-agent-import-sync-item-types = "all"
< external-agent-import-sync-enabled = true
< conversationDetailMode = "STEPS_PROSE"
< sansFontSize = 14
< codeFontSize = 13
< ambient-suggestions-enabled = true
<
< [marketplaces.openai-bundled]
< source_type = "local"
< source = "~/.codex/.tmp/bundled-marketplaces/openai-bundled"
<
< [marketplaces.openai-primary-runtime]
< source_type = "local"
< source = "~/.cache/codex-runtimes/codex-primary-runtime/plugins/openai-primary-runtime"
<
< [marketplaces.claude-plugins-official]
< source_type = "git"
< source = "https://github.com/anthropics/claude-plugins-official.git"
<
< [marketplaces.openai-codex]
< source_type = "git"
< source = "https://github.com/openai/codex-plugin-cc.git"
---
> env_vars = ["CBM_CACHE_DIR", "CBM_RUNTIME_DIR"]

Installer output for the Codex section (no warning about the removed tables):

Codex CLI:
  mcp: ~/.codex/config.toml
  instructions: ~/.codex/AGENTS.md (managed activation pointer)
  ...
  hooks: SessionStart + SubagentStart (dynamic graph context)

Where it happens: src/cli/config_toml_edit.c, cbm_toml_upsert_managed_block() copies existing[0 .. begin_line.start), appends the fresh managed block, then copies existing[end_line.full_end ..). Everything between the two marker lines is discarded, and toml_managed_block_conflicts() only inspects the text outside that span, so foreign tables inside it are neither detected nor preserved.

Project scale (if relevant)

Not index related. (For context: the machine's main index is 567 k nodes / 707 k edges, 1.5 GB.)

Related observations from the same upgrade (not the main report)

~/.codex/hooks.json: the two hook-augment entries had been edited locally to prefix CBM_HOOK_DEADLINE_MS=20000 (see next bullet). The installer treated them as foreign, preserved them, and appended its own canonical SessionStart and SubagentStart groups, so Codex ended up with each hook twice. Matching on the binary path + hook-augment token, rather than the exact command string, would avoid the duplicate.
Feature ask: hook-augment has a built-in 2000 ms deadline (only overridable via the CBM_HOOK_DEADLINE_MS env var). On this 1.5 GB index a hook query takes 4–6 s even with a warm daemon, so every hook silently exited empty (127 deadline_exceeded lines in logs/hook-augment-timeouts.log) until the shims were hand-edited — which then makes them "not ours" for the installer. A config set hook_deadline_ms setting honoured by the managed shims would avoid the whole conflict.

Confirmations

I searched existing issues (Installer adds hooks to both hooks.json and config.toml #570, Codex: reinstall appends duplicate [[hooks.SessionStart]] when hook exists in inline form — "duplicate key" breaks config.toml #1432 and Installer/updater scope and safety: no per-client selector, destructive update step before an interactive prompt, hidden ui_enabled, brittle legacy-marker cleanup #1558 are about hook duplication / orphan markers; none cover foreign tables inside the managed MCP region).
The sandboxed steps above do not modify a real Codex configuration; the real-file diff was taken from a snapshot made two minutes before the upgrade and the file has since been restored by hand.

Thank you for the detailed before/after evidence and for separating the confirmed configuration loss from the additional observations. I checked current main (59a05eb): cbm_toml_upsert_managed_block replaces the entire marked span, and its conflict check excludes that same span. That confirms the preservation gap you identified when foreign tables end up inside the markers.

We are treating this as a high-priority installer data-preservation bug and keeping it open. Review needs to cover both preserving unrelated tables and handling extra keys in the owned table, without weakening the existing malformed-marker protections. The hook duplication and deadline-setting suggestions are separate concerns, not prerequisites for addressing the data loss. Please keep your restored configuration and backup; there is no need to repeat the destructive upgrade on your real settings.

Code checked:

codebase-memory-mcp/src/cli/config_toml_edit.c

Lines 1070 to 1134 in 59a05eb

    
           int cbm_toml_upsert_managed_block(const char *file_path, const char *begin_marker, 
        
                                             const char *end_marker, const char *block) { 
        
               size_t block_len = 0U; 
        
               if (!toml_valid_path(file_path) || !toml_valid_marker(begin_marker) || 
        
                   !toml_valid_marker(end_marker) || strcmp(begin_marker, end_marker) == 0 || !block || 
        
                   toml_bounded_length(block, TOML_EDIT_MAX_BYTES, &block_len) != TOML_EDIT_OK || 
        
                   !toml_text_is_safe(block, block_len, 1) || 
        
                   toml_contains_marker_line(block, block_len, begin_marker, end_marker) || 
        
                   toml_validate_lexical_strings(block, block_len) != TOML_EDIT_OK) { 
        
                   return TOML_EDIT_ERR; 
        
               } 
        
               char *existing = NULL; 
        
               size_t existing_len = 0; 
        
               toml_file_snapshot_t snapshot; 
        
               if (toml_read_file(file_path, &existing, &existing_len, &snapshot) != TOML_EDIT_OK || 
        
                   !toml_text_is_safe(existing, existing_len, 1)) { 
        
                   free(existing); 
        
                   return TOML_EDIT_ERR; 
        
               } 
        
               toml_line_t begin_line = {0}, end_line = {0}; 
        
               int has_pair = 0; 
        
               if (toml_find_markers(existing, existing_len, begin_marker, end_marker, &begin_line, &end_line, 
        
                                     &has_pair) != TOML_EDIT_OK) { 
        
                   free(existing); 
        
                   return TOML_EDIT_ERR; 
        
               } 
        
               size_t exclude_start = has_pair ? begin_line.start : SIZE_MAX; 
        
               size_t exclude_end = has_pair ? end_line.full_end : SIZE_MAX; 
        
               if (toml_managed_block_conflicts(existing, existing_len, exclude_start, exclude_end, block, 
        
                                                block_len) != TOML_EDIT_OK) { 
        
                   free(existing); 
        
                   return TOML_EDIT_ERR; 
        
               } 
        
               toml_buffer_t output = {0}; 
        
               size_t prefix_len = has_pair ? begin_line.start : existing_len; 
        
               if (has_pair && prefix_len == 0U && existing_len >= 3U && (unsigned char)existing[0] == 0xefU && 
        
                   (unsigned char)existing[1] == 0xbbU && (unsigned char)existing[2] == 0xbfU) { 
        
                   prefix_len = 3U; 
        
               } 
        
               const char *newline = toml_newline_style(existing, existing_len); 
        
               size_t payload_start = existing_len >= 3U && (unsigned char)existing[0] == 0xefU && 
        
                                              (unsigned char)existing[1] == 0xbbU && 
        
                                              (unsigned char)existing[2] == 0xbfU 
        
                                          ? 3U 
        
                                          : 0U; 
        
               if (toml_buffer_append(&output, existing, prefix_len) != TOML_EDIT_OK || 
        
                   (!has_pair && existing_len > payload_start && existing[existing_len - 1] != '\n' && 
        
                    toml_buffer_append_cstr(&output, newline) != TOML_EDIT_OK) || 
        
                   toml_append_managed(&output, begin_marker, end_marker, block, newline) != TOML_EDIT_OK || 
        
                   (has_pair && toml_buffer_append(&output, existing + end_line.full_end, 
        
                                                   existing_len - end_line.full_end) != TOML_EDIT_OK)) { 
        
                   toml_buffer_dispose(&output); 
        
                   free(existing); 
        
                   return TOML_EDIT_ERR; 
        
               } 
        
               int result = 
        
                   toml_write_atomic(file_path, existing, existing_len, output.data, output.len, &snapshot); 
        
               toml_buffer_dispose(&output); 
        
               free(existing); 
        
               return result; 
        
           }

fix(config): refuse array and dotted-key conflicts in the managed block

@daceconomy

Thank you very much, @daceconomy, for this report. The real before/after diff, the explanation of how Codex Desktop's appends ended up between our markers, and the pointer straight to cbm_toml_upsert_managed_block made it quick to reproduce and fix. We're sorry the upgrade cost you your Codex Desktop settings, and we appreciate you restoring them by hand and still writing it all up so carefully.

What changed: the installer no longer treats everything between # >>> codebase-memory-mcp MCP >>> and # <<< … <<< as its own. It now reads that region table by table:

[mcp_servers.codebase-memory-mcp] is ours. We rewrite only the keys we write (command, args, env_vars). Keys and comments you add there, like your startup_timeout_sec = 90, stay in the table, and so do sub-tables such as [mcp_servers.codebase-memory-mcp.env].
Every other table ([mcp_servers.node_repl] and its .env, [desktop], [marketplaces.*]) is moved byte for byte, in its original order, to just below the closing marker. From then on the markers bracket only our entry, and Codex's later appends land outside them.
Uninstall removes only our table and the markers; everything else stays where it is.
Where ownership can't be decided safely, the installer refuses and leaves the file untouched rather than guessing.

We chose to keep your tables rather than just refuse. Codex keeps appending below a closing marker that ends the file, so a refusing installer would have blocked every later install and uninstall until the TOML was hand-edited, and a hand edit is exactly where tables like these get lost.

The hook duplication and the hook-deadline observations are separate from this data-loss fix; we've noted them and will follow up on them separately. The fix is in #2467. Thanks again for the careful, reproducible report!

@DeusData

@DeusData, this is why #2434 keeps foreign tables in place: moving them past the closing marker introduces another way to lose that marker.

I reproduced this with #2434 (0f308252), #2467 (98e58acb), temporary configs matching this issue's layout, and Codex app-server 0.149.1:

Call cbm_toml_upsert_managed_block() from the respective PR head.

Delete the first foreign table through app-server config/value/write:

{"keyPath":"mcp_servers.node_repl","value":null,"mergeStrategy":"replace"}

Call the same CBM upsert again.

Observed helper results:

#2434: both markers remain;     next upsert =  0
#2467: closing marker is gone;  next upsert = -1 (file unchanged)

In #2467, the closing marker is emitted before the moved sections. Codex deletes it along with node_repl. The other tables and startup_timeout_sec = 90 survive; the failure is the subsequent update refusal. Deleting a later table, replacing the first table, and adding a table all passed on both heads.

I'd keep foreign tables in place and determine ownership from table paths and managed keys, as #2434 does. If we move them, this edit-and-reinstall sequence needs regression coverage.

These checks used the unmodified CBM helpers under ASan/UBSan and the real app-server edit; I did not run the full installer or Desktop GUI.

I rechecked this on Linux x86_64 with full binaries built using scripts/build.sh: #2434 at 0f308252 and #2467 at 98e58acb, with Codex app-server 0.149.1. Each case used a fresh container and temporary config.

Both initial installs and unchanged reinstalls succeeded. After deleting mcp_servers.node_repl through the same config/value/write request, the full install -y --dir=/root/bin --clients=codex returned:

fix(config): recover Codex MCP after closing marker loss #2434: exit 0, both markers retained.
fix(install): keep foreign tables between the managed TOML markers (#2228) #2467: exit 1, closing marker missing. stderr:
```
error: agent_config agent=Codex CLI op=mcp_install path=/root/.codex/config.toml (target: regular file, 961 bytes)
```

The failed step left config.toml byte-identical; the installer kept the binary it had already published. Deleting the later desktop table instead passed on both PRs.

Merge pull request #2467 from DeusData/fix/issue-2228-toml-foreign-ta…

fix(install): keep foreign tables between the managed TOML markers (D…

@daceconomy

Thank you again, @daceconomy, for this careful report, and @astandrik for the follow-up testing.

#2467 landed on main in 8c12de9. The installer now reads the managed region table by table. It keeps your keys in [mcp_servers.codebase-memory-mcp], moves every other table byte for byte to just below the closing marker, and refuses rather than guessing when ownership is unclear.

Reopening, because @astandrik showed one case it does not cover yet. When Codex itself deletes a table through config/value/write, the closing marker can disappear with it, and the next install then refuses (exit 1, file left untouched) instead of recovering. That follow-up is being worked out on #2434. This issue stays open until it lands.

Keep cbm install working after Codex edits config.toml (#302)

Navigation Menu

Repository navigation

install deletes foreign tables placed between the managed MCP markers in ~/.codex/config.toml (0.10.4 → 0.11.0 upgrade) #2228

Version

Platform

Install channel

Binary variant

What happened, and what did you expect?

Reproduction

Logs

Project scale (if relevant)

Related observations from the same upgrade (not the main report)

Confirmations

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	int cbm_toml_upsert_managed_block(const char file_path, const char begin_marker,
	const char end_marker, const char block) {
	size_t block_len = 0U;
	if (!toml_valid_path(file_path) \|\| !toml_valid_marker(begin_marker) \|\|
	!toml_valid_marker(end_marker) \|\| strcmp(begin_marker, end_marker) == 0 \|\| !block \|\|
	toml_bounded_length(block, TOML_EDIT_MAX_BYTES, &block_len) != TOML_EDIT_OK \|\|
	!toml_text_is_safe(block, block_len, 1) \|\|
	toml_contains_marker_line(block, block_len, begin_marker, end_marker) \|\|
	toml_validate_lexical_strings(block, block_len) != TOML_EDIT_OK) {
	return TOML_EDIT_ERR;
	}

	char *existing = NULL;
	size_t existing_len = 0;
	toml_file_snapshot_t snapshot;
	if (toml_read_file(file_path, &existing, &existing_len, &snapshot) != TOML_EDIT_OK \|\|
	!toml_text_is_safe(existing, existing_len, 1)) {
	free(existing);
	return TOML_EDIT_ERR;
	}

	toml_line_t begin_line = {0}, end_line = {0};
	int has_pair = 0;
	if (toml_find_markers(existing, existing_len, begin_marker, end_marker, &begin_line, &end_line,
	&has_pair) != TOML_EDIT_OK) {
	free(existing);
	return TOML_EDIT_ERR;
	}
	size_t exclude_start = has_pair ? begin_line.start : SIZE_MAX;
	size_t exclude_end = has_pair ? end_line.full_end : SIZE_MAX;
	if (toml_managed_block_conflicts(existing, existing_len, exclude_start, exclude_end, block,
	block_len) != TOML_EDIT_OK) {
	free(existing);
	return TOML_EDIT_ERR;
	}

	toml_buffer_t output = {0};
	size_t prefix_len = has_pair ? begin_line.start : existing_len;
	if (has_pair && prefix_len == 0U && existing_len >= 3U && (unsigned char)existing[0] == 0xefU &&
	(unsigned char)existing[1] == 0xbbU && (unsigned char)existing[2] == 0xbfU) {
	prefix_len = 3U;
	}
	const char *newline = toml_newline_style(existing, existing_len);
	size_t payload_start = existing_len >= 3U && (unsigned char)existing[0] == 0xefU &&
	(unsigned char)existing[1] == 0xbbU &&
	(unsigned char)existing[2] == 0xbfU
	? 3U
	: 0U;
	if (toml_buffer_append(&output, existing, prefix_len) != TOML_EDIT_OK \|\|
	(!has_pair && existing_len > payload_start && existing[existing_len - 1] != '\n' &&
	toml_buffer_append_cstr(&output, newline) != TOML_EDIT_OK) \|\|
	toml_append_managed(&output, begin_marker, end_marker, block, newline) != TOML_EDIT_OK \|\|
	(has_pair && toml_buffer_append(&output, existing + end_line.full_end,
	existing_len - end_line.full_end) != TOML_EDIT_OK)) {
	toml_buffer_dispose(&output);
	free(existing);
	return TOML_EDIT_ERR;
	}

	int result =
	toml_write_atomic(file_path, existing, existing_len, output.data, output.len, &snapshot);
	toml_buffer_dispose(&output);
	free(existing);
	return result;
	}

install deletes foreign tables placed between the managed MCP markers in ~/.codex/config.toml (0.10.4 → 0.11.0 upgrade) #2228

Description

Version

Platform

Install channel

Binary variant

What happened, and what did you expect?

Reproduction

Logs

Project scale (if relevant)

Related observations from the same upgrade (not the main report)

Confirmations

Activity

DeusData commented on Sep 19, 2026

DeusData commented on Oct 1, 2026

astandrik commented on Oct 2, 2026

astandrik commented on Oct 2, 2026

DeusData commented on Oct 4, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions