Skip to content

feat(cli, term, rfd): Sanitize untrusted terminal output - #1217

Open
JeanMertz wants to merge 23 commits into
mainfrom
issue-1201
Open

JeanMertz wants to merge 23 commits into
mainfrom
issue-1201

Conversation

@JeanMertz

Copy link
Copy Markdown
Collaborator

Escape sequences in content JP relays no longer run on your terminal.
Before your messages, the assistant's replies and reasoning, tool calls,
tool results, and custom style.parameters output are shown, JP drops
sequences that move the cursor, clear the screen, retitle the window, or
write the clipboard, along with control characters other than line feeds
and tabs. This holds live and in jp conversation print, and for the
conversation titles and matched lines that jp conversation ls, show,
and grep print to a terminal. Your messages and tool output keep their
colors; the assistant's text keeps none, since it styles text with
markdown, and a character reference such as  cannot bring an
escape back.

The new style.sanitize setting decides what happens to those
sequences: strip (the default) removes them, visualize shows a ␛
in their place, and off shows content as written, with true and
false accepted as strip and off. Stored conversations and what the
model receives keep the original bytes, as do titles and matched lines
in the JSON output of ls, show, and grep. Some protections hold
under every setting, off included. Styling that a message or tool
result leaves open ends with it, so a truncated result no longer colors
the note below it or the reasoning that follows. The window title and
link targets JP writes carry no control characters, and a tool's
question is always shown as plain text.

style.code.file_link and style.code.copy_link accept true and
false through --cfg and JP_CFG_*, as config files already did.
The configuration docs describe style.sanitize, and the glossary tells
display sanitization apart from storage sanitization and stream repair.

Implements: RFD 096
Fixes: #1201

A running tool's or starting MCP server's stderr, shown in a status
region, can no longer change how the terminal reports keys. The
region's line filter now reads escape sequences the way a terminal
does: `\x1b[>4;2m` is a key-reporting setting rather than styling, so
it is dropped instead of forwarded, and `\x1b[08m` is removed as
conceal, the same as `\x1b[8m`.

That parsing comes from the new `jp_term::sanitize` module, which also
holds `ContentWriter`, the display filter RFD 096 applies to untrusted
content. It keeps text, line feeds, and tabs, keeps or drops SGR
styling depending on where the content comes from, and drops every
other escape sequence and control character, including OSC, DCS, and
APC strings with their payloads, even when a sequence is split across
writes. Content that may carry styling is closed with a reset, so
styling it leaves open ends with it. Nothing renders through it yet.

This is step 1 of RFD 096.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
A tool result that opens a color and never closes it no longer colors
what JP writes after it. A colored diff cut short by inline-result
truncation kept its red through the truncation note, the next tool
header, and the reasoning that followed, because the reset that would
have ended it was in the part of the result JP did not show.

JP now closes a tool result with a reset right after the lines it
shows, before its own truncation note and closing fence, and does the
same for a custom formatter's output. The reset comes before the final
line break, so a background the tool left open does not paint the row
below the result either.

The tool's output is otherwise written unchanged, and nothing is
stored: the reset exists only on screen.

This is step 2 of RFD 096.

Fixes: #1201
Signed-off-by: Jean Mertz <git@jeanmertz.com>
A conversation title can no longer write to the terminal outside the
window-title sequence JP puts it in. The title is usually generated by
the model, and one containing a `BEL` or an `ESC \` ended that
sequence early, so whatever followed reached the terminal as raw
input: a title of `fix\x07\x1b[2J` cleared the screen.

JP now removes every control character (C0, DEL, and C1) from a window
title before writing it, and from the target of every hyperlink it
prints, such as the "open in editor" link under a tool result. The
text a link displays is left as it is, styling included. No setting
turns this off: neither a title nor a link target has a use for
control characters.

This is step 3 of RFD 096.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
Signed-off-by: Jean Mertz <git@jeanmertz.com>
A tool's question can no longer move the cursor, erase lines, or
restyle the prompt where you approve what the tool does. The patch a
file-editing tool shows above its "apply the patch?" question is built
from text the model wrote, and escape sequences in that text ran on the
terminal: a cursor movement and a line erase could hide part of the
patch you were about to approve.

JP now removes every control character from what a tool puts in a
question before showing it: the question itself, the text shown above
it (keeping its line breaks and tabs), the options of a selection, and
the default of a text question. The rest of an escape sequence stays
visible, so `\x1b[2J` shows as `[2J`. No setting turns this off.

What the tool gets back is unchanged. Choosing an option answers with
the option as the tool wrote it, even when two options look the same
once shown, because the prompt reports the chosen index rather than
its label. Accepting a default answers with the default the tool
offered. Line breaks in a question's text are removed like any other
control character, so a question with more to say belongs in its
pre-amble.

This is step 7 of RFD 096.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
JP gains a setting for how escape sequences in conversation content are
shown. `strip`, the default, removes everything that does more than
style text; `visualize` also marks each removal with `␛`; and `off`
shows content exactly as written. It can be set in a config file, with
`--cfg style.sanitize=off`, or with `JP_CFG_STYLE_SANITIZE`, and any
other value is refused.

Nothing reads the setting yet. The chat and tool-result renderers start
honoring it in the commits that follow, which land together with this
one, as RFD 096 plans.

A conversation records the setting with the rest of its configuration,
which is why each provider fixture's stored config gains a line.
Conversations created before this change do not record it, and use
`strip` unless a config file says otherwise.

This is step 4 of RFD 096.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
`style.sanitize` now takes a boolean as well as a mode name: `true`
means `strip` and `false` means `off`. That matches the other style
settings that can be turned off, such as `style.code.file_link` and
`stderr_rows`, so `sanitize = false` in a config file, `--cfg
style.sanitize=false`, and `JP_CFG_STYLE_SANITIZE=false` all work.

A conversation still records the mode by name, so a stored config reads
`"off"` whichever spelling set it. The schema lists the boolean form
and the `"true"` and `"false"` spellings beside the three mode names.

Part of RFD 096.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
Escape sequences in the assistant's replies, its reasoning, and your
own messages no longer run on the terminal when JP shows them, live or
through `jp conversation print`. A raw terminal log pasted into a
message used to replay its cursor movement and line erasure every time
the conversation was shown, and a model could clear the screen or
redraw text you had already read, even by writing `&#27;` in place of
an escape character, which markdown decodes into one.

`style.sanitize` decides what you see instead. Under `strip`, the
default, the assistant's text loses every escape sequence and control
character, so the markdown JP renders is its only styling. Your own
messages keep their colors and bold, since a pasted colored log should
look like the log, and JP ends each message with a reset so a color it
leaves open stops there. `visualize` shows a `␛` where something was
removed, and `off` shows everything as written; the reset after a
message stays either way.

Stored conversations are unchanged, and the model still receives the
original text. A truncated reasoning display counts only the characters
it shows, so an escape sequence no longer spends its budget.

This is step 5 of RFD 096.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
`style.code.file_link` and `style.code.copy_link` now take `true` and
`false` everywhere a config value can be set. A config file already
read `file_link = false` as `off`, but `--cfg style.code.file_link=false`
and `JP_CFG_STYLE_CODE_FILE_LINK=false` were refused with `Unknown enum
variant false.`, so the same spelling worked in one place and not the
other. `true` means `full` and `false` means `off` in all three.

The schema now lists the boolean form beside the style names, so an
editor validating a config file no longer flags `file_link = false` as
invalid. A stored config still records the style by name.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
Tool results and custom-formatter output now go through the
`style.sanitize` filter before JP highlights or truncates them. Their
colors stay, but a result can no longer clear the screen, move the
cursor, retitle the window, or write the clipboard when it is shown,
live or in `jp conversation print`. Under `visualize` a `␛` marks what
was removed, and under `off` the bytes pass through as before. The
styling reset added for #1201 still lands right after the lines that
are shown, so a truncated result cannot color the note below it.

Filtering happens ahead of syntax highlighting: the highlighter could
split an escape sequence across tokens and leave its tail on screen as
text.

The tool call header treats what the model supplied as model output.
The tool name and a `function_call` argument name lose their escape
sequences, and argument values lose the DEL and C1 characters that
JSON leaves unescaped. Control characters JSON does escape stay
visible as `\u001b`, so an approval shows what the tool will receive.
The row naming tools whose arguments are still streaming drops every
control character, whatever the setting, since a line break or tab
there would throw off the printer's row count.

What JP stores, what the model receives, and the file behind a
result's link keep the tool's exact output.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
On a terminal, `jp conversation ls`, `jp conversation show`, and
`jp conversation grep` show titles and matched lines the way
`style.sanitize` asks. An escape sequence stored in a conversation (an
erase, a window title, a clipboard write) is dropped instead of run,
and JP styles the text itself. A long title in the `ls` table or a
`grep` heading is cut on its visible text, so a stored escape no longer
spends columns or gets split by the cut. `grep` finds each match again
in the text it shows, so the highlight still lands on the match, and
`grep --output text` on a terminal is filtered the same way.

Only pretty output is filtered. Plain-text output was already stripped
of escape sequences by the printer and is unchanged, and JSON output
carries titles and hit text exactly as stored, since scripts read it as
data.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
The configuration docs gain a section on terminal escape sequences:
what `style.sanitize` does with them under `strip`, `visualize`, and
`off`, which content keeps its colors, what reaches storage and the
assistant unchanged, and the protections that stay on under `off`.
The README's privacy and security page points to it.

The glossary defines display sanitization, content classes, and
content spans, and tells display sanitization apart from the two other
operations JP calls sanitize: storage sanitization (RFD 052) and
stream repair. The glossary's table of contents also gains the
Tracking Ticket entry it was missing.

This completes the implementation plan of RFD 096.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
Structured output (`jp query --schema`) now goes through the same
`style.sanitize` filter as the assistant's messages and reasoning, live
and in `jp conversation print`. Before, a response whose value is a
string, such as an answer to a `{"type": "string"}` schema or a
response that failed to parse as JSON, was printed as stored, so a
`\u001b[31m` the model wrote in it colored the terminal. Parsed values
also lose the DEL and C1 characters that JSON leaves unescaped.

A sequence split across two streamed fragments is still recognized, and
one the stream leaves unfinished is dropped before the closing fence
rather than swallowing it. Stored responses keep the model's exact
output.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
When a tool asks a text question with a default, JP shows that default
with its control characters removed. Typing the shown text is now the
answer as typed. Before, JP could not tell typing it apart from pressing
Enter, and replaced it with the tool's original default: a default of
`"main\n"` shows as `main`, and a user who typed `main` sent `"main\n"`
to the tool anyway.

Pressing Enter on the empty input still answers with the default exactly
as the tool offered it, so a default carrying a line break or a control
character reaches the tool intact. The prompt looks the same as before.

This bumps the `inquire` fork to a revision whose text prompt reports
whether the answer is its default.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
Text a command prints no longer reaches the terminal unfiltered when the
command forgot to filter it. Every line JP prints in pretty output now
passes a floor on its way out: styling, erase to the end of the line,
carriage returns, and OSC 8 links go through, and everything else is
dropped. Stored text can no longer clear the screen, move the cursor off
its line, switch terminal modes, retitle the window, or write the
clipboard, whichever command prints it.

The floor follows `style.sanitize`: `strip` removes what it drops,
`visualize` shows a `␛` in its place, and `off` lets everything through.
A turn rendered under its own config, as `jp conversation print` does
for each turn, switches the floor with it. Lines a prompt prints to
explain its question are filtered too. A prompt widget's own drawing is
not, since it moves the cursor on purpose.

The renderers still filter content by where it comes from, so a model's
reply keeps no styling and a truncated tool result ends its colors
where JP stops writing it. The floor is what remains when a command
does none of that.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
A conversation's title can no longer reach the terminal unfiltered from
the places that still printed it as stored: the conversation picker
(`jp c use ?` and other `?` targets), the `jp c title` picker and the
titles it prints, the notices from `jp c use` and `jp c label`, the
details shown before `jp c rm` and `jp c archive` ask for confirmation,
and the active conversation in `jp workspace show`. They show the title
the way `jp c ls` already does, following `style.sanitize`, and a
picker row is always one line of plain text so it can be redrawn in
place.

Titles are stored, serialized, and handed to plugins and the macOS app
exactly as before. A title is now its own type in JP's code, with no
way to format it for display except through the filter or by asking
for the stored text by name. A command that prints a title it has not
filtered no longer compiles, so the places above cannot quietly come
back.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
RFD 096 describes two protections added after its first eight steps
landed. Everything JP prints in a pretty format passes an output floor
that keeps only what JP's own output uses, so a command that forgets to
filter stored text cannot take over the terminal. Conversation titles
are a type that cannot be printed without choosing how to filter them.

The RFD's rejection of filtering at the printer now reads as rejecting
it as the place for the per-content policy, with the floor kept beneath
that policy, and it records why filtering titles once on load was not
chosen. The configuration docs say what the floor guarantees, and the
glossary names it.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
Signed-off-by: Jean Mertz <git@jeanmertz.com>
Truncating or wrapping styled text to a column budget no longer counts
escape sequences as columns or cuts one in half. A conversation title
or a `grep` match that carries its own styling is cut on its visible
text, keeping each sequence whole, and a reset that directly follows
the last character kept stays with it. Before, the cut could land in
the middle of a sequence, and the half left behind swallowed the text
JP wrote after it: the end of a `grep` heading, or the start of the
next row in `jp conversation ls`.

These functions previously required text with no escape sequences, which
every caller had to arrange by filtering first.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
…yling

Text that reaches the terminal without being filtered by where it came
from, such as a conversation title or a line `jp conversation grep`
found, can no longer return to the start of its row. The output floor
drops a carriage return like any other control character, so a stored
line cannot overwrite the turn and role JP prints in front of a `grep`
hit, or the ID at the start of an `ls` row. An erase to the end of the
line still passes, and without the carriage return it can only fill
the rest of the row.

Styling that such text opens and never closes is closed when JP exits,
so a color left on by a stored title or matched line no longer carries
over to the shell prompt. Under `style.sanitize = "off"` the floor
does not read what it passes, and nothing is closed.

`Printer::erase_line` is removed. Status regions draw their own rows,
so nothing called it, and through the floor it could no longer return
to the start of the row.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
Conversation titles and the lines `jp conversation grep` finds are
printed as stored again, and the printer's output floor is what keeps
them from taking over the terminal. A command that prints a title or a
matched line needs no filtering of its own, so one written later cannot
forget it. The floor drops every sequence that moves the cursor,
returns to the start of the row, clears the screen, switches terminal
modes, or retitles the window, and closes styling left open when JP
exits. The titles `jp conversation ls` and `grep` cut to fit a column
are cut on their visible text, so an escape sequence in one neither
spends columns nor is split.

Titles and matched lines keep their own colors on a terminal; before,
`style.sanitize` removed them. Pretty output is still filtered the way
`style.sanitize` asks, and JSON output still carries titles and matched
lines exactly as stored.

Three places write through a prompt widget, which the floor lets
through so it can redraw itself: the conversation picker, the
`jp conversation title` picker, and the details shown before
`jp conversation rm` or `archive` asks for confirmation. They show a
title as one line of plain text, under every `style.sanitize` setting.

A conversation's title is a plain string again, as it is stored,
serialized, and handed to plugins and the macOS app.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
RFD 096 drops derived strings as a content class. Conversation titles,
the lines `jp conversation grep` finds, and a message's author name are
printed as stored and bounded by the output floor, which drops a
carriage return and closes open styling when JP exits. The width
functions cut around escape sequences, and the three places that hand a
title to a prompt widget show it as one plain line. The typed-titles
step is replaced, and the alternatives record why neither a per-command
class nor a `Title` type is worth its cost once the floor is in place.

The `style.sanitize` description, the configuration guide, and the
glossary entry say the same: titles and matched lines keep their
colors, and nothing they carry can move the cursor, return to the start
of a line, or clear the screen.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
Signed-off-by: Jean Mertz <git@jeanmertz.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tool call response truncation breaks syntax highlighting

1 participant