Skip to content

Security: Zoltu/orchestration-builder

Security

docs/security.md

Security

Threat model

The executor runs a language model against a Guild that exposes tools such as file system access and shell command execution. The model is a trusted component: it carries out the operator's intent and is not assumed to emit attacks. The untrusted input is the workspace — it may contain malicious scripts, binaries, package manifests, or prompt-injection payloads (a file or task description that tries to override the system prompt, ignore safety rules, or exfiltrate data). The goal is to prevent the workspace from causing the model to do bad things, and to contain any damage to the mounted workspace.

The defense is primarily upstream: isolate the workspace, canonicalize file paths, restrict network egress, and design Guild prompts that resist injection. Where model output is displayed in the web UI, a light defense-in-depth backstop exists for the case where a prompt-injection attempt from the workspace succeeds in coercing the model's output — not because the model is an adversary. The model is not over-defended: it is not sandboxed, its output is not quarantined, and its prose is rendered as the Markdown it is intended to be.

Attack surface

  • Model-generated shell commands. run_shell runs arbitrary commands. A confused model can run destructive commands by mistake (misinterpreting the task, not acting maliciously). This is the same risk as any local coding agent that runs generated code.
  • Prompt injection. A task description or a file in the workspace can attempt to override system prompts, instruct the model to ignore safety rules, or exfiltrate data. This is the primary path by which untrusted workspace content reaches the trusted model. The executor cannot prevent all prompt injection, so workspace isolation and egress restriction are the defense.
  • Network egress. A model whose instructions have been subverted by prompt injection — or one that makes a mistake with a network-capable command — could read workspace files and send them to a remote host. Egress should be restricted in production.
  • File traversal. File read/write tools must not escape the workspace.
  • Supply-chain inputs. A workspace may contain scripts, binaries, or package manifests. The executor treats these as untrusted inputs.
  • Vendored web client dependencies. The web client (source/web/static/app.js) vendors three single self-contained files under source/web/static/vendor/ (no runtime import of anything outside the file, no node_modules in the deployed image): hyperapp@2.0.0 (the UI renderer), showdown@2.1.0 (Markdown → HTML), and highlight.js@11.11.1 (syntax highlighting inside fenced code), plus the highlight.js GitHub theme CSS. These are an approved, one-time exception to the no-dependencies policy; they are small, widely-audited, and have no transitive runtime dependencies. They are not listed in package.json runtime or dev dependencies — the web client is a static bundle served verbatim, not a Node program — and they are added to the image only as vendored static assets. The unminified browser builds are vendored (not minified bundles) so the operator can read the source; showdown is a fully-wrapped single file (no minification, no data blobs larger than a few hundred chars per line) and highlight.js is the unminified cdn-release build. Version pins: showdown@2.1.0 (MIT), highlight.js@11.11.1 (BSD-3-Clause), hyperapp@2.0.0 (MIT). Their licenses are vendored alongside them (vendor/*.LICENSE).

Web client rendering pipeline

The agent's prose fields — task, result summary, ask_human question text and context, and the surfaced run error message — are Markdown the UI renders as formatted text (headings, lists, code blocks) with syntax-highlighted fenced code, instead of the literal punctuation a textContent-only render would show. The model is a trusted component and its prose is not assumed to be an attack; rendering it as Markdown is the intended UX.

The residual concern is not a malicious model but prompt injection: a malicious file in the workspace could coerce the model into emitting markup that would be dangerous if inserted into the DOM as raw HTML. The primary defense is upstream (preventing injection from the workspace); the rendering pipeline below is a light defense-in-depth backstop for the case where an injection attempt succeeds in coercing the model's output. It never inserts HTML via innerHTML. It is, in order:

  1. showdown (window.showdown, a classic deferred <script> global) parses the Markdown to an HTML string via a Converter configured for GFM tables and strikethrough (header-id generation disabled). Fenced code blocks are then highlighted by replacing each <pre><code> block with highlight.js (window.hljs) output, emitting <pre><code class="hljs language-X">…</code></pre>.
  2. DOMParser parses that HTML string with text/html into a neutral node tree. text/html parsing never executes scripts, so a <script> in the Markdown becomes an inert element node.
  3. The sanitizer (source/web/static/markdown.js, imported by app.js) walks the tree and keeps only what an allowlist permits. This is the single source of truth for what may render and is unit-tested in-memory by source/web/markdown.test.ts.
  4. The surviving tree is turned back into hyperapp vnodes — text becomes text-node children, elements become h(tag, attributes, children). hyperapp places content into text nodes and DOM properties/attributes, never into markup, so the rendered tree cannot break out of the DOM.

Because strings only ever become text nodes or allowlisted attributes (never innerHTML), and because the sanitizer strips anything not on the allowlist before the vnode step, markup coerced into the model's output is defanged before it reaches the DOM even if an upstream injection defense fails. Machine fields (tool names, log payloads, raw-payload detail, timestamps, role names, run ids, the one-line current-activity summary, the operator's question answer, artifact paths) stay textContent.

The UI's visual polish pass added only CSS and structural class hooks on trusted (non-agent) DOM — the active run's list item (is-active), the create-run form's busy state (is-busy), and layout wrappers. It introduced no new untrusted-content insertion paths: agent prose still flows only through the sanitized Markdown pipeline above, and every other field remains a text node. No class hook is applied to agent-authored content. The run-list task is now rendered through that same sanitized Markdown path (it is the same task field the per-run view renders), removing the prior compact-listing exception so the task is uniformly sanitized Markdown wherever it appears; the multiline task editor itself is trusted operator input that the server stores verbatim and the client renders only through the sanitizer. The interrupt form follows the same rule: the operator's message is posted to the server and never rendered back into the page (the form's copy and outcome notice are fixed trusted strings), so it adds no untrusted-content path.

The flow-graph run view (source/web/static/flow-view.js, the run view's centerpiece) renders the run from a shared InteractionModel (see docs/visualization.md). It renders only machine fields and trusted guild labels as SVG <text> textContent — node labels, sublabels, invocation counters, cost figures, and top-bar history counts are all strings placed into text nodes, never markup. It introduces no agent-prose insertion path: the task summary, role summaries, and error messages that are agent Markdown stay in the sanitized Markdown panels and never reach the SVG. Edge routing uses straight anchor-to-anchor cubic curves (no pathfinding text), so the graph carries no content beyond the labeled nodes. The flow animation (marching-ants on active edges, the active-node pulse, and node enter/depart lifecycle transitions) is CSS-only — it toggles class hooks on the same <path>/<g> elements and adds no new text or content surface, so the textContent invariant is unchanged.

The ask_human question modal (source/web/static/question-modal.js) renders the question text and its optional context as sanitized Markdown through the same shared pipeline (source/web/static/markdown-render.js → markdown.js allowlist) as the rest of the agent prose — never raw HTML. The answer input is trusted operator input: it is sent verbatim to the backend (the product client POSTs to /api/answer; the dev harness advances the fixture frame) and is never rendered back as Markdown from an untrusted source. The modal's structural chrome (the run label naming which run is asking, the heading) carries only machine/trusted fields as text nodes. The modal is scoped to the run view (an overlay over the flow area, not a page-wide mask) so it cannot be confused with content outside the active run.

The result modal (source/web/static/result-modal.js) renders the run's terminal result on completion. The result summary and the surfaced error message are agent Markdown and flow through the same shared sanitized pipeline (markdown-render.js → markdown.js allowlist) as the rest of the prose — never raw HTML. The artifacts list is plain text paths placed as textContent (machine fields, never Markdown) so a workspace-derived path cannot carry markup. The machine error kind (an internal taxonomy like llm_unavailable) is intentionally not surfaced to the user — a non-developer cannot act on it — and reaches the DOM only through the "copy raw" button, which hands the full error object to the operator as a JSON string placed on the clipboard (trusted operator output), never rendered as Markdown. The honest framing line above the error block is fixed trusted text, not agent prose. The modal's status, heading, and run label are machine/trusted fields as text nodes. The modal is scoped to the run view like the question modal.

The flow-graph tooltip (source/web/static/tooltip.js) is the inspector that appears when an operator hovers a node or edge, and disappears when the pointer leaves it. It is an HTML overlay card (not an SVG primitive): SVG cannot host the sanitized-Markdown vnodes (HTML <p>/<ul>/…) the prose path produces, so the card follows the question/result modal pattern. Each labeled section is rendered by kind — pretty-printed JSON as a <pre> text node (JSON.stringify(content, null, 2), indented, so its newlines are real line breaks never literal \n), agent prose (summaries, error messages, task text, question text) through the same shared sanitized Markdown pipeline (markdown-render.js → markdown.js allowlist), and plain text for scalars (status words, invocation counts, formatted times). The executor stores tool arguments as a JSON string; the tooltip probes such a string with JSON.parse and pretty-prints it as JSON when it parses to an object/array, so {"path":"…"} becomes legible rather than rendering as a one-line Markdown paragraph. The card is read-only and carries no buttons: it is pointer-events: auto with user-select: text so the operator can move the pointer into the card to select and copy its contents, and the wiring dismisses it (after a short grace timer) once the pointer is over neither the node/edge nor the card. No untrusted content reaches the DOM as markup except via the sanitized Markdown path; every other field is a text node.

The sequence diagram view (source/web/static/sequence-diagram.js, rendered behind the Flow/Sequence toggle) is the debug/investigation surface. It renders the same InteractionModel as the flow view. It renders only machine fields and trusted guild labels as SVG <text> textContent — column headers (the friendly role/tool labels, or "The human"/"Tools" for the special columns). It introduces no agent-prose insertion path: message lines are <path> elements with a d attribute, terminal nodes are <rect> elements, and both carry a native SVG <title> text child for hover, never markup. The active/flowing highlight is CSS-class-only (seq-message--flowing/returning/error, seq-node--active/success/error) applied to the same <path>/<rect> elements — it toggles stroke color and a marching-ants/pulse animation and adds no new text or content surface, so the textContent invariant is unchanged (the same pattern the flow view's graph-edge--*/graph-node--* classes follow). The inspector (the tooltip.js card the flow view also uses) renders an operation's details markdown through the same shared sanitized Markdown pipeline (markdown-render.js → markdown.js allowlist) — never as raw HTML — and its other fields (the resolved label, invocation counts, formatted times) are text nodes. No workspace-derived prose is interpolated into the graph.

The "now" caption (deriveNowCaption) and the ambient cost strip (deriveCostStrip) are pure derivations over the InteractionModel: the caption resolves the active operation's label through the trusted label registry (source/web/static/labels.js) into a single textContent line, and the cost strip formats summed tokens and the latest elapsed as textContent spans. Neither interpolates an operation's details or any other runtime string, so the textContent-for-identifiers-and-counts / sanitized-Markdown-for-details split holds across every visualization surface. See docs/visualization.md for the model and view design.

The backend InteractionModel adapter (source/web/interaction-model-adapter.ts, served at GET /api/runs/:id/flow and the /api/run/flow alias) is the single place that turns the executor's LogEvent stream into the model the view modules render. It serves structured JSON only: participant ids/roles/kinds, operation kinds/stacks/sources/destinations, timestamps, outcomes, and the per-invocation metrics counters (tokens, cached prompt tokens, elapsed seconds) are JSON values the client places as textContent; the only agent-authored content is each operation's details field, an adapter-formatted Markdown string (delegation task text, pretty-printed tool arguments and results, ask_human question text and context, the operator's answer text, role summaries, and inquiry answers). The server does not sanitize details — it must not serve pre-rendered HTML that would bypass the client — so the invariant reduces to the one the view surfaces already enforce: details reaches the DOM only through the step-27 sanitized Markdown pipeline (markdown-render.js → markdown.js allowlist), and every other field stays a text node. The model carries no display prose (labels are a client concern resolved from the live /api/config), so no other agent string crosses the endpoint. The operator's own inquiry question text likewise travels only as a details markdown field (it is trusted operator input, still passed through the same sanitizer before render).

The product run view (source/web/static/app.js) consumes that live endpoint alongside the live /api/config, /api/questions, and /api/runs/:id routes — the same view modules (flow-view.js, sequence-diagram.js, question-modal.js, result-modal.js) now read live data. The data-source swap changes no rendering surface, so the invariant the surfaces above already enforce holds unchanged for live (untrusted) data: a live run's operation.details, task text, surfaced error message, and ask_human question text/context flow only through the sanitized Markdown pipeline, and every machine field (identifiers, counters, costs, timestamps, status words, run ids) stays a textContent node. Live data is untrusted exactly as fixture data was — the model is the same shape, only its origin changed — so no new untrusted-content insertion path is introduced by the hookup.

The run-view inspector is wired onto both live surfaces (flow-view.js and sequence-diagram.js) by the product client: hovering a node or edge resolves the target's data-operation / data-participant / data-role off the live InteractionModel and label resolver, and a single shared derivation in tooltip.js (deriveOperationTooltip, deriveParticipantTooltip, deriveRoleTooltip) produces the { title, sections } the Tooltip card renders. The card is a hyperapp-managed HTML overlay (the question/result modal pattern), position: fixed and pointer-events: auto so it floats above the stage and the operator can move the pointer into it to select text; mouseover/mouseleave/click on the .pb-flow stage drive a tooltip descriptor in state, and a short grace timer keeps the card open while the pointer travels between a node and the card, dismissing it once the pointer is over neither. Moving between a sequence message and its terminal node — both carry the same data-operation — is the same inspector target, so the card stays open without a dismiss/reopen flicker. The invariant holds for the live inspector exactly as described for the tooltip card above: the resolved label, participant kind, invocation counts, total time/tokens, and status words are textContent, and the only prose a section carries is an operation's details markdown, routed through formatTooltipContent → the shared sanitized Markdown pipeline — no details string is interpolated into an SVG attribute or innerHTML. The derivations are pure functions of (model, label resolver, id), exercised in-memory by source/web/tooltip.test.ts, so the inspector's content decisions stay in the testable surface area while the event → derivation → card wiring in app.js is the thin integration shell.

After the legacy-panel retirement and the page-structure redesign (single-screen console: top bar + Watch/History/Compose screens), the product run view's complete untrusted-content surface is the flow view, the sequence view, the question/result/interrupt modals, the run-view inspector, the plan disclosure, and the History screen. Every agent-authored string across these surfaces reaches the DOM only through the step-27 sanitized Markdown pipeline (markdown-render.js → markdown.js allowlist): the History row's expanded details (full task, result summary, error message), the interrupt modal's and interrupt answer card's answer Markdown, the result modal's summary/error Markdown, and the plan disclosure's Markdown body (the run's plan document, plan on the run view; its toggle row is fixed trusted text). Every identifier, counter, cost, timestamp, status word, run id, artifact path, the operator's answer, the operator's own interrupt messages, the top bar's viewed-run label, and the History row's primary line stay textContent nodes (the interrupt message the operator typed is posted to the server and rendered back only as plain text). The generated one-line run summary (summary.txt, served as RunSummary.summary and preferred for the History row and top-bar labels) is agent prose like any model output, but it reaches the DOM only as textContent — it is never routed through the Markdown pipeline, so it introduces no markup path. The retired role-activity, raw-log, inline-questions, config, and run-summary panels introduced no longer-reachable surface — their removal deleted code paths, not the invariant — so the surface set above is complete and the invariant holds for the final live view.

Sanitizer allowlist

  • Allowed tags: p, br, hr, h1–h6, ul, ol, li, pre, code, blockquote, em, strong, del, s, a, span, table, thead, tbody, tr, th, td. span is allowed because highlight.js wraps syntax tokens in <span class="hljs-…">. img, script, iframe, object, embed, svg, form, input, link, meta, and every other tag are not allowed.
  • Disallowed tags are either dropped wholesale (a denylist of non-prose/dangerous elements: script, style, iframe, object, embed, svg, form, input, link, meta, audio, video, etc. — their content is removed, not unwrapped, so a <script> body never becomes visible text) or unwrapped (every other disallowed tag: the tag is removed but its sanitized children are kept, so agent-authored <div>text</div> still yields the inner text).
  • Allowed attributes: a may carry href and title only; code, pre, and span may carry class only (for highlight.js token classes and showdown's language-… code classes; class values are CSS-only and not executable); every other tag carries no attributes. Every on* event handler, style, src, srcset, and any other attribute is stripped by omission.
  • Allowed URL schemes: http, https, mailto. javascript:, data:, vbscript:, and any other absolute scheme are rejected; relative, anchor, and protocol-relative URLs pass. The scheme is read after stripping leading and embedded control characters (tab, newline, NUL — the bytes browsers ignore before resolving a scheme), so java\tscript: cannot smuggle past.

If showdown or highlight.js fails to load or throws, a prose field falls back to its raw text rendered as a single text node, so the UI stays readable instead of blank.

Mitigations

  • Prompt-injection resistance (primary). Because the workspace is untrusted and the model is trusted, the main defense is preventing workspace content from coercing the model: workspace isolation, egress restriction, and Guild prompts designed to resist injection (ignoring embedded instructions to override safety rules, treating file contents as data rather than commands). The display-layer sanitizer in the web client is a secondary backstop, not the primary line.
  • Workspace isolation. The executor modifies the mounted project at /workspace in place. File tools canonicalize paths and reject any that resolve outside the workspace. Runs are sequential (one at a time), so there is no concurrent-run isolation concern.
  • Path canonicalization. File tools resolve paths relative to the workspace, canonicalize them, and reject any that escape.
  • Run-bookkeeping visibility. The file tools refuse the top-level .orchestration directory (permission_denied), and at startup and run start the executor adds .orchestration/ to the workspace's git exclude file (.git/info/exclude, following worktree gitdir links), so run logs stay out of the agent's file-tool view and out of accidental commits. This is visibility hygiene, not containment: run_shell runs arbitrary commands and can still read anything the workspace user can, so it remains the trust boundary.
  • Tool exposure is a Guild decision. The executor only exposes tools to a role if the role explicitly lists them. A Guild author can remove run_shell if it is not needed.
  • Inspection tools are read-only and bounded. The agent-activity inspection tools (list_role_messages, read_message_window, search_role_blocks, recent_role_tool_calls) never mutate the target role's state, and they never return a full message's content or reasoning — only a compact index, hard-capped windows, capped search matches, and a hash-based tool-call trace. A handler (e.g. the loop detector) therefore cannot have its own context window flooded by the target's 256k conversation, and a compromised handler role has no write path into another role's history (the only write is trigger_interrupt's single marked redirect message or abort decision, which is the platform's intended effect).
  • run_shell containment. Containment comes from the deployment environment (non-root user, restricted egress, read-only filesystem outside the workspace mount), not from an in-tool command allowlist. Per-run environment isolation (scoped PATH/HOME) is Foundry work that hardens benchmark evaluation; an in-tool allowlist remains a deferred enhancement, not the v1 path.
  • Secrets. API keys are passed through environment variables or Docker secrets (/run/secrets), never stored in the Guild, the deployment file, or the workspace.

What the design does not prevent

  • A deliberately destructive task given by a legitimate user. The executor does not second-guess the user; it only contains execution to the workspace.
  • A model that destructively modifies files inside the workspace. This is expected behavior for coding tasks; isolation prevents damage elsewhere.
  • The model itself acting maliciously. The model is a trusted component and is not treated as an adversary; if it were compromised that would be a trusted-component failure outside this threat model. The design defends against the workspace coercing the model, not against the model.
  • Resource exhaustion within budget limits. The executor enforces timeouts and budgets, but a malicious workspace input could still trigger expensive computations within those limits.

Foundry implications

The Foundry may propose Guild changes that expose new tools or broaden existing ones. A branch that weakens workspace isolation or prompt-injection resistance should fail validation before reaching the baseline. Review of Foundry reports should include checking which tools were added or removed, and whether prompt changes make the model more susceptible to coercion by workspace content.

There aren't any published security advisories