Skip to content

Stop Markdown in doc attributes from leaking past the closing quotes - #21

Merged
robertoaloi merged 2 commits into
mainfrom
fix/doc-markdown-leak
Sep 30, 2026
Merged

robertoaloi merged 2 commits into
mainfrom
fix/doc-markdown-leak

Conversation

@robertoaloi

Copy link
Copy Markdown
Member

When a -doc or -moduledoc attribute contains malformed Markdown, such as an unclosed code fence, the highlighting doesn't stop at the closing """. The rest of the module is highlighted as Markdown:

-doc """
Unclosed code fence:
```erlang
foo() -> ok.
""".
f() -> ok.   % highlighted as Markdown

Stop Markdown in doc attributes from leaking past the closing quotes

doc-directive included text.html.markdown directly between its begin and end patterns. The end pattern is only tried once the Markdown grammar hands control back, which it never does while a code fence is open. The include is now wrapped in a begin/while rule whose while pattern stops at the first line starting with """. The while pattern is checked at the start of every line, before the Markdown grammar runs, so the Markdown ends there whatever state the Markdown grammar is in. This is the approach suggested in microsoft/vscode-markdown-tm-grammar#175 (comment).

The fix has been in the ELP VS Code extension's copy of this grammar since March 2026 (WhatsApp/erlang-language-platform@ddeaba9b90). This PR brings it upstream.

Load a Markdown stand-in grammar in the tests

Until now, the tests had no coverage of doc-directive. They ran without a grammar for text.html.markdown, and when a rule's only include can't be found, vscode-textmate silently drops the rule. -doc and -moduledoc were therefore highlighted as generic directives, and the docstring and sigil snapshots recorded that.

This PR adds a minimal grammar for text.html.markdown under tests/grammars/, loads it with -g in both test commands, and regenerates the two snapshots. The only snapshot changes are doc attributes now being highlighted as meta.directive.doc.erlang, as they are in an editor that has the Markdown grammar.

Tests

tests/doc_markdown.erl covers an unclosed ``` fence in -doc and an unclosed ~~~ fence in -moduledoc. Without the fix, it fails: the closing """ and the next function head are highlighted as Markdown.

Co-authored with @bgsmeta.

robertoaloi and others added 2 commits September 30, 2026 10:50
The doc-directive rule only includes text.html.markdown. The tests ran
without a grammar for that scope, and vscode-textmate drops a rule whose
only patterns cannot be resolved, so -doc and -moduledoc attributes fell
back to the generic directive rule and the doc-directive rule was never
exercised. The docstring and sigil snapshots recorded that fallback.

Add a minimal grammar for text.html.markdown under tests/grammars, load
it in both test commands and regenerate the two snapshots. Their only
changes are doc attributes now being scoped meta.directive.doc.erlang,
as they are in an editor with the Markdown grammar available.
The doc-directive rule included text.html.markdown directly between its
begin and end patterns. The end pattern is only tried once the embedded
grammar hands control back, so malformed Markdown such as an unclosed
code fence kept consuming lines past the closing """ and highlighted the
rest of the module as Markdown.

Wrap the include in a begin/while rule whose while pattern stops at the
first line starting with """. The while condition is checked at the
start of every line before the embedded grammar runs, so the Markdown
region ends there regardless of the state the Markdown grammar is in.
This is the approach suggested in
microsoft/vscode-markdown-tm-grammar#175 (comment)

Co-authored-by: Balaji S <bgs@meta.com>
@robertoaloi
robertoaloi merged commit 151758b into main Sep 30, 2026
2 checks passed
meta-codesync Bot pushed a commit to WhatsApp/erlang-language-platform that referenced this pull request Sep 30, 2026
Summary:
Sync both copies of the Erlang TextMate grammar (the ELP extension and the internal `nuclide.erlang` extension) with https://github.com/erlang-ls/grammar at `49f7a9a`. Both copies now match upstream byte for byte.

Changes since the last sync:

- erlang-ls/grammar#19: named fun expressions (`fun Name(...) -> ... end`) are highlighted correctly, and no longer scope the rest of the file as an implicit fun. Zero-arity fun expressions such as `fun() -> ok end` are keywords instead of function types. The function type rule now only applies inside types (`-type`, `-opaque`, `-spec`, `-callback`, typed record fields), via a new `meta.type.erlang` scope. Fixes pgourlain/vscode_erlang#316 and #317.
- erlang-ls/grammar#20: a fun type after a top-level range (`-type t() :: 1..10 | fun(() -> ok).`) or in a `-nominal` type no longer scopes the rest of the file as a fun expression.
- erlang-ls/grammar#21: upstreams the fix for Markdown leaking out of `-doc`/`-moduledoc` attributes (D98300495), which only lived in these copies. No behaviour change here.
- erlang-ls/grammar#22: follow-ups from the review of this diff. Named funs called `_` (`fun _(A) -> A end`) are highlighted as named funs. `README.md` typos fixed (`ignode`, `./test/snap`) and the snapshot update command loads the Markdown test grammar. Upstream also adds tests showing that a comma nested in a record field type does not end the `meta.type.erlang` region, and that a default value stays an expression.

`README.md` is synced too (test instructions and references); `LICENSE` is unchanged.

Reviewed By: jcpetruzza

Differential Revision: D122532634

fbshipit-source-id: cabfee7809fcd0e0977d700bc71b8b50db8cd02f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants