fix(haskell): stop escaped char literals at the closing quote (#2439)

[Zhuoxi2000]

What does this PR do?

Fixes #2439

When a Haskell definition contains an escaped char literal ('\n') and the next definition has a primed name (g'), the first definition swallows the second. With the repro from #2439, f spans lines 3-5 and g' gets no node. The parse also reports has_error even though the module is valid. The same bug has a one-line form: ['\n','\t'] lexes as a single char token and reports no error.

Cause. The scanner is not the culprit. Patching take_char_literal in scanner.c changed nothing, because that path already handles '\n' correctly. The fault is in the grammar's internal lexer. grammar/literal.js defines char: choice(/'[^']'/, /'\\[^ ]*'/). In the escape branch, [^ ]* also matches newlines and quotes, and tree-sitter keeps the longest match, so the token runs to the last ' before the next space. In the repro that is the prime in g'. The regex is the same at our pin 7fa19f195803 and on upstream master (97288e585b0b, 2026-09-30), so there is no upstream fix to pull.

Change. This is a hand patch to the generated char lex states in ts_lex in internal/cbm/vendored/grammars/haskell/parser.c. The diff is +2/-9.

• State 28 (escape body): also stops at \t..\r, not only at space.
• States 112, 114 and 116 (after the closing quote, optionally after #): no longer loop back into the escape body.
• State 114: takes one more ' and moves to state 115. That keeps '\'' a single token.
• Resulting token: '\ [^'\s]* ' '?, followed by up to two # as before.

The patch is recorded as a row in vendored/grammars/MANIFEST.md under "Local source patches". scripts/security-vendored.sh --update refreshes the MANIFEST.md and parser.c lines in scripts/vendored-checksums.txt. No other checksum line changes.

Literals that still lex as one token, unchanged: '\'', '\\', '\x41', '\SOH', '\^A', '\1234', 'a', '"', MagicHash '\n'#, and TH quotes ('foo, ''Bar).

One trade-off, also noted in the MANIFEST row: an escaped char followed directly by a stray ', as in '\n'', is taken as one token. That input is not valid Haskell.

If you would rather fix this in the grammar, the alternative is to narrow the regex in literal.js, for example to /'\\('|[^'\s]+)'/, and regenerate parser.c with tree-sitter 0.25.10. I can rework the PR that way, or send the regex change to tree-sitter-haskell as well.

Checklist

• Every commit is signed off (git commit -s). scripts/check-dco.sh origin/main..HEAD reports OK for both commits.
• Tests pass locally, with one caveat: the ASan+UBSan runner from make -f Makefile.cbm test hangs on my machine during ASan shadow-memory init, before main() runs. That is a host issue (macOS 26.5, Darwin 25.5, Apple clang 17.0.0) and unrelated to this change. An unsanitized runner was used instead, make -f Makefile.cbm -j4 SANITIZE= BUILD_DIR=build/nosan build/nosan/test-runner, and the sanitized run is left to CI. Results with the fix:
  • extraction: 410 passed
  • pipeline: 318 passed
  • language: 241 passed
  • lang_contract: 41 passed
  • node_creation_probe: 80 passed
  • grammar_regression: 1 passed
  • grammar_labels: 2 passed
  • grammar_imports: 1 passed
  • security: 38 passed
  • scripts/security-vendored.sh: passed (1083 files checked, Vendored integrity check passed)
  • tests/test_vendored_integrity_contract.sh: OK
• Lint passes:
  • make -f Makefile.cbm lint-cppcheck passed with cppcheck 2.20.0, built from the tag CI uses, on the pre-rebase base. This PR changes no file in LINT_SRCS, so it cannot change the cppcheck result on main.
  • clang-format 20.1.8 with --dry-run --Werror on tests/test_extraction.c reports one violation. It already exists on main (the parseJsonBody test) and is outside the lines this PR adds.
  • lint-no-suppress and scripts/lint-memory-core.py pass.
  • clang-tidy was not run; it is the local pre-commit linter and not part of lint-ci.
• New behavior is covered by a test (reproduce-first). Both tests are in tests/test_extraction.c, and the first commit adds only these tests:
  • haskell_escaped_char_does_not_swallow_primed_def is the Haskell: an escaped char literal followed by a primed name swallows the next definition (vendored tree-sitter-haskell) #2439 repro. It asserts f ends on line 3, g' and h exist, and there is no parse error.
  • haskell_escaped_char_list_lexes_each_literal checks the one-line form. It asserts that ['\n','\t','\'','\\','\x41','\SOH','a'] gives seven separate char tokens with no parse error.

Test evidence

Before (test commit only, on main 268a9d8):

$ make -f Makefile.cbm -j4 SANITIZE= BUILD_DIR=build/nosan build/nosan/test-runner
$ ./build/nosan/test-runner extraction
  haskell_escaped_char_does_not_swallow_primed_def         FAIL tests/test_extraction.c:1458: (int)f->end_line == 5, expected 3 == 3
  haskell_escaped_char_list_lexes_each_literal             FAIL tests/test_extraction.c:1507: "'\n','\t','\'','\\','\x41','\SOH','a' " != "'\n' '\t' '\'' '\\' '\x41' '\SOH' 'a' "
  408 passed, 2 failed

After (this PR):

$ ./build/nosan/test-runner extraction
  haskell_function                                       PASS
  haskell_escaped_char_does_not_swallow_primed_def       PASS
  haskell_escaped_char_list_lexes_each_literal           PASS
  haskell_data_type                                      PASS
  410 passed

This PR and #2457 (Haskell infix names) both add tests near the Haskell block in tests/test_extraction.c, so whichever merges second will need a small rebase. I'll handle that.

AI assistance: this change was drafted with an AI coding assistant (Claude) and verified locally with the tests above.

[github-actions]

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

• It will not be closed for inactivity. No stale bot touches pull requests here.
• It may still sit a while before a human reads it. That is on us, not on you.
• Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

• Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
• Get CI green, or say which failures you believe are pre-existing.
• Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
• Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

[icecold009]

AI-assisted review using OpenAI Codex and TypeSafe Jev (jev-1.13.0), on behalf of @icecold009. This is independent community feedback, not a maintainer decision.

I reviewed all four changed files and traced the generated lexer transitions for whitespace/newline stopping, escaped apostrophe handling, and the optional # suffix. I found no actionable defect in the patch. Both added regression cases ran in the hosted extraction suite (410 passed, 0 failed) on this head.

The current required PR / ci-ok check is failing because the Windows CLANG64 shard fails daemon_conflict_log_windows_concurrent_appends_are_not_dropped (tests/test_daemon_version.c:686). That failing test is outside the changed files; I could not establish a link to this Haskell parser patch. Please confirm it reproduces on the base or rerun before merge.

[Zhuoxi2000]

Drafted with an AI coding assistant (Claude).

Thanks @icecold009 for going through the lexer changes.

On the red ci-ok: the only failure is daemon_conflict_log_windows_concurrent_appends_are_not_dropped (tests/test_daemon_version.c:686) in the Windows CLANG64 1/2 shard. This PR only touches the vendored Haskell grammar, its checksums and tests/test_extraction.c, so it doesn't reach the daemon conflict log. The same shard passed on the eight most recent PR runs I checked from 10-05 (#514 NestJS DI, the three Scala PRs, the Swift and TS ones, among others), so it looks like an intermittent failure in that concurrency test rather than something this change triggers. I can't re-run jobs on this repo; could a maintainer re-run the failed shard when convenient?