fs: implement glob natively

[avivkeller]

Ref: https://openjs-foundation.slack.com/archives/C019Y2T6STH/p1787069174254509

(AI assistance used to help port minimatch to C++ and take improvements from other implementations, e.g. Rust's fast-glob [oxc] and picomatch, etc).

I'd love some pretty pedantic reviews since this is substantial. It's 1:1 with the existing glob, so it's non-breaking, but please tear this apart just in case. This implementation is over 2x faster than the old one.

Benchmarking Result

                                                                                                       confidence improvement accuracy (*)    (**)   (***)
fs/bench-glob-options.js options='exclude-callback' mode='callback' pattern='**/*.js' dir='lib' n=1000        ***    115.93 %      ±19.24% ±26.59% ±36.73%
fs/bench-glob-options.js options='exclude-callback' mode='promise' pattern='**/*.js' dir='lib' n=1000         ***    115.54 %      ±20.02% ±27.26% ±36.78%
fs/bench-glob-options.js options='exclude-callback' mode='sync' pattern='**/*.js' dir='lib' n=1000            ***     69.66 %      ±14.00% ±18.97% ±25.42%
fs/bench-glob-options.js options='exclude-pattern' mode='callback' pattern='**/*.js' dir='lib' n=1000         ***     37.86 %      ±12.27% ±16.90% ±23.19%
fs/bench-glob-options.js options='exclude-pattern' mode='promise' pattern='**/*.js' dir='lib' n=1000          ***     41.06 %      ±11.48% ±15.49% ±20.60%
fs/bench-glob-options.js options='exclude-pattern' mode='sync' pattern='**/*.js' dir='lib' n=1000                      3.81 %      ±11.93% ±16.14% ±21.56%
fs/bench-glob-options.js options='none' mode='callback' pattern='**/*.js' dir='lib' n=1000                    ***    153.86 %      ±20.91% ±28.39% ±38.12%
fs/bench-glob-options.js options='none' mode='promise' pattern='**/*.js' dir='lib' n=1000                     ***    150.49 %      ±18.20% ±24.64% ±32.95%
fs/bench-glob-options.js options='none' mode='sync' pattern='**/*.js' dir='lib' n=1000                        ***     98.16 %      ±11.23% ±15.15% ±20.14%
fs/bench-glob-options.js options='withFileTypes' mode='callback' pattern='**/*.js' dir='lib' n=1000           ***    111.98 %      ±19.57% ±26.99% ±37.14%
fs/bench-glob-options.js options='withFileTypes' mode='promise' pattern='**/*.js' dir='lib' n=1000            ***    111.56 %      ±17.41% ±23.58% ±31.53%
fs/bench-glob-options.js options='withFileTypes' mode='sync' pattern='**/*.js' dir='lib' n=1000               ***     76.30 %      ±15.82% ±21.46% ±28.80%
fs/bench-glob.js mode='callback' pattern='*.js' dir='lib' n=1000                                              ***     64.04 %       ±6.85%  ±9.31% ±12.51%
fs/bench-glob.js mode='callback' pattern='**/*.{js,json}' dir='lib' n=1000                                    ***    181.99 %       ±5.73%  ±7.78% ±10.46%
fs/bench-glob.js mode='callback' pattern='**/*.js' dir='lib' n=1000                                           ***    162.05 %       ±7.08%  ±9.77% ±13.47%
fs/bench-glob.js mode='callback' pattern='**/*' dir='lib' n=1000                                              ***    167.88 %       ±9.26% ±12.60% ±16.98%
fs/bench-glob.js mode='promise' pattern='*.js' dir='lib' n=1000                                               ***     46.39 %       ±2.77%  ±3.78%  ±5.12%
fs/bench-glob.js mode='promise' pattern='**/*.{js,json}' dir='lib' n=1000                                     ***    167.71 %       ±7.07%  ±9.74% ±13.39%
fs/bench-glob.js mode='promise' pattern='**/*.js' dir='lib' n=1000                                            ***    158.22 %      ±13.81% ±19.17% ±26.64%
fs/bench-glob.js mode='promise' pattern='**/*' dir='lib' n=1000                                               ***    156.03 %      ±11.17% ±15.48% ±21.47%
fs/bench-glob.js mode='sync' pattern='*.js' dir='lib' n=1000                                                  ***     29.58 %       ±4.99%  ±6.92%  ±9.59%
fs/bench-glob.js mode='sync' pattern='**/*.{js,json}' dir='lib' n=1000                                        ***    112.35 %       ±6.09%  ±8.24% ±11.01%
fs/bench-glob.js mode='sync' pattern='**/*.js' dir='lib' n=1000                                               ***     93.18 %       ±5.53%  ±7.60% ±10.38%
fs/bench-glob.js mode='sync' pattern='**/*' dir='lib' n=1000                                                  ***     85.16 %      ±13.12% ±18.20% ±25.31%

Be aware that when doing many comparisons the risk of a false-positive
result increases. In this case, there are 24 comparisons, you can thus
expect the following amount of false-positive results:
  1.20 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.24 false positives, when considering a   1% risk acceptance (**, ***),
  0.02 false positives, when considering a 0.1% risk acceptance (***)

AI-Assisted-By: Claude Fable 5, Claude Opus 5

[nodejs-github-bot]

Review requested:

• @nodejs/actions
• @nodejs/performance
• @nodejs/security-wg
• @nodejs/tsc

[avivkeller]

cc @isaacs @nodejs/fs

[codecov]

Codecov Report

❌ Patch coverage is 77.74495% with 904 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.98%. Comparing base (f955c20) to head (1d62b51).
⚠️ Report is 52 commits behind head on main.

Files with missing lines	Patch %	Lines
src/glob/glob_parser.cc	73.97%	149 Missing and 106 partials ⚠️
src/glob/glob_matcher.cc	66.80%	82 Missing and 84 partials ⚠️
src/glob/glob_walker.cc	85.77%	47 Missing and 112 partials ⚠️
src/glob/glob_program.cc	80.48%	78 Missing and 44 partials ⚠️
src/glob/node_glob.cc	75.45%	60 Missing and 47 partials ⚠️
src/glob/glob_unicode.cc	40.17%	64 Missing and 3 partials ⚠️
src/permission/permission.cc	79.31%	1 Missing and 5 partials ⚠️
src/glob/glob_ast.h	85.29%	2 Missing and 3 partials ⚠️
src/glob/glob_unicode.h	33.33%	4 Missing ⚠️
src/glob/node_glob.h	42.85%	4 Missing ⚠️
... and 4 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #65392      +/-   ##
==========================================
- Coverage   90.17%   89.98%   -0.19%     
==========================================
  Files         771      784      +13     
  Lines      265172   268265    +3093     
  Branches    50364    51099     +735     
==========================================
+ Hits       239119   241411    +2292     
- Misses      17010    17409     +399     
- Partials     9043     9445     +402     

Files with missing lines	Coverage Δ
lib/fs.js	97.31% <100.00%> (+<0.01%)	⬆️
lib/internal/fs/watchers.js	88.06% <100.00%> (-0.22%)	⬇️
lib/internal/test_runner/runner.js	94.94% <100.00%> (-0.01%)	⬇️
src/glob/glob_program.h	100.00% <100.00%> (ø)
src/glob/glob_walker.h	100.00% <100.00%> (ø)
src/lru_cache-inl.h	90.47% <100.00%> (+1.58%)	⬆️
src/node_binding.cc	70.76% <ø> (+0.27%)	⬆️
src/node_external_reference.h	100.00% <ø> (ø)
src/node_snapshotable.cc	73.47% <ø> (ø)
src/node_util.cc	81.96% <ø> (-0.19%)	⬇️
... and 19 more

... and 55 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[aduh95]

take improvements from other implementations, e.g. Rust's fast-glob [oxc] and picomatch, etc

I would feel more comfortable adopting one of those instead of having to maintain our own, have you explored that route?

[avivkeller]

I would feel more comfortable adopting one of those instead of having to maintain our own, have you explored that route?

I have, but the perf wins here are dependent on this being in native, and not in JS:

1. (Minor) We are calling libuv directly, so we don't need to flip-flop between C++ and JS

2. (Major) We are writing code highly optimized for Node.js's usage. Existing JS implementations compile to a RegEx and rely on V8 to optimize it, however, we can skip that whole step by not relying on V8 at all, and manually compiling the glob to an AST.

   If a directory has 100k entries and 500 match, existing implementations are forced to create and discard ~99.5k strings to get there (since they are written in JS), in a native implementation, we don't need to create, allocate, or garbage collect anything inside of V8, we can handle it all natively.

I also think this can bring some improvements to things like the test runner. For instance, --test-coverage-include/exclude is globbed on the native side (like a supercharged version of #65371), we should be able to bring this performance improvement elsewhere. Maybe this can help in similar ways in other places to?

[avivkeller]

(Whoops!)

[isaacs]

Here's the results of node-glob's benchmark script with this branch, as of fd617c2: https://gist.github.com/isaacs/abf2714edd59cb6803d36ff6b3407d22

Notes:

1. Correctness does not seem to be impacted in any way. That's great! Agents often take shortcuts in tasks like this, so it's good to see that the tests kept it well behaved ;)
2. The fs.globSync performance is a bit faster than node-glob. That's not unexpected, since it was already comparable, but it does seem to have improved.
3. The fs.glob performance (async) is still considerably slower. The only explanation I can imagine for this is that it's still stacking waits serially rather than parallelizing, or the more aggressive caching with path-scurry is saving node-glob from a lot of repetitive computations. (I know the node impl does cache, but it might not be caching as much or as aggressively; path-scurry effectively caches every path and fs operation for the duration of the walk.)
4. Performance of fast-glob and globby are there for scale, but they can be safely ignored; these implementations sacrifice correctness for speed, and thus are not apples-to-apples comparable.

Given the deep cuts made here, I'd expect to see the native implementation get at least 5 to 10 times faster than node-glob. No longer being 6x slower is indeed a significant improvement, perhaps enough to make it worth landing this as-is and making further improvements later. But it's definitely still not what I'd call "done", by quite a bit.

[aduh95]

I have, but the perf wins here are dependent on this being in native, and not in JS:

It's not what I meant though, my question is whether you explored adopting one of the native alternatives your AI based itself on (you mentioned Rust, surely that is not JS)

[avivkeller]

It's not what I meant though, my question is whether you explored adopting one of the native alternatives your AI based itself on (you mentioned Rust, surely that is not JS)

https://github.com/oxc-project/fast-glob, the Rust implementation I mentioned, isn't compliant with minimatch (e.g. negated patterns don't work). A lot of the really fast implementations aren't compatible with our current set up (as they match different globs). As @isaacs said about a different fast-glob (a different library of the same name), "these implementations sacrifice correctness for speed, and thus are not apples-to-apples comparable." That can be said about several implementations, so working from scratch was the best way to both match the existing behavior and take the performance benefits from the existing implementations.

[mcollina]

lgtm

[mcollina]

Minimatch had its own tests that covered it extensively. Those tests have not been ported over to this codebase, which I think should be to guarantee backward-compatibility and no regressions.

[avivkeller]

cc @nodejs/build, node -e const m = require('node:module'); if (m.builtinModules.includes('node:ffi')) process.exit(1); try { require('node:ffi'); } catch (e) { if (e.code === 'ERR_UNKNOWN_BUILTIN_MODULE') process.exit(0); } process.exit(1); is the reason these tests failed, but I didn't touch FFI?

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/76833/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/76888/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/76904/

[mcollina]

RSLGTM

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/77045/

[gurgunday]

Nice work! I've been playing with the implementation this morning and it lgtm overall

I just have one question/concern

[gurgunday]

lgtm

[louiellan]

hello, unsolicited review here ;)
I'm experiencing the same issue #61499 here in the native glob implementation but the problem for this is how the glob traverses through directories and instead of an empty match, it's unneeded permission error

IMO, I think that it is only reasonable for us to --allow-fs-read a parent directory if we were to have a followSymlinks set to true, or that we have ../,or that we are traversing the parent directory with the ** or *, otherwise, having --allow-fs-read=. for a specific directory traversal seems broad for me

But if it is intended that way that it needs to have granted access to the parent directory, I think it would be helpful for us to clarify it in the documentation

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/77313/

[avivkeller]

Landed in a8f2bf0