vfs: add --vfs-mount and --vfs-load startup flags - #65748
Conversation
|
Review requested:
|
The zip support in node:zlib is released (v26.8.0) and ZipProvider merged today as nodejs/node#64915, so the old framing — "three things that are not in any release", pointing at #64339 and pipobscure/node#3 — was wrong in both directions: it undersold what has landed and misnamed what has not. What is actually outstanding is nodejs/node#65748, the --vfs-mount / --vfs-load flags and the vfs.registerProvider() that ships with them, plus nodejs/node#65680 for loading native addons out of a mount. Reading those two turned up semantics this repo's prose had wrong: --vfs-load runs the *first* mount (or --vfs-load=<index>), not the last; --vfs-mount takes no target, since node assigns a reserved mount point; argv[1] is the mounted source's real path rather than the mount point; and registered providers are now offered directories as well as files, which retires the constraint that made tools/observe.ts a runner rather than a preload. The deck's status slide said "This is in Node" with the flags marked as landed. It now says "landing", carries the two open PRs and the addon work as their own rows, and the speaker notes say plainly that the keystone is still a pull request. Deck republished to the artifact link in the README.
nodejs/node#65748 changed the flag on 2 Sep: --vfs-load used to select one of the --vfs-mount sources by 0-based index, which meant counting mounts out by hand and an optional-valued flag the options parser could only express as an alias onto a hidden --vfs-load-index. It now takes the source itself and mounts it, so `--vfs-load --vfs-mount X` is gone and `--vfs-load=X` does both. The launcher prefix was the breaking part: shell-base still exec'd node with the two-flag form, which a current build rejects outright with "--vfs-load requires an argument", so every archive signed with --launcher would have failed to run. It is now `--vfs-load="$0"`, and 77 bytes rather than 89. mountArgv() and the test helper follow, along with the command lines in the comments, the README, HISTORY, the deck and the release workflow's header. Two consequences worth recording rather than only fixing: the self-mounting shebang is one flag now, since the kernel-appended path becomes --vfs-load's value, and NODE_OPTIONS mounts sort before the command line's rather than after, because with no index to protect the order cannot change what runs. Verified against a node built from that branch (v27.0.0-pre): the whole suite passes, 136 of 136, and a signed launcher runs by name with dash-leading arguments reaching the program. While in there: tools/observe.ts explained itself with a constraint that no longer holds — registered providers are offered directories too — so it now gives the real reason it is a runner, which is that it wants the mount point.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #65748 +/- ##
==========================================
+ Coverage 89.95% 90.62% +0.66%
==========================================
Files 759 771 +12
Lines 258637 269564 +10927
Branches 49015 55308 +6293
==========================================
+ Hits 232665 244281 +11616
+ Misses 17018 16480 -538
+ Partials 8954 8803 -151
🚀 New features to boost your workflow:
|
Mounting a virtual file system requires calling vfs.mount() from inside the program, so a program cannot itself be served from one: something already running has to mount the VFS first. Add two startup flags. --vfs-mount=<source> mounts a directory or an archive as a virtual file system, and may be repeated. --vfs-load=<source> mounts that source the same way and additionally runs the entry point and all subsequent require()/import resolution against it rather than against the real file system, so an application can be run straight out of a directory or a ZIP archive: node --experimental-vfs --vfs-load=my-app.zip Both options append to one list, so mounts happen in the order written: node --experimental-vfs --vfs-mount=a --vfs-load=b --vfs-mount=c mounts a, b and c in that order and runs b. Mounting the same source twice mounts it twice, at two mount points, and the entry point comes from the mount --vfs-load contributed rather than the earlier one. The provider backing a source is chosen from the source itself rather than from its name: a directory is served by RealFSProvider and a file whose bytes are a ZIP archive by ZipProvider, so an archive can carry any extension. vfs.registerProvider() registers a provider for formats there is no built-in for; selection is deferred until -r and --import preloads have run, so a preloaded module can register one before its source is claimed. The entry point comes from the mount the way `node <directory>` takes one, from package.json "main" or index.js. Nothing is consumed as an entry point argument, so every positional reaches the program unchanged from argv[2] on, and argv[1] reports the named source rather than the reserved mount point, which is an opaque implementation detail. --vfs-load may be given at most once, and is not permitted in NODE_OPTIONS: which entry point runs is the command line's decision, and NODE_OPTIONS is parsed first, so an environment variable could otherwise redirect any invocation. --vfs-mount is permitted there. Workers inherit the same mounts in the same order but not the loading: a worker mounts what the parent mounted and runs its own entry point. Signed-off-by: Philipp Dunkel <pip@pipobscure.com>
|
Rebased because of documentation conflict. |
|
As per our practice and documentation those should be |
This comment was marked as outdated.
This comment was marked as outdated.
They are already behind --experimental-vfs to begin with, so does that still hold? Happy to oblige if so, just double checking. Precedence: It's --allow-vfs-fs and not --experimental-allow-vfs-fs |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
it looks like there is something on CI that‘s repeating to fail. can someone let me know where that is and if there is something I can do about it? |
The module loader manufactures paths under the reserved VFS root that no layer owns: resolving a mount point as a directory first probes the sibling names `<mount>.js`, `<mount>.json` and `<mount>.node`, and a package.json walk-up passes the parents of the mount point. The lookup declined those because their layer segment is not a plain id, so they fell through to the native loader and the real file system. On POSIX that is harmless (ENOTDIR under /dev/null), but on Windows the root sits under `\\.\nul`, and `\\.\nul\<anything>` opens the NUL device: libuv reports it as a character device and a read returns nothing. The loader therefore picked `\\.\nul\vfs\<id>.js` as an existing file, and the native walk-up above it then read the device as an empty package.json and failed with ERR_INVALID_PACKAGE_CONFIG for `\\.\nul\package.json`. Any require() of a mount point hits this on Windows. Distinguish "under the root but unowned" from "outside the root" in the lookup and have every loader override report the former as not found: stat gives ENOENT, reads and realpath throw ENOENT, the package.json lookups return their "no package.json" results, and upward walks stop at the reserved root. Paths outside the root still go to the native loader as before. Refs: nodejs#65748 Signed-off-by: Philipp Dunkel <pip@pipobscure.com>
87aee89 to
0c4f4fc
Compare
|
It looks like the CI for this exercises the path fixed in #65814 so this PRs CI depends on being rebased on that. |
vfs: add --vfs-mount and --vfs-load startup flags
Mounting a virtual file system requires calling
vfs.mount()from inside theprogram, so a program cannot itself be served from one — something already
running has to mount the VFS first.
This adds two startup flags:
--vfs-mount=<source>mounts a directory or an archive as a virtual filesystem. May be repeated.
--vfs-load=<source>mounts<source>exactly as--vfs-mountdoes, andadditionally runs the entry point and all subsequent
require()/importresolution against that mount rather than the real file system.
Together they let an application run straight out of a directory or a ZIP
archive:
$ node --experimental-vfs --vfs-load=my-app.zipMount order
Both options append to one list, so mounts happen in the order written:
$ node --experimental-vfs --vfs-mount=a --vfs-load=b --vfs-mount=cmounts
a,bandcin that order and runsb. Mounting the same sourcetwice mounts it twice, at two separate mount points; the entry point comes from
the mount
--vfs-loadcontributed, not from an earlier--vfs-mountof thesame source.
Choosing a provider
The provider backing a source is chosen from the source itself rather than from
its file name: a directory is served by
RealFSProvider, and a file whose bytesare a ZIP archive by
ZipProvider, so an archive can carry any extension.vfs.registerProvider()registers a provider for formats there is no built-infor. Selection is deferred until
-rand--importpreloads have run, so apreloaded module can register one before its source is claimed.
Entry point and arguments
The entry point is taken from the mount the same way
node <directory>takesone: the mount's own
package.json"main", orindex.js. Because the entrypoint comes from the mount, no positional argument is consumed as one — every
positional reaches the program unchanged from
argv[2]onward.process.argv[1]reports the named source rather than the reserved mount point, which is an
opaque implementation detail.
Constraints
--vfs-loadmay be given at most once; a second is rejected at startup.--vfs-loadis not permitted inNODE_OPTIONS: which entry point runs is thecommand line's decision, and
NODE_OPTIONSis parsed first, so an environmentvariable could otherwise redirect any invocation.
--vfs-mountis permittedthere and its mounts precede the command line's.
worker mounts what the parent mounted and runs its own entry point.
--experimental-vfs.Notes for reviewers
--vfs-mountand--vfs-loadshare onevfs_mountslist so ordering ispreserved by construction; a bracketed internal boolean set via
Implies()records that a load was requested, and which entry it contributed is recovered
from
execArgv.in
EnvironmentOptions::CheckOptions(), which runs at the end of every parse —otherwise
NODE_OPTIONS=--vfs-mount=x node --experimental-vfsis rejected foran
--experimental-vfsthat has not been read yet.separately and is independent of this change.
For context what this gives us see bundling tools and slides