httpx version:
v1.10.0
Current Behavior:
Passing -json (or -csv/-fpt) unconditionally initializes the dit page-type classifier (runner.go, if options.JSONOutput || options.CSVOutput || len(options.OutputFilterPageType) > 0 { dit.New() }). On a machine with no cached model, dit.New() downloads a ~92 MB model.json from Hugging Face (ModelURL in happyhackingspace/dit), logging INFO Model downloaded size=92.6MB to stderr.
There is no flag or env var to opt out.
Expected Behavior:
A flag (e.g. -no-classify) to use -json/-csv output without page-type classification, leaving ditClassifier unset classifyPage already early-returns when r.ditClassifier == nil, so it looks like a one-line guard.
Steps To Reproduce:
- Remove any cached model: rm -rf ~/.dit
- Run: echo https://example.com | httpx -json
- Observe httpx downloads the ~92 MB model and logs Model downloaded size=... — with no way to disable it.
Anything else:
We consume only core JSON fields (not page_type/forms), but must pass -json to parse output, which forces the classifier and its model download.
Workaround is pre-seeding ~/.dit/model.json (even {}), but an opt-out flag would be cleaner.
httpx version:
v1.10.0
Current Behavior:
Passing -json (or -csv/-fpt) unconditionally initializes the dit page-type classifier (runner.go, if options.JSONOutput || options.CSVOutput || len(options.OutputFilterPageType) > 0 { dit.New() }). On a machine with no cached model, dit.New() downloads a ~92 MB model.json from Hugging Face (ModelURL in happyhackingspace/dit), logging INFO Model downloaded size=92.6MB to stderr.
There is no flag or env var to opt out.
Expected Behavior:
A flag (e.g. -no-classify) to use -json/-csv output without page-type classification, leaving ditClassifier unset classifyPage already early-returns when r.ditClassifier == nil, so it looks like a one-line guard.
Steps To Reproduce:
Anything else:
We consume only core JSON fields (not page_type/forms), but must pass -json to parse output, which forces the classifier and its model download.
Workaround is pre-seeding ~/.dit/model.json (even {}), but an opt-out flag would be cleaner.