fix(index): skip unrecognized index types

[lance-gatefixer]

Summary

• exclude index detail types this build cannot resolve from automatic vector index selection
• skip unrecognized logical indices during optimize_indices with a warning so supported indices can still be maintained
• preserve the foreign segment and cover vector fallback plus unrelated BTree maintenance in a regression test

Root cause

load_indices intentionally retains metadata for unknown index types, but vector planning selected any index on the queried field and index optimization attempted every non-system logical index without checking whether IndexDetails::index_version() could resolve its reader. Both paths therefore tried to open a type this build could not read.

Validation

• cargo test -p lance test_unknown_index_type_does_not_block_queries_or_optimization
• cargo fmt --all -- --check
• cargo clippy --all --tests --benches -- -D warnings

Fixes #8528

[lance-gatekeeper]

✅ Gate recommendation: approve.

The feature patch is unchanged after the beta.20 base refresh. Exact, case-insensitive reader identities—including released 0.36 scalar aliases—continue to preserve opaque metadata and legacy fallback while supported queries and index maintenance remain available.

[lance-gatefixer]

Blocked: The current-head Rust linux-arm check is failing from an inherited DataFusion memory-pool exhaustion outside this repair diff.

At remote head c86443bd668cb5b2b2eeec6a162fbd7b2d255034, the linux-arm job 96953077588 failed its test step when an external-sort merge requested 40.0 MB with only 30.0 MB available in a shared 150.0 MB pool. Current main tip 55f5ac5cdd778df673598bd58e531b54b74e1ec3 is an ancestor of this head. The failing JSON BTree test and allocation path are unchanged from main, and the exact test passes under the locked CI profile with every lance-index feature, so the evidence does not support an attributable in-scope code change.

Re-run the failed linux-arm job as the smallest next action. If it reproduces, address the shared DataFusion test memory or concurrency configuration in a dedicated fix; reducing the test runtime sort-spill reservation is a practical alternative.