Skip to content

fix(io): normalize OpenDAL listed paths - #8654

Open
lance-gatefixer[bot] wants to merge 1 commit into
mainfrom
gatekeeper/fix-8652-1
Open

fix(io): normalize OpenDAL listed paths#8654
lance-gatefixer[bot] wants to merge 1 commit into
mainfrom
gatekeeper/fix-8652-1

Conversation

@lance-gatefixer

Copy link
Copy Markdown
Contributor

Summary

  • route every OpenDAL-backed provider through a Lance adapter that preserves raw listed paths
  • normalize recursive, offset, and delimiter listing results with Path::from_url_path
  • retain native start-after pushdown where supported and compare normalized paths in the bridge fallback
  • add an in-memory regression test covering a dataset path containing ~

Root cause

object_store_opendal rebuilds OpenDAL listing entries with Path::from, which percent-encodes reserved characters. Lance deliberately constructs the dataset base with Path::from_url_path, leaving the decoded UTF-8 path in Path. The encoded listing location therefore could not match or be made relative to the raw dataset base.

Validation

  • cargo test -p lance-io object_store::opendal_store::tests::test_list_preserves_raw_locations
  • cargo test -p lance-io --lib -- --skip uring::tests (201 passed)
  • cargo check -p lance-io --all-features --tests
  • cargo fmt --all -- --check
  • cargo clippy --all --tests --benches -- -D warnings

The complete lance-io library test command cannot finish in this container because io_uring initialization is denied by the environment; the non-io_uring library suite passes.

Fixes #8652

@github-actions github-actions Bot added A-encoding Encoding, IO, file reader/writer bug Something isn't working labels Aug 20, 2026
@lance-gatefixer

Copy link
Copy Markdown
Contributor Author

Blocked: the current head is held by an unrelated intermittent macOS BQ test failure that this App cannot rerun.

Remote head 1d7f8e3 contains the current main tip b945b3e. The mac-build (stable) Check failed vector::bq::storage::tests::test_accurate_approx_mode_reduces_binary_lut_quantization_error with accurate_error=0.07851982 and normal_error=0.06913757; this PR changes only lance-io, and the identical base SHA passed the macOS Check.

I fetched and verified base ancestry, inspected the live log, ran both the failed BQ test and the OpenDAL regression successfully, and pushed the assigned branch, which was already up to date. A failed-workflow rerun was requested, but the GitHub App lacks Actions rerun permission.

The smallest unblock is for a maintainer to rerun the failed mac-build (stable) job; alternatively, rerun the full Rust workflow on this head.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-encoding Encoding, IO, file reader/writer bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: OpenDAL-backed stores list percent-encoded paths while dataset.base stays raw

0 participants