Skip to content

feat(format): add IndexMetadata.covering_fields for covering indexes - #8535

Merged
jackye1995 merged 1 commit into
lance-format:mainfrom
vivek-bharathan:vb/covering-format
Aug 21, 2026
Merged

feat(format): add IndexMetadata.covering_fields for covering indexes#8535
jackye1995 merged 1 commit into
lance-format:mainfrom
vivek-bharathan:vb/covering-format

Conversation

@vivek-bharathan

@vivek-bharathan vivek-bharathan commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Adds repeated int32 covering_fields = 11 -- the columns an index stores
alongside its own data so a covered query can skip the base-table take. These
columns are listed in fields too, as its trailing entries. fields answers
what invalidates the index; the keyed prefix,
fields.len() - covering_fields.len(), answers what it can serve.

Nothing populates the field yet; the creation API follows separately.

Note: this is additive on the wire. Field 11 is ignored by readers that predate
it, and manifests written before it decode to an empty declaration

@github-actions github-actions Bot added A-python Python bindings A-java Java bindings + JNI A-format On-disk format: protos and format spec docs A-namespace Namespace impls enhancement New feature or request breaking-change labels Aug 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

  • Start a vote following the Lance community voting process.
    Format specification modifications need 3 binding +1 votes (excluding the
    proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
  • Once the vote passes, link the completed vote in this PR. It should not be
    merged until the vote is linked.

lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 13, 2026
@codecov

codecov Bot commented Aug 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@lance-gatekeeper lance-gatekeeper Bot added K-changes Latest Gatekeeper recommendation requests changes. and removed K-changes Latest Gatekeeper recommendation requests changes. labels Aug 13, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Aug 13, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 13, 2026
@vivek-bharathan vivek-bharathan changed the title feat(format)!: declare covering (included) columns across the metadata contract feat(format): add IndexMetadata.included_fields for covering indexes Aug 14, 2026
@github-actions

Copy link
Copy Markdown
Contributor

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Aug 14, 2026
@vivek-bharathan vivek-bharathan changed the title feat(format): add IndexMetadata.included_fields for covering indexes feat(format): add IndexMetadata.included_fields for covering indexes Aug 14, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 14, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Aug 14, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 14, 2026
@vivek-bharathan vivek-bharathan changed the title feat(format)!: add IndexMetadata.included_fields for covering indexes feat(format): add IndexMetadata.included_fields for covering indexes Aug 20, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 20, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Aug 20, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 20, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 21, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Aug 21, 2026
@lance-gatekeeper lance-gatekeeper Bot removed K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Aug 21, 2026
Comment thread docs/src/format/index/index.md
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Aug 21, 2026
Adds `repeated int32 covering_fields = 11` -- the columns an index stores
alongside its own data so a covered query can skip the base-table take. These
columns are listed in `fields` too, as its trailing entries. `fields` answers
what invalidates the index; the keyed prefix,
`fields.len() - covering_fields.len()`, answers what it can serve.

No index builder writes carried values yet; the creation API follows
separately.

Note: this is additive on the wire. Field 11 is ignored by readers that predate
it, and manifests written before it decode to an empty declaration.
@lance-gatekeeper lance-gatekeeper Bot removed K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Aug 21, 2026
@vivek-bharathan vivek-bharathan changed the title feat(format): add IndexMetadata.included_fields for covering indexes feat(format): add IndexMetadata.covering_fields for covering indexes Aug 21, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Gate recommendation: approve with a non-blocking risk.

The documentation now makes explicit that carried columns participate in fragment invalidation and overlay staleness masking. That matches the existing full-fields dependency contract, and the non-documentation implementation is unchanged from the previously reviewed revision.

The preview-release risk is unchanged: v11.0.0-beta.4 through beta.17 treated reclaimed bit 128 as the retired MemWAL flag, so those builds can open a newly covered dataset and misread carried fields as keyed or erase the new declaration on rewrite. Keeping bit 256 would avoid that exposure, but preview releases explicitly carry no stability guarantee, so no further change is requested here.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Aug 21, 2026

@jackye1995 jackye1995 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

@jackye1995

Copy link
Copy Markdown
Contributor

The vote has passed, merging!

@jackye1995
jackye1995 merged commit 285eeb7 into lance-format:main Aug 21, 2026
41 of 42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-format On-disk format: protos and format spec docs A-java Java bindings + JNI A-namespace Namespace impls A-python Python bindings breaking-change enhancement New feature or request K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants