feat(format): add IndexMetadata.covering_fields for covering indexes - #8535
Conversation
|
Important This PR touches the Lance format specification. Substantive changes to the format specification — the If this is a meaningful format change:
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
7fd9be0 to
4fd22c3
Compare
4fd22c3 to
f3e41ae
Compare
|
ACTION NEEDED The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. For details on the error please inspect the "PR Title Check" action. |
f3e41ae to
a301d96
Compare
a301d96 to
6c0fce1
Compare
1c5f2b7 to
9dac14a
Compare
9dac14a to
314ce5a
Compare
314ce5a to
a850532
Compare
Adds `repeated int32 covering_fields = 11` -- the columns an index stores alongside its own data so a covered query can skip the base-table take. These columns are listed in `fields` too, as its trailing entries. `fields` answers what invalidates the index; the keyed prefix, `fields.len() - covering_fields.len()`, answers what it can serve. No index builder writes carried values yet; the creation API follows separately. Note: this is additive on the wire. Field 11 is ignored by readers that predate it, and manifests written before it decode to an empty declaration.
a850532 to
820bac7
Compare
There was a problem hiding this comment.
The documentation now makes explicit that carried columns participate in fragment invalidation and overlay staleness masking. That matches the existing full-fields dependency contract, and the non-documentation implementation is unchanged from the previously reviewed revision.
The preview-release risk is unchanged: v11.0.0-beta.4 through beta.17 treated reclaimed bit 128 as the retired MemWAL flag, so those builds can open a newly covered dataset and misread carried fields as keyed or erase the new declaration on rewrite. Keeping bit 256 would avoid that exposure, but preview releases explicitly carry no stability guarantee, so no further change is requested here.
|
The vote has passed, merging! |
Adds
repeated int32 covering_fields = 11-- the columns an index storesalongside its own data so a covered query can skip the base-table take. These
columns are listed in
fieldstoo, as its trailing entries.fieldsanswerswhat invalidates the index; the keyed prefix,
fields.len() - covering_fields.len(), answers what it can serve.Nothing populates the field yet; the creation API follows separately.
Note: this is additive on the wire. Field 11 is ignored by readers that predate
it, and manifests written before it decode to an empty declaration