Skip to content

perf(arrow/array): validate dictionary indices by valid runs - #1181

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-dictionary-valid-runs
Open

perf(arrow/array): validate dictionary indices by valid runs#1181
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-dictionary-valid-runs

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Validate nullable dictionary indices using contiguous valid runs.
  • Keep the existing whole-slice min/max path for arrays without nulls.
  • Fall back to the whole-slice path for fragmented validity when all physical values are already in range.
  • Ignore out-of-range physical values at null positions.

Why

The current bounds check scans every physical index, including null positions. This does unnecessary work for clustered nullable arrays and can reject garbage payloads that are hidden by the validity bitmap.

The existing TODO points to the set-bit run reader for this case. This change uses it for up to eight valid runs. More fragmented bitmaps keep the vectorized whole-slice fast path. If that scan finds an invalid physical value, valid runs are checked again so null payloads are still ignored.

Benchmark

1M int32 indices on an Apple M1 Pro, 300 ms per sample, 6 samples:

Case Before After Change
all valid 88.96 us/op 92.68 us/op no significant change
10% valid, clustered 88.08 us/op 45.53 us/op 48.3% faster
50% valid, clustered 90.51 us/op 81.28 us/op 10.2% faster
alternating 86.83 us/op 88.58 us/op no significant change

All cases remain at 0 allocations/op.

Checks

  • go test ./arrow/...
  • go vet ./arrow/array
  • git diff --check

No public API changes.

@fallintoplace
fallintoplace marked this pull request as draft August 12, 2026 11:08
@fallintoplace
fallintoplace marked this pull request as ready for review August 18, 2026 18:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant