Skip to content

perf(parquet): decode levels directly into int16 batches - #1176

Merged
zeroshade merged 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-definition-level-int16
Aug 28, 2026
Merged

perf(parquet): decode levels directly into int16 batches#1176
zeroshade merged 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-definition-level-int16

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What changed

  • decode RLE and bit-packed levels directly into []int16
  • count max definition levels during decoding
  • remove the intermediate uint64 level batch and its second conversion pass
  • add benchmarks for all-defined, mostly-defined, and alternating levels
  • check partial value counts when a literal run is truncated

Benchmarks

Apple M1 Pro, -benchtime=300ms -count=5:

RLE workload Before After Change
1,024 all defined 2,137 ns/op 1,495 ns/op -30%
1,024 mostly defined 5,740 ns/op 5,329 ns/op -7%
65,536 all defined 61,213 ns/op 24,601 ns/op -60%
65,536 alternating 120,994 ns/op 89,543 ns/op -26%

Allocations were unchanged.

@fallintoplace
fallintoplace marked this pull request as draft August 11, 2026 21:46
@fallintoplace
fallintoplace marked this pull request as ready for review August 18, 2026 18:21

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The direct int16 level-decoding path preserves RLE and bit-packed semantics across partial reads, maximum levels, split decoding, and malformed inputs. Package tests, race tests, the full Parquet suite, randomized differential probes, benchmarks, and CI all pass.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow Go maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.

More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.

@zeroshade
zeroshade merged commit 85cf9f6 into apache:main Aug 28, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants