Skip to content

[#2783] fix(spark): correct compression ratio and failure counters in Spark UI - #2784

Open
wForget wants to merge 2 commits into
apache:masterfrom
wForget:UNIFFLE-2783
Open

[#2783] fix(spark): correct compression ratio and failure counters in Spark UI#2784
wForget wants to merge 2 commits into
apache:masterfrom
wForget:UNIFFLE-2783

Conversation

@wForget

@wForget wForget commented Aug 21, 2026

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

  • Calculate the Spark UI compression ratio using floating-point division.
    • Return 0.0 when the compressed shuffle size is zero.
    • Convert uncompressedShuffleBytes to Double before division.
  • Initialize failedTaskNumber and failedTaskMaxAttemptNumber to 0 instead of -1.

Why are the changes needed?

The compression ratio was calculated by dividing two Long values, so Scala performed integer division and discarded the fractional part.

For example, 150 / 100 was calculated as 1 and displayed as 1.0 instead of the expected 1.5.

Additionally, initializing the failure count to -1 caused the first failure to increment the count to 0, resulting in an incorrect failure summary. Starting the failure-related fields at 0 correctly represents the initial state and counts the first failure.

Fix: #2783

Does this PR introduce any user-facing change?

Yes.

  • Spark UI now displays fractional compression ratios correctly.
  • Shuffle failure counters start at zero and include the first failure.

How was this patch tested?

@github-actions

github-actions Bot commented Aug 21, 2026

Copy link
Copy Markdown

Test Results

 3 829 files   - 4   3 829 suites   - 4   8h 1m 5s ⏱️ - 10m 10s
 1 266 tests ±0   1 255 ✅ ±0  11 💤 ±0  0 ❌ ±0 
19 149 runs   - 7  19 101 ✅  - 7  48 💤 ±0  0 ❌ ±0 

Results for commit 10b0f10. ± Comparison against base commit b1ac32f.

♻️ This comment has been updated with latest results.

@wForget wForget changed the title [#2783] fix(spark): calculate compression ratio using floating-point division [#2783] fix(spark): correct compression ratio and failure counters in Spark UI Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Spark UI truncates the compression ratio

1 participant