fix(export): match ProRes swscale output to the declared full colour range

[DPS0340]

Problem

The ProRes encoder declares full range in the stream metadata:

// crates/enc-ffmpeg/src/video/prores.rs
encoder.set_colorspace(color::Space::BT709);
encoder.set_color_range(color::Range::JPEG);   // full range, 0-255

…but the RGBA → YUVA444P10LE conversion feeding it is a plain scaling::Context::get(...) with no colorspace details set, and swscale defaults to limited range output (16-235).

So the pixels written are limited-range while the container says full-range. A player reads Range::JPEG, assumes 16-235 was never applied, and expands the samples again — blacks lift, whites clip, and the whole image shifts. It's the classic double-range-conversion look.

Evidence

Reproduced the swscale default outside of Cap, so this doesn't depend on reading our pipeline correctly. A 2×2 image of pure white and pure black, converted with ffmpeg:

default sws (what prores.rs does today):     luma bytes = [233, 16, 233, 16]
explicit full range (in_range=full:out_range=full): luma bytes = [253, 0, 253, 0]

Pure white lands on 235 and pure black on 16 under the default — i.e. limited range — while the encoder is telling the container those same samples are full range. Confirms the two sides disagree.

Why ProRes specifically

h264.rs and hevc.rs both declare Range::MPEG and also use swscale's limited-range default, so those two are internally consistent — I checked before touching anything, and deliberately left them alone.

prores.rs is the only encoder declaring Range::JPEG, which makes it the only one where the declaration and the conversion disagree. ProRes 4444 is a full-range format, so the declaration is right and the conversion was the side that was wrong — hence fixing the scaler rather than downgrading the metadata to MPEG.

Fix

Set the swscale colorspace details so source and destination are both full range, using the BT.709 coefficients that match the declared Space::BT709:

let coefficients = ffmpeg::ffi::sws_getCoefficients(ffmpeg::ffi::SWS_CS_ITU709);
ffmpeg::ffi::sws_setColorspaceDetails(
    context.as_mut_ptr(),
    coefficients, 1,   // src full range
    coefficients, 1,   // dst full range
    brightness, contrast, saturation,
);

Existing brightness/contrast/saturation are read back via sws_getColorspaceDetails and preserved rather than reset to zero, and the whole thing is guarded on that call succeeding, so a failure leaves the previous behaviour intact instead of silently mangling the conversion.

Scope

This affects MOV/ProRes export (crates/export/src/mov.rs), which builds frames as RawVideoFormat::Rgba and hands them straight to this encoder. MP4/H.264 export is untouched.

Relevant to #1914, which reports colour shift on exported video — though that report is about an external capture card and also describes stutter and compression artifacts, so I'd expect this to be one contributing factor rather than the whole story. I didn't want to claim it closes that issue without a capture card to verify against.

Verification

• cargo check -p cap-enc-ffmpeg — clean.
• cargo build -p cap-enc-ffmpeg — clean.
• cargo clippy -p cap-enc-ffmpeg — no warnings on this file.
• cargo fmt applied.

Diff: 1 file.

[DPS0340]

@tembo please review

[tembo]

Nice catch on swscale's default range mismatch. One small thing: you can avoid the const/mut pointer casts here by matching the FFI signature directly.

Suggested change

	let mut inv_table: *const i32 = std::ptr::null();
	let mut table: *const i32 = std::ptr::null();
	let mut src_range: i32 = 0;
	let mut dst_range: i32 = 0;
	let mut brightness: i32 = 0;
	let mut contrast: i32 = 0;
	let mut saturation: i32 = 0;
	if ffmpeg::ffi::sws_getColorspaceDetails(
	context.as_mut_ptr(),
	&mut inv_table as *mut _ as *mut *mut i32,
	&mut src_range,
	&mut table as *mut _ as *mut *mut i32,
	&mut dst_range,
	&mut brightness,
	&mut contrast,
	&mut saturation,
	) >= 0
	let mut inv_table: *mut i32 = std::ptr::null_mut();
	let mut table: *mut i32 = std::ptr::null_mut();
	let mut src_range: i32 = 0;
	let mut dst_range: i32 = 0;
	let mut brightness: i32 = 0;
	let mut contrast: i32 = 0;
	let mut saturation: i32 = 0;
	if ffmpeg::ffi::sws_getColorspaceDetails(
	context.as_mut_ptr(),
	&mut inv_table,
	&mut src_range,
	&mut table,
	&mut dst_range,
	&mut brightness,
	&mut contrast,
	&mut saturation,
	) >= 0

[DPS0340]

Adopted — pushed. You're right, sws_getColorspaceDetails takes *mut *mut i32, so declaring the locals as *mut i32 and passing &mut inv_table directly matches the signature with no cast. cargo check and clippy both clean afterwards.

I'd defensively written them as *const on the assumption the coefficient tables were read-only output, but the FFI signature is the authority there and it disagrees.

[DPS0340]

@tembo please re-review

[tembo]

Might be worth guarding the FFI calls here: check sws_getCoefficients for null and sws_setColorspaceDetails for an error return. If either ever fails, a warn log will make the fallback behavior obvious.

Suggested change

	let coefficients = ffmpeg::ffi::sws_getCoefficients(ffmpeg::ffi::SWS_CS_ITU709);
	ffmpeg::ffi::sws_setColorspaceDetails(
	context.as_mut_ptr(),
	coefficients,
	1,
	coefficients,
	1,
	brightness,
	contrast,
	saturation,
	);
	let coefficients = ffmpeg::ffi::sws_getCoefficients(ffmpeg::ffi::SWS_CS_ITU709);
	if coefficients.is_null() {
	tracing::warn!("sws_getCoefficients returned null for ITU709");
	} else {
	let ret = ffmpeg::ffi::sws_setColorspaceDetails(
	context.as_mut_ptr(),
	coefficients,
	1,
	coefficients,
	1,
	brightness,
	contrast,
	saturation,
	);
	if ret < 0 {
	tracing::warn!("sws_setColorspaceDetails failed: {ret}");
	}
	}

[DPS0340]

Adopted — pushed b217944a7.

Took the guards, and split them so each failure mode names itself instead of sharing one message:

• sws_getColorspaceDetails < 0 — previously this silently skipped the whole block. Now it warns.
• sws_getCoefficients null — warns and leaves swscale untouched.
• sws_setColorspaceDetails < 0 — warns with the return code.

One deliberate difference from the suggestion: the getColorspaceDetails failure path now logs too. It was already an early-exit in my original code, so that branch was the one silent failure that could produce exactly the bug this PR fixes — limited-range pixels in a stream that declares Range::JPEG — with nothing in the logs to point at it. Both < 0 messages say that explicitly so the symptom is searchable.

Kept them at warn! rather than propagating an error: the fallback is the current shipping behaviour, so failing the export outright would be a regression for anyone whose ffmpeg build returns an error here.

cargo check -p cap-enc-ffmpeg, cargo clippy -p cap-enc-ffmpeg, cargo fmt -p cap-enc-ffmpeg -- --check all clean.

[DPS0340]

@tembo please re-review

[tembo]

One thought on the failure path: if sws_getColorspaceDetails/sws_setColorspaceDetails fails, we’ll still declare Range::JPEG later, which reintroduces the mismatch you’re fixing (just with a warning). Might be worth tracking a declare_full_range flag here and falling back to Range::MPEG when swscale can’t be configured, so the file stays internally consistent.

[DPS0340]

Right, and this is the better version of my own fix — adopted in bb913fd88.

My warnings described the mismatch instead of preventing it: the log said "output will stay limited-range while the stream declares full range", which is just an accurate description of the original bug with better observability. Tracking the flag makes the file internally consistent either way.

let mut declare_full_range = true;
// ... any of the three failure paths sets it to false
encoder.set_color_range(if declare_full_range { color::Range::JPEG } else { color::Range::MPEG });

So the two possible outcomes are now: swscale configured full-range + Range::JPEG declared (the fix), or swscale left at its limited-range default + Range::MPEG declared (correct, just not the wider range). Neither declares something the pixels don't match.

One thing I checked before wiring the flag, since it decides whether the fallback is reachable at all: declare_full_range is only ever cleared inside the converter branch, so I wanted to confirm the None branch isn't a silent third case. It isn't reachable in practice — ProResEncoder::input_format() is RawVideoFormat::Rgba and output_format is YUVA444P10LE, so input_config.pixel_format != output_format always holds and a converter is always built. The flag stays true on a path that can't occur, and if that input format ever changes there'd be no swscale conversion to disagree with.

cargo check, cargo clippy, cargo fmt --check all clean for cap-enc-ffmpeg.

[DPS0340]

@tembo please re-review

[DPS0340]

Closing for queue size — trimming my nine open PRs down to five so the ones closing real user reports are not buried behind them.

This is the most expensive of my PRs to review and the least urgent, which is a bad combination when you have a backlog. It uses unsafe FFI (sws_getColorspaceDetails / sws_setColorspaceDetails) to make swscale emit full range, and the diff is +78/-3 in an encoder path. That deserves real scrutiny from someone who knows the ProRes pipeline, not a fast merge.

The finding itself I still believe is correct and worth recording, so it is not lost if this stays closed:

encoder.set_color_range(color::Range::JPEG);   // we declare full range 0-255

but the RGBA → YUVA444P10LE conversion feeding it is a bare scaling::Context::get(...), and swscale defaults to limited range (16-235). So full-range pixels get squeezed into limited range, and players then expand them again as if full range — crushed contrast and shifted colour on ProRes 4444 export.

No user has reported this, which is why it goes rather than one of the four that close user-filed issues. Happy to reopen against current main if it is worth having, or to hand the analysis to whoever owns that encoder.