Repository navigation
fix(serde): correct Value end offset with utf8_lossy and invalid UTF-8 - #253
Merged
Merged
Conversation
yuxi-liu-wired
force-pushed
the
pr/utf8-lossy-offset
branch
from
October 1, 2026 20:23
f7e6c60 to
f5974e1
Compare
With `utf8_lossy`, deserializing a `Value` from input that contains invalid UTF-8 parses a lossy copy (`String::from_utf8_lossy`), where each invalid sequence became the 3-byte U+FFFD. The end offset `n` returned for that copy was then used as an offset into the original input (`read.eat(n)`), so: - valid input failed. With the `utf8_lossy` feature on, plain `from_slice::<Value>(b"[\"a\xa0b\"]")` returned "Invalid UTF-8 characters" (the documented use of the feature), and so did `Deserializer::utf8_lossy()` + `deserialize::<Value>()` + `end()`. - trailing characters were skipped: `[\"a\xa0b\"]xx` was accepted by `end()`, because the offset overshot the real end. Deserializing the same input into any other type (String, Vec, serde_json::Value, ...) was right. The consumed length in the original is now its length minus what the parser left unread in the copy (the tail after the value is identical in both); an overshoot into the padding (unterminated input) is kept, so it is still EOF. The next-invalid-UTF-8 position is then recomputed from that point, as the invalid UTF-8 inside the value was replaced. Test: `test_utf8_lossy_value_trailing` (fails before this change). Suite: 249 (default), 246 (utf8_lossy, main 245), 248 (all features, main 247); Miri clean on the utf8_lossy tests. Found by a differential fuzzer (serde_json, simd-json, sonic-rs, jiter) extended with sonic-rs's utf8_lossy mode against a decode-then-parse reference.
liuq19
force-pushed
the
pr/utf8-lossy-offset
branch
from
October 10, 2026 07:58
f5974e1 to
5472469
Compare
liuq19
approved these changes
Oct 10, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #253 +/- ##
==========================================
+ Coverage 72.73% 72.83% +0.09%
==========================================
Files 42 42
Lines 9944 9979 +35
==========================================
+ Hits 7233 7268 +35
Misses 2711 2711 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(utf8_lossy): Value end offset when the input has invalid UTF-8
With
utf8_lossy, deserializing aValuefrom input that containsinvalid UTF-8 parses a lossy copy (
String::from_utf8_lossy), whereeach invalid sequence became the 3-byte U+FFFD. The end offset
nreturned for that copy was then used as an offset into the original
input (
read.eat(n)), so:utf8_lossyfeature on, plainfrom_slice::<Value>(b"[\"a\xa0b\"]")returned "Invalid UTF-8characters" (the documented use of the feature), and so did
Deserializer::utf8_lossy()+deserialize::<Value>()+end().[\"a\xa0b\"]xxwas accepted byend(), because the offset overshot the real end.Deserializing the same input into any other type (String, Vec,
serde_json::Value, ...) was right.
The consumed length in the original is now its length minus what the
parser left unread in the copy (the tail after the value is identical
in both); an overshoot into the padding (unterminated input) is kept,
so it is still EOF. The next-invalid-UTF-8 position is then recomputed
from that point, as the invalid UTF-8 inside the value was replaced.
Test:
test_utf8_lossy_value_trailing(fails before this change).Suite: 249 (default), 246 (utf8_lossy, main 245), 248 (all features,
main 247); Miri clean on the utf8_lossy tests.
Testing
Each commit adds a regression test that fails before the change. The full test suite passes and
cargo fmt --checkis clean on the changed files.Found with a differential fuzzer comparing serde_json, simd-json, sonic-rs and jiter.