Implement streaming replication - #4164
Open
cthulhu-rider wants to merge 3 commits into
Open
cthulhu-rider wants to merge 3 commits into
cthulhu-rider wants to merge 3 commits into
Conversation
cthulhu-rider
force-pushed
the
replication-stream
branch
from
September 1, 2026 18:32
4ecd0e6 to
53550c7
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #4164 +/- ##
=======================================
Coverage 31.48% 31.49%
=======================================
Files 677 678 +1
Lines 41369 41447 +78
=======================================
+ Hits 13025 13053 +28
- Misses 28344 28394 +50 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
cthulhu-rider
force-pushed
the
replication-stream
branch
3 times, most recently
from
September 2, 2026 15:52
a1bacbc to
8c4e822
Compare
Refactoring purpose, except that the status messages are updated naturally. Make header verification, payload verification and local save separately. This will be useful for #4157. For ReplicateV2 handler, header is needed to be checked before payload reading. Also, streaming write into local storage is needed. Signed-off-by: Leonard Liubich <leonard@morphbits.io>
Implementation relies on full payload buffering, which is something we want to get rid of in ReplicateV2 (#4157). At the same time, the implementation is no-op for regular objects. Signed-off-by: Leonard Liubich <leonard@morphbits.io>
cthulhu-rider
force-pushed
the
replication-stream
branch
2 times, most recently
from
September 23, 2026 12:58
6702af9 to
e62eca4
Compare
Stream them directly into the local storage instead. Refs #4157. Signed-off-by: Leonard Liubich <leonard@morphbits.io>
cthulhu-rider
force-pushed
the
replication-stream
branch
from
September 23, 2026 12:59
e62eca4 to
6cbae14
Compare
Contributor
Author
|
code is rdy for review. Gonna make performance runs and add unit tests in the background |
cthulhu-rider
marked this pull request as ready for review
September 23, 2026 13:26
cthulhu-rider
requested review from
End-rey,
carpawell and
roman-khimov
as code owners
September 23, 2026 13:26
| return x.local.Search(ctx, cID, fs, attrs, cursor, count) | ||
| } | ||
|
|
||
| func (x storageForObjectService) VerifyAndStoreObjectLocally(ctx context.Context, obj object.Object) error { |
| if isStream { | ||
| // TODO: can be optimized from two sides: | ||
| // 1. header structure decoding can be done without unmarshaling (e.g. via protoscan funcs) | ||
| // 2. since header is already serialized in the original request, io.WriterTo can be implemented around it |
Member
There was a problem hiding this comment.
2 should be easy to do, isn't it? Although PUTs are relatively slow (and streamed ones are relatively big), so not likely to be noticeable.
Contributor
Author
There was a problem hiding this comment.
yes, it's not very difficult, but given the general case of unordered fields, the logic will have to branch. Therefore, I still suggest doing this separately
| if len(chunk) == 0 { | ||
| return nil, newBadRequestStatus("empty payload chunk"), nil | ||
| var st *protostatus.Status | ||
| gotHash, gotPayloadLen, st, err = readReplicatedObjectPayload(obj.PayloadSize(), recvChunkFn, func(chunk []byte) *protostatus.Status { |
Member
There was a problem hiding this comment.
Why not passing a simple (optional) io.Writer there? This callback doesn't do a lot, can be simplified this way.
|
|
||
| var st *protostatus.Status | ||
| gotHash, gotPayloadLen, st, err = readReplicatedObjectPayload(obj.PayloadSize(), recvChunkFn, func(chunk []byte) *protostatus.Status { | ||
| payload = append(payload, chunk...) |
Member
There was a problem hiding this comment.
bytes.Buffer can be used here to provide io.Writer.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.