Skip to content

object: speed up split streaming#4014

Merged
roman-khimov merged 1 commit into
masterfrom
2791-speed-up-split-object-streaming
Jun 22, 2026
Merged

object: speed up split streaming#4014
roman-khimov merged 1 commit into
masterfrom
2791-speed-up-split-object-streaming

Conversation

@End-rey

@End-rey End-rey commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Closes #2791.

@End-rey End-rey self-assigned this Jun 9, 2026
@codecov

codecov Bot commented Jun 9, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.81818% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 26.76%. Comparing base (f1fd400) to head (8d87a18).

Files with missing lines Patch % Lines
pkg/services/object/get/assemble_pipeline.go 83.05% 5 Missing and 5 partials ⚠️
pkg/services/object/get/assembly_v2.go 57.89% 5 Missing and 3 partials ⚠️
pkg/services/object/get/exec.go 83.78% 5 Missing and 1 partial ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #4014      +/-   ##
==========================================
+ Coverage   26.58%   26.76%   +0.18%     
==========================================
  Files         687      688       +1     
  Lines       46591    46710     +119     
==========================================
+ Hits        12384    12501     +117     
+ Misses      33065    33062       -3     
- Partials     1142     1147       +5

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

roman-khimov
roman-khimov reviewed Jun 10, 2026
View reviewed changes
Comment thread pkg/services/object/get/assemble.go Outdated
Comment thread pkg/services/object/get/assemble_pipeline.go Outdated
Comment thread pkg/services/object/get/assembly_v2.go Outdated
Comment thread pkg/services/object/get/assemble_pipeline.go
@End-rey End-rey force-pushed the 2791-speed-up-split-object-streaming branch 2 times, most recently from 9d6e409 to aeecac4 Compare June 10, 2026 14:09
@End-rey End-rey requested a review from roman-khimov June 10, 2026 14:36
roman-khimov
roman-khimov reviewed Jun 11, 2026
View reviewed changes

@roman-khimov roman-khimov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have any test results from real cluster for this? It looks ok in general, but I'd like to check that it has some real effect.

Comment thread pkg/services/object/get/assemble.go
exec.statusError = exec.streamChildrenPipelined(len(children), at, !withRng && checkRight, prefetchWindow)
return
}

@roman-khimov roman-khimov Jun 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What worries me a bit is that everything down below effectively becomes a dead code. Most likely it can be dropped in future.

Comment thread pkg/services/object/get/assemble_pipeline.go
Comment thread pkg/services/object/get/assemble_pipeline.go Outdated
Comment thread pkg/services/object/get/assemble_pipeline.go Outdated
Comment thread pkg/services/object/get/assemble.go Outdated
@End-rey End-rey force-pushed the 2791-speed-up-split-object-streaming branch 2 times, most recently from ad698c7 to df89b93 Compare June 11, 2026 15:24
@roman-khimov roman-khimov mentioned this pull request Jun 15, 2026
Merged
@End-rey

End-rey commented Jun 18, 2026

Copy link
Copy Markdown
Contributor Author
Policy Test Size, GB Threads per Node 0.25.0 speed up split Δ
REP3 PUT 1 1 0.54 obj/s 0.52 obj/s -3.7%
REP3 PUT 1 2 0.80 obj/s 0.78 obj/s -2.5%
REP3 PUT 1 4 0.93 obj/s 0.92 obj/s -1.1%
REP3 GET 1 1 1.61 obj/s 1.65 obj/s +2.5%
REP3 GET 1 2 2.27 obj/s 2.27 obj/s 0%
REP3 GET 1 4 2.59 obj/s 2.55 obj/s -1.5%

The observed numbers are within the margin of error. Most likely, the change does not show a noticeable improvement in this setup because the potential gain is masked by other, more resource-intensive operations currently dominating the request path.

@End-rey End-rey force-pushed the 2791-speed-up-split-object-streaming branch 2 times, most recently from d7c1b95 to 8d87a18 Compare June 22, 2026 09:35
@End-rey
object: speed up split streaming
8d87a18 
Assemble split objects through a pipeline: while one child streams to the
client, up to prefetchWindow-1 others are fetched concurrently, hiding
inter-node setup/TTFB latency. Children are still written in order.

```
goos: linux
goarch: amd64
pkg: github.com/nspcc-dev/neofs-node/pkg/services/object/get
cpu: AMD Ryzen 7 PRO 4750U with Radeon Graphics
BenchmarkAssembleSplitObject
BenchmarkAssembleSplitObject/window=1
BenchmarkAssembleSplitObject/window=1-16   58	  20309812 ns/op	   60215 B/op	     549 allocs/op
BenchmarkAssembleSplitObject/window=2
BenchmarkAssembleSplitObject/window=2-16   94	  11778861 ns/op	   54929 B/op	     603 allocs/op
BenchmarkAssembleSplitObject/window=4
BenchmarkAssembleSplitObject/window=4-16   162	   7308957 ns/op	   62799 B/op	     603 allocs/op
```

Closes #2791.

Signed-off-by: Andrey Butusov <andrey@nspcc.io>
@roman-khimov roman-khimov merged commit c54d045 into master Jun 22, 2026
22 checks passed
@roman-khimov roman-khimov deleted the 2791-speed-up-split-object-streaming branch June 22, 2026 13:50
@roman-khimov roman-khimov mentioned this pull request Jun 23, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@roman-khimov roman-khimov roman-khimov approved these changes
@carpawell carpawell Awaiting requested review from carpawell carpawell is a code owner
@cthulhu-rider cthulhu-rider Awaiting requested review from cthulhu-rider cthulhu-rider is a code owner

Assignees

@End-rey End-rey

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Speed up split object streaming

2 participants

@End-rey @roman-khimov