feat(query-engine): prototype streaming query engine by forshev · Pull Request #360 · ozontech/seq-db

forshev · 2026-02-20T11:08:34Z

Summary

Introduces the foundation for a new streaming search engine that will enable searching by non-indexed fields.

Key Changes

New Query Engine with streaming RecordProducer interface
SeqQL parser extensions: filter, sort, limit, stats, fields pipes
gRPC streaming API for one-phase search (store side)

Architecture

Instead of the traditional two-phase approach (find IDs → fetch documents), this engine streams documents directly through a chain of executors (filter → sort → limit → project), enabling efficient search on non-indexed fields.

I have read and followed all requirements in CONTRIBUTING.md;
I used LLM/AI assistance to make this pull request;

If you have used LLM/AI assistance please provide model name and full prompt:

Model: {{model-name}}
Prompt: {{prompt}}

codecov-commenter · 2026-02-20T11:14:36Z

Codecov Report

❌ Patch coverage is 58.38028% with 591 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.90%. Comparing base (23d27e1) to head (5b47f6e).
⚠️ Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
proxyapi/grpc_one_phase_search.go	0.00%	175 Missing ⚠️
proxyapi/grpc_v1.go	14.28%	80 Missing and 4 partials ⚠️
storeapi/grpc_one_phase_search.go	60.76%	64 Missing and 18 partials ⚠️
proxy/search/one_phase_search.go	53.98%	63 Missing and 12 partials ⚠️
parser/seqql_pipes.go	71.71%	32 Missing and 24 partials ⚠️
query/exec/data_source.go	76.19%	21 Missing and 14 partials ⚠️
storeapi/client.go	0.00%	25 Missing ⚠️
proxyapi/grpc_async_search.go	0.00%	16 Missing ⚠️
query/exec/merger.go	89.79%	9 Missing and 1 partial ⚠️
query/record.go	79.48%	6 Missing and 2 partials ⚠️
... and 7 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #360      +/-   ##
==========================================
- Coverage   70.83%   69.90%   -0.93%     
==========================================
  Files         225      236      +11     
  Lines       17512    18922    +1410     
==========================================
+ Hits        12404    13227     +823     
- Misses       4183     4691     +508     
- Partials      925     1004      +79

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-02-20T11:16:49Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`AggWide/size=10000-4`	`8aefcc`	`afad98`
	`47761.00 ns/op`	`58671.00 ns/op`	`1.23`	`🔴`
`AggWide/size=1000000-4`	`8aefcc`	`afad98`
	`516.00 B/op`	`623.00 B/op`	`1.21`	`🔴`
	`5419714.00 ns/op`	`6613404.00 ns/op`	`1.22`	`🔴`
`FindSequence_Random/small-4`	`8aefcc`	`afad98`
	`6638.44 MB/s`	`5400.47 MB/s`	`0.81`	`🔴`

github-actions · 2026-02-20T11:25:00Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`AggDeep/size=1000-4`	`8aefcc`	`5e2344`
	`4817.00 ns/op`	`5772.00 ns/op`	`1.20`	`🔴`
`AggDeep/size=1000000-4`	`8aefcc`	`5e2344`
	`4837044.00 ns/op`	`5950444.00 ns/op`	`1.23`	`🔴`
`AggWide/size=10000-4`	`8aefcc`	`5e2344`
	`47761.00 ns/op`	`57932.00 ns/op`	`1.21`	`🔴`
`FindSequence_Random/small-4`	`8aefcc`	`5e2344`
	`6638.44 MB/s`	`5084.65 MB/s`	`0.77`	`🔴`

github-actions · 2026-05-25T13:58:47Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`AggWide/size=1000000-4`	`2c48a6`	`6a5e64`
	`467.00 B/op`	`545.00 B/op`	`1.17`	`🔴`

… in one phase search

…ion to disable one ohase search

cheb0 · 2026-07-10T07:50:54Z

+
+	getNextTs := getAutoTsGenerator(time.Now(), -time.Nanosecond)
+
+	toBulk := []string{


if it's a single bulk, then we don' test in a cluster env?

cheb0 · 2026-07-10T07:51:42Z

+  bool explain = 6;
+  bool with_total = 7;
+  Order order = 8;
+  string offset_id = 9;


TODO: offset_id not supported

cheb0 · 2026-07-10T07:54:16Z

+  google.protobuf.Timestamp from = 2;
+  google.protobuf.Timestamp to = 3;
+  int64 size = 4;
+  int64 offset = 5;


TODO: we need to establish the API for query engine. Currently, I can set both limit and size and they work both at the same time, which is odd.

cheb0 · 2026-07-10T08:23:14Z

+	getNextTs := getAutoTsGenerator(time.Now(), -time.Nanosecond)
+
+	toBulk := []string{
+		fmt.Sprintf(`{"ts":%q, "level":3, "service":"service1", "unindexed":"1", "sorted":"1"}`, getNextTs()),


I would like to match the behaviour of existing search API. Like service:travel | limit 100 must match ordinary search with size set to 100 with both orders. Ideally both on low and large (thousands) amount of docs.

Currently, I'm running a cluster env and unable to get same results using both old and new API for limit.

cheb0 · 2026-07-10T08:40:07Z

+	}
+	fraction := setupFraction(t, mapping, docs)
+
+	queryString := "level:in(3, 5)"


I think it looks like something we would put to fraction_test. We could add tests there (or make a new test) where we would test different expressions written in a new language against different fractions.

However, we will still tests most stuff in integration tests because cluster env tests are a must.

cheb0 · 2026-07-10T11:06:53Z

 	ErrTooManyFractionTokens     = errors.New("aggregation has too many fraction tokens")
 	ErrTooManyFractionsHit       = errors.New("too many fractions hit")
 	ErrMemoryLimitExceeded       = errors.New("memory limit exceeded")
+	ErrResourceDisabled          = errors.New("resource is disabled")


what's the concrete meaning of resource here?

cheb0 · 2026-07-10T11:31:14Z

 		// If zero then there is no limit.
 		MaxRegexTokensCheck int `config:"max_regex_tokens_check" default:"0"`
+		// EnableOnePhaseSearch enanles experimental one phase search method that supports new query engine.
+		EnableOnePhaseSearch bool `config:"enable_one_phase_search"`


Even with disabled toggle I can issue a new query service:payment* AND level:5 | sort level asc | limit 100 using complex search and I do not have parse error, but I think I should.

cheb0 · 2026-07-10T12:40:17Z

    };
  }
+
+  rpc OnePhaseSearch(SearchRequest) returns (ComplexSearchResponse) {


Duplicating an issue here - currently we must set size even in presence of limit pipe.

cheb0 · 2026-07-10T12:49:26Z

+  google.protobuf.Timestamp from = 2;
+  google.protobuf.Timestamp to = 3;
+  int64 size = 4;
+  int64 offset = 5;


TODO: offset is NOP

cheb0 · 2026-07-10T12:58:30Z

+
+	curLeft, curRight *query.Record
+
+	colIdx   int


since colIdx is exact same for all docs, how do we guarantee all docs have field, let's say amount, on same column index?

forshev self-assigned this Feb 20, 2026

forshev added the feature New feature or request label Feb 20, 2026

forshev force-pushed the 0-one-phase-search branch from ba8080f to 4918b79 Compare February 20, 2026 11:10

forshev force-pushed the 0-one-phase-search branch 2 times, most recently from cf22e58 to f9534c5 Compare April 14, 2026 06:28

forshev force-pushed the 0-one-phase-search branch from a1659f5 to 64a9b26 Compare April 24, 2026 12:21

forshev force-pushed the 0-one-phase-search branch from 7a58d66 to 8d3d79d Compare May 4, 2026 12:23

forshev force-pushed the 0-one-phase-search branch from d167fff to 695b657 Compare May 14, 2026 11:15

forshev force-pushed the 0-one-phase-search branch 3 times, most recently from 4877d35 to 21cd3f4 Compare May 25, 2026 13:50

forshev force-pushed the 0-one-phase-search branch 6 times, most recently from b417d52 to 67e6e8a Compare June 4, 2026 11:48

forshev added 9 commits June 15, 2026 16:19

feat: one phase search

02947b3

feat: add stats pipe

74c7753

feat(one phase search): new grcp api on storeapi/

e8b1f33

feat(query engine): init

436f2d2

feat(query engine): init data source

882c591

feat(query engine): json filter

c521d48

chore(query engine): RecordProducer returns Metadata

3dd5ee5

feat(query engine): onePhaseSearch method uses new executors

3d43a6d

feat(query engine): add doc projector executor

86e8db5

forshev added 7 commits June 15, 2026 16:19

feat(query engine): add filter and limit pipes to seq-ql and use them…

bf8b523

… in one phase search

feat(query engine): add sort pipe to seq-ql, add sorter executor

1b2924a

chore(query engine): merge stores responses

a461ae1

chore(query engine): merge stores responses with aggs

0e3e7d8

chore(query engine): move agg to data source on store

c30509f

chore(query engine): add release method for executors, add config opt…

d2b697c

…ion to disable one ohase search

chore(query engine): add integration test

5b47f6e

forshev force-pushed the 0-one-phase-search branch from 67e6e8a to 5b47f6e Compare June 17, 2026 14:07

eguguchkin changed the title ~~feat: one phase search~~ feat(query-engine): prototype streaming query engine Jul 3, 2026

eguguchkin added the epic/query-engine label Jul 3, 2026

eguguchkin requested review from cheb0, dkharms, eguguchkin and moflotas and removed request for cheb0 July 3, 2026 11:41

cheb0 reviewed Jul 10, 2026

View reviewed changes


		getNextTs := getAutoTsGenerator(time.Now(), -time.Nanosecond)

		toBulk := []string{

Uh oh!

Conversation

forshev commented Feb 20, 2026 • edited by eguguchkin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Changes

Architecture

Uh oh!

codecov-commenter commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Feb 20, 2026

🔴 Performance Degradation

Uh oh!

github-actions Bot commented Feb 20, 2026

🔴 Performance Degradation

Uh oh!

github-actions Bot commented May 25, 2026

🔴 Performance Degradation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

forshev commented Feb 20, 2026 •

edited by eguguchkin

Loading

codecov-commenter commented Feb 20, 2026 •

edited

Loading