Skip to content

IB CI fixes: correct CQ-full warning bound, drain in-flight posts on stopProxy, expand multi-QP perf tests#808

Merged
chhwang merged 4 commits into
mainfrom
chhwang/ib-ci-fix
Jul 15, 2026
Merged

IB CI fixes: correct CQ-full warning bound, drain in-flight posts on stopProxy, expand multi-QP perf tests#808
chhwang merged 4 commits into
mainfrom
chhwang/ib-ci-fix

Conversation

@chhwang

@chhwang chhwang commented May 20, 2026

Copy link
Copy Markdown
Contributor

Corrects the send-CQ overflow warning, closes a proxy-shutdown leak, and reworks the multi-QP perf tests. No production API change.

Changes

  • IbQp (src/core/ib.cc, src/core/include/ib.hpp): track the send CQ size explicitly via new maxSendCqSize_ and use it for the "CQ almost full" warning instead of sendCq_->cqe. Fix warning message spacing.
  • ProxyService::stopProxy (src/core/port_channel.cc): after draining pending TriggerSync flushes, flush remaining in-flight signaled posts per connection (5 s timeout) so shutdown leaves no unflushed sends. Reword the drain comment.
  • Multi-QP perf tests (test/mp_unit/port_channel_tests.cu): rewrite kernelMultiQpBandwidth to one thread per QP with parallel waits; split the combined {1,2,4} loops into individually named PERF_TESTs (Single/Two/Three/Multi QP); tune nIters and extend the size sweeps; report aggregate GB/s in the flush-stress test.

@chhwang

chhwang commented May 21, 2026

Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 5 pipeline(s).

@chhwang
chhwang force-pushed the chhwang/ib-ci-fix branch from 2d85947 to 4c624ce Compare June 23, 2026 17:02
@chhwang chhwang changed the title Change DefaultMaxCqPollNum to 256 & minor IB CI fixes IB CI fixes: correct CQ-full warning bound, drain in-flight posts on stopProxy, expand multi-QP perf tests Jul 15, 2026
@chhwang
chhwang requested a review from a team July 15, 2026 12:54

@mahdiehghazim mahdiehghazim left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@chhwang
chhwang merged commit 6a648fa into main Jul 15, 2026
16 checks passed
@chhwang
chhwang deleted the chhwang/ib-ci-fix branch July 15, 2026 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants