Skip to content

feat: add Prometheus counter for dropped subscriber updates#7643

Open
falloficaruss wants to merge 3 commits into
flyteorg:mainfrom
falloficaruss:feature/dropped-updates-counter
Open

feat: add Prometheus counter for dropped subscriber updates#7643
falloficaruss wants to merge 3 commits into
flyteorg:mainfrom
falloficaruss:feature/dropped-updates-counter

Conversation

@falloficaruss

@falloficaruss falloficaruss commented Jul 8, 2026

Copy link
Copy Markdown

Tracking issue

Closes #7641

Why are the changes needed?

Currently, when a subscriber channel is full, dropped updates are only logged at the Warnf level. This makes it difficult to detect chronic subscriber backpressure in production — logs are ephemeral, and there is no metric to alert on.

Adding a Prometheus counter allows operators to:

  1. Monitor subscriber channel capacity over time via dashboards.
  2. Configure alerting (e.g., PagerDuty) when drop rates exceed a threshold.
  3. Distinguish between transient drops (acceptable) and sustained backpressure (indicating a subscriber is too slow or the buffer is too small).

What changes were proposed in this pull request?

This PR adds a dropped_updates Prometheus counter to the ActionsClient struct in actions/k8s/client.go:

  1. New fielddroppedUpdates prometheus.Counter added to the ActionsClient struct.
  2. Initialization — The counter is created via scope.MustNewCounter("dropped_updates", ...) in NewActionsClient.
  3. Increment on dropc.droppedUpdates.Inc() is called in the default branch of the subscriber notification select, right before the existing Warnf log.

No behavioral changes beyond the metric emission. All existing tests continue to pass.

How was this patch tested?

Existing tests (TestSubscribeIsScopedByRun, TestUnsubscribeScopedByRun) call c.notifySubscribers directly with buffered channels that never fill, so they never hit the default branch and the nil counter is unreachable, they continue to pass.

Labels

  • added

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Stack

If you do use git town to manage PR Stacks, the stack relevant to this PR
will show below. Otherwise, you can ignore this section.

Docs link

Track dropped updates via a  counter in the ActionsClient
so administrators can monitor subscriber channel capacity.

Signed-off-by: Abhishek Shinde <norizzabhii@gmail.com>
@github-actions github-actions Bot added the flyte2 label Jul 8, 2026
falloficaruss and others added 2 commits July 10, 2026 19:58
Signed-off-by: Abhishek Shinde <139879930+falloficaruss@users.noreply.github.com>
Signed-off-by: Abhishek Shinde <norizzabhii@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Prometheus counter for dropped subscriber updates

1 participant