Skip to content

fix: dispatch OnStateChange outside mutex to prevent deadlock#119

Open
nghiack7 wants to merge 1 commit into
sony:masterfrom
nghiack7:fix/onStateChange-deadlock
Open

fix: dispatch OnStateChange outside mutex to prevent deadlock#119
nghiack7 wants to merge 1 commit into
sony:masterfrom
nghiack7:fix/onStateChange-deadlock

Conversation

@nghiack7

@nghiack7 nghiack7 commented May 9, 2026

Copy link
Copy Markdown

Problem

Fixes #37

OnStateChange was called from inside setState while cb.mutex was held. Any CircuitBreaker method invoked from within the callback — Counts(), State(), Execute() — tries to acquire the same mutex and dead-locks.

// Before — deadlock if callback calls cb.Counts()
OnStateChange: func(name string, from, to gobreaker.State) {
    counts := cb.Counts()  // ← acquires mutex → deadlock
    log.Printf("state changed: %v counts: %v", to, counts)
},

Root cause

setState is always called while cb.mutex is held (via beforeRequest, afterRequest, or State). It called cb.onStateChange synchronously under that lock.

Fix

Record the pending notification in a new pendingNotify field (set under the lock in setState), then dispatch it after unlocking in the three public entry points that can trigger a state change (State, beforeRequest, afterRequest).

The notification is captured and cleared while the lock is still held, so concurrent goroutines cannot steal each other's notifications. Dispatching outside the lock allows the callback to freely call back into the breaker.

// After — safe to call any CircuitBreaker method in the callback
OnStateChange: func(name string, from, to gobreaker.State) {
    counts := cb.Counts()  // ✓ no deadlock
    log.Printf("state: %s → %s  counts: %v", from, to, counts)
},

Tests

Added TestOnStateChangeNoDeadlock: creates a breaker whose OnStateChange callback calls cb.Counts() and cb.State(), triggers a closed→open transition, then waits for the test to complete within 2 seconds. If the deadlock were present the test would time out.

OnStateChange was invoked while the internal sync.Mutex was held.
Any CircuitBreaker method called from within the callback (e.g.
Counts, State, Execute) tried to acquire the same mutex and dead-locked.

Fix: instead of calling onStateChange directly in setState, record a
pending notification in the CircuitBreaker struct.  The three public
entry points that can trigger a state change (State, beforeRequest,
afterRequest) capture and clear the pending notification while still
holding the lock, then dispatch it after unlocking.

This is race-safe: the notification is both written (in setState) and
captured/cleared (in the entry points) under the same mutex.
Dispatching outside the lock allows callbacks to freely re-enter the
breaker.

Fixes sony#37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

It will cause a deadlock when call breaker.Counts() method in the OnStateChange() method

1 participant