fix: dispatch OnStateChange outside mutex to prevent deadlock#119
Open
nghiack7 wants to merge 1 commit into
Open
fix: dispatch OnStateChange outside mutex to prevent deadlock#119nghiack7 wants to merge 1 commit into
nghiack7 wants to merge 1 commit into
Conversation
OnStateChange was invoked while the internal sync.Mutex was held. Any CircuitBreaker method called from within the callback (e.g. Counts, State, Execute) tried to acquire the same mutex and dead-locked. Fix: instead of calling onStateChange directly in setState, record a pending notification in the CircuitBreaker struct. The three public entry points that can trigger a state change (State, beforeRequest, afterRequest) capture and clear the pending notification while still holding the lock, then dispatch it after unlocking. This is race-safe: the notification is both written (in setState) and captured/cleared (in the entry points) under the same mutex. Dispatching outside the lock allows callbacks to freely re-enter the breaker. Fixes sony#37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #37
OnStateChangewas called from insidesetStatewhilecb.mutexwas held. AnyCircuitBreakermethod invoked from within the callback —Counts(),State(),Execute()— tries to acquire the same mutex and dead-locks.Root cause
setStateis always called whilecb.mutexis held (viabeforeRequest,afterRequest, orState). It calledcb.onStateChangesynchronously under that lock.Fix
Record the pending notification in a new
pendingNotifyfield (set under the lock insetState), then dispatch it after unlocking in the three public entry points that can trigger a state change (State,beforeRequest,afterRequest).The notification is captured and cleared while the lock is still held, so concurrent goroutines cannot steal each other's notifications. Dispatching outside the lock allows the callback to freely call back into the breaker.
Tests
Added
TestOnStateChangeNoDeadlock: creates a breaker whoseOnStateChangecallback callscb.Counts()andcb.State(), triggers a closed→open transition, then waits for the test to complete within 2 seconds. If the deadlock were present the test would time out.