go-redis/redis · warning

circuit breaker is open, failing fast

Error message

circuit breaker is open, failing fast

What it means

Returned by performConnectionHandoff (handoff_worker.go:373) and CircuitBreaker.Execute (circuit_breaker.go:110,113,120) when the per-endpoint circuit breaker is Open — the target endpoint has failed >= CircuitBreakerFailureThreshold times and the reset timeout (CircuitBreakerResetTimeout) has not elapsed. The handoff fails fast (shouldRetry=false) rather than dialing a known-bad endpoint.

Source

Thrown at maintnotifications/errors.go:65

	// and should not be used until the handoff is complete
	ErrConnectionMarkedForHandoff = errors.New(logs.ConnectionMarkedForHandoffErrorMessage)
	// ErrConnectionMarkedForHandoffWithState is returned when a connection is marked for handoff
	// and should not be used until the handoff is complete
	ErrConnectionMarkedForHandoffWithState = errors.New(logs.ConnectionMarkedForHandoffErrorMessage + " with state")
	// ErrConnectionInvalidHandoffState is returned when a connection is in an invalid state for handoff
	ErrConnectionInvalidHandoffState = errors.New(logs.ConnectionInvalidHandoffStateErrorMessage)
)

// shutdown errors
var (
	// ErrShutdown is returned when the maintnotifications manager is shutdown
	ErrShutdown = errors.New(logs.ShutdownError())
)

// circuit breaker errors
var (
	// ErrCircuitBreakerOpen is returned when the circuit breaker is open
	ErrCircuitBreakerOpen = errors.New(logs.CircuitBreakerOpenErrorMessage)
)

// circuit breaker configuration errors
var (
	// ErrInvalidCircuitBreakerFailureThreshold is returned when the circuit breaker failure threshold is invalid
	ErrInvalidCircuitBreakerFailureThreshold = errors.New(logs.InvalidCircuitBreakerFailureThresholdError())
	// ErrInvalidCircuitBreakerResetTimeout is returned when the circuit breaker reset timeout is invalid
	ErrInvalidCircuitBreakerResetTimeout = errors.New(logs.InvalidCircuitBreakerResetTimeoutError())
	// ErrInvalidCircuitBreakerMaxRequests is returned when the circuit breaker max requests is invalid
	ErrInvalidCircuitBreakerMaxRequests = errors.New(logs.InvalidCircuitBreakerMaxRequestsError())
)

View on GitHub (pinned to 36d97525cd)

Solutions

1. Verify the endpoint is actually reachable (the circuit is opening for a reason — check connectivity/DNS/firewall).
2. If failures are transient, raise CircuitBreakerFailureThreshold (default 5) and/or lower CircuitBreakerResetTimeout (default 60s) for faster recovery probing.
3. Correct EndpointType if the returned address is unroutable from the client.
4. Monitor via GetCircuitBreakerStats() to see which endpoints are repeatedly opening.

Example fix

// before
cfg := maintnotifications.DefaultConfig()
// FailureThreshold 5, ResetTimeout 60s -> opens quickly on flaky net

// after
cfg := maintnotifications.DefaultConfig()
cfg.CircuitBreakerFailureThreshold = 20
cfg.CircuitBreakerResetTimeout = 15 * time.Second

if errors.Is(err, maintnotifications.ErrCircuitBreakerOpen) {
    // endpoint temporarily blocked; back off then retry. 
    // Investigate why the endpoint keeps failing.
    time.Sleep(cfg.CircuitBreakerResetTimeout / 4)
    return retryCommand(ctx, cmd)
}

When it happens

Trigger: Repeated handoff failures to a specific new endpoint open that endpoint's circuit breaker; subsequent MOVING handoffs to the same endpoint short-circuit with ErrCircuitBreakerOpen until resetTimeout passes, after which it transitions to half-open (circuit_breaker.go:100) to probe recovery.

Common situations: A target node is genuinely down/unreachable; wrong EndpointType returns an address the client can't reach, so every handoff fails and trips the breaker; failure threshold too low for a noisy network; reset timeout too long so recovery is slow.

Related errors

• max handoff retries reached
• handoff queue is full, cannot queue new handoff requests - c
• connection is in invalid state for handoff
• MaxWorkers must be greater than or equal to 0
• handoff queue size must be greater than 0

AI-assisted analysis of go-redis/redis@36d97525cd (2026-08-06). Data as JSON: /data/errors/cb7e4d3175e038be.json. Report an issue: GitHub.