What causes "request failed due to circuit breaker triggered" in go-kratos/kratos?

Look behind the rejection: check the target service's health/logs for the 5xx errors that opened the breaker, and fix those first

go-kratos/kratos · error · github.com/go-kratos/kratos/v3/errors.Error

CIRCUITBREAKER

CIRCUITBREAKER

Error message

request failed due to circuit breaker triggered

What it means

ErrNotAllowed (middleware/circuitbreaker/circuitbreaker.go:14) is a typed kratos error (HTTP 503, reason CIRCUITBREAKER) returned by the Client circuitbreaker middleware. Before invoking the handler it calls breaker.Allow() for the operation (grouped per operation name from transport.FromClientContext); on failure it marks the breaker failed again (deliberately, to keep the drop ratio high per the code comment) and rejects locally with ErrNotAllowed - the request never leaves the process. Failures counted by the breaker are handler errors that are 5xx/ServiceUnavailable/GatewayTimeout.

Source

Thrown at middleware/circuitbreaker/circuitbreaker.go:14

package circuitbreaker

import (
	"context"

	"github.com/go-kratos/kratos/v3/errors"
	internalbreaker "github.com/go-kratos/kratos/v3/internal/circuitbreaker"
	"github.com/go-kratos/kratos/v3/internal/group"
	"github.com/go-kratos/kratos/v3/middleware"
	"github.com/go-kratos/kratos/v3/transport"
)

// ErrNotAllowed is request failed due to circuit breaker triggered.
var ErrNotAllowed = errors.New(503, "CIRCUITBREAKER", "request failed due to circuit breaker triggered")

// CircuitBreaker is a circuit breaker.
type CircuitBreaker = internalbreaker.CircuitBreaker

// Option is circuit breaker option.
type Option func(*options)

// WithBreakerFactory configures a factory used to lazily create one circuit breaker per operation.
func WithBreakerFactory(factory func() CircuitBreaker) Option {
	return func(o *options) {
		if factory == nil {
			return
		}
		o.group = group.NewGroup(factory)
	}
}

type options struct {

View on GitHub (pinned to 668db92c2c)

Solutions

Look behind the rejection: check the target service's health/logs for the 5xx errors that opened the breaker, and fix those first
Handle the error explicitly and serve a fallback (cached/default response) during the open window
Retry later with backoff sized to the breaker cooldown rather than immediately
Tune the breaker via circuitbreaker.WithBreakerFactory to supply a breaker with more forgiving window/success-ratio settings for that operation
In tests/mocks, mark handler errors as non-5xx where appropriate so the breaker does not open spuriously

Example fix

// before
reply, err := userClient.GetUser(ctx, req)
if err != nil { return nil, err } // surfaces 503 CIRCUITBREAKER to callers

// after
reply, err := userClient.GetUser(ctx, req)
if e := kerrors.FromError(err); e.Reason == "CIRCUITBREAKER" && e.Code == 503 {
    return cachedUser(req.Id), nil // fallback while breaker is open
}

Defensive patterns

Strategy: fallback

Try / catch

reply, err := handler(ctx, req)
if e := kerrors.FromError(err); e.Code == 503 && e.Reason == "CIRCUITBREAKER" {
    reply, err = serveFallback(ctx, req) // cached/default response during open window
}
return reply, err

Prevention

Fix the downstream 5xx root cause first - the breaker only reports it
Pair circuitbreaker.Client with a fallback path (cache, default, queue-and-retry-later)
Tune breaker parameters per operation via WithBreakerFactory instead of accepting defaults for low-traffic routes
Remember rejections also MarkFailed by design: do not expect the breaker to close while you keep retrying hard
In tests, return non-5xx errors from mocked handlers so the breaker stays closed

When it happens

Trigger: Wrapping a client with circuitbreaker.Client(): once the downstream error rate for an operation crosses the SRE-breaker threshold, Allow() fails and every call in the cooldown window returns ErrNotAllowed immediately. Also triggered transiently right at boundary conditions with low request volume (min-sample style gates) where a couple of 500s open the breaker.

Common situations: A degraded dependency returning 500/503/504 causes all calls to be rejected locally, masking the original downstream error; a single flaky endpoint opening the breaker for an operation shared by many callers; tests asserting on real transport errors but receiving the 503 CIRCUITBREAKER rejection instead; breaker never seeming to close because rejections themselves MarkFailed (by design per the NOTE comment).

Related errors

AI-assisted analysis of go-kratos/kratos@668db92c2c (2026-08-16). Data as JSON: /api/errors/ce0648a34f836167. Report an issue: GitHub.