gastownhall/beads · error

%w (circuit breaker tripped)

Error message

%w (circuit breaker tripped)

What it means

The DoltStore's circuit breaker recorded a connection failure and its state is now circuitOpen, so the store returns a permanent (non-retryable) error marked 'circuit breaker tripped'. The store stops hammering a dead/unresponsive database and fails fast until the breaker's cooldown elapses and it half-opens again.

Source

Thrown at internal/storage/dolt/store.go:851

			return tripped
		}
		return err // backoff will retry
	}
	return backoff.Permanent(err) // non-retryable — stop immediately
}

// recordRetryFailure records a connection-level failure to the breaker. It
// returns a permanent "circuit breaker tripped" error when this failure trips
// the breaker — signaling the retry loop to stop — and nil otherwise, including
// when err is not a connection error or no breaker is configured.
func (s *DoltStore) recordRetryFailure(ctx context.Context, err error) error {
	if s.breaker == nil || !isConnectionError(err) {
		return nil
	}
	s.breaker.RecordFailure()
	if s.breaker.State() == circuitOpen {
		doltMetrics.circuitTrips.Add(ctx, 1)
		return backoff.Permanent(fmt.Errorf("%w (circuit breaker tripped)", err))
	}
	return nil
}

// doltTracer is the OTel tracer for SQL-level spans.
// It uses the global provider, which is a no-op until telemetry.Init() is called.
var doltTracer = otel.Tracer("github.com/steveyegge/beads/storage/dolt")

// doltMetrics holds OTel metric instruments for the dolt storage backend.
// Instruments are registered against the global delegating provider at init time,
// so they automatically forward to the real provider once telemetry.Init() runs.
var doltMetrics struct {
	retryCount           metric.Int64Counter
	lockWaitMs           metric.Float64Histogram
	circuitTrips         metric.Int64Counter
	circuitRejected      metric.Int64Counter
	serializationErrors  metric.Int64Counter
	writeRetries         metric.Int64Counter

View on GitHub (pinned to 71377f2769)

Solutions

1. Restore database connectivity (restart the embedded Dolt server / check bd doctor) and wait for the breaker cooldown, then retry.
2. Inspect the wrapped cause to see the original connection error.
3. If trips recur, check system resources (memory, file descriptors) and reduce concurrency against the store.

Example fix

// before
for {
	if err := store.SlotClear(ctx, id, key, actor); err != nil {
		continue // hot-retries against an open breaker
	}
}
// after
if err := store.SlotClear(ctx, id, key, actor); err != nil {
	if strings.Contains(err.Error(), "circuit breaker tripped") {
		time.Sleep(breakerCooldown) // let the breaker half-open
	}
}

if err := ctx.Err(); err != nil {
	return err
}
// verify DB reachable before resuming after a trip
if err := store.Ping(ctx); err != nil {
	return fmt.Errorf("database still unreachable: %w", err)
}

func isCircuitOpen(err error) bool {
	return err != nil && strings.Contains(err.Error(), "circuit breaker tripped")
}

if err := op(ctx); err != nil {
	if isCircuitOpen(err) {
		time.Sleep(cooldown) // fail fast until the breaker half-opens
		return op(ctx)
	}
	return err
}

When it happens

Trigger: Repeated connection-class errors (isConnectionError) on writes — embedded Dolt server down or hung, connection refused, repeated timeouts — pushing the breaker over its failure threshold.

Common situations: Dolt server crashed mid-session; resource exhaustion making every connection attempt time out; network partition with a remote Dolt; a burst of concurrent requests after an outage triggers the trip.

Related errors

• failed to open server connection: %w
• server not reachable: %w
• dolt server connection failed: %w
• failed to reach the workspace identity: %v
• ErrCircuitOpen

AI-assisted analysis of gastownhall/beads@71377f2769 (2026-08-30). Data as JSON: /api/errors/049d010b601d0887. Report an issue: GitHub.