{"record":{"id":"049d010b601d0887","repo":"gastownhall/beads","slug":"w-circuit-breaker-tripped","errorCode":null,"errorMessage":"%w (circuit breaker tripped)","messagePattern":"%w \\(circuit breaker tripped\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/storage/dolt/store.go","lineNumber":851,"sourceCode":"\t\t\treturn tripped\n\t\t}\n\t\treturn err // backoff will retry\n\t}\n\treturn backoff.Permanent(err) // non-retryable — stop immediately\n}\n\n// recordRetryFailure records a connection-level failure to the breaker. It\n// returns a permanent \"circuit breaker tripped\" error when this failure trips\n// the breaker — signaling the retry loop to stop — and nil otherwise, including\n// when err is not a connection error or no breaker is configured.\nfunc (s *DoltStore) recordRetryFailure(ctx context.Context, err error) error {\n\tif s.breaker == nil || !isConnectionError(err) {\n\t\treturn nil\n\t}\n\ts.breaker.RecordFailure()\n\tif s.breaker.State() == circuitOpen {\n\t\tdoltMetrics.circuitTrips.Add(ctx, 1)\n\t\treturn backoff.Permanent(fmt.Errorf(\"%w (circuit breaker tripped)\", err))\n\t}\n\treturn nil\n}\n\n// doltTracer is the OTel tracer for SQL-level spans.\n// It uses the global provider, which is a no-op until telemetry.Init() is called.\nvar doltTracer = otel.Tracer(\"github.com/steveyegge/beads/storage/dolt\")\n\n// doltMetrics holds OTel metric instruments for the dolt storage backend.\n// Instruments are registered against the global delegating provider at init time,\n// so they automatically forward to the real provider once telemetry.Init() runs.\nvar doltMetrics struct {\n\tretryCount           metric.Int64Counter\n\tlockWaitMs           metric.Float64Histogram\n\tcircuitTrips         metric.Int64Counter\n\tcircuitRejected      metric.Int64Counter\n\tserializationErrors  metric.Int64Counter\n\twriteRetries         metric.Int64Counter","sourceCodeStart":833,"sourceCodeEnd":869,"githubUrl":"https://github.com/gastownhall/beads/blob/71377f276968b452ee607177637970a4ff888584/internal/storage/dolt/store.go#L833-L869","documentation":"The DoltStore's circuit breaker recorded a connection failure and its state is now circuitOpen, so the store returns a permanent (non-retryable) error marked 'circuit breaker tripped'. The store stops hammering a dead/unresponsive database and fails fast until the breaker's cooldown elapses and it half-opens again.","triggerScenarios":"Repeated connection-class errors (isConnectionError) on writes — embedded Dolt server down or hung, connection refused, repeated timeouts — pushing the breaker over its failure threshold.","commonSituations":"Dolt server crashed mid-session; resource exhaustion making every connection attempt time out; network partition with a remote Dolt; a burst of concurrent requests after an outage triggers the trip.","solutions":["Restore database connectivity (restart the embedded Dolt server / check bd doctor) and wait for the breaker cooldown, then retry.","Inspect the wrapped cause to see the original connection error.","If trips recur, check system resources (memory, file descriptors) and reduce concurrency against the store."],"exampleFix":"// before\nfor {\n\tif err := store.SlotClear(ctx, id, key, actor); err != nil {\n\t\tcontinue // hot-retries against an open breaker\n\t}\n}\n// after\nif err := store.SlotClear(ctx, id, key, actor); err != nil {\n\tif strings.Contains(err.Error(), \"circuit breaker tripped\") {\n\t\ttime.Sleep(breakerCooldown) // let the breaker half-open\n\t}\n}","handlingStrategy":"retry","validationCode":"if err := ctx.Err(); err != nil {\n\treturn err\n}\n// verify DB reachable before resuming after a trip\nif err := store.Ping(ctx); err != nil {\n\treturn fmt.Errorf(\"database still unreachable: %w\", err)\n}","typeGuard":"func isCircuitOpen(err error) bool {\n\treturn err != nil && strings.Contains(err.Error(), \"circuit breaker tripped\")\n}","tryCatchPattern":"if err := op(ctx); err != nil {\n\tif isCircuitOpen(err) {\n\t\ttime.Sleep(cooldown) // fail fast until the breaker half-opens\n\t\treturn op(ctx)\n\t}\n\treturn err\n}","preventionTips":["Monitor circuitTrips metrics to catch database instability early.","Fix the underlying connection problem before hammering retries — the breaker is non-retryable by design.","Ensure the embedded Dolt server has adequate memory and file descriptors.","Throttle concurrent writers to avoid failure-threshold bursts."],"tags":["circuit-breaker","connection","dolt","resilience"],"backgroundTag":"circuit-breaker-open","analyzedSha":"71377f276968b452ee607177637970a4ff888584","analyzedAt":"2026-08-30T18:55:39.744Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}