{"record":{"id":"b825c2ccaff602c6","repo":"thanos-io/thanos","slug":"invalid-service-state-v-expected-v-failure","errorCode":null,"errorMessage":"invalid service state: %v, expected: %v, failure: %w","messagePattern":"invalid service state: (.+?), expected: (.+?), failure: %w","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/cortex/util/services/basic_service.go","lineNumber":80,"sourceCode":"\tfailureCase error\n\tlisteners   []chan func(l Listener)\n\tserviceName string\n\n\t// closed when state reaches Running, Terminated or Failed state\n\trunningWaitersCh chan struct{}\n\t// closed when state reaches Terminated or Failed state\n\tterminatedWaitersCh chan struct{}\n\n\tserviceContext context.Context\n\tserviceCancel  context.CancelFunc\n}\n\nfunc invalidServiceStateError(state, expected State) error {\n\treturn fmt.Errorf(\"invalid service state: %v, expected: %v\", state, expected)\n}\n\nfunc invalidServiceStateWithFailureError(state, expected State, failure error) error {\n\treturn fmt.Errorf(\"invalid service state: %v, expected: %v, failure: %w\", state, expected, failure)\n}\n\n// NewBasicService returns service built from three functions (using BasicService).\nfunc NewBasicService(start StartingFn, run RunningFn, stop StoppingFn) *BasicService {\n\treturn &BasicService{\n\t\tstartFn:             start,\n\t\trunningFn:           run,\n\t\tstoppingFn:          stop,\n\t\tstate:               New,\n\t\trunningWaitersCh:    make(chan struct{}),\n\t\tterminatedWaitersCh: make(chan struct{}),\n\t}\n}\n\n// WithName sets service name, if service is still in New state, and returns service to allow\n// usage like NewBasicService(...).WithName(\"service name\").\nfunc (b *BasicService) WithName(name string) *BasicService {\n\t// Hold lock to make sure state doesn't change while setting service name.","sourceCodeStart":62,"sourceCodeEnd":98,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/internal/cortex/util/services/basic_service.go#L62-L98","documentation":"invalidServiceStateWithFailureError is like invalidServiceStateError but additionally wraps the underlying failure (with %w) that caused the service to leave the expected state. awaitState uses it when a service transitions to Failed (or an unexpected state) while callers are waiting for it to become Running/Stopping.","triggerScenarios":"awaitState observes the service enter Failed during StartAsync/AwaitRunning; the underlying starting/running function returned an error which is carried in the failure field of the message.","commonSituations":"Dependency initialization failures (DB unreachable, bucket credentials invalid, ring not ready) surfacing through a dependent service's AwaitRunning; misconfigured module parameters; resource exhaustion during startup.","solutions":["Unwrap the wrapped failure (errors.Unwrap / %w chain) to find the true root cause and fix it first.","Check that all dependencies the service requires (ring, store, bucket) are healthy and configured.","Increase startup timeouts only after the root cause is understood, not as a substitute.","Add failure logging via observability on the service's StartingFn/RunningFn to capture the original error."],"exampleFix":"// before\nif err := services.AwaitRunning(ctx, svc); err != nil {\n    return err\n}\n// after\nif err := services.AwaitRunning(ctx, svc); err != nil {\n    var stateErr *services.FailureError\n    if errors.As(err, &stateErr) {\n        return fmt.Errorf(\"service failed to start: %w\", stateErr.Cause)\n    }\n    return err\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := services.AwaitRunning(ctx, svc); err != nil {\n    root := errors.Unwrap(err)\n    level.Error(logger).Log(\"msg\", \"service failed\", \"root\", root)\n    return err\n}","preventionTips":["Always unwrap to the root failure before reacting.","Keep StartingFn errors descriptive with context.","Ensure dependencies are healthy before starting the service.","Set startup timeouts proportional to dependency readiness probes."],"tags":["lifecycle","state-machine","wrapped-error"],"backgroundTag":"invalid-state-transition","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}