thanos-io/thanos · error

invalid service state

Error message

invalid service state: %v, expected: %v, failure: %w

What it means

invalidServiceStateWithFailureError is like invalidServiceStateError but additionally wraps the underlying failure (with %w) that caused the service to leave the expected state. awaitState uses it when a service transitions to Failed (or an unexpected state) while callers are waiting for it to become Running/Stopping.

Solutions

1. Unwrap the wrapped failure (errors.Unwrap / %w chain) to find the true root cause and fix it first.
2. Check that all dependencies the service requires (ring, store, bucket) are healthy and configured.
3. Increase startup timeouts only after the root cause is understood, not as a substitute.
4. Add failure logging via observability on the service's StartingFn/RunningFn to capture the original error.

Example fix

// before
if err := services.AwaitRunning(ctx, svc); err != nil {
    return err
}
// after
if err := services.AwaitRunning(ctx, svc); err != nil {
    var stateErr *services.FailureError
    if errors.As(err, &stateErr) {
        return fmt.Errorf("service failed to start: %w", stateErr.Cause)
    }
    return err
}

if err := services.AwaitRunning(ctx, svc); err != nil {
    root := errors.Unwrap(err)
    level.Error(logger).Log("msg", "service failed", "root", root)
    return err
}

When it happens

Trigger: awaitState observes the service enter Failed during StartAsync/AwaitRunning; the underlying starting/running function returned an error which is carried in the failure field of the message.

Common situations: Dependency initialization failures (DB unreachable, bucket credentials invalid, ring not ready) surfacing through a dependent service's AwaitRunning; misconfigured module parameters; resource exhaustion during startup.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• invalid service state
• get file
• read file
• failed to run pre compaction callback for plan
• lookup IP addresses

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/b825c2ccaff602c6. Report an issue: GitHub.

Appendix: source

Thrown at internal/cortex/util/services/basic_service.go:80

	failureCase error
	listeners   []chan func(l Listener)
	serviceName string

	// closed when state reaches Running, Terminated or Failed state
	runningWaitersCh chan struct{}
	// closed when state reaches Terminated or Failed state
	terminatedWaitersCh chan struct{}

	serviceContext context.Context
	serviceCancel  context.CancelFunc
}

func invalidServiceStateError(state, expected State) error {
	return fmt.Errorf("invalid service state: %v, expected: %v", state, expected)
}

func invalidServiceStateWithFailureError(state, expected State, failure error) error {
	return fmt.Errorf("invalid service state: %v, expected: %v, failure: %w", state, expected, failure)
}

// NewBasicService returns service built from three functions (using BasicService).
func NewBasicService(start StartingFn, run RunningFn, stop StoppingFn) *BasicService {
	return &BasicService{
		startFn:             start,
		runningFn:           run,
		stoppingFn:          stop,
		state:               New,
		runningWaitersCh:    make(chan struct{}),
		terminatedWaitersCh: make(chan struct{}),
	}
}

// WithName sets service name, if service is still in New state, and returns service to allow
// usage like NewBasicService(...).WithName("service name").
func (b *BasicService) WithName(name string) *BasicService {
	// Hold lock to make sure state doesn't change while setting service name.

View on GitHub (pinned to 35b8b99117)