thanos-io/thanos · error

invalid service state

Error message

invalid service state: %v, expected: %v

What it means

invalidServiceStateError reports that a BasicService (Cortex services framework) was observed in a different lifecycle state than required by StartAsync or awaitState. The message includes the actual state and the expected one, e.g. calling StartAsync on a service that is already running or was terminated.

Solutions

1. Log/inspect the actual vs expected state in the message; if actual is Failed, find the root cause via invalidServiceStateWithFailureError or service failure logs.
2. Ensure StartAsync is called exactly once per service instance, from a single owner (module manager).
3. Use service.Manager / observers to wait for dependencies to reach Running before starting dependents.
4. Do not call StartAsync after Terminate; create a fresh service instance instead.

Example fix

// before
svc.StartAsync(ctx)
svc.AwaitRunning(ctx) // panics/errors if svc already started
// after
if svc.State() == services.New {
    svc.StartAsync(ctx)
    if err := svc.StartAsyncErr(); err != nil { /* handle */ }
    if err := services.AwaitRunning(ctx, svc); err != nil { /* handle */ }
}

if svc.State() != services.New {
    return fmt.Errorf("cannot start service in state %v", svc.State())
}
svc.StartAsync(ctx)

func canStart(s services.Service) bool { return s.State() == services.New }

if err := services.AwaitRunning(ctx, svc); err != nil {
    return fmt.Errorf("service did not reach Running: %w", err)
}

When it happens

Trigger: StartAsync called on a service not in New state; awaitState observed a state transition to anything other than the expected state (e.g., the service failed or was stopped while waiting for Running).

Common situations: Double-starting a module by wiring the same service into a module manager twice; a dependency service failing during startup causing dependents to see Failed instead of Running; tests restarting services without proper termination; race between StopAsync and StartAsync.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• invalid service state
• failed to run pre compaction callback for plan
• hashring config watcher stopped unexpectedly
• block querier already closed

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/c3eff433da3db64a. Report an issue: GitHub.

Appendix: source

Thrown at internal/cortex/util/services/basic_service.go:76

	// everything below is protected by this mutex
	stateMu     sync.RWMutex
	state       State
	failureCase error
	listeners   []chan func(l Listener)
	serviceName string

	// closed when state reaches Running, Terminated or Failed state
	runningWaitersCh chan struct{}
	// closed when state reaches Terminated or Failed state
	terminatedWaitersCh chan struct{}

	serviceContext context.Context
	serviceCancel  context.CancelFunc
}

func invalidServiceStateError(state, expected State) error {
	return fmt.Errorf("invalid service state: %v, expected: %v", state, expected)
}

func invalidServiceStateWithFailureError(state, expected State, failure error) error {
	return fmt.Errorf("invalid service state: %v, expected: %v, failure: %w", state, expected, failure)
}

// NewBasicService returns service built from three functions (using BasicService).
func NewBasicService(start StartingFn, run RunningFn, stop StoppingFn) *BasicService {
	return &BasicService{
		startFn:             start,
		runningFn:           run,
		stoppingFn:          stop,
		state:               New,
		runningWaitersCh:    make(chan struct{}),
		terminatedWaitersCh: make(chan struct{}),
	}
}

View on GitHub (pinned to 35b8b99117)