hashicorp/nomad · error

unexpected state %s

Error message

unexpected state %s

What it means

The task lifecycle Coordinator is a finite state machine (FSM); nextStateLocked computes the next state from currentState via an exhaustive switch. This panic fires when currentState holds a value outside the known state set — by design unreachable in correct code, so it signals a corrupted or uninitialized FSM state rather than a user-facing failure.

Source

Thrown at client/allocrunner/tasklifecycle/coordinator.go:210

	case coordinatorStatePoststart:
		if !c.isPoststartDone(states) {
			return coordinatorStatePoststart
		}
		return coordinatorStateWaitAlloc

	case coordinatorStateWaitAlloc:
		if !c.isAllocDone(states) {
			return coordinatorStateWaitAlloc
		}
		return coordinatorStatePoststop

	case coordinatorStatePoststop:
		return coordinatorStatePoststop
	}

	// If the code reaches here it's a programming error, since the switch
	// statement should cover all possible states and return the next state.
	panic(fmt.Sprintf("unexpected state %s", c.currentState))
}

// enterStateLocked updates the current state of the Coordinator FSM and
// executes any action necessary for the state transition.
// The currentStateLock must be held before calling this method.
func (c *Coordinator) enterStateLocked(state coordinatorState) {
	c.logger.Trace("state transition", "from", c.currentState, "to", state)

	switch state {
	case coordinatorStateInit:
		c.block(lifecycleStagePrestartEphemeral)
		c.block(lifecycleStagePrestartSidecar)
		c.block(lifecycleStageMain)
		c.block(lifecycleStagePoststartEphemeral)
		c.block(lifecycleStagePoststartSidecar)
		c.block(lifecycleStagePoststop)

	case coordinatorStatePrestart:

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Add a case in nextStateLocked for the missing coordinatorState value
2. Check the state value printed in the panic message against the coordinatorState enum to identify the unknown state
3. Ensure Coordinator instances are created via the constructor so currentState is always initialized
4. File/inspect a bug report with the state string from the panic — this should never happen in production

Example fix

// before
switch c.currentState {
case coordinatorStateInit:
  ...
}
// after
switch c.currentState {
case coordinatorStateInit:
  ...
case coordinatorStateNewState:
  return coordinatorStateNewState // newly added state now handled
}

// before triggering transitions
func validState(s coordinatorState) bool {
    switch s {
    case coordinatorStateInit, coordinatorStatePrestart,
        coordinatorStateRunning, coordinatorStatePoststop:
        return true
    }
    return false
}
if !validState(c.currentState) { return }

// Go: recover around the coordinator call
defer func() {
    if r := recover(); r != nil {
        logger.Error("coordinator FSM invalid state", "panic", r, "state", fmt.Sprint(c.currentState))
    }
}()

When it happens

Trigger: TaskStateUpdated -> nextStateLocked is called while c.currentState is a zero-value/uninitialized coordinatorState or one added to the enum without a corresponding case in the switch.

Common situations: A developer adds a new coordinatorState constant but forgets to add a case in nextStateLocked; state struct initialized without going through NewCoordinator; memory corruption or unsafe state writes bypassing currentStateLock.

Related errors

• Unexpected task coordinator state %d
• cannot open cpuset
• invalid acl policy: %v
• cannot prepare consul tokens, no task specified
• host volume updates already batched

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/dbc52ae762f8cd3c. Report an issue: GitHub.