temporalio/temporal · warning

ErrStaleState

ErrStaleState

Error message

%w: %w

What it means

Fallback validation (used when transition history is disabled or the node has no transition history) treats a missing HSM node as ErrStaleState only when potentialStaleState is true — i.e. the mutable state itself may be behind the ref (TaskID==0 indicates mutable state possibly not yet replicated). It signals the caller that waiting/re-checking may resolve it.

Source

Thrown at service/history/statemachine_environment.go:300

		// fallback to the old validation logic.
		return e.validateStateMachineRefWithoutTransitionHistory(ms, ref, potentialStaleState)
	}
	if ref.Validate == nil {
		return nil
	}
	return ref.Validate(ref.StateMachineRef, node)
}

func (e *stateMachineEnvironment) validateStateMachineRefWithoutTransitionHistory(ms historyi.MutableState, ref hsm.Ref, potentialStaleState bool) error {
	// Ignore potentialStaleState if the reference cannot reference stale state (e.g if it came from task executor and
	// not an API request).
	potentialStaleState = potentialStaleState && ref.TaskID == 0

	node, err := ms.HSM().Child(ref.StateMachinePath())
	if err != nil {
		if errors.Is(err, hsm.ErrStateMachineNotFound) {
			if potentialStaleState {
				return fmt.Errorf("%w: %w", consts.ErrStaleState, err)
			}
			// We checked above that mutable state is up-to-date with our ref. If we can't find the state machine node,
			// we must assume the reference is stale.
			// This isn't bulletproof since the ref may have been generated on a different cluster and come from an API
			// request before the state has been replicated to the current cluster.
			// We accept the imperfection here and plan to solve it with the introduction of transition history.
			return fmt.Errorf("%w: %w", consts.ErrStaleReference, err)
		}
		return fmt.Errorf("%w: %w", serviceerror.NewInternal("node lookup failed"), err)
	}

	if node.InternalRepr().InitialVersionedTransition.NamespaceFailoverVersion !=
		ref.StateMachineRef.MachineInitialVersionedTransition.NamespaceFailoverVersion {
		if potentialStaleState {
			return fmt.Errorf("%w: state machine ref initial failover version mismatch", consts.ErrStaleState)
		}
		return fmt.Errorf("%w: state machine ref initial failover version mismatch", consts.ErrStaleReference)
	}

View on GitHub (pinned to bde624efd1)

Solutions

1. Retry after replication catches up; this may be a transient cross-cluster lag condition
2. Verify cluster replication health (replication queue depth, DLQ) if it recurs
3. Enable transition history so unambiguous stale-reference detection is used instead of the heuristic fallback

func isStaleState(err error) bool { return errors.Is(err, consts.ErrStaleState) }

if err := validateStateMachineRef(ms, ref, potentialStale); err != nil {
	if errors.Is(err, consts.ErrStaleState) {
		// transient: replication may not have caught up; retry with backoff
		return retryLater(ctx, task)
	}
	return err
}

When it happens

Trigger: validateStateMachineRefWithoutTransitionHistory, called from validateStateMachineRef when TransitionCount==0; Child() returns ErrStateMachineNotFound and potentialStaleState was true (mutable state possibly stale AND ref.TaskID==0).

Common situations: Cross-cluster reads where the target state machine hasn't replicated yet; task delivered to a standby cluster before replication of the node; replication lag spikes.

Related errors

• ErrStaleReference
• %w: %w
• Failed to get HSM operation log: %v
• ErrInvalidTransition
• failed to deserialize component: %w

AI-assisted analysis of temporalio/temporal@bde624efd1 (2026-09-01). Data as JSON: /api/errors/e33f7eab73f7c8a9. Report an issue: GitHub.