{"record":{"id":"e33f7eab73f7c8a9","repo":"temporalio/temporal","slug":"errstalestate","errorCode":"ErrStaleState","errorMessage":"%w: %w","messagePattern":"%w: %w","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"service/history/statemachine_environment.go","lineNumber":300,"sourceCode":"\t\t// fallback to the old validation logic.\n\t\treturn e.validateStateMachineRefWithoutTransitionHistory(ms, ref, potentialStaleState)\n\t}\n\tif ref.Validate == nil {\n\t\treturn nil\n\t}\n\treturn ref.Validate(ref.StateMachineRef, node)\n}\n\nfunc (e *stateMachineEnvironment) validateStateMachineRefWithoutTransitionHistory(ms historyi.MutableState, ref hsm.Ref, potentialStaleState bool) error {\n\t// Ignore potentialStaleState if the reference cannot reference stale state (e.g if it came from task executor and\n\t// not an API request).\n\tpotentialStaleState = potentialStaleState && ref.TaskID == 0\n\n\tnode, err := ms.HSM().Child(ref.StateMachinePath())\n\tif err != nil {\n\t\tif errors.Is(err, hsm.ErrStateMachineNotFound) {\n\t\t\tif potentialStaleState {\n\t\t\t\treturn fmt.Errorf(\"%w: %w\", consts.ErrStaleState, err)\n\t\t\t}\n\t\t\t// We checked above that mutable state is up-to-date with our ref. If we can't find the state machine node,\n\t\t\t// we must assume the reference is stale.\n\t\t\t// This isn't bulletproof since the ref may have been generated on a different cluster and come from an API\n\t\t\t// request before the state has been replicated to the current cluster.\n\t\t\t// We accept the imperfection here and plan to solve it with the introduction of transition history.\n\t\t\treturn fmt.Errorf(\"%w: %w\", consts.ErrStaleReference, err)\n\t\t}\n\t\treturn fmt.Errorf(\"%w: %w\", serviceerror.NewInternal(\"node lookup failed\"), err)\n\t}\n\n\tif node.InternalRepr().InitialVersionedTransition.NamespaceFailoverVersion !=\n\t\tref.StateMachineRef.MachineInitialVersionedTransition.NamespaceFailoverVersion {\n\t\tif potentialStaleState {\n\t\t\treturn fmt.Errorf(\"%w: state machine ref initial failover version mismatch\", consts.ErrStaleState)\n\t\t}\n\t\treturn fmt.Errorf(\"%w: state machine ref initial failover version mismatch\", consts.ErrStaleReference)\n\t}","sourceCodeStart":282,"sourceCodeEnd":318,"githubUrl":"https://github.com/temporalio/temporal/blob/bde624efd13fbd3843654058db6d9c716166318b/service/history/statemachine_environment.go#L282-L318","documentation":"Fallback validation (used when transition history is disabled or the node has no transition history) treats a missing HSM node as ErrStaleState only when potentialStaleState is true — i.e. the mutable state itself may be behind the ref (TaskID==0 indicates mutable state possibly not yet replicated). It signals the caller that waiting/re-checking may resolve it.","triggerScenarios":"validateStateMachineRefWithoutTransitionHistory, called from validateStateMachineRef when TransitionCount==0; Child() returns ErrStateMachineNotFound and potentialStaleState was true (mutable state possibly stale AND ref.TaskID==0).","commonSituations":"Cross-cluster reads where the target state machine hasn't replicated yet; task delivered to a standby cluster before replication of the node; replication lag spikes.","solutions":["Retry after replication catches up; this may be a transient cross-cluster lag condition","Verify cluster replication health (replication queue depth, DLQ) if it recurs","Enable transition history so unambiguous stale-reference detection is used instead of the heuristic fallback"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":"func isStaleState(err error) bool { return errors.Is(err, consts.ErrStaleState) }","tryCatchPattern":"if err := validateStateMachineRef(ms, ref, potentialStale); err != nil {\n\tif errors.Is(err, consts.ErrStaleState) {\n\t\t// transient: replication may not have caught up; retry with backoff\n\t\treturn retryLater(ctx, task)\n\t}\n\treturn err\n}","preventionTips":["Bound retries and fall back to discarding after the replication window lapses","Monitor replication queue lag to size the retry window","Prefer enabling transition history to remove the heuristic"],"tags":["go","chasm","hsm","replication","stale-state"],"backgroundTag":"stale-state-machine-reference","analyzedSha":"bde624efd13fbd3843654058db6d9c716166318b","analyzedAt":"2026-09-01T07:18:39.080Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}