temporalio/temporal · critical

Failed to get HSM operation log

Error message

Failed to get HSM operation log: %v

What it means

When collecting tombstones for a workflow's HSM (hierarchical state machine) nodes, the code calls ms.stateMachineNode.OpLog() to read the pending operation log. If that returns an error — meaning the in-memory HSM tree node is corrupted or the operation log cannot be read — the code panics with "Failed to get HSM operation log: %v" instead of continuing, because proceeding without the op log would drop delete tombstones and silently corrupt replication.

Solutions

1. Capture the underlying error from the panic message and the workflow execution identifiers, then report it — this indicates a CHASM internal bug, not user error.
2. Retry the operation; if state is rebuilt from persistence the op log may load cleanly.
3. Check whether the workflow history/mutable state was modified by an experimental or custom CHASM feature and disable it.
4. Upgrade to a server version with the relevant CHASM fix if one exists.

// Internal invariant — no user-side pre-check. Server operators can recover state by:
//   temporal --ns <ns> workflow reset --wid <wid> --reset-type LastContinuedAsNew
temporal --ns my-ns workflow reset --wid <workflow-id> --reset-type LastContinuedAsNew

// Server-side pattern (not user-catchable): convert panic to error at the
// transaction boundary via the existing panic-to-error recovery in
// (n *historyEngineImpl).Execute, then surface a 500 INTERNAL to the caller.

When it happens

Trigger: Calling the mutable-state method that gathers StateMachineTombstones (during closeTransaction / generating replication state) when the stateMachineNode's OpLog() errors, i.e. an inconsistent or corrupted CHASM state machine tree in memory.

Common situations: CHASM/HSM internal invariant violations, bugs in state machine tree manipulation, or state restored/reloaded inconsistently mid-transaction.

Related errors

• component is not registered to a library
• ErrStaleReference
• ErrStaleState
• failed to encode chasm path on clean tree
• task is not registered to a library

AI-assisted analysis of temporalio/temporal@bde624efd1 (2026-09-01). Data as JSON: /api/errors/2960ccd2a62f6689. Report an issue: GitHub.

Appendix: source

Thrown at service/history/workflow/mutable_state_impl.go:8207

		// snapshot or mutation.
		return
	}

	if !ms.transitionHistoryEnabled {
		// transition history is not enabled
		return
	}

	if len(ms.executionInfo.TransitionHistory) == 0 {
		// in an unknown state
		return
	}

	var tombstones []*persistencespb.StateMachineTombstone
	if ms.stateMachineNode != nil {
		opLog, err := ms.stateMachineNode.OpLog()
		if err != nil {
			panic(fmt.Sprintf("Failed to get HSM operation log: %v", err))
		}

		for _, op := range opLog {
			if deleteOp, ok := op.(hsm.DeleteOperation); ok {
				path := deleteOp.Path()
				if len(path) == 0 {
					continue // Skip root deletion
				}

				tombstone := &persistencespb.StateMachineTombstone{
					StateMachineKey: &persistencespb.StateMachineTombstone_StateMachinePath{
						StateMachinePath: &persistencespb.StateMachinePath{
							Path: make([]*persistencespb.StateMachineKey, len(path)),
						},
					},
				}

				for i, key := range path {

View on GitHub (pinned to bde624efd1)