temporalio/temporal · critical

Failed to get HSM operation log: %v

Error message

Failed to get HSM operation log: %v

What it means

When collecting tombstones for a workflow's HSM (hierarchical state machine) nodes, the code calls ms.stateMachineNode.OpLog() to read the pending operation log. If that returns an error — meaning the in-memory HSM tree node is corrupted or the operation log cannot be read — the code panics with "Failed to get HSM operation log: %v" instead of continuing, because proceeding without the op log would drop delete tombstones and silently corrupt replication.

Source

Thrown at service/history/workflow/mutable_state_impl.go:8207

		// snapshot or mutation.
		return
	}

	if !ms.transitionHistoryEnabled {
		// transition history is not enabled
		return
	}

	if len(ms.executionInfo.TransitionHistory) == 0 {
		// in an unknown state
		return
	}

	var tombstones []*persistencespb.StateMachineTombstone
	if ms.stateMachineNode != nil {
		opLog, err := ms.stateMachineNode.OpLog()
		if err != nil {
			panic(fmt.Sprintf("Failed to get HSM operation log: %v", err))
		}

		for _, op := range opLog {
			if deleteOp, ok := op.(hsm.DeleteOperation); ok {
				path := deleteOp.Path()
				if len(path) == 0 {
					continue // Skip root deletion
				}

				tombstone := &persistencespb.StateMachineTombstone{
					StateMachineKey: &persistencespb.StateMachineTombstone_StateMachinePath{
						StateMachinePath: &persistencespb.StateMachinePath{
							Path: make([]*persistencespb.StateMachineKey, len(path)),
						},
					},
				}

				for i, key := range path {

View on GitHub (pinned to bde624efd1)

Solutions

  1. Capture the underlying error from the panic message and the workflow execution identifiers, then report it — this indicates a CHASM internal bug, not user error.
  2. Retry the operation; if state is rebuilt from persistence the op log may load cleanly.
  3. Check whether the workflow history/mutable state was modified by an experimental or custom CHASM feature and disable it.
  4. Upgrade to a server version with the relevant CHASM fix if one exists.
Defensive patterns

Strategy: try-catch

Validate before calling

// Internal invariant — no user-side pre-check. Server operators can recover state by:
//   temporal --ns <ns> workflow reset --wid <wid> --reset-type LastContinuedAsNew
temporal --ns my-ns workflow reset --wid <workflow-id> --reset-type LastContinuedAsNew

Try / catch

// Server-side pattern (not user-catchable): convert panic to error at the
// transaction boundary via the existing panic-to-error recovery in
// (n *historyEngineImpl).Execute, then surface a 500 INTERNAL to the caller.

Prevention

When it happens

Trigger: Calling the mutable-state method that gathers StateMachineTombstones (during closeTransaction / generating replication state) when the stateMachineNode's OpLog() errors, i.e. an inconsistent or corrupted CHASM state machine tree in memory.

Common situations: CHASM/HSM internal invariant violations, bugs in state machine tree manipulation, or state restored/reloaded inconsistently mid-transaction.

Related errors

AI-assisted analysis of temporalio/temporal@bde624efd1 (2026-09-01). Data as JSON: /api/errors/2960ccd2a62f6689. Report an issue: GitHub.