{"record":{"id":"2960ccd2a62f6689","repo":"temporalio/temporal","slug":"failed-to-get-hsm-operation-log-v","errorCode":null,"errorMessage":"Failed to get HSM operation log: %v","messagePattern":"Failed to get HSM operation log: (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"service/history/workflow/mutable_state_impl.go","lineNumber":8207,"sourceCode":"\t\t// snapshot or mutation.\n\t\treturn\n\t}\n\n\tif !ms.transitionHistoryEnabled {\n\t\t// transition history is not enabled\n\t\treturn\n\t}\n\n\tif len(ms.executionInfo.TransitionHistory) == 0 {\n\t\t// in an unknown state\n\t\treturn\n\t}\n\n\tvar tombstones []*persistencespb.StateMachineTombstone\n\tif ms.stateMachineNode != nil {\n\t\topLog, err := ms.stateMachineNode.OpLog()\n\t\tif err != nil {\n\t\t\tpanic(fmt.Sprintf(\"Failed to get HSM operation log: %v\", err))\n\t\t}\n\n\t\tfor _, op := range opLog {\n\t\t\tif deleteOp, ok := op.(hsm.DeleteOperation); ok {\n\t\t\t\tpath := deleteOp.Path()\n\t\t\t\tif len(path) == 0 {\n\t\t\t\t\tcontinue // Skip root deletion\n\t\t\t\t}\n\n\t\t\t\ttombstone := &persistencespb.StateMachineTombstone{\n\t\t\t\t\tStateMachineKey: &persistencespb.StateMachineTombstone_StateMachinePath{\n\t\t\t\t\t\tStateMachinePath: &persistencespb.StateMachinePath{\n\t\t\t\t\t\t\tPath: make([]*persistencespb.StateMachineKey, len(path)),\n\t\t\t\t\t\t},\n\t\t\t\t\t},\n\t\t\t\t}\n\n\t\t\t\tfor i, key := range path {","sourceCodeStart":8189,"sourceCodeEnd":8225,"githubUrl":"https://github.com/temporalio/temporal/blob/bde624efd13fbd3843654058db6d9c716166318b/service/history/workflow/mutable_state_impl.go#L8189-L8225","documentation":"When collecting tombstones for a workflow's HSM (hierarchical state machine) nodes, the code calls ms.stateMachineNode.OpLog() to read the pending operation log. If that returns an error — meaning the in-memory HSM tree node is corrupted or the operation log cannot be read — the code panics with \"Failed to get HSM operation log: %v\" instead of continuing, because proceeding without the op log would drop delete tombstones and silently corrupt replication.","triggerScenarios":"Calling the mutable-state method that gathers StateMachineTombstones (during closeTransaction / generating replication state) when the stateMachineNode's OpLog() errors, i.e. an inconsistent or corrupted CHASM state machine tree in memory.","commonSituations":"CHASM/HSM internal invariant violations, bugs in state machine tree manipulation, or state restored/reloaded inconsistently mid-transaction.","solutions":["Capture the underlying error from the panic message and the workflow execution identifiers, then report it — this indicates a CHASM internal bug, not user error.","Retry the operation; if state is rebuilt from persistence the op log may load cleanly.","Check whether the workflow history/mutable state was modified by an experimental or custom CHASM feature and disable it.","Upgrade to a server version with the relevant CHASM fix if one exists."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Internal invariant — no user-side pre-check. Server operators can recover state by:\n//   temporal --ns <ns> workflow reset --wid <wid> --reset-type LastContinuedAsNew\ntemporal --ns my-ns workflow reset --wid <workflow-id> --reset-type LastContinuedAsNew","typeGuard":null,"tryCatchPattern":"// Server-side pattern (not user-catchable): convert panic to error at the\n// transaction boundary via the existing panic-to-error recovery in\n// (n *historyEngineImpl).Execute, then surface a 500 INTERNAL to the caller.","preventionTips":["Avoid experimental CHASM/HSM flags in production unless required.","Upgrade promptly when CHASM fixes are released.","Reset affected workflow executions to recover corrupted in-memory state.","Preserve the panic error text and execution IDs for bug reports."],"tags":["go","panic","hsm","chasm","internal-invariant"],"backgroundTag":"hsm-oplog-read-failed","analyzedSha":"bde624efd13fbd3843654058db6d9c716166318b","analyzedAt":"2026-09-01T07:18:39.080Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}