{"record":{"id":"1c372788e3451aaf","repo":"temporalio/temporal","slug":"cache-encountered-dirty-mutable-state-transaction","errorCode":null,"errorMessage":"Cache encountered dirty mutable state transaction","messagePattern":"Cache encountered dirty mutable state transaction","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"service/history/workflow/cache/cache.go","lineNumber":399,"sourceCode":"\t\t\t\t\t// TODO see issue #668, there are certain type or errors which can bypass the clear\n\t\t\t\t\twfContext.Clear()\n\t\t\t\t\twfContext.Unlock()\n\t\t\t\t\tc.Release(cacheKey)\n\t\t\t\t} else {\n\t\t\t\t\tisDirty := wfContext.IsDirty()\n\t\t\t\t\tif isDirty {\n\t\t\t\t\t\twfContext.Clear()\n\t\t\t\t\t\tsoftassert.Fail(shardContext.GetLogger(), \"Cache encountered dirty mutable state transaction\",\n\t\t\t\t\t\t\ttag.ComponentHistoryCache,\n\t\t\t\t\t\t\ttag.WorkflowNamespaceID(wfContext.GetWorkflowKey().NamespaceID),\n\t\t\t\t\t\t\ttag.WorkflowID(wfContext.GetWorkflowKey().WorkflowID),\n\t\t\t\t\t\t\ttag.WorkflowRunID(wfContext.GetWorkflowKey().RunID),\n\t\t\t\t\t\t)\n\t\t\t\t\t}\n\t\t\t\t\twfContext.Unlock()\n\t\t\t\t\tc.Release(cacheKey)\n\t\t\t\t\tif isDirty {\n\t\t\t\t\t\tpanic(\"Cache encountered dirty mutable state transaction\")\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n}\n\nfunc (c *cacheImpl) validateWorkflowExecutionInfo(\n\tctx context.Context,\n\tshardContext historyi.ShardContext,\n\tnamespaceID namespace.ID,\n\texecution *commonpb.WorkflowExecution,\n\tarchetypeID chasm.ArchetypeID,\n\tlockPriority locks.Priority,\n) error {\n\n\tif err := c.validateWorkflowID(execution.GetWorkflowId()); err != nil {\n\t\treturn err","sourceCodeStart":381,"sourceCodeEnd":417,"githubUrl":"https://github.com/temporalio/temporal/blob/bde624efd13fbd3843654058db6d9c716166318b/service/history/workflow/cache/cache.go#L381-L417","documentation":"The workflow cache's eviction/teardown path asserts that when a workflow context is released from the cache it is not still holding an in-flight (dirty) mutable-state transaction. If isDirty is true at release time the cache would leak or double-apply a transaction, so it panics to surface the lock/ordering invariant violation instead of corrupting workflow state.","triggerScenarios":"A workflow context is evicted (e.g. cache size pressure or Release after processing) while its mutable-state transaction is still uncommitted/dirty — caused by code paths that unlock the context or release it without committing/aborting the transaction first, or a missed error return before release.","commonSituations":"Racing between cache eviction and an ongoing UpdateWorkflowExecution; error-handling paths that Release the cache entry without rolling back; shard unload during in-flight updates; bugs introduced in custom mutable-state update code.","solutions":["Find the code path that called Unlock/Release without commit: ensure every acquisition of the update lock commits or aborts the transaction before releasing the cache entry.","Audit error branches around UpdateWorkflowExecution calls — on error, call Abort/UnlockWorkflow before c.Release(cacheKey).","Reproduce with cache size set very small (force eviction) and concurrent workflow updates to confirm the fix.","Check for known upstream fixes: this invariant panic often follows changes to workflow cache or mutable-state locking; upgrade if a patch addresses it.","Capture the panic stack (history service crash log) and map the goroutine to the release site to identify the missing commit."],"exampleFix":"// before\nwfContext.Unlock()\nc.Release(cacheKey)\n\n// after\nif err != nil {\n    wfContext.Abort() // rolls back the dirty transaction\n} else {\n    wfContext.Commit()\n}\nwfContext.Unlock()\nc.Release(cacheKey)","handlingStrategy":"try-catch","validationCode":"if wfContext.IsUpdateLocked() || wfContext.HasPendingTransaction() {\n    return // do not release while a transaction is in flight\n}","typeGuard":null,"tryCatchPattern":"// This panic is a shard-level invariant crash, not a recoverable error.\n// Catch it only at the service boundary to capture state before exiting:\nfunc safeRelease(wfContext workflow.Context, c cache.Cache, key cache.Key) (err error) {\n    defer func() {\n        if r := recover(); r != nil {\n            err = fmt.Errorf(\"cache release invariant violated: %v\", r)\n            logger.Error(err.Error())\n        }\n    }()\n    wfContext.Unlock()\n    c.Release(key)\n    return nil\n}","preventionTips":["Ensure every acquired update lock is committed or aborted before Unlock/Release","Audit all error branches around UpdateWorkflowExecution for missing aborts","Test with a small cache size to force evictions during concurrent updates","Keep workflow-cache/mutable-state code in sync with upstream fixes"],"tags":["workflow-cache","mutable-state","panic","invariant-violation","concurrency"],"backgroundTag":"dirty-transaction-on-cache-release","analyzedSha":"bde624efd13fbd3843654058db6d9c716166318b","analyzedAt":"2026-09-01T07:18:39.080Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}