temporalio/temporal · critical

Cache encountered dirty mutable state transaction

Error message

Cache encountered dirty mutable state transaction

What it means

The workflow cache's eviction/teardown path asserts that when a workflow context is released from the cache it is not still holding an in-flight (dirty) mutable-state transaction. If isDirty is true at release time the cache would leak or double-apply a transaction, so it panics to surface the lock/ordering invariant violation instead of corrupting workflow state.

Source

Thrown at service/history/workflow/cache/cache.go:399

					// TODO see issue #668, there are certain type or errors which can bypass the clear
					wfContext.Clear()
					wfContext.Unlock()
					c.Release(cacheKey)
				} else {
					isDirty := wfContext.IsDirty()
					if isDirty {
						wfContext.Clear()
						softassert.Fail(shardContext.GetLogger(), "Cache encountered dirty mutable state transaction",
							tag.ComponentHistoryCache,
							tag.WorkflowNamespaceID(wfContext.GetWorkflowKey().NamespaceID),
							tag.WorkflowID(wfContext.GetWorkflowKey().WorkflowID),
							tag.WorkflowRunID(wfContext.GetWorkflowKey().RunID),
						)
					}
					wfContext.Unlock()
					c.Release(cacheKey)
					if isDirty {
						panic("Cache encountered dirty mutable state transaction")
					}
				}
			}
		}
	}
}

func (c *cacheImpl) validateWorkflowExecutionInfo(
	ctx context.Context,
	shardContext historyi.ShardContext,
	namespaceID namespace.ID,
	execution *commonpb.WorkflowExecution,
	archetypeID chasm.ArchetypeID,
	lockPriority locks.Priority,
) error {

	if err := c.validateWorkflowID(execution.GetWorkflowId()); err != nil {
		return err

View on GitHub (pinned to bde624efd1)

Solutions

1. Find the code path that called Unlock/Release without commit: ensure every acquisition of the update lock commits or aborts the transaction before releasing the cache entry.
2. Audit error branches around UpdateWorkflowExecution calls — on error, call Abort/UnlockWorkflow before c.Release(cacheKey).
3. Reproduce with cache size set very small (force eviction) and concurrent workflow updates to confirm the fix.
4. Check for known upstream fixes: this invariant panic often follows changes to workflow cache or mutable-state locking; upgrade if a patch addresses it.
5. Capture the panic stack (history service crash log) and map the goroutine to the release site to identify the missing commit.

Example fix

// before
wfContext.Unlock()
c.Release(cacheKey)

// after
if err != nil {
    wfContext.Abort() // rolls back the dirty transaction
} else {
    wfContext.Commit()
}
wfContext.Unlock()
c.Release(cacheKey)

if wfContext.IsUpdateLocked() || wfContext.HasPendingTransaction() {
    return // do not release while a transaction is in flight
}

// This panic is a shard-level invariant crash, not a recoverable error.
// Catch it only at the service boundary to capture state before exiting:
func safeRelease(wfContext workflow.Context, c cache.Cache, key cache.Key) (err error) {
    defer func() {
        if r := recover(); r != nil {
            err = fmt.Errorf("cache release invariant violated: %v", r)
            logger.Error(err.Error())
        }
    }()
    wfContext.Unlock()
    c.Release(key)
    return nil
}

When it happens

Trigger: A workflow context is evicted (e.g. cache size pressure or Release after processing) while its mutable-state transaction is still uncommitted/dirty — caused by code paths that unlock the context or release it without committing/aborting the transaction first, or a missed error return before release.

Common situations: Racing between cache eviction and an ongoing UpdateWorkflowExecution; error-handling paths that Release the cache entry without rolling back; shard unload during in-flight updates; bugs introduced in custom mutable-state update code.

Related errors

• HistoryEventIterator Next() should return either a history e
• future has already been completed
• expect at least one reservation
• Found key with non-zero pending task count but has no corres
• Unknown category type: %v

AI-assisted analysis of temporalio/temporal@bde624efd1 (2026-09-01). Data as JSON: /api/errors/1c372788e3451aaf. Report an issue: GitHub.