dotnet/orleans · error · InconsistentStateException

DynamoDB transactional state storage conflict. PartitionKey=

Error message

DynamoDB transactional state storage conflict. PartitionKey={key.PartitionKey}. {operations}

What it means

Thrown by BatchOperation.FlushCore when DynamoDB's TransactWriteItemsAsync returns a TransactionCanceledException whose CancellationReasons contain a 'ConditionalCheckFailed' (detected by IsStorageConflict). The library translates this into an InconsistentStateException with a detailed conflict message built by GetConflictMessage listing every operation's role, row key, condition, expected ETag and cancellation reason. It signals that the optimistic ETag precondition on the key row (or a state row) no longer held — another writer committed first.

Source

Thrown at src/AWS/Orleans.Transactions.DynamoDB/TransactionalState/DynamoDBTransactionalStateStorage.cs:591

                {
                    for (var i = 0; i < this.operations.Count; i++)
                    {
                        var operation = this.operations[i];
                        LogTraceBatchOpOk(logger, operation.PartitionKey, operation.RowKey, i);
                    }

                    LogTraceBatchOpOk(logger, key.PartitionKey, key.RowKey, this.operations.Count);
                }

                this.operations.Clear();
                this.operationsSize = 0;
            }
            catch (TransactionCanceledException exception) when (IsStorageConflict(exception))
            {
                LogFailures();
                var conflictMessage = GetConflictMessage(exception, keyPut);
                LogWarningTransactionalStateConflict(logger, conflictMessage);
                throw new InconsistentStateException(
                    conflictMessage,
                    storedEtag: "Unknown",
                    currentEtag: currentETag?.ToString() ?? "null",
                    exception);
            }
            catch
            {
                LogFailures();
                throw;
            }
        }

        private void LogFailures()
        {
            if (!logger.IsEnabled(LogLevel.Trace))
            {
                return;
            }

View on GitHub (pinned to fca799fa70)

Solutions

1. Let the Orleans transaction runtime retry — InconsistentStateException is the expected conflict signal and the framework re-drives the transaction after reloading state.
2. Ensure only one activation owns the grain at a time (deactivation/membership tuning) to remove competing writers.
3. If it persists, lower write contention on the grain (partition it) and add caller-side backoff.
4. Inspect the CancellationReasonCode/Message in the exception text to confirm it is 'ConditionalCheckFailed' and not another (e.g., throttling) reason.

Example fix

// InconsistentStateException is expected under contention; let the tx runtime retry:
// before
try { await grain.DoTx(); } catch (InconsistentStateException) { /* give up */ }

// after: rely on the transaction framework's built-in retry, or wrap with a retry policy
await Policy
  .Handle<InconsistentStateException>()
  .WaitAndRetryAsync(3, i => TimeSpan.FromMilliseconds(50 * i))
  .ExecuteAsync(() => grain.DoTx());

// Conditional-check conflict is the normal optimistic-concurrency signal; retry after reload
try { await grain.TxUpdate(); }
catch (InconsistentStateException ex)
{
    _logger.LogInformation("Transactional ETag conflict (stored={Stored}, current={Current}); will retry",
        ex.StoredEtag, ex.CurrentEtag);
    throw; // let the transaction framework reload + retry, or wrap in a retry policy
}

When it happens

Trigger: Produced during a transaction Confirm/Prepare when the keyPut.ConditionExpression 'ETag = :currentETag' fails because key.ETag advanced since Load(), or a state row condition failed. Triggered by two activations/silos writing the same transactional grain concurrently, or by a stale in-memory ETag after a previous failed flush.

Common situations: Concurrent writers on the same transactional grain across silos; a grain that did not reload after a prior InconsistentStateException; race between activation migration and an in-flight transaction; throttling delaying one writer so another wins.

Related errors

• Could not load a consistent DynamoDB transactional state sna
• The transactional state storage provider name is required.
• Configuration for DynamoDBTransactionalStateStorage {this.na
• Configuration for DynamoDBTransactionalStateStorage {this.na
• Configuration for DynamoDBTransactionalStateStorage {this.na

AI-assisted analysis of dotnet/orleans@fca799fa70 (2026-08-13). Data as JSON: /api/errors/c23d8cd2e9e8b34a. Report an issue: GitHub.