dotnet/orleans · error · InconsistentStateException

Could not load a consistent DynamoDB transactional state sna

Error message

Could not load a consistent DynamoDB transactional state snapshot.

What it means

Thrown by LoadSnapshotAsync when reading a transactional state snapshot from DynamoDB. The method reads the key entity (with its ETag) before and after reading the state entities; if the ETag changed between the two reads it means a concurrent write interleaved, so the snapshot is inconsistent. After MaxSnapshotLoadAttempts (5) consecutive failures to get a stable ETag, it throws InconsistentStateException with the stored and current ETags. It reflects DynamoDB's lack of a multi-item consistent read across the key row plus all state rows.

Source

Thrown at src/AWS/Orleans.Transactions.DynamoDB/TransactionalState/DynamoDBTransactionalStateStorage.cs:365

        return keyEntity ?? new KeyEntity(this.partitionKey);
    }

    private async Task<(KeyEntity Key, List<KeyValuePair<long, StateEntity>> States)> LoadSnapshotAsync()
    {
        KeyEntity keyBefore = null!;
        KeyEntity keyAfter = null!;
        for (var attempt = 0; attempt < MaxSnapshotLoadAttempts; attempt++)
        {
            keyBefore = await LoadKeyEntityAsync().ConfigureAwait(false);
            var stateEntities = await LoadStateEntitiesAsync().ConfigureAwait(false);
            keyAfter = await LoadKeyEntityAsync().ConfigureAwait(false);
            if (keyBefore.ETag == keyAfter.ETag)
            {
                return (keyAfter, stateEntities);
            }
        }

        throw new InconsistentStateException(
            "Could not load a consistent DynamoDB transactional state snapshot.",
            storedEtag: keyBefore.ETag?.ToString() ?? "null",
            currentEtag: keyAfter.ETag?.ToString() ?? "null");
    }

    /// <summary>
    /// Loads all unpublished StateEntity records from DynamoDB.
    /// </summary>
    private async Task<List<KeyValuePair<long, StateEntity>>> LoadStateEntitiesAsync()
    {
        var keyConditionExpression =
            $"{DynamoDBTransactionalStateConstants.PARTITION_KEY_PROPERTY_NAME} = :partitionKey and {DynamoDBTransactionalStateConstants.ROW_KEY_PROPERTY_NAME} between :minRowKeyPrefix and :maxRowKeyPrefix";
        var keys = new Dictionary<string, AttributeValue>
        {
            { ":partitionKey", new AttributeValue { S = this.partitionKey } },
            { ":minRowKeyPrefix", new AttributeValue { S = StateEntity.ROW_KEY_MIN } },
            { ":maxRowKeyPrefix", new AttributeValue { S = StateEntity.ROW_KEY_MAX } },
        };

View on GitHub (pinned to fca799fa70)

Solutions

1. Reduce write contention on the affected transactional grain (split the grain, partition the key, or back off concurrent callers).
2. Increase DynamoDB throughput / switch the table to on-demand (PAY_PER_REQUEST) so LoadStateEntitiesAsync completes faster and the ETag is less likely to change mid-read.
3. Confirm only one silo owns the grain at a time (silo membership/deactivation settings) to eliminate cross-silo writers.
4. If the workload is legitimately contended, let the Orleans transaction framework retry the operation at a higher level (InconsistentStateException is retryable) and add caller-side retry/backoff.

Example fix

// before: many callers hammer the same transactional grain concurrently
await txState.PerformUpdate(...);

// after: serialize access through a single owning grain / partition the key so one writer wins per partition
// and/or raise table capacity to shorten the read window
options.UseProvisionedThroughput = false; // PAY_PER_REQUEST to reduce DynamoDB throttling

// InconsistentStateException is expected under contention; the transaction framework
// reloads and retries. Wrap caller code so transient contention does not surface:
try
{
    await grain.PerformTransactionalOperation();
}
catch (InconsistentStateException ex)
{
    // log storedEtag/currentEtag for diagnostics, then rethrow or retry at a higher level
    _logger.LogWarning(ex, "Transactional snapshot contention: stored={Stored} current={Current}", ex.StoredEtag, ex.CurrentEtag);
    throw;
}

When it happens

Trigger: Produced inside DynamoDBTransactionalStateStorage<TState>.Load() -> LoadSnapshotAsync() when keyBefore.ETag != keyAfter.ETag for all 5 attempts. Triggered by a grain whose transactional state is being written very frequently by another silo/activation while this silo loads (e.g., a hot singleton transactional grain under concurrent load, or a long-running LoadStateEntitiesAsync query racing a writer).

Common situations: High-throughput transactional grains shared across silos; a rebalance/activation spike where two activations touch the same grain state; throttling on the DynamoDB table that stretches LoadStateEntitiesAsync long enough for the ETag to advance; transactional state rows growing large so the query is slow.

Related errors

• Etag does not match
• DynamoDB transactional state storage conflict. PartitionKey=
• The transactional state storage provider name is required.
• Configuration for DynamoDBTransactionalStateStorage {this.na
• Configuration for DynamoDBTransactionalStateStorage {this.na

AI-assisted analysis of dotnet/orleans@fca799fa70 (2026-08-13). Data as JSON: /api/errors/4c6ae884d14b87f1. Report an issue: GitHub.