{"record":{"id":"4c6ae884d14b87f1","repo":"dotnet/orleans","slug":"could-not-load-a-consistent-dynamodb-transactional","errorCode":null,"errorMessage":"Could not load a consistent DynamoDB transactional state snapshot.","messagePattern":"Could not load a consistent DynamoDB transactional state snapshot\\.","errorType":"exception","errorClass":"InconsistentStateException","httpStatus":null,"severity":"error","filePath":"src/AWS/Orleans.Transactions.DynamoDB/TransactionalState/DynamoDBTransactionalStateStorage.cs","lineNumber":365,"sourceCode":"        return keyEntity ?? new KeyEntity(this.partitionKey);\n    }\n\n    private async Task<(KeyEntity Key, List<KeyValuePair<long, StateEntity>> States)> LoadSnapshotAsync()\n    {\n        KeyEntity keyBefore = null!;\n        KeyEntity keyAfter = null!;\n        for (var attempt = 0; attempt < MaxSnapshotLoadAttempts; attempt++)\n        {\n            keyBefore = await LoadKeyEntityAsync().ConfigureAwait(false);\n            var stateEntities = await LoadStateEntitiesAsync().ConfigureAwait(false);\n            keyAfter = await LoadKeyEntityAsync().ConfigureAwait(false);\n            if (keyBefore.ETag == keyAfter.ETag)\n            {\n                return (keyAfter, stateEntities);\n            }\n        }\n\n        throw new InconsistentStateException(\n            \"Could not load a consistent DynamoDB transactional state snapshot.\",\n            storedEtag: keyBefore.ETag?.ToString() ?? \"null\",\n            currentEtag: keyAfter.ETag?.ToString() ?? \"null\");\n    }\n\n    /// <summary>\n    /// Loads all unpublished StateEntity records from DynamoDB.\n    /// </summary>\n    private async Task<List<KeyValuePair<long, StateEntity>>> LoadStateEntitiesAsync()\n    {\n        var keyConditionExpression =\n            $\"{DynamoDBTransactionalStateConstants.PARTITION_KEY_PROPERTY_NAME} = :partitionKey and {DynamoDBTransactionalStateConstants.ROW_KEY_PROPERTY_NAME} between :minRowKeyPrefix and :maxRowKeyPrefix\";\n        var keys = new Dictionary<string, AttributeValue>\n        {\n            { \":partitionKey\", new AttributeValue { S = this.partitionKey } },\n            { \":minRowKeyPrefix\", new AttributeValue { S = StateEntity.ROW_KEY_MIN } },\n            { \":maxRowKeyPrefix\", new AttributeValue { S = StateEntity.ROW_KEY_MAX } },\n        };","sourceCodeStart":347,"sourceCodeEnd":383,"githubUrl":"https://github.com/dotnet/orleans/blob/fca799fa70ecb6ad975224271703ca43221f58de/src/AWS/Orleans.Transactions.DynamoDB/TransactionalState/DynamoDBTransactionalStateStorage.cs#L347-L383","documentation":"Thrown by LoadSnapshotAsync when reading a transactional state snapshot from DynamoDB. The method reads the key entity (with its ETag) before and after reading the state entities; if the ETag changed between the two reads it means a concurrent write interleaved, so the snapshot is inconsistent. After MaxSnapshotLoadAttempts (5) consecutive failures to get a stable ETag, it throws InconsistentStateException with the stored and current ETags. It reflects DynamoDB's lack of a multi-item consistent read across the key row plus all state rows.","triggerScenarios":"Produced inside DynamoDBTransactionalStateStorage<TState>.Load() -> LoadSnapshotAsync() when keyBefore.ETag != keyAfter.ETag for all 5 attempts. Triggered by a grain whose transactional state is being written very frequently by another silo/activation while this silo loads (e.g., a hot singleton transactional grain under concurrent load, or a long-running LoadStateEntitiesAsync query racing a writer).","commonSituations":"High-throughput transactional grains shared across silos; a rebalance/activation spike where two activations touch the same grain state; throttling on the DynamoDB table that stretches LoadStateEntitiesAsync long enough for the ETag to advance; transactional state rows growing large so the query is slow.","solutions":["Reduce write contention on the affected transactional grain (split the grain, partition the key, or back off concurrent callers).","Increase DynamoDB throughput / switch the table to on-demand (PAY_PER_REQUEST) so LoadStateEntitiesAsync completes faster and the ETag is less likely to change mid-read.","Confirm only one silo owns the grain at a time (silo membership/deactivation settings) to eliminate cross-silo writers.","If the workload is legitimately contended, let the Orleans transaction framework retry the operation at a higher level (InconsistentStateException is retryable) and add caller-side retry/backoff."],"exampleFix":"// before: many callers hammer the same transactional grain concurrently\nawait txState.PerformUpdate(...);\n\n// after: serialize access through a single owning grain / partition the key so one writer wins per partition\n// and/or raise table capacity to shorten the read window\noptions.UseProvisionedThroughput = false; // PAY_PER_REQUEST to reduce DynamoDB throttling","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// InconsistentStateException is expected under contention; the transaction framework\n// reloads and retries. Wrap caller code so transient contention does not surface:\ntry\n{\n    await grain.PerformTransactionalOperation();\n}\ncatch (InconsistentStateException ex)\n{\n    // log storedEtag/currentEtag for diagnostics, then rethrow or retry at a higher level\n    _logger.LogWarning(ex, \"Transactional snapshot contention: stored={Stored} current={Current}\", ex.StoredEtag, ex.CurrentEtag);\n    throw;\n}","preventionTips":["Avoid multiple concurrent writers on the same transactional grain; route updates through a single owning activation.","Keep the table well-provisioned (or on-demand) so LoadStateEntitiesAsync is fast and the ETag window is short.","Monitor DynamoDB throttling and fix it before it stretches snapshot reads.","Tune silo grain-placement/deactivation so only one activation owns a contended grain at a time."],"tags":["dynamodb","transactions","concurrency","optimistic-concurrency","snapshot"],"backgroundTag":null,"analyzedSha":"fca799fa70ecb6ad975224271703ca43221f58de","analyzedAt":"2026-08-13T19:55:57.938Z","schemaVersion":2},"datasetVersion":"2026-08-14T00:17:13.853Z"}