dotnet/orleans · error · ArgumentException

Etag does not match

Error message

Etag does not match

What it means

Thrown by DynamoDBTransactionalStateStorage.Store when the cached key ETag does not match the expectedETag supplied by the caller. This is an optimistic-concurrency check: between the caller's Load and Store, another write changed the row's ETag, so the caller's view is stale and the batch must not be applied. The exception is an ArgumentException naming expectedETag.

Source

Thrown at src/AWS/Orleans.Transactions.DynamoDB/TransactionalState/DynamoDBTransactionalStateStorage.cs:159

    /// <inheritdoc />
    public async Task<string> Store(string? expectedETag, TransactionalStateMetaData metadata, List<PendingTransactionState<TState>>? statesToPrepare, long? commitUpTo, long? abortAfter)
    {
        if (this.requiresReload)
        {
            throw new InvalidOperationException("Load must be called after a failed Store before this storage instance can be reused.");
        }

        var batchOperation = new BatchOperation(this.storage, this.tableName, this.key, this.logger);
        var keyWasNew = !this.key.ETag.HasValue;

        try
        {
            var keyETag = key.ETag?.ToString();
            if ((!string.IsNullOrWhiteSpace(keyETag) || !string.IsNullOrWhiteSpace(expectedETag)) &&
                keyETag != expectedETag)
            {
                throw new ArgumentException("Etag does not match", nameof(expectedETag));
            }

            var serializedMetadata = this.ConvertToStorageFormat(metadata);
            var timestamp = DateTimeOffset.UtcNow;
            var committedSequenceId = commitUpTo.HasValue && commitUpTo.Value > key.CommittedSequenceId
                ? commitUpTo.Value
                : key.CommittedSequenceId;
            this.ValidateKeyItemSize(serializedMetadata, timestamp, committedSequenceId);

            // Store mutates the cached entities while constructing the transaction. If any operation
            // fails, Load must restore the cache before this instance can be used again.
            this.requiresReload = true;

            // assemble all storage operations into a single batch
            // these operations must commit in sequence, but not necessarily atomically
            // so we can split this up if needed

            // first, clean up aborted records

View on GitHub (pinned to fca799fa70)

Solutions

1. Treat this as a transient optimistic-concurrency conflict: call Load to get the fresh ETag, recompute the change, and retry Store.
2. Reduce concurrency on the same transactional grain key where possible (single-writer pattern).
3. Ensure the caller always stores using the ETag returned by its most recent Load, not a cached older value.
4. Cap retries with backoff to avoid live-lock under heavy contention.

Example fix

// before
var resp = await store.Load();
// ... another writer commits here ...
await store.Store(resp.ETag, meta, prep, commit, abort); // throws: Etag mismatch

// after
try { await store.Store(resp.ETag, meta, prep, commit, abort); }
catch (ArgumentException ex) when (ex.Message.Contains("Etag"))
{
    resp = await store.Load(); // refresh ETag
    await store.Store(resp.ETag, meta, prep, commit, abort);
}

for (int attempt = 0; ; attempt++)
{
    try { return await store.Store(etag, meta, prep, commit, abort); }
    catch (ArgumentException ex) when (ex.ParamName == "expectedETag" && ex.Message.Contains("Etag"))
    {
        if (attempt >= MaxRetries) throw;
        var fresh = await store.Load(); // refresh ETag
        etag = fresh.ETag;
        // optionally recompute prep/meta against fresh.CommittedSequenceId
        await Task.Delay(Backoff(attempt));
    }
}

When it happens

Trigger: Two concurrent transactions on the same grain/state both read the same ETag, then both attempt to Store; the second sees key.ETag already advanced by the first and is rejected. Also if a caller passes a stale cached ETag after the row was written elsewhere.

Common situations: Concurrent grain activations or transaction participants contending on one transactional state; retrying Store with an ETag obtained before a successful competing write; cross-silo races on the same grain key.

Related errors

• The transactional state storage provider name is required.
• Configuration for DynamoDBTransactionalStateStorage {this.na
• Configuration for DynamoDBTransactionalStateStorage {this.na
• Configuration for DynamoDBTransactionalStateStorage {this.na
• Storage state corrupted: no record for committed state v{thi

AI-assisted analysis of dotnet/orleans@fca799fa70 (2026-08-13). Data as JSON: /api/errors/4486f58340a0089b. Report an issue: GitHub.