dotnet/orleans · error · CosmosConditionNotSatisfiedException

Cosmos DB condition not satisfied. GrainType: {0}, GrainId:

Error message

Cosmos DB condition not satisfied. GrainType: {0}, GrainId: {1}, TableName: {2}, StoredETag: {3}, CurrentETag: {4}

What it means

Thrown by CosmosGrainStorage.WriteStateAsync when a CosmosException occurs with PreconditionFailed (412), Conflict (409), or NotFound (404) status. This is an optimistic ETag concurrency violation during a ReplaceItemAsync — the stored document's ETag does not match the ETag the client sent, indicating a concurrent modification or that the document was deleted.

Source

Thrown at src/Azure/Orleans.Persistence.Cosmos/CosmosGrainStorage.cs:154

            }
            else
            {
                var requestOptions = new ItemRequestOptions { IfMatchEtag = grainState.ETag };
                response = await _executor.ExecuteOperation(
                    static args =>
                    {
                        var (self, entity, pk, requestOptions) = args;
                        return self._container.ReplaceItemAsync(entity, entity.Id, pk, requestOptions);
                    },
                    (this, entity, pk, requestOptions)).ConfigureAwait(false);
            }

            grainState.ETag = response.Resource.ETag;
            grainState.RecordExists = true;
        }
        catch (CosmosException ex) when (ex.StatusCode is HttpStatusCode.PreconditionFailed or HttpStatusCode.Conflict or HttpStatusCode.NotFound)
        {
            throw new CosmosConditionNotSatisfiedException(grainType, grainId, _options.ContainerName, "Unknown", grainState.ETag);
        }
        catch (Exception exc)
        {
            LogErrorWritingState(exc, grainType, id);
            WrappedException.CreateAndRethrow(exc);
            throw;
        }
    }

    public async Task ClearStateAsync<T>(string grainType, GrainId grainId, IGrainState<T> grainState)
    {
        var (id, partitionKey) = await _documentIdProvider.GetDocumentIdentifiers(grainType, grainId);

        LogTraceClearingState(grainType, id, grainId, grainState.ETag, _options.DeleteStateOnClear, _options.ContainerName, partitionKey);

        var pk = new PartitionKey(partitionKey);
        var requestOptions = new ItemRequestOptions { IfMatchEtag = grainState.ETag };
        try

View on GitHub (pinned to fca799fa70)

Solutions

Catch CosmosConditionNotSatisfiedException and re-read/retry the operation.
Ensure single-activation semantics for the grain to avoid concurrent writes from different silos.
Reduce write contention by batching updates within a single grain method call.
Configure the Cosmos executor retry policy for transient conflicts.

Defensive patterns

Strategy: retry

Try / catch

try { await grain.WriteStateAsync(); }
catch (CosmosConditionNotSatisfiedException ex)
{
    logger.LogWarning(ex, "Cosmos ETag conflict on {GrainType} {GrainId}.", ex.GrainType, ex.GrainId);
    // Re-read state and retry the write
    await grain.ReadStateAsync();
    await grain.WriteStateAsync();
}

Prevention

Ensure single-activation semantics to prevent concurrent writes to the same grain.
Minimize the read-modify-write window in grain methods to reduce ETag conflicts.
Monitor Cosmos 412/409 rates as a contention indicator.
Let the Orleans runtime's built-in retry handle transient conflicts when possible.

When it happens

Trigger: Grain reads state (ETag A), a concurrent write changes the document (ETag B), then this grain attempts ReplaceItemAsync with the stale ETag A. Cosmos DB rejects with HTTP 412. Also triggered by 409 Conflict or 404 NotFound if the document was simultaneously modified or removed.

Common situations: Multiple activations of the same grain across silos writing concurrently. Reentrant operations that trigger nested state writes. Race conditions in multi-silo or multi-datacenter deployments. Document deleted between read and write.

Related errors

AI-assisted analysis of dotnet/orleans@fca799fa70 (2026-08-13). Data as JSON: /api/errors/4ad835f6d5746fae. Report an issue: GitHub.