dotnet/orleans · error · InconsistentStateException

Blob storage condition not Satisfied. BlobName: {blob.Name},

Error message

Blob storage condition not Satisfied. BlobName: {blob.Name}, Container: {blob.BlobContainerName}, CurrentETag: {currentETag}

What it means

Thrown as InconsistentStateException by AzureBlobStorage.DoOptimisticUpdate when a blob write fails with a precondition-failed, conflict, or (non-container) not-found RequestFailedException. Orleans grain storage uses ETag-based optimistic concurrency; a mismatch means another caller mutated the blob between the read and the write, so the operation is aborted for the runtime to retry.

Source

Thrown at src/Azure/Orleans.Persistence.AzureStorage/Providers/Storage/AzureBlobStorage.cs:330

            {
                loadedState = this.ConvertFromStorageFormat<T>(new BinaryData(content));
                LogWarningLargePayloadFallback(contentLength, grainType, grainId, eTag, blobName, containerName);
            }

            LogTraceDataRead(grainType, grainId, eTag, blobName, containerName);
            grainState.ETag = eTag;
            return loadedState;
        }

        private static async Task<TResult> DoOptimisticUpdate<TState, TResult>(Func<TState, Task<TResult>> updateOperation, TState state, BlobClient blob, string? currentETag)
        {
            try
            {
                return await updateOperation(state).ConfigureAwait(false);
            }
            catch (RequestFailedException ex) when (ex.IsPreconditionFailed() || ex.IsConflict() || ex.IsNotFound() && !ex.IsContainerNotFound())
            {
                throw new InconsistentStateException($"Blob storage condition not Satisfied. BlobName: {blob.Name}, Container: {blob.BlobContainerName}, CurrentETag: {currentETag}", "Unknown", currentETag, ex);
            }
        }

        public void Participate(ISiloLifecycle lifecycle)
        {
            lifecycle.Subscribe(OptionFormattingUtilities.Name<AzureBlobGrainStorage>(this.name), this.options.InitStage, Init);
        }

        /// <summary> Initialization function for this storage provider. </summary>
        private async Task Init(CancellationToken ct)
        {
            var stopWatch = Stopwatch.StartNew();

            try
            {
                LogDebugInitializing(this.name, this.options.ContainerName);
                if (options.CreateClient is not { } createClient)
                {

View on GitHub (pinned to fca799fa70)

Solutions

Let the Orleans runtime retry (InconsistentStateException is typically retried by the storage layer/caller); ensure ReadStateAsync is re-invoked before re-writing so the ETag refreshes.
Reduce concurrent writers to the same grain (single-writer activation, grain locking) to avoid contention.
Verify no external writer is mutating the grain's blob; align ETag handling if you interoperate.
Increase configured retry counts/backoff if transient under load.

Example fix

// before
var state = await grain.Read();
state.Counter++;
await grain.Write(state); // stale ETag -> InconsistentStateException
// after
for (int attempt = 0; ; attempt++)
{
    var state = await grain.Read(); // fresh ETag each attempt
    state.Counter++;
    try { await grain.Write(state); break; }
    catch (InconsistentStateException) when (attempt < 5) { /* refresh and retry */ }
}

Defensive patterns

Strategy: retry

Validate before calling

// Before writing, re-read fresh state to obtain the current ETag:
var fresh = await grain.ReadStateAsync();
fresh.State = newState;
await grain.WriteStateAsync();

Type guard

static bool IsInconsistentState(Exception ex) => ex is InconsistentStateException;

Try / catch

for (int attempt = 0; attempt < maxRetries; attempt++)
{
    try
    {
        await storage.WriteStateAsync(grainType, grainReference, grainState);
        return;
    }
    catch (InconsistentStateException) when (attempt < maxRetries - 1)
    {
        await Task.Delay(backoff(attempt));
        await storage.ReadStateAsync(grainType, grainReference, grainState); // refresh ETag
    }
}

Prevention

Refresh grain state (and its ETag) immediately before writing on retry.
Single-writer per grain activation to reduce contention.
Ensure no external process mutates the grain's blob.
Tune storage retry/backoff under high concurrency.

When it happens

Trigger: Two silos/activations or two grain calls read the same grain state, then both attempt to write; the second write's If-Match ETag no longer matches the blob's current ETag, so Azure returns PreconditionFailed/412 (or Conflict/409, or NotFound/404 if the blob was deleted), which DoOptimisticUpdate wraps into InconsistentStateException.

Common situations: Concurrent writes to the same grain from multiple silos; a race between read and write within the same grain; an external process modified the blob outside Orleans; the blob was deleted between read and write; retry storms under load.

Related errors

AI-assisted analysis of dotnet/orleans@fca799fa70 (2026-08-13). Data as JSON: /api/errors/c0dca838eaf5bdd4. Report an issue: GitHub.