dotnet/orleans · error · InconsistentStateException

Blob storage condition not Satisfied. BlobName

Error message

Blob storage condition not Satisfied. BlobName: {blob.Name}, Container: {blob.BlobContainerName}, CurrentETag: {currentETag}

What it means

Thrown as InconsistentStateException by AzureBlobStorage.DoOptimisticUpdate when a blob write fails with a precondition-failed, conflict, or (non-container) not-found RequestFailedException. Orleans grain storage uses ETag-based optimistic concurrency; a mismatch means another caller mutated the blob between the read and the write, so the operation is aborted for the runtime to retry.

Solutions

Let the Orleans runtime retry (InconsistentStateException is typically retried by the storage layer/caller); ensure ReadStateAsync is re-invoked before re-writing so the ETag refreshes.
Reduce concurrent writers to the same grain (single-writer activation, grain locking) to avoid contention.
Verify no external writer is mutating the grain's blob; align ETag handling if you interoperate.
Increase configured retry counts/backoff if transient under load.

Example fix

// before
var state = await grain.Read();
state.Counter++;
await grain.Write(state); // stale ETag -> InconsistentStateException
// after
for (int attempt = 0; ; attempt++)
{
    var state = await grain.Read(); // fresh ETag each attempt
    state.Counter++;
    try { await grain.Write(state); break; }
    catch (InconsistentStateException) when (attempt < 5) { /* refresh and retry */ }
}

Defensive patterns

Strategy: retry

Validate before calling

// Before writing, re-read fresh state to obtain the current ETag:
var fresh = await grain.ReadStateAsync();
fresh.State = newState;
await grain.WriteStateAsync();

Type guard

static bool IsInconsistentState(Exception ex) => ex is InconsistentStateException;

Try / catch

for (int attempt = 0; attempt < maxRetries; attempt++)
{
    try
    {
        await storage.WriteStateAsync(grainType, grainReference, grainState);
        return;
    }
    catch (InconsistentStateException) when (attempt < maxRetries - 1)
    {
        await Task.Delay(backoff(attempt));
        await storage.ReadStateAsync(grainType, grainReference, grainState); // refresh ETag
    }
}

Prevention

Refresh grain state (and its ETag) immediately before writing on retry.
Single-writer per grain activation to reduce contention.
Ensure no external process mutates the grain's blob.
Tune storage retry/backoff under high concurrency.

When it happens

Trigger: Two silos/activations or two grain calls read the same grain state, then both attempt to write; the second write's If-Match ETag no longer matches the blob's current ETag, so Azure returns PreconditionFailed/412 (or Conflict/409, or NotFound/404 if the blob was deleted), which DoOptimisticUpdate wraps into InconsistentStateException.

Common situations: Concurrent writes to the same grain from multiple silos; a race between read and write within the same grain; an external process modified the blob outside Orleans; the blob was deleted between read and write; retry storms under load.

Related errors

AI-assisted analysis of dotnet/orleans@fca799fa70 (2026-08-13). Data as JSON: /api/errors/c0dca838eaf5bdd4. Report an issue: GitHub.

Appendix: source

Thrown at src/Azure/Orleans.Persistence.AzureStorage/Providers/Storage/AzureBlobStorage.cs:330

            {
                loadedState = this.ConvertFromStorageFormat<T>(new BinaryData(content));
                LogWarningLargePayloadFallback(contentLength, grainType, grainId, eTag, blobName, containerName);
            }

            LogTraceDataRead(grainType, grainId, eTag, blobName, containerName);
            grainState.ETag = eTag;
            return loadedState;
        }

        private static async Task<TResult> DoOptimisticUpdate<TState, TResult>(Func<TState, Task<TResult>> updateOperation, TState state, BlobClient blob, string? currentETag)
        {
            try
            {
                return await updateOperation(state).ConfigureAwait(false);
            }
            catch (RequestFailedException ex) when (ex.IsPreconditionFailed() || ex.IsConflict() || ex.IsNotFound() && !ex.IsContainerNotFound())
            {
                throw new InconsistentStateException($"Blob storage condition not Satisfied. BlobName: {blob.Name}, Container: {blob.BlobContainerName}, CurrentETag: {currentETag}", "Unknown", currentETag, ex);
            }
        }

        public void Participate(ISiloLifecycle lifecycle)
        {
            lifecycle.Subscribe(OptionFormattingUtilities.Name<AzureBlobGrainStorage>(this.name), this.options.InitStage, Init);
        }

        /// <summary> Initialization function for this storage provider. </summary>
        private async Task Init(CancellationToken ct)
        {
            var stopWatch = Stopwatch.StartNew();

            try
            {
                LogDebugInitializing(this.name, this.options.ContainerName);
                if (options.CreateClient is not { } createClient)
                {

View on GitHub (pinned to fca799fa70)