microsoft/FASTER · warning · OperationCanceledException

storage operation ( ) was canceled

Error message

storage operation {name} ({intent}) was canceled

What it means

StorageOperations.PerformWithRetriesAsync wraps each Azure storage operation in retry logic. When a storage exception is caught and StorageErrorHandler.IsTerminated is true, the operation is not retried; instead the exception is wrapped in an OperationCanceledException with the message 'storage operation {name} ({intent}) was canceled' to indicate the device is shutting down. Like error 156, it is a cooperative-cancellation signal on the termination path.

Solutions

1. Catch OperationCanceledException during shutdown and treat it as expected.
2. Ensure orderly teardown: stop background operations and await in-flight tasks before disposing the device.
3. If seen in normal operation, investigate the original exception that set IsTerminated (auth failure, deleted container, etc.).

Example fix

// before: disposing while operations run
faster.Dispose();
// after: stop first, then dispose
cancellationTokenSource.Cancel();
await faster.StopAsync();
faster.Dispose();

if (storageErrorHandler.IsTerminated)
    return; // do not issue new storage operations after termination

static bool IsStorageOpCanceled(OperationCanceledException ex) =>
    ex.Message.StartsWith("storage operation ") && ex.Message.EndsWith("was canceled");

try
{
    await storageOp.PerformWithRetriesAsync(...);
}
catch (OperationCanceledException ex) when (ex.Message.Contains("was canceled"))
{
    logger.LogInformation("Storage operation canceled during shutdown: {Message}", ex.Message);
}

When it happens

Trigger: Any storage operation performed via PerformWithRetriesAsync (e.g. lease acquisition called by AcquireOwnership) that throws while Faster's storage error handler has been marked terminated, i.e. during StopAsync/disposal or after a terminal storage error.

Common situations: Disposing the Faster instance while background lease renewal or checkpoint storage calls are in flight; service shutdown triggered mid-operation; a previous unrecoverable storage error marked the handler terminated and subsequent calls all cancel.

Related errors

• Lease acquisition was canceled
• log has already been closed
• wrong amount of data received from page blob, expected=
• delay
• Hlog compaction was not successful :( -- retry!

AI-assisted analysis of microsoft/FASTER@321d872eab (2026-09-15). Data as JSON: /api/errors/862032ef69fe9e78. Report an issue: GitHub.

Appendix: source

Thrown at cs/src/devices/AzureStorageDevice/StorageOperations.cs:79

                        long size = await operationAsync(numAttempts).ConfigureAwait(false);

                        stopwatch.Stop();
                        this.StorageTracer?.FasterStorageProgress($"storage operation {name} ({intent}) succeeded on attempt {numAttempts}; target={target} latencyMs={stopwatch.Elapsed.TotalMilliseconds:F1} {data}");

                        if (stopwatch.ElapsedMilliseconds > expectedLatencyBound)
                        {
                            this.TraceHelper.FasterPerfWarning($"storage operation {name} ({intent}) took {stopwatch.Elapsed.TotalSeconds:F1}s on attempt {numAttempts}, which is excessive; {data}");
                        }

                        this.TraceHelper.FasterAzureStorageAccessCompleted(intent, size, name, target, stopwatch.Elapsed.TotalMilliseconds, numAttempts);

                        return;
                    }
                    catch (Exception e) when (this.StorageErrorHandler.IsTerminated)
                    {
                        string message = $"storage operation {name} ({intent}) was canceled";
                        this.StorageTracer?.FasterStorageProgress(message);
                        throw new OperationCanceledException(message, e);
                    }
                    catch (Exception e) when (BlobUtils.IsTransientStorageError(e) && numAttempts < BlobManager.MaxRetries)
                    {
                        stopwatch.Stop();

                        if (BlobUtils.IsTimeout(e))
                        {
                            this.TraceHelper.FasterPerfWarning($"storage operation {name} ({intent}) timed out on attempt {numAttempts} after {stopwatch.Elapsed.TotalSeconds:F1}s, retrying now; target={target} {data}");
                        }
                        else
                        {
                            TimeSpan nextRetryIn = BlobManager.GetDelayBetweenRetries(numAttempts);
                            this.HandleStorageError(name, $"storage operation {name} ({intent}) failed transiently on attempt {numAttempts}, retry in {nextRetryIn}s", target, e, false, true);
                            await Task.Delay(nextRetryIn);
                        }
                        continue;
                    }
                    catch (Azure.RequestFailedException ex) when (BlobUtilsV12.PreconditionFailed(ex) && readETagAsync != null)

View on GitHub (pinned to 321d872eab)