{"record":{"id":"862032ef69fe9e78","repo":"microsoft/FASTER","slug":"storage-operation-name-intent-was-canceled","errorCode":null,"errorMessage":"storage operation {name} ({intent}) was canceled","messagePattern":"storage operation (.+?) \\((.+?)\\) was canceled","errorType":"exception","errorClass":"OperationCanceledException","httpStatus":null,"severity":"warning","filePath":"cs/src/devices/AzureStorageDevice/StorageOperations.cs","lineNumber":79,"sourceCode":"                        long size = await operationAsync(numAttempts).ConfigureAwait(false);\n\n                        stopwatch.Stop();\n                        this.StorageTracer?.FasterStorageProgress($\"storage operation {name} ({intent}) succeeded on attempt {numAttempts}; target={target} latencyMs={stopwatch.Elapsed.TotalMilliseconds:F1} {data}\");\n\n                        if (stopwatch.ElapsedMilliseconds > expectedLatencyBound)\n                        {\n                            this.TraceHelper.FasterPerfWarning($\"storage operation {name} ({intent}) took {stopwatch.Elapsed.TotalSeconds:F1}s on attempt {numAttempts}, which is excessive; {data}\");\n                        }\n\n                        this.TraceHelper.FasterAzureStorageAccessCompleted(intent, size, name, target, stopwatch.Elapsed.TotalMilliseconds, numAttempts);\n\n                        return;\n                    }\n                    catch (Exception e) when (this.StorageErrorHandler.IsTerminated)\n                    {\n                        string message = $\"storage operation {name} ({intent}) was canceled\";\n                        this.StorageTracer?.FasterStorageProgress(message);\n                        throw new OperationCanceledException(message, e);\n                    }\n                    catch (Exception e) when (BlobUtils.IsTransientStorageError(e) && numAttempts < BlobManager.MaxRetries)\n                    {\n                        stopwatch.Stop();\n\n                        if (BlobUtils.IsTimeout(e))\n                        {\n                            this.TraceHelper.FasterPerfWarning($\"storage operation {name} ({intent}) timed out on attempt {numAttempts} after {stopwatch.Elapsed.TotalSeconds:F1}s, retrying now; target={target} {data}\");\n                        }\n                        else\n                        {\n                            TimeSpan nextRetryIn = BlobManager.GetDelayBetweenRetries(numAttempts);\n                            this.HandleStorageError(name, $\"storage operation {name} ({intent}) failed transiently on attempt {numAttempts}, retry in {nextRetryIn}s\", target, e, false, true);\n                            await Task.Delay(nextRetryIn);\n                        }\n                        continue;\n                    }\n                    catch (Azure.RequestFailedException ex) when (BlobUtilsV12.PreconditionFailed(ex) && readETagAsync != null)","sourceCodeStart":61,"sourceCodeEnd":97,"githubUrl":"https://github.com/microsoft/FASTER/blob/321d872eabda6a0345c8bd76419f89723ed864ae/cs/src/devices/AzureStorageDevice/StorageOperations.cs#L61-L97","documentation":"StorageOperations.PerformWithRetriesAsync wraps each Azure storage operation in retry logic. When a storage exception is caught and StorageErrorHandler.IsTerminated is true, the operation is not retried; instead the exception is wrapped in an OperationCanceledException with the message 'storage operation {name} ({intent}) was canceled' to indicate the device is shutting down. Like error 156, it is a cooperative-cancellation signal on the termination path.","triggerScenarios":"Any storage operation performed via PerformWithRetriesAsync (e.g. lease acquisition called by AcquireOwnership) that throws while Faster's storage error handler has been marked terminated, i.e. during StopAsync/disposal or after a terminal storage error.","commonSituations":"Disposing the Faster instance while background lease renewal or checkpoint storage calls are in flight; service shutdown triggered mid-operation; a previous unrecoverable storage error marked the handler terminated and subsequent calls all cancel.","solutions":["Catch OperationCanceledException during shutdown and treat it as expected.","Ensure orderly teardown: stop background operations and await in-flight tasks before disposing the device.","If seen in normal operation, investigate the original exception that set IsTerminated (auth failure, deleted container, etc.)."],"exampleFix":"// before: disposing while operations run\nfaster.Dispose();\n// after: stop first, then dispose\ncancellationTokenSource.Cancel();\nawait faster.StopAsync();\nfaster.Dispose();","handlingStrategy":"try-catch","validationCode":"if (storageErrorHandler.IsTerminated)\n    return; // do not issue new storage operations after termination","typeGuard":"static bool IsStorageOpCanceled(OperationCanceledException ex) =>\n    ex.Message.StartsWith(\"storage operation \") && ex.Message.EndsWith(\"was canceled\");","tryCatchPattern":"try\n{\n    await storageOp.PerformWithRetriesAsync(...);\n}\ncatch (OperationCanceledException ex) when (ex.Message.Contains(\"was canceled\"))\n{\n    logger.LogInformation(\"Storage operation canceled during shutdown: {Message}\", ex.Message);\n}","preventionTips":["Cancel and await background tasks before disposing the device.","Check StorageErrorHandler.IsTerminated before issuing new storage calls.","Treat this exception as expected in shutdown/teardown paths.","Diagnose the root terminal storage error if this appears during normal running."],"tags":["azure-storage","cancellation","shutdown","retry"],"backgroundTag":"operation-canceled","analyzedSha":"321d872eabda6a0345c8bd76419f89723ed864ae","analyzedAt":"2026-09-15T22:18:00.693Z","contentChangedAt":"2026-09-15T22:18:00.693Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}