microsoft/FASTER · warning · OperationCanceledException

Lease acquisition was canceled

Error message

Lease acquisition was canceled

What it means

BlobManager.AcquireOwnership retries blob lease acquisition while the storage error handler reports the device/service is terminating or shutting down. Instead of continuing to retry, it wraps the triggering exception in an OperationCanceledException with the message 'Lease acquisition was canceled' to signal cooperative cancellation. This is a shutdown-path exception, not a lease contention error.

Solutions

1. Treat this OperationCanceledException as benign during shutdown; catch it and log instead of failing.
2. Avoid disposing/stopping the Faster instance while StartAsync is still in flight; await initialization before shutdown.
3. If it appears outside shutdown, check what set IsTerminated (earlier terminal storage error) and fix that underlying fault.

Example fix

// before
task = faster.StartAsync();
// after: tolerate cancellation during shutdown
try { await faster.StartAsync(); }
catch (OperationCanceledException) when (shuttingDown) { /* expected during termination */ }

static bool IsLeaseAcquisitionCanceled(OperationCanceledException ex) =>
    ex.Message == "Lease acquisition was canceled";

try
{
    await device.StartAsync();
}
catch (OperationCanceledException ex) when (ex.Message == "Lease acquisition was canceled")
{
    logger.LogInformation("Lease acquisition canceled due to shutdown; ignoring.");
}

When it happens

Trigger: Calling StartAsync (which acquires the blob lease) while Faster is being terminated: StorageErrorHandler.IsTerminated is true when a storage exception is caught, so acquisition aborts with OperationCanceledException instead of retrying.

Common situations: Graceful application shutdown or disposal racing with Faster initialization; host cancellation token fired while StartAsync was still leasing the container; a kubernetes/service shutdown killing the process mid-start.

Related errors

• storage operation ( ) was canceled
• log has already been closed
• wrong amount of data received from page blob, expected=
• delay

AI-assisted analysis of microsoft/FASTER@321d872eab (2026-09-15). Data as JSON: /api/errors/0aaf28f4b6a29be8. Report an issue: GitHub.

Appendix: source

Thrown at cs/src/devices/AzureStorageDevice/BlobManager.cs:231

                            {
                                // creation race, try from top
                                this.TraceHelper.LeaseProgress("Creation race observed, retrying");
                            }

                            return 1;
                        });

                    continue;
                }
                catch (OperationCanceledException) when (this.StorageErrorHandler.IsTerminated)
                {
                    throw; // o.k. during termination or shutdown
                }
                catch (Exception e) when (this.StorageErrorHandler.IsTerminated)
                {
                    string message = $"Lease acquisition was canceled";
                    this.TraceHelper.LeaseProgress(message);
                    throw new OperationCanceledException(message, e);
                }
                catch (Exception ex) when (numAttempts < BlobManager.MaxRetries
                    && !this.StorageErrorHandler.IsTerminated && BlobUtils.IsTransientStorageError(ex))
                {
                    if (BlobUtils.IsTimeout(ex))
                    {
                        this.TraceHelper.FasterPerfWarning($"Lease acquisition timed out, retrying now");
                    }
                    else
                    {
                        TimeSpan nextRetryIn = BlobManager.GetDelayBetweenRetries(numAttempts);
                        this.TraceHelper.FasterPerfWarning($"Lease acquisition failed transiently, retrying in {nextRetryIn}");
                        await Task.Delay(nextRetryIn);
                    }
                    continue;
                }
                catch (Exception e) when (!Utils.IsFatal(e))
                {

View on GitHub (pinned to 321d872eab)