microsoft/FASTER · error · InvalidOperationException

Recovery of blobs is single-threaded and should not yield…

Error message

Recovery of blobs is single-threaded and should not yield any failure due to concurrency

What it means

InvalidOperationException thrown during AzureStorageDevice.StartAsync recovery when TryAdd of a discovered page-blob into the segment map returns false, meaning the same segmentId was already registered. Recovery enumerates blobs in the container single-threadedly, so a duplicate indicates two blobs mapping to the same segment id — an unexpected storage-state corruption.

Solutions

  1. Inspect the storage container for duplicate/conflicting segment blob names and remove or rename the stale ones
  2. Use a dedicated container per FASTER device instance; do not share containers across devices or concurrent runs
  3. Delete the container (or blobs) and reinitialize the device if leftover state is disposable
  4. Check whether a previous run left orphaned blobs from a crashed checkpoint and clean up before recovery

Example fix

// before
var device = new AzureStorageDevice("accountKey", "account", "shared-container", "dir", ...); // container shared with old runs
// after
var device = new AzureStorageDevice("accountKey", "account", "device-42-dedicated-container", "dir", ...); // unique container per device
Defensive patterns

Strategy: try-catch

Validate before calling

// before constructing the device, list blobs and detect duplicate segment ids
var names = await container.GetBlobsAsync();
var dupes = names.GroupBy(b => ParseSegmentId(b.Name)).Where(g => g.Count() > 1).ToList();
if (dupes.Any()) throw new InvalidOperationException("Duplicate segment blobs: " + dupes.First().Key);

Try / catch

try { await device.StartAsync(...); }
catch (InvalidOperationException ex) when (ex.Message.Contains("Recovery of blobs is single-threaded"))
{ logger.Error("Duplicate segment blobs in container; clean up and retry", ex); }

Prevention

When it happens

Trigger: Running StartAsync (constructor path of AzureStorageDevice) against a container containing blobs whose names map to the same parsed segmentId, or leftover duplicated/conflicting blob names from a prior misconfigured or crashed session.

Common situations: Reusing an Azure container across multiple FASTER devices/instances; manually copied or restored blobs with name collisions; leftover blobs after deleting and recreating a device on the same container; concurrent instance misuse assumptions.

Understand the failure class

Background: "already exists" / EEXIST / FileAlreadyExistsException: what the 'file already exists' error means and how to fix it — this error's family across 37 libraries.

Related errors

AI-assisted analysis of microsoft/FASTER@321d872eab (2026-09-15). Data as JSON: /api/errors/f53386cf90cd47ff. Report an issue: GitHub.

Appendix: source

Thrown at cs/src/devices/AzureStorageDevice/AzureStorageDevice.cs:204

                                .AsPages(continuationToken, 100)
                                .FirstAsync();

                            pageResults = page.Values;
                            continuationToken = page.ContinuationToken;
                            return page.Values.Count; // not accurate, in terms of bytes, but still useful for tracing purposes
                        });

                    foreach (var item in pageResults)
                    {
                        if (Int32.TryParse(item.Name.Replace(prefix, ""), out int segmentId))
                        {
                            this.BlobManager?.StorageTracer?.FasterStorageProgress($"AzureStorageDevice.StartAsync found segment={item.Name}");

                            bool ret = this.blobs.TryAdd(segmentId, new BlobEntry(BlobUtilsV12.GetPageBlobClients(this.pageBlobDirectory.Client, item.Name), item.Properties.ETag.Value, this));

                            if (!ret)
                            {
                                throw new InvalidOperationException("Recovery of blobs is single-threaded and should not yield any failure due to concurrency");
                            }
                        }
                    }
                }
                while (!string.IsNullOrEmpty(continuationToken));

                // make sure we did not lose the lease while iterating to find the blobs
                await this.BlobManager.ConfirmLeaseIsGoodForAWhileAsync();
                this.StorageErrorHandler.Token.ThrowIfCancellationRequested();


                // find longest contiguous sequence at end
                var keys = this.blobs.Keys.ToList();
                if (keys.Count == 0)
                {
                    // nothing has been written to this device so far.
                    this.startSegment = 0;
                    this.endSegment = -1;

View on GitHub (pinned to 321d872eab)