{"record":{"id":"f53386cf90cd47ff","repo":"microsoft/FASTER","slug":"recovery-of-blobs-is-single-threaded-and-should-not-yield","errorCode":null,"errorMessage":"Recovery of blobs is single-threaded and should not yield any failure due to concurrency","messagePattern":"Recovery of blobs is single-threaded and should not yield any failure due to concurrency","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"cs/src/devices/AzureStorageDevice/AzureStorageDevice.cs","lineNumber":204,"sourceCode":"                                .AsPages(continuationToken, 100)\n                                .FirstAsync();\n\n                            pageResults = page.Values;\n                            continuationToken = page.ContinuationToken;\n                            return page.Values.Count; // not accurate, in terms of bytes, but still useful for tracing purposes\n                        });\n\n                    foreach (var item in pageResults)\n                    {\n                        if (Int32.TryParse(item.Name.Replace(prefix, \"\"), out int segmentId))\n                        {\n                            this.BlobManager?.StorageTracer?.FasterStorageProgress($\"AzureStorageDevice.StartAsync found segment={item.Name}\");\n\n                            bool ret = this.blobs.TryAdd(segmentId, new BlobEntry(BlobUtilsV12.GetPageBlobClients(this.pageBlobDirectory.Client, item.Name), item.Properties.ETag.Value, this));\n\n                            if (!ret)\n                            {\n                                throw new InvalidOperationException(\"Recovery of blobs is single-threaded and should not yield any failure due to concurrency\");\n                            }\n                        }\n                    }\n                }\n                while (!string.IsNullOrEmpty(continuationToken));\n\n                // make sure we did not lose the lease while iterating to find the blobs\n                await this.BlobManager.ConfirmLeaseIsGoodForAWhileAsync();\n                this.StorageErrorHandler.Token.ThrowIfCancellationRequested();\n\n\n                // find longest contiguous sequence at end\n                var keys = this.blobs.Keys.ToList();\n                if (keys.Count == 0)\n                {\n                    // nothing has been written to this device so far.\n                    this.startSegment = 0;\n                    this.endSegment = -1;","sourceCodeStart":186,"sourceCodeEnd":222,"githubUrl":"https://github.com/microsoft/FASTER/blob/321d872eabda6a0345c8bd76419f89723ed864ae/cs/src/devices/AzureStorageDevice/AzureStorageDevice.cs#L186-L222","documentation":"InvalidOperationException thrown during AzureStorageDevice.StartAsync recovery when TryAdd of a discovered page-blob into the segment map returns false, meaning the same segmentId was already registered. Recovery enumerates blobs in the container single-threadedly, so a duplicate indicates two blobs mapping to the same segment id — an unexpected storage-state corruption.","triggerScenarios":"Running StartAsync (constructor path of AzureStorageDevice) against a container containing blobs whose names map to the same parsed segmentId, or leftover duplicated/conflicting blob names from a prior misconfigured or crashed session.","commonSituations":"Reusing an Azure container across multiple FASTER devices/instances; manually copied or restored blobs with name collisions; leftover blobs after deleting and recreating a device on the same container; concurrent instance misuse assumptions.","solutions":["Inspect the storage container for duplicate/conflicting segment blob names and remove or rename the stale ones","Use a dedicated container per FASTER device instance; do not share containers across devices or concurrent runs","Delete the container (or blobs) and reinitialize the device if leftover state is disposable","Check whether a previous run left orphaned blobs from a crashed checkpoint and clean up before recovery"],"exampleFix":"// before\nvar device = new AzureStorageDevice(\"accountKey\", \"account\", \"shared-container\", \"dir\", ...); // container shared with old runs\n// after\nvar device = new AzureStorageDevice(\"accountKey\", \"account\", \"device-42-dedicated-container\", \"dir\", ...); // unique container per device","handlingStrategy":"try-catch","validationCode":"// before constructing the device, list blobs and detect duplicate segment ids\nvar names = await container.GetBlobsAsync();\nvar dupes = names.GroupBy(b => ParseSegmentId(b.Name)).Where(g => g.Count() > 1).ToList();\nif (dupes.Any()) throw new InvalidOperationException(\"Duplicate segment blobs: \" + dupes.First().Key);","typeGuard":null,"tryCatchPattern":"try { await device.StartAsync(...); }\ncatch (InvalidOperationException ex) when (ex.Message.Contains(\"Recovery of blobs is single-threaded\"))\n{ logger.Error(\"Duplicate segment blobs in container; clean up and retry\", ex); }","preventionTips":["Use one dedicated Azure container per FASTER device instance","Clean orphaned/duplicated blobs before initializing a device on existing storage","Avoid manual blob copies/restores into an active device container"],"tags":["azure","storage","recovery","duplicate-key"],"backgroundTag":"file-already-exists","analyzedSha":"321d872eabda6a0345c8bd76419f89723ed864ae","analyzedAt":"2026-09-15T22:18:00.693Z","contentChangedAt":"2026-09-15T22:18:00.693Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}