{"record":{"id":"9146e7949f564194","repo":"dotnet/orleans","slug":"azure-blob-journal-wal-ended-before-the-checkpoint","errorCode":null,"errorMessage":"Azure Blob journal WAL ended before the checkpoint offset was reached.","messagePattern":"Azure Blob journal WAL ended before the checkpoint offset was reached\\.","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"critical","filePath":"src/Azure/Orleans.Journaling.AzureStorage/AzureBlobJournalStorageStreamHelpers.cs","lineNumber":45,"sourceCode":"        var buffer = ArrayPool<byte>.Shared.Rent(maxChunkSize);\n\n        try\n        {\n            while (length > 0)\n            {\n                // ArrayPool can return a larger array, so slice it to keep each skip read capped.\n                var toRead = (int)Math.Min(maxChunkSize, length);\n\n                // ReadExactlyAsync guarantees the buffer slice is completely filled before returning,\n                // or it throws an EndOfStreamException if the stream ends early.\n                await input.ReadExactlyAsync(buffer.AsMemory(0, toRead), cancellationToken).ConfigureAwait(false);\n\n                length -= toRead;\n            }\n        }\n        catch (EndOfStreamException ex)\n        {\n            throw new InvalidOperationException(\"Azure Blob journal WAL ended before the checkpoint offset was reached.\", ex);\n        }\n        finally\n        {\n            ArrayPool<byte>.Shared.Return(buffer);\n        }\n    }\n}\n","sourceCodeStart":27,"sourceCodeEnd":53,"githubUrl":"https://github.com/dotnet/orleans/blob/fca799fa70ecb6ad975224271703ca43221f58de/src/Azure/Orleans.Journaling.AzureStorage/AzureBlobJournalStorageStreamHelpers.cs#L27-L53","documentation":"Thrown by AzureBlobJournalStorageStreamHelpers.SkipStreamAsync when the WAL blob stream ends before the requested number of checkpoint-covered bytes were drained. During recovery the helper skips bytes already covered by a checkpoint; an early EOF means the blob is shorter than the checkpoint offset claims, indicating corruption or truncation. The inner EndOfStreamException is preserved.","triggerScenarios":"Recovery reads the WAL and tries to skip 'length' bytes (the checkpoint offset) but the stream returns fewer bytes than requested via ReadExactlyAsync, which throws EndOfStreamException, caught and rewrapped here.","commonSituations":"Blob truncation or corruption (manual edit, partial upload, storage-side issue); a checkpoint offset that points beyond the blob's actual content; concurrent replace that shrank the WAL during recovery.","solutions":["Inspect the WAL blob length in Azure and compare it to the recorded checkpoint offset; if truncated, restore from a backup or force a fresh checkpoint.","Trigger a full recovery/compaction to rebuild the WAL and checkpoint consistently.","If this recurs, verify no external process is modifying the WAL blob concurrently."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try\n{\n    await storage.ReadAsync(consumer, ct);\n}\ncatch (InvalidOperationException ex) when (ex.Message.Contains(\"WAL ended before the checkpoint offset\"))\n{\n    logger.LogCritical(ex, \"WAL truncated; forcing full recovery from last checkpoint.\");\n    await storage.ReplaceAsync(freshSnapshot, metadata, ct);\n}","preventionTips":["Never modify WAL blobs externally; let only the journaling layer write them.","Monitor blob size vs recorded checkpoint offset for drift.","Keep frequent checkpoints so recovery after truncation is bounded."],"tags":["azure-blob-storage","recovery","wal","corruption","invalid-operation"],"backgroundTag":null,"analyzedSha":"fca799fa70ecb6ad975224271703ca43221f58de","analyzedAt":"2026-08-13T19:55:57.938Z","schemaVersion":2},"datasetVersion":"2026-08-14T00:17:13.853Z"}