dotnet/orleans · critical · InvalidOperationException

Azure Blob journal WAL ended before the checkpoint offset wa

Error message

Azure Blob journal WAL ended before the checkpoint offset was reached.

What it means

Thrown by AzureBlobJournalStorageStreamHelpers.SkipStreamAsync when the WAL blob stream ends before the requested number of checkpoint-covered bytes were drained. During recovery the helper skips bytes already covered by a checkpoint; an early EOF means the blob is shorter than the checkpoint offset claims, indicating corruption or truncation. The inner EndOfStreamException is preserved.

Source

Thrown at src/Azure/Orleans.Journaling.AzureStorage/AzureBlobJournalStorageStreamHelpers.cs:45

        var buffer = ArrayPool<byte>.Shared.Rent(maxChunkSize);

        try
        {
            while (length > 0)
            {
                // ArrayPool can return a larger array, so slice it to keep each skip read capped.
                var toRead = (int)Math.Min(maxChunkSize, length);

                // ReadExactlyAsync guarantees the buffer slice is completely filled before returning,
                // or it throws an EndOfStreamException if the stream ends early.
                await input.ReadExactlyAsync(buffer.AsMemory(0, toRead), cancellationToken).ConfigureAwait(false);

                length -= toRead;
            }
        }
        catch (EndOfStreamException ex)
        {
            throw new InvalidOperationException("Azure Blob journal WAL ended before the checkpoint offset was reached.", ex);
        }
        finally
        {
            ArrayPool<byte>.Shared.Return(buffer);
        }
    }
}

View on GitHub (pinned to fca799fa70)

Solutions

Inspect the WAL blob length in Azure and compare it to the recorded checkpoint offset; if truncated, restore from a backup or force a fresh checkpoint.
Trigger a full recovery/compaction to rebuild the WAL and checkpoint consistently.
If this recurs, verify no external process is modifying the WAL blob concurrently.

Defensive patterns

Strategy: try-catch

Try / catch

try
{
    await storage.ReadAsync(consumer, ct);
}
catch (InvalidOperationException ex) when (ex.Message.Contains("WAL ended before the checkpoint offset"))
{
    logger.LogCritical(ex, "WAL truncated; forcing full recovery from last checkpoint.");
    await storage.ReplaceAsync(freshSnapshot, metadata, ct);
}

Prevention

Never modify WAL blobs externally; let only the journaling layer write them.
Monitor blob size vs recorded checkpoint offset for drift.
Keep frequent checkpoints so recovery after truncation is bounded.

When it happens

Trigger: Recovery reads the WAL and tries to skip 'length' bytes (the checkpoint offset) but the stream returns fewer bytes than requested via ReadExactlyAsync, which throws EndOfStreamException, caught and rewrapped here.

Common situations: Blob truncation or corruption (manual edit, partial upload, storage-side issue); a checkpoint offset that points beyond the blob's actual content; concurrent replace that shrank the WAL during recovery.

Related errors

AI-assisted analysis of dotnet/orleans@fca799fa70 (2026-08-13). Data as JSON: /api/errors/9146e7949f564194. Report an issue: GitHub.