dotnet/orleans · error · InvalidOperationException

Azure Table journal batch of {length:N0} bytes exceeds the p

Error message

Azure Table journal batch of {length:N0} bytes exceeds the per-transaction limit of {MaxAppendBytes:N0} bytes (2 MiB). Reduce the operation size or compact more aggressively.

What it means

Thrown by AzureTableJournalStorage.ThrowIfBatchTooLarge when an Append or Replace operation would produce an entity group transaction exceeding 2 MiB (MaxAppendBytes). Azure Table entity group transactions are capped at 4 MiB; the journal reserves headroom for Base64 encoding and overhead, so it fails locally at 2 MiB with actionable guidance rather than letting Azure reject it opaquely.

Source

Thrown at src/Azure/Orleans.Journaling.AzureStorage/AzureTableJournalStorage.cs:570

        finally
        {
            _shared.Instruments.OnOperationCompleted(
                AzureTableJournalStorageInstruments.OperationReplace,
                Stopwatch.GetElapsedTime(startTimestamp),
                value.Length,
                succeeded);
        }
    }

    private static void ThrowIfBatchTooLarge(long length)
    {
        // Azure rejects oversize entity group transactions, so fail locally with the journal-specific guidance.
        if (length <= MaxAppendBytes)
        {
            return;
        }

        throw new InvalidOperationException(
            $"Azure Table journal batch of {length:N0} bytes exceeds the per-transaction limit of {MaxAppendBytes:N0} bytes (2 MiB). " +
            "Reduce the operation size or compact more aggressively.");
    }

    private async ValueTask EnsureHeaderAsync(CancellationToken cancellationToken)
    {
        // Either create the initial header or load the header created by a racing instance, then loop until state is cached.
        while (!HeaderExists)
        {
            try
            {
                var created = await CreateHeaderAsync(callerMetadata: null, cancellationToken).ConfigureAwait(false);
                if (created.ETag != default)
                {
                    SetHeader(created.ETag, created.ProviderState);
                    return;
                }

View on GitHub (pinned to fca799fa70)

Solutions

1. Split the append into smaller batches each under ~1.5 MiB to stay clear of the 2 MiB cap.
2. Trigger compaction (ReplaceAsync with a checkpoint) more frequently so generations stay small.
3. If using ReplaceAsync for a large snapshot, ensure the snapshot itself is under 2 MiB or implement a custom chunking strategy.

Example fix

// before
await storage.AppendAsync(largeBuffer, metadata, ct); // largeBuffer > 2 MiB

// after
foreach (var chunk in SplitIntoChunks(largeBuffer, chunkSize: 1_500_000))
{
    await storage.AppendAsync(chunk, metadata, ct);
}

const long SafeAppendCeiling = 1_500_000; // 1.5 MiB headroom under 2 MiB cap
foreach (var chunk in SplitIntoChunks(value, SafeAppendCeiling))
{
    await storage.AppendAsync(chunk, metadata, ct);
}

static IEnumerable<ReadOnlyMemory<byte>> SplitIntoChunks(ReadOnlyMemory<byte> data, long max)
{
    for (var i = 0; i < data.Length; i += (int)max)
        yield return data.Slice(i, (int)Math.Min(max, data.Length - i));
}

try { await storage.AppendAsync(value, metadata, ct); }
catch (InvalidOperationException ex) when (ex.Message.Contains("exceeds the per-transaction limit"))
{
    logger.LogWarning("Append too large; splitting into smaller batches.");
    foreach (var chunk in SplitIntoChunks(value, 1_500_000))
        await storage.AppendAsync(chunk, metadata, ct);
}

When it happens

Trigger: Calling AppendAsync with a payload that, after chunking into 64 KiB table properties and adding the header fence entity, exceeds MaxAppendBytes (2 MiB). Or calling ReplaceAsync with a snapshot large enough to exceed the limit in a single transaction.

Common situations: Appending a large batch in one call without pre-chunking; insufficient compaction so accumulated rows push a replace over the limit; misjudging payload size after encoding.

Related errors

• Azure Table journal header state does not include a generati
• The journal storage consumer did not read all supplied journ
• Azure Table partition keys must not exceed 1,024 characters.
• Data too large to write to DynamoDB table. Size={dataSize} M
• AzureBlobJournalStorageProvider has not been initialized. En

AI-assisted analysis of dotnet/orleans@fca799fa70 (2026-08-13). Data as JSON: /api/errors/ccb2bfc1d564cb91. Report an issue: GitHub.