microsoft/FASTER · error · FasterException

Size of key-value exceeds max of 2GB:

Error message

Size of key-value exceeds max of 2GB:

What it means

FASTER's GenericAllocator reads a logical key-value's record(s) from disk as one contiguous read into a single buffer, and that read length is stored in an int. When the total byte span of a record (endAddress - startAddress) exceeds int.MaxValue (2GB), it cannot be read/allocated in one piece, so GenericAllocator throws FasterException during an async read (AsyncGetAtAddress path).

Solutions

  1. Reduce the size of individual key-values below 2GB by splitting large values into chunks stored under multiple keys (e.g. key_0..key_n)
  2. Use a smaller serializer output: compress the payload before storing, or switch to a blittable/ref struct type with a smaller fixed footprint
  3. If large objects are inherent, store the blob in external storage (file/object store) and keep only a reference/handle in FASTER
  4. Re-check record overhead: the 2GB limit includes the record header and multiple contiguous records spanned during a read; shrink both value size and record packing

Example fix

// before
faster.Upsert(key, hugeByteArray); // > 2GB value
// after
const int ChunkSize = 512 * 1024 * 1024;
for (int i = 0; i * ChunkSize < hugeByteArray.Length; i++)
{
    var chunk = new byte[Math.Min(ChunkSize, hugeByteArray.Length - i * ChunkSize)];
    Array.Copy(hugeByteArray, i * ChunkSize, chunk, 0, chunk.Length);
    faster.Upsert(new Key(key, i), chunk);
}
Defensive patterns

Strategy: validation

Validate before calling

// Estimate serialized record size before upserting
long approxSize = keySize + valueSize + RecordHeaderSize; // e.g. 4-8 bytes header
if (approxSize > int.MaxValue)
    throw new InvalidOperationException($"Value of {valueSize} bytes exceeds FASTER's 2GB per-record limit; chunk the value.");

Try / catch

try
{
    faster.Upsert(key, largeValue);
}
catch (FasterException ex) when (ex.Message.StartsWith("Size of key-value exceeds max of 2GB"))
{
    // fall back to chunked storage or external blob store
}

Prevention

When it happens

Trigger: Calling Read/ReadAsync (or an operation that pulls a record from disk, e.g. a cache miss during an upsert/read-copy path) when a single serialized key+value spans more than 2GB of contiguous log address space; also hit when scanning adjacent records coalesce into a >2GB span (startAddress of the first record to endAddress of the last).

Common situations: Storing multi-GB blobs/objects as single FASTER values with generic allocator backing devices (local storage device); users migrating from a heap-based store to disk where record size limits were previously implicit; accidentally writing huge arrays as one value instead of chunking.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

AI-assisted analysis of microsoft/FASTER@321d872eab (2026-09-15). Data as JSON: /api/errors/06c06f7670002220. Report an issue: GitHub.

Appendix: source

Thrown at cs/src/core/Allocator/GenericAllocator.cs:1015

                long endAddress = -1;
                if (KeyHasObjects())
                {
                    var x = GetKeyAddressInfo((long)record);
                    startAddress = x->Address;
                    endAddress = x->Address + x->Size;
                }

                if (ValueHasObjects() && !GetInfoFromBytePointer(record).Tombstone)
                {
                    var x = GetValueAddressInfo((long)record);
                    if (startAddress == -1)
                        startAddress = x->Address;
                    endAddress = x->Address + x->Size;
                }

                // We are limited to a 2GB size per key-value
                if (endAddress-startAddress > int.MaxValue)
                    throw new FasterException("Size of key-value exceeds max of 2GB: " + (endAddress - startAddress));

                if (startAddress < 0)
                    startAddress = 0;

                AsyncGetFromDisk(startAddress, (int)(endAddress - startAddress), ctx, ctx.record);
                return false;
            }

            // Parse the key and value objects
            MemoryStream ms = new MemoryStream(ctx.objBuffer.buffer);
            ms.Seek(ctx.objBuffer.offset + ctx.objBuffer.valid_offset, SeekOrigin.Begin);

            if (KeyHasObjects())
            {
                var keySerializer = SerializerSettings.keySerializer();
                keySerializer.BeginDeserialize(ms);
                keySerializer.Deserialize(out ctx.key);
                keySerializer.EndDeserialize();

View on GitHub (pinned to 321d872eab)