{"record":{"id":"06c06f7670002220","repo":"microsoft/FASTER","slug":"size-of-key-value-exceeds-max-of-2gb","errorCode":null,"errorMessage":"Size of key-value exceeds max of 2GB: ","messagePattern":"Size of key-value exceeds max of 2GB: ","errorType":"exception","errorClass":"FasterException","httpStatus":null,"severity":"error","filePath":"cs/src/core/Allocator/GenericAllocator.cs","lineNumber":1015,"sourceCode":"                long endAddress = -1;\n                if (KeyHasObjects())\n                {\n                    var x = GetKeyAddressInfo((long)record);\n                    startAddress = x->Address;\n                    endAddress = x->Address + x->Size;\n                }\n\n                if (ValueHasObjects() && !GetInfoFromBytePointer(record).Tombstone)\n                {\n                    var x = GetValueAddressInfo((long)record);\n                    if (startAddress == -1)\n                        startAddress = x->Address;\n                    endAddress = x->Address + x->Size;\n                }\n\n                // We are limited to a 2GB size per key-value\n                if (endAddress-startAddress > int.MaxValue)\n                    throw new FasterException(\"Size of key-value exceeds max of 2GB: \" + (endAddress - startAddress));\n\n                if (startAddress < 0)\n                    startAddress = 0;\n\n                AsyncGetFromDisk(startAddress, (int)(endAddress - startAddress), ctx, ctx.record);\n                return false;\n            }\n\n            // Parse the key and value objects\n            MemoryStream ms = new MemoryStream(ctx.objBuffer.buffer);\n            ms.Seek(ctx.objBuffer.offset + ctx.objBuffer.valid_offset, SeekOrigin.Begin);\n\n            if (KeyHasObjects())\n            {\n                var keySerializer = SerializerSettings.keySerializer();\n                keySerializer.BeginDeserialize(ms);\n                keySerializer.Deserialize(out ctx.key);\n                keySerializer.EndDeserialize();","sourceCodeStart":997,"sourceCodeEnd":1033,"githubUrl":"https://github.com/microsoft/FASTER/blob/321d872eabda6a0345c8bd76419f89723ed864ae/cs/src/core/Allocator/GenericAllocator.cs#L997-L1033","documentation":"FASTER's GenericAllocator reads a logical key-value's record(s) from disk as one contiguous read into a single buffer, and that read length is stored in an int. When the total byte span of a record (endAddress - startAddress) exceeds int.MaxValue (2GB), it cannot be read/allocated in one piece, so GenericAllocator throws FasterException during an async read (AsyncGetAtAddress path).","triggerScenarios":"Calling Read/ReadAsync (or an operation that pulls a record from disk, e.g. a cache miss during an upsert/read-copy path) when a single serialized key+value spans more than 2GB of contiguous log address space; also hit when scanning adjacent records coalesce into a >2GB span (startAddress of the first record to endAddress of the last).","commonSituations":"Storing multi-GB blobs/objects as single FASTER values with generic allocator backing devices (local storage device); users migrating from a heap-based store to disk where record size limits were previously implicit; accidentally writing huge arrays as one value instead of chunking.","solutions":["Reduce the size of individual key-values below 2GB by splitting large values into chunks stored under multiple keys (e.g. key_0..key_n)","Use a smaller serializer output: compress the payload before storing, or switch to a blittable/ref struct type with a smaller fixed footprint","If large objects are inherent, store the blob in external storage (file/object store) and keep only a reference/handle in FASTER","Re-check record overhead: the 2GB limit includes the record header and multiple contiguous records spanned during a read; shrink both value size and record packing"],"exampleFix":"// before\nfaster.Upsert(key, hugeByteArray); // > 2GB value\n// after\nconst int ChunkSize = 512 * 1024 * 1024;\nfor (int i = 0; i * ChunkSize < hugeByteArray.Length; i++)\n{\n    var chunk = new byte[Math.Min(ChunkSize, hugeByteArray.Length - i * ChunkSize)];\n    Array.Copy(hugeByteArray, i * ChunkSize, chunk, 0, chunk.Length);\n    faster.Upsert(new Key(key, i), chunk);\n}","handlingStrategy":"validation","validationCode":"// Estimate serialized record size before upserting\nlong approxSize = keySize + valueSize + RecordHeaderSize; // e.g. 4-8 bytes header\nif (approxSize > int.MaxValue)\n    throw new InvalidOperationException($\"Value of {valueSize} bytes exceeds FASTER's 2GB per-record limit; chunk the value.\");","typeGuard":null,"tryCatchPattern":"try\n{\n    faster.Upsert(key, largeValue);\n}\ncatch (FasterException ex) when (ex.Message.StartsWith(\"Size of key-value exceeds max of 2GB\"))\n{\n    // fall back to chunked storage or external blob store\n}","preventionTips":["Chunk any value larger than ~1GB before storing in FASTER","Store very large blobs in external storage and keep handles in FASTER","Account for record headers when computing value size limits","Use 64-bit length checks (long) when measuring serialized sizes, not int"],"tags":["faster","allocator","storage","limit-exceeded"],"backgroundTag":"payload-too-large","analyzedSha":"321d872eabda6a0345c8bd76419f89723ed864ae","analyzedAt":"2026-09-15T22:18:00.693Z","contentChangedAt":"2026-09-15T22:18:00.693Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}