{"record":{"id":"12c4e1e8c6d68402","repo":"microsoft/garnet","slug":"estimatedrecordsize-estimatedtotalsize-exceeds","errorCode":null,"errorMessage":"estimatedRecordSize ({estimatedTotalSize}) exceeds max allocated heap size (to use: {allocationSizeToUse}; max: {maxHeapAllocationSize})","messagePattern":"estimatedRecordSize \\((.+?)\\) exceeds max allocated heap size \\(to use: (.+?); max: (.+?)\\)","errorType":"exception","errorClass":"TsavoriteException","httpStatus":null,"severity":"error","filePath":"libs/storage/Tsavorite/cs/src/core/Allocator/DiskLogRecord.cs","lineNumber":411,"sourceCode":"        }\n\n        /// <summary>\n        /// Directly copies a record in inline format to the SpanByteAndMemory. Allocates <see cref=\"SpanByteAndMemory.Memory\"/> if needed.\n        /// </summary>\n        /// <remarks>If <paramref name=\"output\"/>.<see cref=\"SpanByteAndMemory.IsSpanByte\"/>, it points directly to the network buffer so we include the length prefix in the output.</remarks>\n        [MethodImpl(MethodImplOptions.AggressiveInlining)]\n        public static void DirectCopyInlinePortionOfRecord<TSourceLogRecord>(in TSourceLogRecord logRecord, int alignedInlineRecordSize, int estimatedTotalSize, int maxHeapAllocationSize,\n            MemoryPool<byte> memoryPool, ref SpanByteAndMemory output)\n            where TSourceLogRecord : ISourceLogRecord\n        {\n            // See if we have enough space in the SpanByte and, if not, if we would fit in maxHeapAllocationSize.\n            // For SpanByte the recordSize must include the length prefix, which is included in the output stream\n            // if we can write directly to the SpanByte, which is a span in the network buffer.\n            if (!output.IsSpanByte || estimatedTotalSize + sizeof(int) > output.SpanByte.Length || logRecord.DataHeader.ValueIsObject)\n            {\n                var allocationSizeToUse = logRecord.DataHeader.ValueIsObject ? maxHeapAllocationSize : estimatedTotalSize + sizeof(int);\n                if (estimatedTotalSize > allocationSizeToUse)\n                    throw new TsavoriteException($\"estimatedRecordSize ({estimatedTotalSize}) exceeds max allocated heap size (to use: {allocationSizeToUse}; max: {maxHeapAllocationSize})\");\n                output.EnsureHeapMemorySize(allocationSizeToUse, memoryPool);\n            }\n\n            // We must reset the LogRecord's filler size, because we truncated the record down to the (rounded-up) ActualSize if it had been shrunken.\n            var newFillerLength = alignedInlineRecordSize - logRecord.ActualSize;\n            if (output.IsSpanByte)\n            {\n                // TotalSize includes the length prefix. If there is a SpanByte it is a span in the network buffer, so we include the prefix length in the output stream.\n                var outPtr = output.SpanByte.ToPointer();\n                *(int*)outPtr = alignedInlineRecordSize;\n                outPtr += sizeof(int);\n                Buffer.MemoryCopy((byte*)logRecord.PhysicalAddress, outPtr, alignedInlineRecordSize, alignedInlineRecordSize);\n                new LogRecord((long)outPtr).SetFillerLength(newFillerLength);\n            }\n            else\n            {\n                // Do not include the length prefix in the output stream; this is done by the caller before writing the stream to the network buffer.\n                fixed (byte* outPtr = output.MemorySpan)","sourceCodeStart":393,"sourceCodeEnd":429,"githubUrl":"https://github.com/microsoft/garnet/blob/951b0fc6838721f89d102c2bbe1b914e8d39d700/libs/storage/Tsavorite/cs/src/core/Allocator/DiskLogRecord.cs#L393-L429","documentation":"DirectCopyInlinePortionOfRecord copies a log record's inline bytes directly into the output SpanByte (network buffer) when it fits; otherwise it allocates heap memory bounded by maxHeapAllocationSize. The throw fires when the record's estimated total size exceeds that budgeted heap allocation, i.e. the record is larger than the per-operation heap memory the caller authorized. For non-object records allocationSizeToUse is estimatedTotalSize+4 (so the check only trips for object-valued records whose stored size exceeds maxHeapAllocationSize).","triggerScenarios":"Reading/scanning/serializing a record whose inline value plus length prefix and metadata exceeds maxHeapAllocationSize, while the output is heap-backed (not a directly writable SpanByte) or the record is object-valued. Replication/checkpoint serialization of an oversized object record is the typical path.","commonSituations":"Large object values in an object store with maxHeapAllocationSize set too low for the workload; deserializing records written under a larger memory budget; checkpoint or replica copy where the per-record heap budget was not scaled to actual value sizes.","solutions":["Increase the maxHeapAllocationSize passed to the Serialize/DirectCopy call so it covers your largest record.","Reduce stored value size: chunk or offload large blobs instead of keeping them as single object records.","Use a directly-writable SpanByte output (network buffer) so the direct-copy path is taken and the heap limit is bypassed.","For object values, ensure the configured serializer's max object size matches the data actually being stored.","Log estimatedTotalSize vs maxHeapAllocationSize at write time so oversized records are rejected before they are stored."],"exampleFix":"// before\nDiskLogRecord.Serialize(rec, maxHeapAllocationSize: 1 << 20, serializer, pool, ref output);\n\n// after\nDiskLogRecord.Serialize(rec, maxHeapAllocationSize: 1 << 24, serializer, pool, ref output);","handlingStrategy":"validation","validationCode":"// Reject oversized records before the copy\nint est = estimatedTotalSize;\nint budget = logRecord.DataHeader.ValueIsObject ? maxHeapAllocationSize : estimatedTotalSize + sizeof(int);\nif (est > budget)\n    throw new InvalidOperationException($\"record {est} > budget {budget}; raise maxHeapAllocationSize or shrink the value\");","typeGuard":null,"tryCatchPattern":"try { DiskLogRecord.Serialize(rec, maxHeap, ser, pool, ref output); }\ncatch (TsavoriteException ex) when (ex.Message.Contains(\"exceeds max allocated heap size\"))\n{\n    // log est vs budget, drop or chunk the record, surface to caller\n}","preventionTips":["Size maxHeapAllocationSize to your largest expected record at configuration time.","Cap value sizes at the write path so oversized records never get stored.","Prefer a directly-writable SpanByte output (network buffer) to bypass the heap budget."],"tags":["tsavorite","memory","serialization","record-size","heap-allocation"],"backgroundTag":null,"analyzedSha":"951b0fc6838721f89d102c2bbe1b914e8d39d700","analyzedAt":"2026-08-13T19:01:32.939Z","schemaVersion":2},"datasetVersion":"2026-08-13T19:17:28.613Z"}