microsoft/FASTER · error · FasterException

Unable to read object page, total size greater than 2GB:

Error message

Unable to read object page, total size greater than 2GB: 

What it means

Thrown during object-page reads in GenericAllocator when the aligned length of the object-log page/fragment to read exceeds int.MaxValue (~2GB). The in-memory object read buffer and the device ReadAsync API are sized with 32-bit ints, so a larger single read cannot be represented. FASTER throws instead of truncating the read.

Solutions

1. Reduce the object log page size (lower LogSettings.PageSizeBits / LogPageSizeBits) so each read stays under 2GB.
2. Keep individual serialized objects small and ensure objects are distributed across pages rather than concentrated.
3. If possible, re-create the store with default page sizes and re-populate data from the source of truth.
4. Verify sector alignment assumptions — the alignedLength check runs only after alignment; oversized raw pages are the root cause.

Example fix

// before: 31-bit page size -> >2GB reads
var settings = new LogSettings { PageSizeBits = 31 };

// after: keep pages well under 2GB
var settings = new LogSettings { PageSizeBits = 22 }; // 4MB pages

// Ensure configured page sizes cannot yield >2GB object reads
long alignedMax = 1L << settings.PageSizeBits;
if (alignedMax > int.MaxValue)
    throw new InvalidOperationException($"PageSizeBits too large: 2^{settings.PageSizeBits} exceeds int.MaxValue");

static bool PageSizeWithinIntLimit(int pageSizeBits) => (1L << pageSizeBits) <= int.MaxValue;

try
{
    var page = ReadObjectPage(address, alignedLength);
}
catch (FasterException ex) when (ex.Message.Contains("greater than 2GB"))
{
    // Rebuild the store with smaller page sizes or migrate data to a new instance
    log.LogError("Object log page exceeds 2GB; re-create store with smaller PageSizeBits");
    throw;
}

When it happens

Trigger: Reading back an object log whose page/segment size (LogPageSizeBits-derived alignedLength) exceeds 2GB — typically an oversized object log page size or extremely large accumulated object fragments in one page on a restart/recovery scan.

Common situations: Configuring very large page sizes for the object log; upgrading page size bits after logs were written with objects spanning huge regions; recovery of an object log written by a differently tuned instance.

Understand the failure class

Background: "File too large" / "file size exceeds limit" errors: why libraries cap file sizes and how to fix them — this error's family across 46 libraries.

Related errors

• Unexpected entry type
• Unable to set first valid segment to
• Unable to set last valid segment to
• LogSettings.ObjectLogDevice needs to be specified (e.g.…
• Objects in key/value, but object log not provided during…

AI-assisted analysis of microsoft/FASTER@321d872eab (2026-09-15). Data as JSON: /api/errors/c4f20b50d9709456. Report an issue: GitHub.

Appendix: source

Thrown at cs/src/core/Allocator/GenericAllocator.cs:702

            // If we have processed entire page, return
            if (result.untilPtr >= result.maxPtr)
            {
                result.Free();

                // Call the "real" page read callback
                result.callback(errorCode, numBytes, context);
                return;
            }

            // We will now be able to process all records until (but not including) untilPtr
            GetObjectInfo(result.freeBuffer1.GetValidPointer(), ref result.untilPtr, result.maxPtr, ObjectBlockSize, out long startptr, out long alignedLength);

            // Object log fragment should be aligned by construction
            Debug.Assert(startptr % sectorSize == 0);
            Debug.Assert(alignedLength % sectorSize == 0);

            if (alignedLength > int.MaxValue)
                throw new FasterException("Unable to read object page, total size greater than 2GB: " + alignedLength);

            var objBuffer = bufferPool.Get((int)alignedLength);
            result.freeBuffer2 = objBuffer;

            // Request objects from objlog
            result.objlogDevice.ReadAsync(
                (int)((result.page - result.offset) >> (LogSegmentSizeBits - LogPageSizeBits)),
                (ulong)startptr,
                (IntPtr)objBuffer.aligned_pointer, (uint)alignedLength, AsyncReadPageWithObjectsCallback<TContext>, result);
        }

        /// <summary>
        /// Invoked by users to obtain a record from disk. It uses sector aligned memory to read 
        /// the record efficiently into memory.
        /// </summary>
        /// <param name="fromLogical"></param>
        /// <param name="numBytes"></param>
        /// <param name="callback"></param>

View on GitHub (pinned to 321d872eab)