What causes "{length} exceeds buffer capacity" in dotnet/machinelearning?

Split the data into multiple smaller buffers/columns, each under MaxCapacity bytes (the DataFrameColumn APIs do this automatically)

dotnet/machinelearning · error · ArgumentException

{length} exceeds buffer capacity

Error message

{length} exceeds buffer capacity

What it means

The ReadOnlyDataFrameBuffer constructor validates that length * Size (bytes per element) does not exceed MaxCapacity (int.MaxValue bytes). If you request more elements than fit in a single managed byte array, the constructor throws ArgumentException with parameter name 'length'. Buffers are backed by single arrays, so a buffer cannot span the 2GB limit.

Source

Thrown at src/Microsoft.Data.Analysis/ReadOnlyDataFrameBuffer.cs:53

        protected static int Size = Unsafe.SizeOf<T>();

        protected int Capacity => ReadOnlyBuffer.Length / Size;

        public static int MaxCapacity => ArrayUtility.ArrayMaxSize / Size;

        public ReadOnlySpan<T> ReadOnlySpan
        {
            [MethodImpl(MethodImplOptions.AggressiveInlining)]
            get => (MemoryMarshal.Cast<byte, T>(ReadOnlyBuffer.Span)).Slice(0, Length);
        }

        public int Length { get; protected set; }

        public ReadOnlyDataFrameBuffer(int length = 0)
        {
            if ((long)length * Size > MaxCapacity)
            {
                throw new ArgumentException($"{length} exceeds buffer capacity", nameof(length));
            }
            _readOnlyBuffer = new byte[length * Size];
            Length = length;
        }

        public ReadOnlyDataFrameBuffer(ReadOnlyMemory<byte> buffer, int length)
        {
            _readOnlyBuffer = buffer;
            Length = length;
        }

        internal virtual T this[int index]
        {
            get
            {
                if (index >= Length)
                    throw new ArgumentOutOfRangeException(nameof(index));

View on GitHub (pinned to 7b76e69cf9)

Solutions

Split the data into multiple smaller buffers/columns, each under MaxCapacity bytes (the DataFrameColumn APIs do this automatically)
Reduce the requested length so length * sizeof(T) <= int.MaxValue
Check your length units — pass element count, not byte count, and verify Size for the element type
For datasets near 2GB, avoid in-place buffer construction and stream/append via the column APIs

Example fix

// before
var buffer = new ReadOnlyDataFrameBuffer<double>(300_000_000); // 2.4GB > MaxCapacity -> ArgumentException
// after
const int maxElements = int.MaxValue / sizeof(double); // ~268M
var buffer = new ReadOnlyDataFrameBuffer<double>(Math.Min(length, maxElements));
// keep the remainder in a second buffer

Defensive patterns

Strategy: validation

Validate before calling

static bool BufferLengthFits(long length, int elementSize) =>
    length >= 0 && length * elementSize <= int.MaxValue;

Type guard

bool CanCreateBuffer<T>(int length) => (long)length * System.Runtime.CompilerServices.Unsafe.SizeOf<T>() <= int.MaxValue;

Try / catch

try { var buffer = new ReadOnlyDataFrameBuffer<double>(length); }
catch (ArgumentException ex) when (ex.ParamName == "length") {
    // split into multiple buffers, each <= int.MaxValue / sizeof(double) elements
}

Prevention

Always pass element counts, never byte counts, to buffer constructors
Remember MaxCapacity is per-buffer (int.MaxValue bytes); columns split data across buffers automatically
Use the DataFrameColumn append APIs instead of constructing giant buffers manually
Compute maxElements = int.MaxValue / sizeof(T) as a named constant in buffer-heavy code

When it happens

Trigger: new ReadOnlyDataFrameBuffer<byte>(length) (or a derived buffer such as DataFrameBuffer<T>/ReadOnlyDataFrameBuffer<T>) with length*Size > ~2,147,483,591 bytes; e.g. a byte buffer with length > int.MaxValue, or a double buffer with more than ~268M elements requested in ONE buffer.

Common situations: Loading very large files or datasets into a single buffer instead of the column's multi-buffer layout; misreading Length as element capacity vs bytes; constructing a buffer manually with an inflated length instead of letting the column split data across buffers.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/b65f347061fb1031. Report an issue: GitHub.