What causes "Current buffer is full" in dotnet/machinelearning?

Stop appending once Length approaches MaxCapacity and flush/paginate the data

dotnet/machinelearning · error · ArgumentException

Current buffer is full

Error message

Current buffer is full

What it means

EnsureCapacity throws when appending numberOfValues more items would push Length past MaxCapacity. The buffer is full and cannot grow further, so the append is rejected with an ArgumentException naming numberOfValues.

Source

Thrown at src/Microsoft.Data.Analysis/DataFrameBuffer.cs:77

        {
            EnsureCapacity(1);

            RawSpan[Length] = value;
            Length++;
        }

        public void IncreaseSize(int numberOfValues)
        {
            EnsureCapacity(numberOfValues);
            Length += numberOfValues;
        }

        public void EnsureCapacity(int numberOfValues)
        {
            long newLength = Length + (long)numberOfValues;
            if (newLength > MaxCapacity)
            {
                throw new ArgumentException("Current buffer is full", nameof(numberOfValues));
            }

            if (newLength > Capacity)
            {
                //Double buffer size, but not higher than MaxByteCapacity
                var doubledSize = (int)Math.Min((long)ReadOnlyBuffer.Length * 2, ArrayUtility.ArrayMaxSize);
                var newCapacity = Math.Max(newLength * Size, doubledSize);

                var memory = new Memory<byte>(new byte[newCapacity]);
                _memory.CopyTo(memory);
                _memory = memory;
            }
        }

        internal override T this[int index]
        {
            set
            {

View on GitHub (pinned to 7b76e69cf9)

Solutions

Stop appending once Length approaches MaxCapacity and flush/paginate the data
Split ingestion into multiple buffers or DataFrame chunks
Pre-check Length + numberOfValues <= MaxCapacity before appending

Example fix

// before
foreach (var row in hugeSource) column.Append(row.Value);
// after
foreach (var row in hugeSource) {
    if (column.Length + 1 > DataFrameBuffer<int>.MaxCapacity) FlushAndReset();
    column.Append(row.Value);
}

Defensive patterns

Strategy: validation

Validate before calling

if (buffer.Length + numberOfValues > DataFrameBuffer<T>.MaxCapacity) FlushAndReset();

Try / catch

try { buffer.Append(item); }
catch (ArgumentException ex) when (ex.Message.Contains("Current buffer is full")) { /* flush, allocate new buffer, retry item */ }

Prevention

Track total appended count and paginate before hitting MaxCapacity
Stream large inputs into chunked buffers instead of one giant buffer
Add capacity assertions in ingestion loops

When it happens

Trigger: Calling Append or IncreaseSize (via EnsureCapacity, src/Microsoft.Data.Analysis/DataFrameBuffer.cs:77) when Length + numberOfValues > MaxCapacity; long-running streaming appends that never cap total size.

Common situations: Ingesting very large files into a single column buffer; aggregation loops that accumulate unbounded rows; underestimating dataset size versus the buffer max.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/60b1ca82477badba. Report an issue: GitHub.