dotnet/machinelearning · error · ArgumentException

Strings.SpansMultipleBuffers

Error message

Strings.SpansMultipleBuffers

What it means

PrimitiveDataFrameColumn<T>.ToArrowArray throws ArgumentException with the message Strings.SpansMultipleBuffers when the requested [startIndex, numberOfRows] slice is not contained within a single internal ReadOnlyDataFrameBuffer. Arrow arrays must be contiguous, so a range that would cross a buffer boundary (buffer capacity is ReadOnlyDataFrameBuffer<T>.MaxCapacity) is rejected via nameof(numberOfRows).

Source

Thrown at src/Microsoft.Data.Analysis/PrimitiveDataFrameColumn.cs:136

        private int GetNullCount(long startIndex, int numberOfRows)
        {
            int nullCount = 0;
            for (long i = startIndex; i < numberOfRows; i++)
            {
                if (!IsValid(i))
                    nullCount++;
            }
            return nullCount;
        }

        protected internal override Apache.Arrow.Array ToArrowArray(long startIndex, int numberOfRows)
        {
            int bufferIndex = numberOfRows == 0 ? 0 : _columnContainer.GetIndexOfBufferContainingRowIndex(startIndex);
            int offset = (int)(startIndex - bufferIndex * ReadOnlyDataFrameBuffer<T>.MaxCapacity);

            if (numberOfRows != 0 && numberOfRows > _columnContainer.Buffers[bufferIndex].Length - offset)
            {
                throw new ArgumentException(Strings.SpansMultipleBuffers, nameof(numberOfRows));
            }

            int nullCount = GetNullCount(startIndex, numberOfRows);

            //DateTime requires convertion
            if (this.DataType == typeof(DateTime))
            {
                if (numberOfRows == 0)
                    return new Date64Array(ArrowBuffer.Empty, ArrowBuffer.Empty, numberOfRows, nullCount, offset);

                ReadOnlyDataFrameBuffer<T> valueBuffer = (numberOfRows == 0) ? null : _columnContainer.Buffers[bufferIndex];
                ReadOnlyDataFrameBuffer<byte> nullBuffer = (numberOfRows == 0) ? null : _columnContainer.NullBitMapBuffers[bufferIndex];

                ReadOnlySpan<DateTime> valueSpan = MemoryMarshal.Cast<T, DateTime>(valueBuffer.ReadOnlySpan);
                Date64Array.Builder builder = new Date64Array.Builder().Reserve(valueBuffer.Length);

                for (int i = 0; i < valueBuffer.Length; i++)
                {

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Clamp numberOfRows so startIndex + numberOfRows stays within the containing buffer: numberOfRows = min(numberOfRows, bufferLength - offset)
2. Use GetMaxRecordBatchLength(startIndex) to pick a batch size that does not span buffers
3. Split the export into multiple Arrow arrays, one per buffer, and concatenate at the consumer level
4. Call ToArrowArray with numberOfRows == 0 only for empty arrays; avoid mixing zero and boundary-crossing requests

Example fix

// before
var array = column.ToArrowArray(startIndex, numberOfRows); // may span buffers
// after
var maxLen = column.GetMaxRecordBatchLength(startIndex);
var array = column.ToArrowArray(startIndex, Math.Min(numberOfRows, (int)maxLen));

long maxBatch = column.GetMaxRecordBatchLength(startIndex);
int safeRows = (int)Math.Min(numberOfRows, maxBatch);
var array = column.ToArrowArray(startIndex, safeRows);

try { return column.ToArrowArray(startIndex, numberOfRows); }
catch (ArgumentException ex) when (ex.ParamName == nameof(PrimitiveDataFrameColumn<int>.ToArrowArray) && ex.Message.Contains("buffers")) { numberOfRows = (int)column.GetMaxRecordBatchLength(startIndex); return column.ToArrowArray(startIndex, numberOfRows); }

When it happens

Trigger: Calling ToArrowArray(startIndex, numberOfRows) where numberOfRows exceeds the remaining length of the buffer that contains startIndex, i.e. startIndex + numberOfRows crosses a MaxCapacity boundary while numberOfRows != 0.

Common situations: Exporting large columns (over one buffer capacity) to Arrow with slices computed without regard to internal buffer boundaries; custom record-batch writers choosing oversized batch lengths; using GetMaxRecordBatchLength incorrectly.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• {fieldType.Name}
• Exception of type 'System.ArgumentException' was thrown.
• offsetBuffer
• Strings.SpansMultipleBuffers
• Strings.InconsistentNullBitMapAndLength

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/50f17ee927ca015c. Report an issue: GitHub.