hiyouga/LlamaFactory · error · ValueError

Buffer size exceeds max buffer size {self._max_buffer_size}.

Error message

Buffer size exceeds max buffer size {self._max_buffer_size}.

What it means

ValueError from the token buffer's put() (objects.py:44): adding the batch's tokens would push the buffered token count past _max_buffer_size. The buffer counts tokens (sum of input_ids lengths), not samples, and the check is all-or-nothing: an oversized single put() raises even on an empty buffer.

Source

Thrown at src/llamafactory/v1/utils/objects.py:44

        self._buffer_size: int = 0
        self._max_buffer_size: int = max_buffer_size

    def __len__(self) -> int:
        return len(self._buffer)

    @property
    def size(self) -> int:
        return self._buffer_size

    @property
    def samples(self) -> list[ModelInput]:
        return self._buffer

    def put(self, samples: list[ModelInput]) -> None:
        """Add samples to the buffer."""
        num_tokens = sum(len(sample["input_ids"]) for sample in samples)
        if self._buffer_size + num_tokens > self._max_buffer_size:
            raise ValueError(f"Buffer size exceeds max buffer size {self._max_buffer_size}.")

        self._buffer.extend(samples)
        self._buffer_size += num_tokens

    def get(self, value: int) -> list[ModelInput]:
        """Get samples from the buffer and remove them."""
        samples = self._buffer[:value]
        self._buffer_size -= sum(len(sample["input_ids"]) for sample in samples)
        del self._buffer[:value]
        return samples

    def clear(self) -> None:
        """Clear the buffer."""
        self._buffer = []
        self._buffer_size = 0

    def state_dict(self) -> dict:
        """Returns the state of the buffer."""

View on GitHub (pinned to f28afaf635)

Solutions

1. Drain the buffer (call get()/clear()) before putting more samples so _buffer_size is near zero.
2. Put samples in smaller chunks so each chunk's token sum fits within the remaining capacity.
3. Raise the buffer's max size at construction if throughput requires larger bursts.
4. Compute the chunk's total tokens first and split it to fit: while tokens remain, put only what fits.

Example fix

# before
buffer.put(all_samples)  # may exceed max_buffer_size

# after
i = 0
while i < len(all_samples):
    chunk = []
    used = 0
    while i < len(all_samples) and used + len(all_samples[i]['input_ids']) <= buffer.max_buffer_size - buffer.size:
        chunk.append(all_samples[i]); used += len(all_samples[i]['input_ids']); i += 1
    buffer.put(chunk)

tokens = sum(len(s["input_ids"]) for s in samples)
assert buffer.size + tokens <= buffer._max_buffer_size, f"put of {tokens} tokens would exceed max {buffer._max_buffer_size}"

When it happens

Trigger: Calling put(samples) where sum(len(sample['input_ids'])) plus the current _buffer_size exceeds max buffer size; common in generation/offline pipelines that accumulate batches of tokens, e.g. after raising batch size or sequence length.

Common situations: Larger cutoff_len or per-device batch size increases tokens per put; consumer falling behind so the buffer stays near capacity; a burst of long samples arriving together.

Related errors

• Please enable `predict_with_generate` to save model predicti
• Invalid input type {input_item.type}.
• Invalid tools
• Invalid request
• Unknown backend: {model_args.infer_backend}

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/2e63062e5947aaa0. Report an issue: GitHub.