run-llama/llama_index · error · ValueError

Single token exceeded chunk size

Error message

Single token exceeded chunk size

What it means

Error "Single token exceeded chunk size" thrown in run-llama/llama_index.

Source

Thrown at llama-index-core/llama_index/core/node_parser/text/sentence.py:280

            new_chunk = True

            # add overlap to the next chunk using the last one first
            if len(last_chunk) > 0:
                last_index = len(last_chunk) - 1
                while (
                    last_index >= 0
                    and cur_chunk_len + last_chunk[last_index][1] <= self.chunk_overlap
                ):
                    overlap_text, overlap_length = last_chunk[last_index]
                    cur_chunk_len += overlap_length
                    cur_chunk.insert(0, (overlap_text, overlap_length))
                    last_index -= 1

        split_idx = 0
        while split_idx < len(splits):
            cur_split = splits[split_idx]
            if cur_split.token_size > chunk_size:
                raise ValueError("Single token exceeded chunk size")
            if cur_chunk_len + cur_split.token_size > chunk_size and not new_chunk:
                # if adding split to current chunk exceeds chunk size: close out chunk
                close_chunk()
            else:
                # If this is a new chunk with overlap, and adding the split would
                # exceed chunk_size, remove overlap to make room
                if new_chunk and cur_chunk_len + cur_split.token_size > chunk_size:
                    # Remove overlap from the beginning until split fits
                    while (
                        len(cur_chunk) > 0
                        and cur_chunk_len + cur_split.token_size > chunk_size
                    ):
                        _, length = cur_chunk.pop(0)
                        cur_chunk_len -= length

                if (
                    cur_split.is_sentence
                    or cur_chunk_len + cur_split.token_size <= chunk_size

View on GitHub (pinned to afd0fef371)

Solutions

1. Increase chunk_size so the largest single token/word fits.
2. Pre-split very long unbroken strings in the input text before parsing.

When it happens

Trigger: Thrown at llama-index-core/llama_index/core/node_parser/text/sentence.py:280 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/ea6da9bd5d9a2d35. Report an issue: GitHub.