deepset-ai/haystack · error · ValueError

Overlap cannot be greater than or equal to the chunk size.

Error message

Overlap cannot be greater than or equal to the chunk size.

What it means

RecursiveSplitter enforces split_overlap < split_length. If overlap equals or exceeds the chunk size, windows would not advance (infinite loop) or would produce degenerate output, so __init__ rejects it.

Source

Thrown at haystack/components/preprocessors/recursive_splitter.py:119

        """
        Warm up the sentence tokenizer and tiktoken tokenizer if needed.
        """
        if self._is_warmed_up:
            return
        if "sentence" in self.separators:
            self.nltk_tokenizer = self._get_custom_sentence_tokenizer(self.sentence_splitter_params)
        if self.split_units == "token":
            tiktoken_imports.check()
            self.tiktoken_tokenizer = tiktoken.get_encoding("o200k_base")
        self._is_warmed_up = True

    def _check_params(self) -> None:
        if self.split_length < 1:
            raise ValueError("Split length must be at least 1 character.")
        if self.split_overlap < 0:
            raise ValueError("split_overlap must be greater than or equal to 0.")
        if self.split_overlap >= self.split_length:
            raise ValueError("Overlap cannot be greater than or equal to the chunk size.")
        if not all(isinstance(separator, str) for separator in self.separators):
            raise ValueError("All separators must be strings.")

    @staticmethod
    def _get_custom_sentence_tokenizer(sentence_splitter_params: dict[str, Any]) -> Any:
        from haystack.components.preprocessors.sentence_tokenizer import SentenceSplitter

        return SentenceSplitter(**sentence_splitter_params)

    def _split_chunk(self, current_chunk: str) -> tuple[str, str]:
        """
        Splits a chunk based on the split_length and split_units attribute.

        :param current_chunk: The current chunk to be split.
        :returns:
            A tuple containing the current chunk and the remaining chunk.
        """
        if self.split_units == "word":

View on GitHub (pinned to e318778c9b)

Solutions

1. Ensure split_overlap < split_length, e.g. keep overlap around 10-20% of split_length.
2. Recompute overlap whenever split_length changes: overlap = min(overlap, split_length - 1).
3. Set split_overlap=0 if overlap is not needed.

Example fix

// before
splitter = RecursiveSplitter(split_length=100, split_overlap=100)
// after
splitter = RecursiveSplitter(split_length=100, split_overlap=20)

if split_overlap >= split_length:
    raise ValueError(f'split_overlap ({split_overlap}) must be < split_length ({split_length})')

def are_valid_chunk_params(length: int, overlap: int) -> bool:
    return isinstance(length, int) and isinstance(overlap, int) and length >= 1 and 0 <= overlap < length

try:
    splitter = RecursiveSplitter(split_length=length, split_overlap=overlap)
except ValueError as e:
    logger.error('Invalid chunk params: %s', e)
    splitter = RecursiveSplitter(split_length=length, split_overlap=min(overlap, length - 1))

When it happens

Trigger: RecursiveSplitter(split_length=100, split_overlap=100) or overlap > length; changing split_length downward in config without lowering overlap; swapped parameter values.

Common situations: Tuning chunk sizes where overlap was set as a fraction of an older, larger split_length; copy-paste between splitter configs with different scales.

Understand the failure class

Background: "must be positive", "Invalid value": how libraries reject invalid parameter values (ValueError, ArgumentError, INVALID_PARAMETER_VALUE) — this error's family across 28 libraries.

Related errors

• header_split_levels must be a non-empty list.
• min_effective_lines must be at least 1.
• max_effective_lines must be at least 1.
• expected_chars_per_line must be at least 1.
• oversized_factor must be at least 1.

AI-assisted analysis of deepset-ai/haystack@e318778c9b (2026-08-30). Data as JSON: /api/errors/da3a8936bf0801dd. Report an issue: GitHub.