crewAIInc/crewAI · error · ValueError

Chunk overlap ({chunk_overlap}) cannot be >= chunk size ({ch

Error message

Chunk overlap ({chunk_overlap}) cannot be >= chunk size ({chunk_size})

What it means

BaseChunker (RecursiveCharacterTextSplitter wrapper) rejects chunk_overlap >= chunk_size at construction. With overlap >= size, the splitter could never advance through the text (each chunk would re-cover the overlap), producing infinite loops or empty chunks, so it fails fast with ValueError.

Source

Thrown at lib/crewai-tools/src/crewai_tools/rag/chunkers/base_chunker.py:23

    """A text splitter that recursively splits text based on a hierarchy of separators."""

    def __init__(
        self,
        chunk_size: int = 4000,
        chunk_overlap: int = 200,
        separators: list[str] | None = None,
        keep_separator: bool = True,
    ) -> None:
        """Initialize the RecursiveCharacterTextSplitter.

        Args:
            chunk_size: Maximum size of each chunk
            chunk_overlap: Number of characters to overlap between chunks
            separators: List of separators to use for splitting (in order of preference)
            keep_separator: Whether to keep the separator in the split text
        """
        if chunk_overlap >= chunk_size:
            raise ValueError(
                f"Chunk overlap ({chunk_overlap}) cannot be >= chunk size ({chunk_size})"
            )

        self._chunk_size = chunk_size
        self._chunk_overlap = chunk_overlap
        self._keep_separator = keep_separator

        self._separators = separators or [
            "\n\n",
            "\n",
            " ",
            "",
        ]

    def split_text(self, text: str) -> list[str]:
        """Split the input text into chunks.

        Args:

View on GitHub (pinned to 754d7323be)

Solutions

Set overlap strictly less than chunk_size — a common ratio is 10-20% of chunk_size (e.g. 1000/150)
Derive overlap from size programmatically: chunk_overlap = min(chunk_overlap, chunk_size // 2)
Validate config at load time before constructing the chunker

Example fix

# before
chunker = BaseChunker(chunk_size=512, chunk_overlap=512)
# after
chunk_overlap = min(chunk_overlap, chunk_size // 2)
chunker = BaseChunker(chunk_size=512, chunk_overlap=chunk_overlap)

Defensive patterns

Strategy: validation

Validate before calling

if chunk_overlap >= chunk_size:
    raise ValueError('chunk_overlap must be < chunk_size')
# or auto-fix:
chunk_overlap = min(chunk_overlap, chunk_size // 2)

Type guard

def is_valid_chunk_config(size: int, overlap: int) -> bool:
    return size > 0 and 0 <= overlap < size

Try / catch

try:
    chunker = BaseChunker(chunk_size=cfg['chunk_size'], chunk_overlap=cfg['chunk_overlap'])
except ValueError:
    cfg['chunk_overlap'] = cfg['chunk_size'] // 5
    chunker = BaseChunker(**cfg)

Prevention

Derive overlap as a fraction of chunk_size (10-20%) instead of hardcoding both
Validate RAG config with Pydantic constraints (overlap < size) at load
When tuning chunk_size down, recompute overlap in the same change

When it happens

Trigger: Calling BaseChunker(chunk_size=500, chunk_overlap=500) or any overlap >= size, e.g. config with chunk_size=1000/overlap=1000 or downsizing chunk_size while keeping a large overlap (chunk_size=512 with overlap=1024 inherited from a preset).

Common situations: Copying RAG config from a project that used a larger chunk_size; tuning chunk_size down for recall but forgetting overlap; YAML presets where the two values are edited independently; equal values passed as a misreading of 'overlap must be smaller'.

Related errors

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/74f05ce4d67aa40c. Report an issue: GitHub.