{"record":{"id":"74f05ce4d67aa40c","repo":"crewAIInc/crewAI","slug":"chunk-overlap-chunk-overlap-cannot-be-chunk","errorCode":null,"errorMessage":"Chunk overlap ({chunk_overlap}) cannot be >= chunk size ({chunk_size})","messagePattern":"Chunk overlap \\((.+?)\\) cannot be >= chunk size \\((.+?)\\)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"lib/crewai-tools/src/crewai_tools/rag/chunkers/base_chunker.py","lineNumber":23,"sourceCode":"    \"\"\"A text splitter that recursively splits text based on a hierarchy of separators.\"\"\"\n\n    def __init__(\n        self,\n        chunk_size: int = 4000,\n        chunk_overlap: int = 200,\n        separators: list[str] | None = None,\n        keep_separator: bool = True,\n    ) -> None:\n        \"\"\"Initialize the RecursiveCharacterTextSplitter.\n\n        Args:\n            chunk_size: Maximum size of each chunk\n            chunk_overlap: Number of characters to overlap between chunks\n            separators: List of separators to use for splitting (in order of preference)\n            keep_separator: Whether to keep the separator in the split text\n        \"\"\"\n        if chunk_overlap >= chunk_size:\n            raise ValueError(\n                f\"Chunk overlap ({chunk_overlap}) cannot be >= chunk size ({chunk_size})\"\n            )\n\n        self._chunk_size = chunk_size\n        self._chunk_overlap = chunk_overlap\n        self._keep_separator = keep_separator\n\n        self._separators = separators or [\n            \"\\n\\n\",\n            \"\\n\",\n            \" \",\n            \"\",\n        ]\n\n    def split_text(self, text: str) -> list[str]:\n        \"\"\"Split the input text into chunks.\n\n        Args:","sourceCodeStart":5,"sourceCodeEnd":41,"githubUrl":"https://github.com/crewAIInc/crewAI/blob/754d7323beb2fd042e33444a115ea2d5a47193f0/lib/crewai-tools/src/crewai_tools/rag/chunkers/base_chunker.py#L5-L41","documentation":"BaseChunker (RecursiveCharacterTextSplitter wrapper) rejects chunk_overlap >= chunk_size at construction. With overlap >= size, the splitter could never advance through the text (each chunk would re-cover the overlap), producing infinite loops or empty chunks, so it fails fast with ValueError.","triggerScenarios":"Calling BaseChunker(chunk_size=500, chunk_overlap=500) or any overlap >= size, e.g. config with chunk_size=1000/overlap=1000 or downsizing chunk_size while keeping a large overlap (chunk_size=512 with overlap=1024 inherited from a preset).","commonSituations":"Copying RAG config from a project that used a larger chunk_size; tuning chunk_size down for recall but forgetting overlap; YAML presets where the two values are edited independently; equal values passed as a misreading of 'overlap must be smaller'.","solutions":["Set overlap strictly less than chunk_size — a common ratio is 10-20% of chunk_size (e.g. 1000/150)","Derive overlap from size programmatically: chunk_overlap = min(chunk_overlap, chunk_size // 2)","Validate config at load time before constructing the chunker"],"exampleFix":"# before\nchunker = BaseChunker(chunk_size=512, chunk_overlap=512)\n# after\nchunk_overlap = min(chunk_overlap, chunk_size // 2)\nchunker = BaseChunker(chunk_size=512, chunk_overlap=chunk_overlap)","handlingStrategy":"validation","validationCode":"if chunk_overlap >= chunk_size:\n    raise ValueError('chunk_overlap must be < chunk_size')\n# or auto-fix:\nchunk_overlap = min(chunk_overlap, chunk_size // 2)","typeGuard":"def is_valid_chunk_config(size: int, overlap: int) -> bool:\n    return size > 0 and 0 <= overlap < size","tryCatchPattern":"try:\n    chunker = BaseChunker(chunk_size=cfg['chunk_size'], chunk_overlap=cfg['chunk_overlap'])\nexcept ValueError:\n    cfg['chunk_overlap'] = cfg['chunk_size'] // 5\n    chunker = BaseChunker(**cfg)","preventionTips":["Derive overlap as a fraction of chunk_size (10-20%) instead of hardcoding both","Validate RAG config with Pydantic constraints (overlap < size) at load","When tuning chunk_size down, recompute overlap in the same change"],"tags":["rag","chunking","validation","config"],"backgroundTag":null,"analyzedSha":"754d7323beb2fd042e33444a115ea2d5a47193f0","analyzedAt":"2026-08-15T04:06:56.746Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}