agentscope-ai/agentscope · error · ValueError

overlap must be less than chunk_size, got overlap={self.over

Error message

overlap must be less than chunk_size, got overlap={self.overlap}, chunk_size={self.chunk_size}.

What it means

ApproxTokenChunker's Parameters model validator rejects configurations where overlap >= chunk_size, since overlapping chunks must be strictly smaller than the chunk size to make progress and avoid infinite/duplicate chunking.

Source

Thrown at src/agentscope/rag/_chunker/_approx_token_chunker.py:63

            title="Chunk Size",
            description="Maximum number of approximate tokens per chunk.",
        )
        overlap: int = Field(
            default=50,
            ge=0,
            title="Overlap",
            description=(
                "Number of approximate tokens shared between "
                "consecutive chunks."
            ),
        )

        @model_validator(mode="after")
        def _overlap_less_than_chunk_size(
            self,
        ) -> "ApproxTokenChunker.Parameters":
            if self.overlap >= self.chunk_size:
                raise ValueError(
                    "overlap must be less than chunk_size, got "
                    f"overlap={self.overlap}, chunk_size={self.chunk_size}.",
                )
            return self

    def __init__(
        self,
        parameters: "ApproxTokenChunker.Parameters | None" = None,
        **kwargs: Any,
    ) -> None:
        """Initialize the approx token chunker.

        Args:
            parameters (`ApproxTokenChunker.Parameters | None`, optional):
                The chunker parameters (``chunk_size`` and ``overlap``).
                Defaults to ``Parameters()`` when not provided.
            **kwargs (`Any`):
                Deprecated. ``chunk_size`` and ``overlap`` are still

View on GitHub (pinned to e90f1c7592)

Solutions

Set overlap strictly less than chunk_size, e.g. chunk_size=512, overlap=50
If computing overlap dynamically, clamp: overlap = min(overlap, chunk_size - 1)
A common safe ratio is overlap ≈ 10–20% of chunk_size

Example fix

# before
chunker = ApproxTokenChunker(parameters=Parameters(chunk_size=200, overlap=200))

# after
chunker = ApproxTokenChunker(parameters=Parameters(chunk_size=512, overlap=64))

Defensive patterns

Strategy: validation

Validate before calling

overlap = min(overlap, chunk_size - 1)
params = ApproxTokenChunker.Parameters(chunk_size=chunk_size, overlap=overlap)

Prevention

Keep overlap at 10-20% of chunk_size
Compute overlap from chunk_size programmatically, never hardcode both independently

When it happens

Trigger: Constructing Parameters(chunk_size=200, overlap=200) or overlap > chunk_size; also chunk_size=0 with overlap=0 (0 >= 0).

Common situations: Copying chunking configs from RAG tutorials with different defaults; computing overlap as a percentage but passing it as absolute tokens; setting chunk_size equal to overlap by accident.

Related errors

AI-assisted analysis of agentscope-ai/agentscope@e90f1c7592 (2026-08-28). Data as JSON: /api/errors/5fffe804a3438cee. Report an issue: GitHub.