run-llama/llama_index · error · ValueError

Metadata length ({metadata_len}) is longer than chunk size (

Error message

Metadata length ({metadata_len}) is longer than chunk size ({self.chunk_size}). Consider increasing the chunk size or decreasing the size of your metadata to avoid this.

What it means

TokenTextSplitter.split_text_metadata_aware reserves room inside each chunk for the serialized metadata plus a fixed format overhead (DEFAULT_METADATA_FORMAT_LEN). It computes effective_chunk_size = chunk_size - (len(tokenizer(metadata_str)) + DEFAULT_METADATA_FORMAT_LEN) and raises when that value is <= 0, i.e. the metadata alone does not fit in a chunk. This usually surfaces indirectly when a MetadataAwareTextSplitter or node parser runs in metadata-aware mode with large document metadata.

Source

Thrown at llama-index-core/llama_index/core/node_parser/text/token.py:122

            separator=separator,
            backup_separators=backup_separators,
            keep_whitespaces=keep_whitespaces,
            callback_manager=callback_manager,
            include_metadata=include_metadata,
            include_prev_next_rel=include_prev_next_rel,
            id_func=id_func,
        )

    @classmethod
    def class_name(cls) -> str:
        return "TokenTextSplitter"

    def split_text_metadata_aware(self, text: str, metadata_str: str) -> List[str]:
        """Split text into chunks, reserving space required for metadata str."""
        metadata_len = len(self._tokenizer(metadata_str)) + DEFAULT_METADATA_FORMAT_LEN
        effective_chunk_size = self.chunk_size - metadata_len
        if effective_chunk_size <= 0:
            raise ValueError(
                f"Metadata length ({metadata_len}) is longer than chunk size "
                f"({self.chunk_size}). Consider increasing the chunk size or "
                "decreasing the size of your metadata to avoid this."
            )
        elif effective_chunk_size < 50:
            print(
                f"Metadata length ({metadata_len}) is close to chunk size "
                f"({self.chunk_size}). Resulting chunks are less than 50 tokens. "
                "Consider increasing the chunk size or decreasing the size of "
                "your metadata to avoid this.",
                flush=True,
            )

        return self._split_text(text, chunk_size=effective_chunk_size)

    def split_text(self, text: str) -> List[str]:
        """Split text into chunks."""
        return self._split_text(text, chunk_size=self.chunk_size)

View on GitHub (pinned to afd0fef371)

Solutions

1. Increase chunk_size above len(tokenizer(metadata_str)) + DEFAULT_METADATA_FORMAT_LEN (check DEFAULT_METADATA_FORMAT_LEN in llama_index.core.node_parser.text.token, it accounts for the 'path/to/key value ' template overhead)
2. Trim the metadata that gets serialized: remove large keys from the metadata passed to the splitter or shorten metadata values before indexing
3. Simplify custom metadata_str formats so less of the chunk is consumed by metadata
4. Tokenize the metadata first and assert chunk_size - metadata_len > 0 before calling the splitter

Example fix

# before
splitter = TokenTextSplitter(chunk_size=256)
chunks = splitter.split_text_metadata_aware(doc.text, metadata_str)  # ValueError: metadata ~300 tokens

# after
splitter = TokenTextSplitter(chunk_size=512)
chunks = splitter.split_text_metadata_aware(doc.text, metadata_str)

from llama_index.core.node_parser.text.token import DEFAULT_METADATA_FORMAT_LEN

metadata_len = len(splitter._tokenizer(metadata_str)) + DEFAULT_METADATA_FORMAT_LEN
if splitter.chunk_size - metadata_len <= 0:
    raise ValueError(
        f"metadata needs {metadata_len} tokens but chunk_size={splitter.chunk_size}; "
        "raise chunk_size or trim metadata before splitting"
    )
chunks = splitter.split_text_metadata_aware(text, metadata_str)

def is_metadata_fits_chunk(splitter: TokenTextSplitter, metadata_str: str) -> bool:
    needed = len(splitter._tokenizer(metadata_str)) + DEFAULT_METADATA_FORMAT_LEN
    return splitter.chunk_size - needed > 0

try:
    chunks = splitter.split_text_metadata_aware(text, metadata_str)
except ValueError as e:
    if "Metadata length" in str(e):
        raise ValueError(f"chunk_size={splitter.chunk_size} too small for metadata; trim metadata or increase chunk_size") from e
    raise

When it happens

Trigger: Calling split_text_metadata_aware(text, metadata_str) (or building an index with a chunk_size set low, e.g. 128/256) where the tokenized metadata_str plus the format template exceeds chunk_size; e.g. a metadata dict with a long 'file_description' or many excluded_embed_metadata_keys serialized into metadata_str.

Common situations: Lowering chunk_size for embedding-model context limits (e.g. 256 for BGE-small) while keeping rich metadata; using custom metadata templates in SentenceSplitter; documents carrying large extracted metadata (whole summaries, base64 fields); upgrading code that previously ignored metadata in chunking.

Related errors

• Configured node parser does not have chunk size.
• Configured node parser does not have chunk overlap.
• embed_model must start with str 'local' or of type BaseEmbed
• Must pass in cohere api key or specify via COHERE_API_KEY en
• Invalid metric name: {metric}

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/ffd67172623fe69d. Report an issue: GitHub.