{"record":{"id":"ffd67172623fe69d","repo":"run-llama/llama_index","slug":"metadata-length-metadata-len-is-longer-than-ch-ffd671","errorCode":null,"errorMessage":"Metadata length ({metadata_len}) is longer than chunk size ({self.chunk_size}). Consider increasing the chunk size or decreasing the size of your metadata to avoid this.","messagePattern":"Metadata length \\((.+?)\\) is longer than chunk size \\((.+?)\\)\\. Consider increasing the chunk size or decreasing the size of your metadata to avoid this\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/node_parser/text/token.py","lineNumber":122,"sourceCode":"            separator=separator,\n            backup_separators=backup_separators,\n            keep_whitespaces=keep_whitespaces,\n            callback_manager=callback_manager,\n            include_metadata=include_metadata,\n            include_prev_next_rel=include_prev_next_rel,\n            id_func=id_func,\n        )\n\n    @classmethod\n    def class_name(cls) -> str:\n        return \"TokenTextSplitter\"\n\n    def split_text_metadata_aware(self, text: str, metadata_str: str) -> List[str]:\n        \"\"\"Split text into chunks, reserving space required for metadata str.\"\"\"\n        metadata_len = len(self._tokenizer(metadata_str)) + DEFAULT_METADATA_FORMAT_LEN\n        effective_chunk_size = self.chunk_size - metadata_len\n        if effective_chunk_size <= 0:\n            raise ValueError(\n                f\"Metadata length ({metadata_len}) is longer than chunk size \"\n                f\"({self.chunk_size}). Consider increasing the chunk size or \"\n                \"decreasing the size of your metadata to avoid this.\"\n            )\n        elif effective_chunk_size < 50:\n            print(\n                f\"Metadata length ({metadata_len}) is close to chunk size \"\n                f\"({self.chunk_size}). Resulting chunks are less than 50 tokens. \"\n                \"Consider increasing the chunk size or decreasing the size of \"\n                \"your metadata to avoid this.\",\n                flush=True,\n            )\n\n        return self._split_text(text, chunk_size=effective_chunk_size)\n\n    def split_text(self, text: str) -> List[str]:\n        \"\"\"Split text into chunks.\"\"\"\n        return self._split_text(text, chunk_size=self.chunk_size)","sourceCodeStart":104,"sourceCodeEnd":140,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/node_parser/text/token.py#L104-L140","documentation":"TokenTextSplitter.split_text_metadata_aware reserves room inside each chunk for the serialized metadata plus a fixed format overhead (DEFAULT_METADATA_FORMAT_LEN). It computes effective_chunk_size = chunk_size - (len(tokenizer(metadata_str)) + DEFAULT_METADATA_FORMAT_LEN) and raises when that value is <= 0, i.e. the metadata alone does not fit in a chunk. This usually surfaces indirectly when a MetadataAwareTextSplitter or node parser runs in metadata-aware mode with large document metadata.","triggerScenarios":"Calling split_text_metadata_aware(text, metadata_str) (or building an index with a chunk_size set low, e.g. 128/256) where the tokenized metadata_str plus the format template exceeds chunk_size; e.g. a metadata dict with a long 'file_description' or many excluded_embed_metadata_keys serialized into metadata_str.","commonSituations":"Lowering chunk_size for embedding-model context limits (e.g. 256 for BGE-small) while keeping rich metadata; using custom metadata templates in SentenceSplitter; documents carrying large extracted metadata (whole summaries, base64 fields); upgrading code that previously ignored metadata in chunking.","solutions":["Increase chunk_size above len(tokenizer(metadata_str)) + DEFAULT_METADATA_FORMAT_LEN (check DEFAULT_METADATA_FORMAT_LEN in llama_index.core.node_parser.text.token, it accounts for the 'path/to/key value ' template overhead)","Trim the metadata that gets serialized: remove large keys from the metadata passed to the splitter or shorten metadata values before indexing","Simplify custom metadata_str formats so less of the chunk is consumed by metadata","Tokenize the metadata first and assert chunk_size - metadata_len > 0 before calling the splitter"],"exampleFix":"# before\nsplitter = TokenTextSplitter(chunk_size=256)\nchunks = splitter.split_text_metadata_aware(doc.text, metadata_str)  # ValueError: metadata ~300 tokens\n\n# after\nsplitter = TokenTextSplitter(chunk_size=512)\nchunks = splitter.split_text_metadata_aware(doc.text, metadata_str)","handlingStrategy":"validation","validationCode":"from llama_index.core.node_parser.text.token import DEFAULT_METADATA_FORMAT_LEN\n\nmetadata_len = len(splitter._tokenizer(metadata_str)) + DEFAULT_METADATA_FORMAT_LEN\nif splitter.chunk_size - metadata_len <= 0:\n    raise ValueError(\n        f\"metadata needs {metadata_len} tokens but chunk_size={splitter.chunk_size}; \"\n        \"raise chunk_size or trim metadata before splitting\"\n    )\nchunks = splitter.split_text_metadata_aware(text, metadata_str)","typeGuard":"def is_metadata_fits_chunk(splitter: TokenTextSplitter, metadata_str: str) -> bool:\n    needed = len(splitter._tokenizer(metadata_str)) + DEFAULT_METADATA_FORMAT_LEN\n    return splitter.chunk_size - needed > 0","tryCatchPattern":"try:\n    chunks = splitter.split_text_metadata_aware(text, metadata_str)\nexcept ValueError as e:\n    if \"Metadata length\" in str(e):\n        raise ValueError(f\"chunk_size={splitter.chunk_size} too small for metadata; trim metadata or increase chunk_size\") from e\n    raise","preventionTips":["Set chunk_size with headroom: budget len(tokenizer(metadata_str)) + DEFAULT_METADATA_FORMAT_LEN + at least 100 tokens of text","Keep large fields out of chunked metadata (store them in a separate store keyed by node id)","Run a pre-flight check on one representative document before bulk indexing"],"tags":["chunking","tokenization","configuration","node-parser"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}