{"record":{"id":"bce8e262b855e040","repo":"open-mmlab/mmdetection","slug":"inputting-a-text-that-is-too-long-will-result-in-p-bce8e2","errorCode":null,"errorMessage":"Inputting a text that is too long will result in poor prediction performance. Please reduce the --chunked-size.","messagePattern":"Inputting a text that is too long will result in poor prediction performance\\. Please reduce the --chunked-size\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"mmdet/models/detectors/grounding_dino.py","lineNumber":286,"sourceCode":"            list(range(1,\n                       len(original_caption) + 1)), chunked_size)\n\n        positive_map_label_to_token_chunked = []\n        caption_string_chunked = []\n        positive_map_chunked = []\n        entities_chunked = []\n\n        for i in range(len(ids_chunked)):\n            if enhanced_text_prompts is not None:\n                caption_string, tokens_positive = self.to_enhance_text_prompts(\n                    original_caption_chunked[i], enhanced_text_prompts)\n            else:\n                caption_string, tokens_positive = self.to_plain_text_prompts(\n                    original_caption_chunked[i])\n            tokenized = self.language_model.tokenizer([caption_string],\n                                                      return_tensors='pt')\n            if tokenized.input_ids.shape[1] > self.language_model.max_tokens:\n                warnings.warn('Inputting a text that is too long will result '\n                              'in poor prediction performance. '\n                              'Please reduce the --chunked-size.')\n            positive_map_label_to_token, positive_map = self.get_positive_map(\n                tokenized, tokens_positive)\n\n            caption_string_chunked.append(caption_string)\n            positive_map_label_to_token_chunked.append(\n                positive_map_label_to_token)\n            positive_map_chunked.append(positive_map)\n            entities_chunked.append(original_caption_chunked[i])\n\n        return positive_map_label_to_token_chunked, \\\n            caption_string_chunked, \\\n            positive_map_chunked, \\\n            entities_chunked\n\n    def forward_transformer(\n        self,","sourceCodeStart":268,"sourceCodeEnd":304,"githubUrl":"https://github.com/open-mmlab/mmdetection/blob/cfd5d3a985b0249de009b67d04f37263e11cdf3d/mmdet/models/detectors/grounding_dino.py#L268-L304","documentation":"Grounding DINO's chunked prompt tokenizer warns that a chunk's tokenized caption exceeds the language model's max_tokens, degrading embedding/prediction quality; it recommends reducing the chunk size.","triggerScenarios":"get_tokens_positive_and_prompts_chunked with a chunk whose tokenized input_ids length > language_model.max_tokens.","commonSituations":"Grounding DINO demos/inference with many categories and a large chunked-size value.","solutions":["Lower the --chunked-size so each chunk fits max_tokens","Shorten/normalize category text"],"exampleFix":"# before\npython demo/image_demo.py img.jpg grounding_dino config ckpt --texts 'many categories ...' --chunked-size 10\n# after\npython demo/image_demo.py img.jpg grounding_dino config ckpt --texts 'many categories ...' --chunked-size 3","handlingStrategy":"validation","validationCode":"tok = detector.language_model.tokenizer\nwhile any(tok([c], return_tensors='pt').input_ids.shape[1] > detector.language_model.max_tokens for c in chunks):\n    chunked_size = max(1, chunked_size // 2)\n    chunks = rechunk(categories, chunked_size)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Auto-halve chunked_size when token overflow is detected","Keep category names concise"],"tags":["mmdet","grounding-dino","chunked-inference","token-limit"],"backgroundTag":"input-exceeds-token-limit","analyzedSha":"cfd5d3a985b0249de009b67d04f37263e11cdf3d","analyzedAt":"2026-08-27T20:54:20.183Z","schemaVersion":2},"datasetVersion":"2026-08-28T00:17:15.603Z"}