{"record":{"id":"f84af6ef18741064","repo":"open-mmlab/mmdetection","slug":"inputting-a-text-that-is-too-long-will-result-in-p-f84af6","errorCode":null,"errorMessage":"Inputting a text that is too long will result in poor prediction performance. Please reduce the --chunked-size.","messagePattern":"Inputting a text that is too long will result in poor prediction performance\\. Please reduce the --chunked-size\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"mmdet/models/detectors/glip.py","lineNumber":394,"sourceCode":"            list(range(1,\n                       len(original_caption) + 1)), chunked_size)\n\n        positive_map_label_to_token_chunked = []\n        caption_string_chunked = []\n        positive_map_chunked = []\n        entities_chunked = []\n\n        for i in range(len(ids_chunked)):\n            if enhanced_text_prompts is not None:\n                caption_string, tokens_positive = self.to_enhance_text_prompts(\n                    original_caption_chunked[i], enhanced_text_prompts)\n            else:\n                caption_string, tokens_positive = self.to_plain_text_prompts(\n                    original_caption_chunked[i])\n            tokenized = self.language_model.tokenizer([caption_string],\n                                                      return_tensors='pt')\n            if tokenized.input_ids.shape[1] > self.language_model.max_tokens:\n                warnings.warn('Inputting a text that is too long will result '\n                              'in poor prediction performance. '\n                              'Please reduce the --chunked-size.')\n            positive_map_label_to_token, positive_map = self.get_positive_map(\n                tokenized, tokens_positive)\n\n            caption_string_chunked.append(caption_string)\n            positive_map_label_to_token_chunked.append(\n                positive_map_label_to_token)\n            positive_map_chunked.append(positive_map)\n            entities_chunked.append(original_caption_chunked[i])\n\n        return positive_map_label_to_token_chunked, \\\n            caption_string_chunked, \\\n            positive_map_chunked, \\\n            entities_chunked\n\n    def loss(self, batch_inputs: Tensor,\n             batch_data_samples: SampleList) -> Union[dict, list]:","sourceCodeStart":376,"sourceCodeEnd":412,"githubUrl":"https://github.com/open-mmlab/mmdetection/blob/cfd5d3a985b0249de009b67d04f37263e11cdf3d/mmdet/models/detectors/glip.py#L376-L412","documentation":"Chunked variant of GLIP's prompt tokenization: warns when an individual chunk's token count exceeds language_model.max_tokens, and suggests reducing the chunk size so each chunk fits.","triggerScenarios":"Using get_tokens_positive_and_prompts_chunked (chunked-size too large) so a chunk's input_ids.shape[1] > max_tokens.","commonSituations":"Grounding many categories with chunked inference where chunked-size divides categories into still-too-long text groups; default chunk size too big for verbose category names.","solutions":["Decrease the --chunked-size argument so each chunk's caption fits within max_tokens","Shorten category names/phrases"],"exampleFix":"# before\ntexts, chunked_size = prompt, 5\n# after\ntexts, chunked_size = prompt, 2","handlingStrategy":"validation","validationCode":"tok = detector.language_model.tokenizer\nfor chunk in chunks:\n    assert tok([chunk], return_tensors='pt').input_ids.shape[1] <= detector.language_model.max_tokens, 'reduce chunked-size'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Start with a small chunked-size and increase only while token counts fit","Monitor token counts per chunk in preprocessing"],"tags":["mmdet","glip","chunked-inference","token-limit"],"backgroundTag":"input-exceeds-token-limit","analyzedSha":"cfd5d3a985b0249de009b67d04f37263e11cdf3d","analyzedAt":"2026-08-27T20:54:20.183Z","schemaVersion":2},"datasetVersion":"2026-08-28T00:17:15.603Z"}